← Tech Guides

Python & pip

Complete reference for Python CLI, pip, virtual environments, and modern Python tooling

01

Quick Reference

Essential Python CLI Commands

python script.py # Run a script python -c "print('hello')" # Execute one-liner python -m module # Run module as script python -i script.py # Run script then enter REPL python -u script.py # Unbuffered output python -O script.py # Optimize (remove assert) python -m venv myenv # Create virtual environment python -m pdb script.py # Debug with pdb python -m trace --trace script.py # Trace execution python -m cProfile script.py # Profile performance

Essential pip Commands

pip install package # Install package pip install -r requirements.txt # Install from file pip install package[extra] # Install with extras pip install -e . # Editable install pip uninstall package # Remove package pip list # List installed packages pip show package # Show package info pip freeze > requirements.txt # Export dependencies pip install --upgrade package # Upgrade package pip cache purge # Clear cache

Virtual Environment Quick Start

# Create and activate (Linux/Mac) python -m venv myenv source myenv/bin/activate # Create and activate (Windows) python -m venv myenv myenv\Scripts\activate # Deactivate (all platforms) deactivate
02

Python CLI

Common Flags

Flag Description Example
-c command Execute Python code directly python -c "print('hello')"
-m module Run module as script python -m http.server 8000
-i Interactive mode after script execution python -i script.py
-u Unbuffered output (critical for Docker/CI) python -u script.py
-O Optimize (remove assert statements) python -O script.py
-v Verbose mode (show import info) python -v script.py
-E Ignore environment variables python -E script.py
-B Don't write .pyc files python -B script.py

REPL (Python 3.13+ Features)

Python 3.13+ includes enhanced REPL with multi-line editing, syntax highlighting, and improved history.

  • Multi-line editing with syntax highlighting
  • Color support (disable with NO_COLOR=1)
  • Improved history and auto-completion
  • Better interactive experience

Useful One-Liners

# Print Python version python -c "import sys; print(sys.version)" # Quick calculations python -c "print(sum(range(100)))" # JSON processing echo '{"key":"value"}' | python -m json.tool # Simple HTTP server python -m http.server 8080 # Generate random password python -c "import secrets; print(secrets.token_urlsafe(16))" # Base64 encode/decode echo "hello" | python -c "import sys, base64; print(base64.b64encode(sys.stdin.read().encode()).decode())"
03

Virtual Environments

Virtual environments isolate project dependencies, preventing conflicts between packages and ensuring reproducible builds.

venv (Built-in)

Standard library tool, recommended for most projects

# Create python -m venv myenv # Activate source myenv/bin/activate # Linux/Mac myenv\Scripts\activate # Windows # Deactivate deactivate
  • Part of Python standard library
  • Lightweight, uses symlinks
  • Cannot create environments for different Python versions

virtualenv

More flexible alternative to venv

# Install pip install virtualenv # Create virtualenv myenv virtualenv -p python3.9 myenv # Activate (same as venv) source myenv/bin/activate
  • Works with Python 2 and Python 3
  • Can create environments for any installed Python version
  • Copies Python executables (not symlinks)

conda

Full-featured environment and package manager for data science

# Create conda create -n myenv python=3.11 # Activate conda activate myenv # Deactivate conda deactivate # Export conda env export > environment.yml
  • Manages both Python versions AND packages
  • Can install non-Python dependencies
  • Best for data science and scientific computing

pyenv - Python Version Manager

Manage multiple Python versions on your system. Uses shim executables to intercept Python commands and route to the correct version.

# Install pyenv (Linux/Mac) curl https://pyenv.run | bash # List available Python versions pyenv install --list # Install Python version pyenv install 3.11.8 pyenv install 3.12.2 # Set global Python version pyenv global 3.11.8 # Set local version (per-directory) pyenv local 3.12.2 # Creates .python-version file # List installed versions pyenv versions

Best Practices

When to use each tool:

  • venv: Simple projects, standard Python development, minimal dependencies
  • virtualenv: Need Python 2 support or more control than venv
  • conda: Data science, scientific computing, complex native dependencies
  • pyenv: Need to switch between Python versions frequently
  • pyenv + virtualenv: Multiple projects with different Python versions
04

pip Essentials

Installation Commands

# Install package pip install package pip install package==1.2.3 # Specific version pip install "package>=1.2,<2.0" # Version range pip install package --upgrade # Upgrade to latest # Install from requirements.txt pip install -r requirements.txt # Install with extras (optional dependencies) pip install "package[extra]" pip install "httpx[http2]" # Single extra pip install "httpx[http2,cli]" # Multiple extras # Editable/development install pip install -e . # Current directory pip install -e /path/to/package # Specific path # Install from different sources pip install package # PyPI (default) pip install git+https://github.com/user/repo.git pip install https://example.com/package.tar.gz pip install ./local-package-0.1.0.tar.gz # Install with constraints pip install -c constraints.txt package # Dry run (show what would be installed) pip install --dry-run package

Information Commands

pip list
List all installed packages
pip list --outdated
Show packages needing updates
pip show package
Show package information
pip show -f package
Show package with file list
pip check
Check for broken dependencies
pip freeze
Output installed packages in requirements format

Upgrade Commands

# Upgrade pip itself python -m pip install --upgrade pip # More reliable # Upgrade package pip install --upgrade package pip install -U package # Shorthand # Upgrade all in requirements.txt pip install -r requirements.txt --upgrade

Advanced Options

Option Description
--no-deps Install without dependencies
--force-reinstall Force reinstall of package
--user Install to user directory (no sudo needed)
-v, -vv, -vvv Verbose output levels
-q, -qq Quiet mode levels
--no-cache-dir Disable cache for this operation
05

Dependency Management

pip-tools (Recommended)

Separates direct dependencies from transitive (sub)dependencies. Industry standard for traditional pip workflow.

# Install pip-tools pip install pip-tools # Create requirements.in (direct dependencies only) cat > requirements.in << EOF django>=4.2 requests psycopg2-binary EOF # Compile to requirements.txt (with all transitive deps) pip-compile requirements.in pip-compile requirements.in --upgrade # Update all packages pip-compile requirements.in --upgrade-package django # Update specific package # Sync environment to match requirements.txt exactly pip-sync requirements.txt pip-sync requirements.txt dev-requirements.txt # Multiple files # Advanced options pip-compile --generate-hashes # Add hashes for security pip-compile --allow-unsafe # Include pip, setuptools, wheel

Workflow Example

# 1. Edit requirements.in with your direct dependencies echo "requests>=2.28" >> requirements.in # 2. Compile to lock file pip-compile requirements.in # 3. Install exact versions pip-sync requirements.txt # 4. To upgrade everything pip-compile --upgrade requirements.in pip-sync requirements.txt

Constraints Files

Control which versions are allowed without triggering installation. Use cases: monorepos, CI/CD pipelines, shared version requirements.

# constraints.txt - versions allowed, but don't install cat > constraints.txt << EOF requests==2.28.0 django<5.0 EOF # Use constraints with install pip install -c constraints.txt package pip install -r requirements.txt -c constraints.txt

Extras (Optional Dependencies)

Define optional feature sets in pyproject.toml for modular dependency management.

[project] name = "mypackage" dependencies = [ "requests>=2.28", ] [project.optional-dependencies] dev = [ "pytest>=7.0", "black>=22.0", ] docs = [ "sphinx>=5.0", "sphinx-rtd-theme>=1.0", ] all = [ "mypackage[dev,docs]", # Include other extras ]

Install extras:

pip install "mypackage[dev]" # Install with dev extras pip install "mypackage[dev,docs]" # Multiple extras pip install -e ".[dev]" # Editable install with extras
06

Modern Tools

uv

Ultra-fast pip replacement built in Rust. 10-100x faster than pip.

# Install curl -LsSf https://astral.sh/uv/install.sh | sh # Use as pip replacement uv pip install package uv pip install -r requirements.txt # Create virtual environment uv venv uv venv --python 3.11 # Install Python versions uv python install 3.11 uv python install 3.12 # Run commands uv run python script.py uv run pytest
  • 10-100x faster than pip
  • Deterministic resolution
  • Automatic venv management
  • Built-in Python version management

pipx

Install Python CLI tools in isolated environments. Like npm's npx or macOS's brew.

# Install pipx pip install pipx pipx ensurepath # Install applications pipx install black pipx install pytest pipx install poetry # Run without installing pipx run black . # Upgrade pipx upgrade black pipx upgrade-all # Inject dependencies pipx inject poetry poetry-plugin-export
  • Each app in isolated environment
  • Apps available globally in PATH
  • No dependency conflicts
  • Easy to upgrade/uninstall

Poetry

All-in-one dependency management and packaging. Easiest for publishing libraries.

# Install (via pipx recommended) pipx install poetry # Create new project poetry new myproject # Initialize existing project poetry init # Add dependencies poetry add requests poetry add pytest --group dev # Install all dependencies poetry install # Update dependencies poetry update # Build and publish poetry build poetry publish
  • Complete dependency management
  • Automatic virtual environment
  • Easy package publishing
  • Lock file for reproducibility

PDM (PEP 582)

Modern package manager following PEP standards. Uses __pypackages__ directory instead of virtual environments.

# Install PDM pipx install pdm # Initialize project pdm init # Add dependencies pdm add requests pdm add -d pytest # Dev dependency pdm add -G test pytest # Named group # Install dependencies pdm install pdm install --prod # Skip dev dependencies # Run scripts pdm run python script.py pdm run pytest

Tool Comparison

Tool Use Case Best For
uv Everything Speed, modern projects, single tool
pip + venv Basic projects Simple needs, standard library
pip-tools Requirements management Traditional workflow, enterprise
pipx CLI tools Global tools, no conflicts
Poetry Full projects Libraries, publishing, ease of use
PDM PEP-compliant projects Standards compliance, no venv
07

pyproject.toml

Modern standard for Python project metadata (PEP 621). Replaces setup.py and setup.cfg.

Complete Example

[build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "mypackage" version = "0.1.0" description = "A short description" readme = "README.md" requires-python = ">=3.9" license = {text = "MIT"} authors = [ {name = "Your Name", email = "you@example.com"} ] keywords = ["example", "package"] classifiers = [ "Development Status :: 4 - Beta", "Programming Language :: Python :: 3", ] dependencies = [ "requests>=2.28.0", "click>=8.0.0", ] [project.optional-dependencies] dev = [ "pytest>=7.0", "black>=22.0", "mypy>=0.990", ] docs = [ "sphinx>=5.0", ] [project.urls] Homepage = "https://github.com/user/repo" Documentation = "https://mypackage.readthedocs.io" Repository = "https://github.com/user/repo.git" [project.scripts] mycli = "mypackage.cli:main" [tool.setuptools.packages.find] where = ["src"] [tool.black] line-length = 88 [tool.pytest.ini_options] testpaths = ["tests"] python_files = "test_*.py" [tool.mypy] python_version = "3.11" warn_return_any = true

Build and Install

pip install . # Install current project pip install -e . # Editable install pip install -e ".[dev]" # With extras python -m build # Build wheel and sdist

Modern Best Practice

  • Use pyproject.toml for all new projects
  • Keep setup.py only for complex build processes
  • Use MANIFEST.in only if needed for non-Python files
  • Tool configurations (black, pytest, mypy) live in [tool.*] sections
08

Environment Management

pyenv - Complete Guide

# Install pyenv (Linux/Mac) curl https://pyenv.run | bash # Add to shell startup (~/.bashrc, ~/.zshrc) export PYENV_ROOT="$HOME/.pyenv" export PATH="$PYENV_ROOT/bin:$PATH" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)" # For pyenv-virtualenv plugin # List available Python versions pyenv install --list pyenv install --list | grep 3.11 # Filter by version # Install Python version pyenv install 3.11.8 pyenv install 3.12.2 pyenv install 3.13.0 # List installed versions pyenv versions # Set Python version pyenv global 3.11.8 # System-wide default pyenv local 3.12.2 # Current directory (creates .python-version) pyenv shell 3.13.0 # Current shell session only # Uninstall version pyenv uninstall 3.11.8 # Refresh shims (after installing packages with entry points) pyenv rehash

How pyenv Works

pyenv intercepts Python commands via shims, determining the correct version based on:

  • PYENV_VERSION environment variable
  • .python-version file in current or parent directory
  • ~/.pyenv/version (global)
# Example flow: python script.py ↓ /home/user/.pyenv/shims/python ↓ pyenv determines version (e.g., 3.11.8) ↓ /home/user/.pyenv/versions/3.11.8/bin/python script.py

pyenv-virtualenv Plugin

Combine version management with virtual environments for maximum flexibility.

# Install plugin git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv # Create virtualenv pyenv virtualenv 3.11.8 myproject-env # List virtualenvs pyenv virtualenvs # Activate pyenv activate myproject-env # Deactivate pyenv deactivate # Auto-activate based on directory cd ~/projects/myproject pyenv local myproject-env # Creates .python-version # Now cd'ing into directory auto-activates

Version Switching Strategies

Option 1: pyenv

Recommended for development

# Per-project version with auto-activation cd ~/projects/old-project pyenv local 3.9.18 python --version # Python 3.9.18 cd ~/projects/new-project pyenv local 3.12.2 python --version # Python 3.12.2

Option 2: Direct invocation

Simple approach

# Use specific Python directly python3.11 -m venv venv source venv/bin/activate python --version

Option 3: Docker

Isolation + reproducibility

FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . CMD ["python", "app.py"]
09

Debugging & Profiling

pdb (Python Debugger)

# Run script with debugger python -m pdb script.py # Start debugger from code (Python 3.7+) breakpoint() # Common pdb commands (Pdb) h # Help (Pdb) l # List source code (Pdb) ll # List entire current function (Pdb) w # Print stack trace (Pdb) n # Next line (step over) (Pdb) s # Step into function (Pdb) c # Continue until breakpoint (Pdb) p variable # Print variable (Pdb) pp variable # Pretty-print variable (Pdb) b 42 # Set breakpoint at line 42 (Pdb) q # Quit debugger

breakpoint() (Python 3.7+)

Modern debugging entry point. Configurable via PYTHONBREAKPOINT environment variable.

# Instead of: import pdb; pdb.set_trace() # Use: breakpoint() # Configure debugger via PYTHONBREAKPOINT export PYTHONBREAKPOINT=ipdb.set_trace # Use ipdb instead export PYTHONBREAKPOINT=0 # Disable all breakpoints export PYTHONBREAKPOINT=mymodule.debug # Custom debugger

cProfile (Performance Profiling)

# Profile script python -m cProfile script.py # Sort by time spent in function python -m cProfile -s time script.py python -m cProfile -s cumulative script.py # Save profile data python -m cProfile -o profile.stats script.py # Analyze saved profile python -m pstats profile.stats >>> sort time >>> stats 10 # Show top 10 >>> sort cumulative >>> stats func_name # Stats for specific function
Column Meaning
ncalls Number of calls
tottime Time in function (excluding subfunctions)
cumtime Time in function (including subfunctions)
percall Per-call average

timeit (Micro-benchmarking)

Accurate timing of small code snippets. Automatically runs code multiple times for reliable measurements.

# From command line python -m timeit "sum(range(100))" python -m timeit -n 1000 -r 5 "sum(range(100))" # 1000 loops, 5 repeats # Setup code python -m timeit -s "import math" "math.sqrt(144)" # In code import timeit # Time a function time = timeit.timeit("sum(range(100))", number=10000) print(f"Time: {time:.6f} seconds") # With setup time = timeit.timeit( stmt="sorted(numbers)", setup="import random; numbers = [random.random() for _ in range(1000)]", number=1000 ) # Repeat and get all results results = timeit.repeat("sum(range(100))", number=10000, repeat=5) print(f"Best: {min(results):.6f} seconds")

python -m trace (Execution Tracing)

# Trace execution (prints every line) python -m trace --trace script.py # Generate coverage report python -m trace --count script.py # Creates .cover files showing execution counts # Show missing lines python -m trace --count --missing script.py # Combined options python -m trace \ --trace \ # Print lines as executed --count \ # Count executions --report \ # Show coverage report --missing \ # Show lines not executed --ignore-dir=/usr/lib/python3.11 \ # Ignore stdlib script.py

Advanced Profiling Tools

memory_profiler
Track memory consumption line-by-line. Install: pip install memory_profiler
line_profiler
Detailed line-by-line performance analysis. Install: pip install line_profiler
py-spy
Sampling profiler for running processes. No code changes needed.
scalene
High-performance CPU, GPU, and memory profiler with AI-powered optimization suggestions.
10

Pro Tips

__main__.py Patterns

Make packages executable with python -m package

mypackage/ ├── __init__.py ├── __main__.py # Entry point for -m └── cli.py # mypackage/__main__.py from mypackage.cli import main if __name__ == "__main__": main() # Now runnable as: python -m mypackage

Why use __main__.py?

  • Provides CLI interface for packages
  • Works with python -m (not just direct execution)
  • Properly handles imports (no sys.path hacks needed)
  • Professional package structure

Prefer -m Module Execution

Good

# Uses current environment's tools python -m pip install package python -m http.server 8000 python -m pytest tests/ python -m black .

Bad

# Might use different Python pip install package pytest tests/ black .

Why?

  • Ensures correct Python interpreter
  • Works consistently across environments
  • Adds current directory to sys.path
  • More explicit about what's running

Useful Modules to Run with -m

python -m http.server 8000
Start HTTP server in current directory
python -m json.tool file.json
Pretty-print JSON file
python -m venv myenv
Create virtual environment
python -m pip install package
Package management
python -m pytest
Run tests
python -m black .
Format code
python -m pdb script.py
Debug script
python -m site
Show site configuration

Wheels vs sdist

Source Distribution (sdist)

  • .tar.gz or .zip file
  • Contains source code
  • Requires build step during installation
  • May need compiler if C extensions present
  • Slower to install
  • More compatible (works on any platform)

Wheel (wheel)

  • .whl file (actually a ZIP)
  • Pre-built distribution
  • No build step needed
  • Platform and Python version specific
  • Faster to install (10-100x)
  • Filename encodes compatibility
# Wheel filename format: {package}-{version}-{python}-{abi}-{platform}.whl # Example: numpy-1.26.0-cp311-cp311-manylinux_2_17_x86_64.whl │ │ │ │ └─ Platform (manylinux on x86_64) │ │ │ └─ ABI (CPython 3.11) │ │ └─ Python implementation (CPython 3.11) │ └─ Version └─ Package name # Universal wheel (pure Python, any version): requests-2.28.0-py3-none-any.whl
# pip prefers wheels over sdist (faster) pip install package # Tries wheel first, falls back to sdist # Force source installation pip install --no-binary :all: package # Build from source # Build wheels pip install build wheel python -m build # Creates wheel and sdist

pip Caching

# Find cache directory pip cache dir # Linux: ~/.cache/pip # Mac: ~/Library/Caches/pip # Windows: %LocalAppData%\pip\Cache # View cache info pip cache info # List cached packages pip cache list # Remove specific package from cache pip cache remove numpy pip cache remove "numpy*" # Pattern matching # Purge entire cache pip cache purge # Removes ALL cached files # Install without using cache pip install --no-cache-dir package

Why caching matters:

  • Speeds up repeated installs (CI/CD, multiple environments)
  • Saves bandwidth
  • Enables offline installs (if package previously cached)
  • Stores HTTP responses and built wheels

Environment Variable Configuration

# pip configuration via environment variables export PIP_INDEX_URL=https://pypi.org/simple/ # Default PyPI index export PIP_EXTRA_INDEX_URL=https://my.pypi.org/simple/ # Additional index export PIP_TRUSTED_HOST=my.pypi.org # Trust HTTP host export PIP_NO_CACHE_DIR=1 # Disable cache export PIP_REQUIRE_VIRTUALENV=1 # Only allow venv installs # Python configuration export PYTHONPATH=/path/to/modules # Add to sys.path export PYTHONDONTWRITEBYTECODE=1 # Don't create .pyc files export PYTHONUNBUFFERED=1 # Unbuffered output export PYTHONBREAKPOINT=ipdb.set_trace # Custom debugger export PYTHONWARNINGS=ignore # Suppress warnings

Troubleshooting Common Issues

Issue Solution
pip: command not found python -m pip install package (use -m instead)
Permission denied pip install --user package or use virtual environment
SSL certificate errors pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org package
Behind corporate proxy Set HTTP_PROXY and HTTPS_PROXY environment variables
Dependency conflicts pip check to show broken dependencies
Broken environment pip install --force-reinstall package
Debug pip itself pip install -vvv package (debug level verbosity)

Performance Tips

  • Use wheels (not sdist): pip install --only-binary :all: package
  • Parallel downloads (pip 20.3+): Install multiple packages at once
  • Use pip-tools for deterministic installs: pip-sync faster than pip install -r
  • Prefer uv for speed: 10-100x faster than pip
  • Cache aggressively in CI/CD: Use actions/cache for GitHub Actions