Eric Campbell 5 min read Python

How to Package and Distribute Python Libraries

A well-structured Python package follows conventions that tools expect. Here's the standard layout:

Key Insights

  • Modern Python packaging uses pyproject.toml as the single source of truth, replacing the fragmented setup.py/setup.cfg/requirements.txt approach that plagued earlier ecosystems.
  • Distribution via PyPI requires understanding the difference between source distributions (sdist) and wheels—wheels are pre-built and install faster, while sdist provides maximum compatibility across platforms.
  • Automated publishing through GitHub Actions eliminates manual release errors and ensures consistent versioning, testing, and deployment workflows for every package release.

Project Structure and Essential Files

A well-structured Python package follows conventions that tools expect. Here’s the standard layout:

my-package/
├── src/
│   └── mypackage/
│       ├── __init__.py
│       ├── core.py
│       └── utils/
│           ├── __init__.py
│           └── helpers.py
├── tests/
│   ├── __init__.py
│   └── test_core.py
├── pyproject.toml
├── README.md
├── LICENSE
└── .gitignore

The src/ layout is superior to placing your package at the root. It prevents accidental imports of the local development version when running tests, forcing proper installation testing. Each package directory needs an __init__.py file, even if empty, though you should use it strategically:

# src/mypackage/__init__.py
"""MyPackage - A fantastic library for doing things."""

__version__ = "0.1.0"

from .core import main_function, HelperClass
from .utils.helpers import utility_function

__all__ = ["main_function", "HelperClass", "utility_function"]

This pattern controls your public API. Users can from mypackage import main_function without knowing internal structure. The __all__ list explicitly defines what from mypackage import * includes, though wildcard imports should be discouraged in documentation.

Your LICENSE file isn’t optional—it’s legally required for others to use your code. MIT and Apache 2.0 are popular permissive choices. The README.md serves as your package’s front page on PyPI and GitHub, so invest time making it clear and comprehensive.

Creating pyproject.toml and Package Metadata

The pyproject.toml file is the modern standard, defined in PEP 518 and PEP 621. It consolidates all package configuration:

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.1.0"
description = "A fantastic library for doing things"
readme = "README.md"
license = {text = "MIT"}
authors = [
    {name = "Your Name", email = "you@example.com"}
]
requires-python = ">=3.8"
classifiers = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
]
dependencies = [
    "requests>=2.28.0,<3.0.0",
    "click>=8.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "black>=22.0.0",
    "mypy>=0.990",
]
docs = [
    "sphinx>=5.0.0",
    "sphinx-rtd-theme>=1.0.0",
]

[project.urls]
Homepage = "https://github.com/yourusername/mypackage"
Documentation = "https://mypackage.readthedocs.io"
Repository = "https://github.com/yourusername/mypackage"
Changelog = "https://github.com/yourusername/mypackage/blob/main/CHANGELOG.md"

[project.scripts]
mypackage-cli = "mypackage.cli:main"

[tool.setuptools.packages.find]
where = ["src"]

Version constraints matter. Use >=2.28.0,<3.0.0 for dependencies to allow patch and minor updates while preventing breaking major version changes. The requires-python field prevents installation on incompatible Python versions, saving users from cryptic runtime errors.

Classifiers are metadata tags that help users filter packages on PyPI. Choose them carefully from the official list. They affect how your package appears in searches.

The [project.scripts] section creates console commands. When users install your package, mypackage-cli becomes an executable command in their PATH that calls your main() function.

Building Distribution Files

Python packages distribute in two formats: source distributions (sdist) and wheels. Source distributions contain raw source code and require build steps during installation. Wheels are pre-built, platform-specific packages that install almost instantly.

Always build both:

# Install build tools
pip install build twine

# Build distributions
python -m build

# This creates:
# dist/mypackage-0.1.0.tar.gz (source distribution)
# dist/mypackage-0.1.0-py3-none-any.whl (wheel)

The wheel filename encodes important information: mypackage-0.1.0-py3-none-any.whl means Python 3, no specific ABI, any platform. If your package includes C extensions, you’ll need platform-specific wheels for Windows, macOS, and Linux.

Inspect what’s included in your distributions:

# List contents of wheel
unzip -l dist/mypackage-0.1.0-py3-none-any.whl

# Extract and inspect source distribution
tar -tzf dist/mypackage-0.1.0.tar.gz

Common mistake: including unnecessary files. Create a MANIFEST.in if you need fine control over included files, but the defaults usually work with proper .gitignore configuration.

Version your packages semantically: MAJOR.MINOR.PATCH. Increment MAJOR for breaking changes, MINOR for new features, PATCH for bug fixes. Consider using setuptools-scm to derive versions automatically from git tags:

[build-system]
requires = ["setuptools>=61.0", "setuptools-scm>=7.0"]

[tool.setuptools_scm]
write_to = "src/mypackage/_version.py"

Publishing to PyPI

Create accounts on TestPyPI and PyPI. TestPyPI is a separate instance for testing uploads without polluting the real index. Configure API tokens for secure authentication:

# Create ~/.pypirc
[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username = __token__
password = pypi-your-api-token-here

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-your-testpypi-token-here

Upload to TestPyPI first:

twine upload --repository testpypi dist/*

Verify the package page looks correct on TestPyPI, then upload to production PyPI:

twine upload dist/*

Automate this with GitHub Actions. Create .github/workflows/publish.yml:

name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"
      
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install build twine          
      
      - name: Build package
        run: python -m build
      
      - name: Publish to PyPI
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
        run: twine upload dist/*

Store your PyPI API token in GitHub repository secrets as PYPI_API_TOKEN. Now creating a GitHub release automatically publishes to PyPI.

Testing Your Package Installation

Before announcing your package, test installation in isolated environments:

# Create fresh virtual environment
python -m venv test-env
source test-env/bin/activate  # On Windows: test-env\Scripts\activate

# Install from TestPyPI
pip install --index-url https://test.pypi.org/simple/ \
    --extra-index-url https://pypi.org/simple/ \
    mypackage

# Test import and functionality
python -c "import mypackage; print(mypackage.__version__)"

The --extra-index-url is necessary because your dependencies likely aren’t on TestPyPI.

If you defined console scripts, test them:

# src/mypackage/cli.py
import click
from mypackage import main_function

@click.command()
@click.option('--name', default='World', help='Name to greet')
def main(name):
    """Simple CLI tool."""
    result = main_function(name)
    click.echo(result)

if __name__ == '__main__':
    main()

After installation, mypackage-cli --name Alice should work from anywhere in the terminal.

Test in multiple Python versions using tox:

# tox.ini
[tox]
envlist = py38,py39,py310,py311

[testenv]
deps = pytest
commands = pytest tests/

Run tox to test across all specified Python versions automatically.

Documentation and Best Practices

Your README is the most important documentation. Structure it clearly:

# MyPackage

Brief one-line description.

## Installation

```bash
pip install mypackage

Quick Start

from mypackage import main_function

result = main_function("example")
print(result)

Features

  • Feature one with brief explanation
  • Feature two with brief explanation

Documentation

Full documentation at: https://mypackage.readthedocs.io

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE file for details.


Maintain a `CHANGELOG.md` following [Keep a Changelog](https://keepachangelog.com/) format:

```markdown
# Changelog

## [0.2.0] - 2024-01-15
### Added
- New feature X for handling Y cases
### Changed
- Improved performance of core algorithm by 40%
### Deprecated
- `old_function()` will be removed in 1.0.0, use `new_function()` instead

## [0.1.0] - 2024-01-01
### Added
- Initial release with core functionality

Handle deprecations gracefully using warnings:

import warnings

def old_function():
    warnings.warn(
        "old_function is deprecated and will be removed in version 1.0.0. "
        "Use new_function instead.",
        DeprecationWarning,
        stacklevel=2
    )
    return new_function()

Pin your development dependencies but not your package dependencies. In pyproject.toml, use loose version constraints for dependencies but create a requirements-dev.txt with exact versions for reproducible development environments.

Finally, add badges to your README for build status, coverage, and PyPI version. They provide instant credibility and useful information at a glance. Your package is now ready for the world.

Liked this? There's more.

Every week: one practical technique, explained simply, with code you can use immediately.