Python Packaging

Where to start?

With the holidays fast approaching, I thought I would dedicate a blog post to packaging, specifically Python packaging. I’ll go ahead and say it, packaging Python projects is confusing. If we go to the official Python documentation and take a look at the Distributing Python Modules section we get the following information about these projects:

• Distutils - Added to the Python standard library in 1998. The direct use of Distutils is currently being phased out.
• Setuptools - A drop-in replacement for Distutils since 2004 and the current recommendation for packaging Python projects. Setuptools offers support for more recent packaging standards across a wide range of Python versions.
• Wheel - A project that allows Distutils/Setuptools to produce a cross platform binary packaging format called wheels or wheel files. Wheels allow Python libraries to be installed on systems without having to be built locally.

The documentation then goes on to state, “The standard library does not include build tools that support modern Python packaging standards, as the core development team has found that it is important to have standard tools that work consistently, even on older versions of Python.”

Alright so we could use Distutils to package projects, but we should most likely use Setuptools and Wheel if possible. Where to go from here then? The documentation then provides a link to the Python Packaging User Guide which describes the use of Setuptools with a setup.py file to build Python projects.

I’m going to suggest instead we go directly to the Setuptools documentation and took a look at their most up-to-date packaging guide.

Project Structure

Let’s start an example project called meowproject. Meowproject is a simple Python package that will print “Meow!” when run. We want a directory structure with the following standard format:

meowpkg
├── meowpkg
│   ├── __init__.py
│   └── meow.py
├── pyproject.toml
└── setup.cfg

Let’s go through these files one by one. If we follow the instructions on the Setuptools quickstart guide we should first create a simple pyproject.toml file:

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

Our init.py file will remain empty and we’ll create our simple meow.py script:

def main():
  print("Meow!")

Next we’ll create a setup.cfg which will contain the meat of our packaging configuration:

[metadata]
name = meowpkg
version = 0.0.1

[options]
packages = find:
install_requires =
  python_version >= "3.8"

[options.entry_points]
console_scripts =
  meow = meowpkg.meow:main

Automatic package discovery

In our setup.cfg we specified packages = find: under [options]. This line allows for automatic package discovery and ends up making our setup.cfg file a bit less confusing.

Entry points

Adding an [options.entry_points] section allows for one or more scripts to be callable from the installed Python package. This means any number of helper Python files can be generated alongside meow.py. When the meowpkg project is installed, a meow script is installed from meowpkg.meow:main.

Dependency management

Setuptools allows for dependencies to be automatically installed when a package is installed. Any number of arguments can be added to the install_requires line, following the package name, operator (<, >, <=, >=, ==, or !=) and a version identifier syntax.

Venv

Before we move on I thought I would mention the module Venv, which allows for the creation of virtual environments, each having their own Python binary and independent set of installed Python Packages.

Let’s create a virtual environment for our new package:

$ cd ~/meowpkg
python -m venv venv
source venv/bin/activate

Here we set up our new virtual environment in a directory called venv. I like this method because it separates environment files from the rest of the project and reduces clutter. We then activate our environment with source venv/bin/activate to enter the environment, which can be seen with a change to in our terminal (venv).

Note: I prefer using the built-in pip module as opposed to the system’s pip.

Setuptools

We’ll need to install a pep517 compatible builder. Even though it’s a fairly new project, let’s install build, build meowpkg in the current directory, and install the generated wheel file in the resulting dist directory:

$ (venv) python -m pip install build
$ (venv) python -m build .
$ (venv) cd dist
$ (venv) python -m pip install meowpkg-0.0.1-py3-none-any.whl

Finally let’s test out our new program

(venv) $ meow
Meow!

Success!

Distribute

With our package built we could upload either the tar.gz or .whl files to the Python Package Index (PyPI), but again I’m going to suggest a different approach.

Zipapp

Zipapp is a module used to create zip files of Python code, allowing them to be directly run by the Python interpreter. This allows for self-contained and executable single file Python programs, which only Python itself. In this case a single file could be uploaded to a public or private repo, such Github and downloaded directly.

Here we have a similar file and directory structure as the previous example, but we are missing the venv directory, and the pyproject.toml and setup.cfg files as they are not necessary. Let’s generate an executable archive from our meowpkg directory:

$ python -m zipapp meowpkg -p "/usr/bin/env python3" -m "meow:main" -o meow_0.0.1

Here we are calling the zipapp module, specifying the “meowpkg” directory to archive, specifying the correct Python interpreter to use, what script to use as the entrypoint, and an output filename.

Now let’s test out our self-contained package in a fresh environment. We’ll go ahead and use a container for this:

$ podman run -it --rm -v ./meow_0.0.1:/meow:Z docker.io/python:slim /meow_0.0.1
Meow!

Note: If you specify the interpreter as “/usr/bin/python” or “/usr/bin/python3” issues could arise depending on how specific distributions name their Python interpreters.

Alright what if our package has dependencies? Simply install the dependencies in the source directory and then build:

$ python -m pip install PyYaml --target .
$ ls
build  meow_0.0.1  meowpkg.egg-info  PyYAML-5.3.1.dist-info  venv
dist   meowpkg     pyproject.toml    setup.cfg               yaml

We could alternatively create a requirements.txt if the project had a large number of dependencies:

$ python -m pip install -r requirements.txt --target .

Hopefully that cleared up some of the confusion with creating Python packages. With a solid foundation, modern Python package tooling is actually pretty straight forward to use.