fromager

Fromager is a tool for completely re-building a dependency tree of Python wheels from source.

The goals are to support guaranteeing

1. The binary package someone is installing was built from source in a known build environment compatible with their own environment
2. All of the package’s dependencies were also built from source -- any binary package installed will have been built from source
3. All of the build tools used to build these binary packages will also have been built from source
4. The build can be customized for the packager's needs, including patching out bugs, passing different compilation options to support build "variants", etc.

The basic design tenet is to automate everything with a default behavior that works for most PEP-517 compatible packages, but support overriding all of the actions for special cases, without encoding those special cases directly into fromager.

Modes

Fromager has different modes for bootstrapping and production builds. The bootstrap mode recursively processes all dependencies starting from the requirements specifications given to determine what needs to be built and what order to build it. The production build commands separate these steps and avoid recursive processing so each step can be performed in isolation.

Bootstrapping

The bootstrap command

• Creates an empty package repository for wheels
• Downloads the source distributions for the input packages and places them under sdists-repo/downloads/.
• Recurses through the dependencies
  • Firstly, any build system dependency specified in the pyproject.toml build-system.requires section as per PEP517
  • Secondly, any build backend dependency returned from the get_requires_for_build_wheel() build backend hook (PEP517 again)
  • Lastly, any install-time dependencies of the project as per the wheel’s core metadata Requires-Dist list.
• As each wheel is built, it is placed in a PEP503 "simple" package repository under wheels-repo/simple generated by pypi-mirror.
• The order the dependencies need to be built bottom-up is written to build-order.json.

Wheels are built by running pip wheel configured so it will only download dependencies from the local wheel repository. This ensures that all dependencies are being built in the correct order.

Production Builds

Production builds use separate commands for the steps described as part of bootstrapping, and accept arguments to control the servers that are used for downloading source or built wheels.

All-in-one commands

Two commands support building wheels from source.

The build command takes as input the distribution name and version to build, the variant, and the URL where it is acceptable to download source distributions. The server URL is usually a simple index URL for an internal package index. The outputs are one patched source distribution and one built wheel.

The build-sequence command takes a build-order file, the variant, and the source distribution server URL. The outputs are patched source distributions and built wheels for each item in the build-order file.

Step-by-step commands

Occasionally it is necessary to perform additional tasks between build steps, or to run the different steps in different configurations (with or without network access, for example). Using the step subcommands, it is possible to script the same operations performed by the build and build-sequence commands.

The step download-source-archive command finds the source distribution for a specific version of a dependency on the specified package index and downloads it. It will be common to run this step with pypi.org, but for truly isolated and reproducible builds a private index server is more robust.

The step prepare-source command unpacks the source archive downloaded from the previous step and applies any patches (refer to customization for details about patching).

The step prepare-build command creates a virtualenv with the build dependencies for building the wheel. It expects a --wheel-server-url as argument to control where built wheels can be downloaded.

The step build-sdist command turns the prepared source tree into a new source distribution ("sdist"), including any patches or vendored code.

The step build-wheel command creates a wheel using the build environment and prepared source, compiling any extensions using the appropriate override environment settings (refer to customization for details about overrides).

Using private registries

Fromager uses the requests library and pip at different points for talking to package registries. Both support authenticating to remote servers in various ways. The simplest way to integrate the authentication with fromager is to have a netrc file with a valid entry for the host. The file will be read from ~/.netrc by default. Another location can be specified by setting the NETRC environment variable.

For example, to use a gitlab package registry, use a personal access token as documented in this issue:

machine gitlab.com login oauth2 password $token

Determining versions via GitHub tags

In some cases, the builder might have to use tags on GitHub to determine the version of a project instead of looking at pypi.org. To avoid rate limit or to access private GitHub repository, a personal access token can be passed to fromager by setting the following environment variable:

GITHUB_TOKEN=<access_token>

Additional docs

• Package build customization instructions
• Developer instructions

What's with the name?

Python's name comes from Monty Python, the group of comedians. One of their skits is about a cheese shop that has no cheese in stock. The original Python Package Index (pypi.org) was called The Cheeseshop, in part because it hosted metadata about packages but no actual packages. The wheel file format was selected because cheese is packaged in wheels. And "fromager" is the French word for someone who makes or sells cheese.