pypackagery 1.0.4

Requirement Specification

As already mentioned, the requirements are expected to follow Pip format (see pip documentation) and live in requirements.txt at the root of the code base. The mapping from modules to requirements is expected in module_to_requirement.tsv also at the root of the code base.

Assume that the code base lives in ~/workspace/some-project.

Here is an excerpt from ~/workspace/some-project/requirements.txt:

pillow==5.2.0
pytz==2018.5
pyzmq==17.1.2

And here is an excerpt from ~/workspace/some-project/module_to_requirement.tsv (mind that it’s tab separated):

PIL pillow
PIL.Image   pillow
PIL.ImageFile       pillow
PIL.ImageOps        pillow
PIL.ImageStat       pillow
PIL.ImageTk pillow
cv2 opencv-python

Directory

Assume that the code base lives in ~/workspace/some-project and we are interested to bundle everything in pipeline/out directory.

To determine the subset of the files and requirements, run the following command line:

pypackagery \
    --root_dir ~/workspace/some-project \
    --initial_set ~/workspace/some-project/pipeline/out

This gives us a verbose, human-readable output like:

External dependencies:
Package name | Requirement spec
-------------+---------------------
pyzmq        | 'pyzmq==17.1.2'
temppathlib  | 'temppathlib==1.0.3'

Local dependencies:
pipeline/out/__init__.py
common/__init__.py
common/logging.py
common/proc.py

If we want to get the same output in JSON, we need to call:

pypackagery \
    --root_dir ~/workspace/some-project \
    --initial_set ~/workspace/some-project/pipeline/out \
    --format json

which gives us a JSON-encoded dependency graph:

{
  "requirements": {
    "pyzmq": {
      "name": "pyzmq",
      "line": "pyzmq==17.1.2\n"
    },
    "temppathlib": {
      "name": "temppathlib",
      "line": "temppathlib==1.0.3\n"
    }
  },
  "rel_paths": [
    "pipeline/out/__init__.py",
    "common/__init__.py",
    "common/logging.py",
    "common/proc.py"
  ],
  "unresolved_modules": []
}

Files

Assume again that the code base lives in ~/workspace/some-project. We would like to get a subset of the code base required by a list of scripts. We need to specify the initial set as a list of files:

pypackagery \
    --root_dir ~/workspace/some-project \
    --initial_set \
        ~/workspace/some-project/pipeline/input/receivery.py \
        ~/workspace/some-project/pipeline/input/snapshotry.py

which gives us:

External dependencies:
Package name | Requirement spec
-------------+-------------------
icontract    | 'icontract==1.5.1'
pillow       | 'pillow==5.2.0'
protobuf     | 'protobuf==3.5.1'
pytz         | 'pytz==2018.5'
pyzmq        | 'pyzmq==17.1.2'
requests     | 'requests==2.19.1'

Local dependencies:
pipeline/__init__.py
pipeline/input/receivery.py
pipeline/input/snapshotry.py
common/__init__.py
common/img.py
common/logging.py
protoed/__init__.py
protoed/pipeline_pb2.py

Unresolved Modules

If there is a module which could not be resolved (neither in built-ins, nor specified in the requirements nor living in the code base), the pypackagery will return a non-zero return code.

If you specify --dont_panic, the return code will be 0 even if there are unresolved modules.

Module packagery

Pypackagery provides a module packagery which can be used to programmatically determine the dependencies of the subset of the code base. For example, this is particularly useful for deployments to a remote machine where you want to deploy only a part of the code base depending on some given configuration.

Here is an example:

import pathlib

import packagery

root_dir = pathlib.Path('/some/codebase')

rel_pths = [
    pathlib.Path("some/dir/file1.py"),
    pathlib.Path("some/other/dir/file2.py")]

requirements_txt = root_dir / "requirements.txt"
module_to_requirement_tsv = root_dir / "module_to_requirement.tsv"

requirements = packagery.parse_requirements(
    text=requirements_txt.read_text())

module_to_requirement = packagery.parse_module_to_requirement(
    text=module_to_requirement_tsv.read_text(),
    filename=module_to_requirement_tsv.as_posix())

pkg = packagery.collect_dependency_graph(
    root_dir=root_dir,
    rel_paths=rel_pths,
    requirements=requirements,
    module_to_requirement=module_to_requirement)

# do something with pkg ...

Mind that relative paths (given as rel_paths argument) all need to be files, not directories.

Pre-commit Checks

We provide a set of pre-commit checks that lint and check code for formatting.

Namely, we use:

• yapf to check the formatting.

• The style of the docstrings is checked with pydocstyle.

• Static type analysis is performed with mypy.

• Various linter checks are done with pylint.

• Doctests are executed using the Python doctest module.

Run the pre-commit checks locally from an activated virtual environment with development dependencies:

./precommit.py

• The pre-commit script can also automatically format the code:

./precommit.py  --overwrite