Skip to main content

Manage a cache of Python environments

Project description

Manage a cache of virtual environments based on changes in requirements files.

An example usage of this tool might be a CI system that executes a testsuite. If there are no changes in requirements, a cached virtual environment already available in the CI might be used. This environment can have all the dependencies already installed to speed up CI jobs.

If a new virtual environment is created, this virtual environment can be added to the cache to speed up next CI jobs.

This tool automatically manages the cache, lifetime of cache entries, and additional cache parameters based on the supplied configuration file.

Installation

You can install the latest version from PyPI:

pip install virtualenv-cache

If you wish to run the latest version from the Git repository:

pip install git+https://github.com/fridex/virtualenv-cache@latest

Usage

First, there needs to be generated a configuration file:

virtualenv-cache init

The command above will create a configuration file in the current directory (by default) called .virtualenv_cache.toml. Check its configuration options as described below to match your desired behavior. This file has be generated per project that should be using the cache and should be part of the project code base - e.g. committed to the Git repository so that it can be used on a clone.

Next, your configuration in a CI can look similar to the code snipped bellow:

cd project-root/
virtualenv-cache restore
[ $? -eq 1 ] && ( python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt && virtualenv-cache store )

The code snipped above will try to restore a virtual environment from cache. If there is no matching cached virtual environment, the restore command exists with exit code of 1 indicating cache miss. In that case, the virtual environment can be created and prepared for the next runs which will result in a cache hit (unless there is a change in requirements files which results in a new virtual environment creation that will get cached again). Just make sure you keep all the requirements installed in your virtual environment and have them stated in requirements_lock_paths.

Configuration file

The configuration file can be generated using virtualenv-cache init. An example of such configuration file:

[virtualenv-cache]
cache_size = 25
cache_path = "${HOME}/.virtualenv-cache/my-project/cache/"
virtualenv_path = ".venv"
requirements_lock_paths = [
    "requirements.txt",
    "requirements-dev.txt",
    "requirements-typing.txt"
]

cache_size

The number of virtual environments cached. If the cache_size is reached, the cache is trimmed based on use of virtual environments - only the most used virtual environments based on datetime are kept in the cache.

cache_path

A path where cached virtual environments should be stored. If you use virtualenv-cache to manage cache for multiple projects, make sure you define different cache_path for each of them.

The path configuration value can state environment variables which get expanded.

virtualenv_path

A path where the virtual environment is created for the project. This path is used to copy the virtual environment into the cache or restored from the cache.

The path configuration value can state environment variables which get expanded.

requirements_lock_paths

A list of requirements files that affect installed dependencies in the virtual environment. There can be stated lock files, such as requirements.txt as produced by pip-tools, a poetry.lock file as produced by Poetry, a pdm.lock file as produced by PDM, or a Pipfile.lock as produced by Pipenv.

Note there are internally computed hashes of these files on their content without taking into account semantics. That means any change, even a new line, added to the file affects a new cache entry creation. Generally, this does not create any issues as the old cache entries will get removed over time based on the cache_size configuration option. This also mean that you can add any other file which content potentially affects virtual environment to this listing.

Commands

The tool can be run with the following sub-commands:

  • virtualenv-cache store - store the curent virtual environment into the cache

  • virtualenv-cache restore - restore the matching virtual environment from the cache

  • virtualenv-cache init - initialize the configuration file

  • virtualenv-cache list - list entries in the cache with their additional metadata, such as the last access time

  • virtualenv-cache erase - drop all cached virtual environments

See --help for more information and options available.

Additional notes

All the CLI parameters can be supplied as environment variables:

  • VIRTUALENV_CACHE_CONFIG_PATH - a path to the virtualenv-cache configuration file

  • VIRTUALENV_CACHE_FORMAT - format used to print output to terminal

  • VIRTUALENV_CACHE_WORK_DIR - a working directory for the CLI

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

virtualenv-cache-0.0.2.tar.gz (28.9 kB view details)

Uploaded Source

Built Distribution

virtualenv_cache-0.0.2-py3-none-any.whl (22.4 kB view details)

Uploaded Python 3

File details

Details for the file virtualenv-cache-0.0.2.tar.gz.

File metadata

  • Download URL: virtualenv-cache-0.0.2.tar.gz
  • Upload date:
  • Size: 28.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for virtualenv-cache-0.0.2.tar.gz
Algorithm Hash digest
SHA256 ea8467854e1659b8eebc7b9d51181dd26bcb6819c532d2d18b69dfffd0b4117e
MD5 2f2002ee8d19b9546dc063859b32fc82
BLAKE2b-256 453b6d5e680540b70b661821d00c805ac08d70aa119264eef379f79bbe41446e

See more details on using hashes here.

File details

Details for the file virtualenv_cache-0.0.2-py3-none-any.whl.

File metadata

File hashes

Hashes for virtualenv_cache-0.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 ba3472638969b27b4b8068c78a186fd118fd7c9d09e80e5fe5636d2a8086a424
MD5 119b5c25042137f428288541830eb7b0
BLAKE2b-256 53248f187cf1d101048a3b78f783a8ca24e397d0b29f2f862c46db4262241c44

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page