:rocket: conda-join - Unified Conda and Pip Requirements Management :rocket:

conda_join is a Python package designed to streamline the management and combination of multiple requirements.yaml files into a single Conda environment.yaml, whilest also being able to import the requirements.yaml file in setup.py where it will add the Python PyPI dependencies to requires. This tool is ideal for projects with multiple subcomponents, each having its own dependencies, where some are only available on conda and some on PyPI (pip), simplifying the process of creating a unified Conda environment, while being pip installable with the Python only dependencies. 🖥️🔥

:books: Table of Contents

• :page_facing_up: Requirements File Structure
  • Basic Structure
  • Example
  • Explanation
• :package: Installation
• :memo: Usage with pyproject.toml or setup.py
• :memo: Usage as a CLI
  • :wrench: Advanced Configuration
  • :scroll: Output Options
  • :question: Help Menu
• :warning: Limitations

:page_facing_up: Requirements File Structure

The requirements.yaml files that conda_join processes should follow a specific structure for the tool to correctly interpret and combine them. Here's an overview of the expected format:

Basic Structure

Each requirements.yaml file should contain the following key elements:

• name: (Optional) A name for the environment. This is not used in the combined output but can be helpful for documentation purposes.
• channels: A list of channels from which packages will be sourced. Commonly includes channels like conda-forge.
• dependencies: A list of package dependencies. This can include both Conda and Pip packages.

Example

Here is an example of a typical requirements.yaml file:

name: example_environment
channels:
  - conda-forge
dependencies:
  - numpy  # same name on conda and pip
  - pandas  # same name on conda and pip
  - conda: scipy  # different name on conda and pip
    pip: scipy-package
  - pip: package3  # only available on pip
  - conda: mumps  # only available on conda

Explanation

• Dependencies listed as simple strings (e.g., - numpy) are assumed to be Conda packages.
• If a package is available through both Conda and Pip but with different names, you can specify both using the conda: <conda_package> and pip: <pip_package> format.
• Packages only available through Pip should be listed with the pip: prefix.

conda_join will combine these dependencies into a single environment.yaml file, structured as follows:

name: some_name
channels:
  - conda-forge
dependencies:
  - numpy
  - pandas
  - scipy
  pip:
    - scipy-package
    - package3

:package: Installation

To install conda_join, run the following command:

pip install -U conda_join

Or just copy the script to your computer:

wget https://raw.githubusercontent.com/basnijholt/requirements.yaml/main/conda_join.py

:memo: Usage with pyproject.toml or setup.py

To use conda_join in your project, you can configure it in pyproject.toml. This setup works alongside a requirements.yaml file located in the same directory. The behavior depends on your project's setup:

• When using only pyproject.toml: The dependencies field in pyproject.toml will be automatically populated based on the contents of requirements.yaml.
• When using setup.py: The install_requires field in setup.py will be automatically populated, reflecting the dependencies defined in requirements.yaml.

Here's an example pyproject.toml configuration:

[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "wheel", "conda_join"]

In this configuration, conda_join is included as a build requirement, allowing it to process the Python dependencies in the requirements.yaml file and update the project's dependencies accordingly.

:memo: Usage as a CLI

After installation, you can use conda_join to scan directories for requirements.yaml files and combine them into an environment.yaml file. Basic usage is as follows:

conda_join -d [DIRECTORY] --depth [DEPTH] -o [OUTPUT_FILE]

• -d or --directory: Specify the base directory to scan (default is current directory).
• --depth: Specify the depth for scanning subdirectories (default is 1).
• -o or --output: Specify the output file for the combined environment (default is environment.yaml).

:wrench: Advanced Configuration

conda_join allows advanced configurations such as verbose output and printing to stdout instead of a file.

• To enable verbose output, use the -v or --verbose flag.
• To print the combined environment to stdout instead of saving to a file, use the --stdout flag.

Example with advanced options:

conda_join -d src --depth 2 -o dev_environment.yaml --verbose

:scroll: Output Options

• The output environment.yaml file will contain a unified list of dependencies from all scanned requirements.yaml files.
• If the --stdout flag is used, the combined environment will be printed to the console.

:question: Help Menu

For more options, use:

usage: conda-join [-h] [-d DIRECTORY] [--depth DEPTH] [-o OUTPUT] [-v]
                  [--stdout]

Unified Conda and Pip requirements management.

options:
  -h, --help            show this help message and exit
  -d DIRECTORY, --directory DIRECTORY
                        Base directory to scan for requirements.yaml files, by
                        default `.`
  --depth DEPTH         Depth to scan for requirements.yaml files, by default
                        1
  -o OUTPUT, --output OUTPUT
                        Output file for the conda environment, by default
                        `environment.yaml`
  -v, --verbose         Print verbose output
  --stdout              Output to stdout instead of a file

:warning: Limitations

• The current version of conda_join does not support conflict resolution between different versions of the same package in multiple requirements.yaml files.
• Designed primarily for use with Conda environments; may not fully support other package management systems.

Try conda_join today for a streamlined approach to managing your Conda environment dependencies across multiple projects! 🎉👏