Snakemake-like pipeline manager for reproducible Jupyter Notebooks
Project description
Snakemake-like pipelines for Jupyter Notebooks, producing interactive pipeline reports like this:
Install & general remarks
These are still early days of this software so please bear in mind that it is not ready for production yet. Note: for simplicity I assume that you are using a recent Ubuntu with git installed.
pip install nbpipelineGraphiz is required for static SVG plots:
sudo apt-get install graphviz libgraphviz-dev graphviz-devDevelopment install
To install the latest development version you may use:
git clone https://github.com/krassowski/nbpipeline
cd nbpipeline
pip install -r requirements.txt
ln -s $(pwd)/nbpipeline/nbpipeline.py ~/bin/nbpipelineQuickstart
Create pipeline.py file with list of rules for your pipeline. For example:
from nbpipeline.rules import NotebookRule
NotebookRule(
'Extract protein data', # a nice name for the step
input={'protein_data_path': 'data/raw/data_from_wetlab.xlsx'},
output={'output_path': 'data/clean/protein_levels.csv'},
notebook='analyses/Data_extraction.ipynb',
group='Proteomics' # this is optional
)
NotebookRule(
'Quality control and PCA on proteins',
input={'protein_levels_path': 'data/clean/protein_levels.csv'},
output={'qc_report_path': 'reports/proteins_failing_qc.csv'},
notebook='analyses/Exploration_and_quality_control.ipynb',
group='Proteomics'
)the keys of the input and output variables should correspond to variables in one of the first cells in the corresponding notebook, which should be tagged as “parameters”. It can be done easily in JupyterLab:
If you forget to add them, a warning will be displayed.
Alternativaly, you can create a dedicated cell for input paths definitions and tag it “inputs” and a separate one for output paths definitions, tagging it “outputs”, which allows to omit input and output keywords when creating a NotebookRule. However, only simple variable definitions will be deduced (parsing uses regular expressions to avoid potential dangers of eval).
For more details, please see the example pipeline and notebooks in the examples directory.
Run the pipeline:
nbpipelineOn any consecutive run the notebooks which did not change will not be run again. To disable this cache, use --disable_cache switch.
To generate an interactive diagram of the rules graph, together with reproducibility report add -i switch:
nbpipeline -iThe software defaults to google-chrome for graph visualization display, which can be changed with a CLI option.
If you named your definition files differently (e.g. my_rules.py instead of pipeline.py), use:
nbpipeline --definitions_file my_rules.pyTo display all command line options use:
nbpipeline -hTroubleshooting
If you see ModuleNotFoundError: No module named 'name_of_your_local_module', you may need to enforce the path, running nbpipeline with:
PYTHONPATH=/path/to/the/parent/of/local/module:$PYTHONPATH nbpipelineOftentimes the path is the same as the current directory, so the following command may work:
PYTHONPATH=$(pwd):$PYTHONPATH nbpipelineRelease history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file nbpipeline-0.2.1.tar.gz.
File metadata
- Download URL: nbpipeline-0.2.1.tar.gz
- Upload date:
- Size: 23.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.8.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ba862e5d9ba71c35da279070f1f5bb54f7dbced615dac11dbbb32e684b763d7b |
|
| MD5 | c9c3f629ed4a646f24136eb104e9bd56 |
|
| BLAKE2b-256 | 52fbf84e50256a1e9bb0306dd1daf879abc2065862ca7d6bf87e2f12056d75b1 |
