Skip to main content

Reproducible machine learning pipelines using mlflow.

Project description

mlf-core logo

mlf-core

Github Workflow Build mlf-core Status Github Workflow Tests Status PyPI Discord Documentation Status Dependabot Enabled

Fully GPU deterministic machine learning project templates using MLflow.

mlf-core create gif

Features

  • Jumpstart your machine learning project with fully fledged, multi GPU enabled mlflow project templates

  • Pytorch, Tensorflow, XGBoost supported

  • mlflow templates are fully GPU deterministic with system-intelligence

  • Conda and Docker support out of the box

mlf-core summary

mlf-core enables deterministic machine learning. MLflow and a provided Read the Docs setup ensure that all hyperparameters, metrics and model details are well documented. Reproducible environments are provided with the use of Conda and Docker. Finally, the mlf-core ecosystem ensures that all library specific settings required for determinism are enabled, no non-deterministic algorithms are used and that the used hardware is tracked.

Credits

Primary idea and main development by Lukas Heumos. This package was created with cookietemple based on a modified audreyr/cookiecutter-pypackage project template using Cookiecutter.

Changelog

This project adheres to Semantic Versioning.

1.7.6 (2020-11-22)

Added

Fixed

  • Github project creation support due to Github’s new main branch

Dependencies

Deprecated

1.7.5 (2020-11-18)

Added

Fixed

sync workflow set-env

Dependencies

Deprecated

1.7.4 (2020-11-11)

Added

Fixed

  • Sync now compares against the development branch and not the master branch.

Dependencies

Deprecated

1.7.3 (2020-11-09)

Added

Fixed

  • Added CHANGELOG.rst to blacklisted files

Dependencies

Deprecated

1.7.2 (2020-11-07)

Added

Fixed

  • Removed redundant print in xgboost

Dependencies

Deprecated

1.7.1 (2020-11-07)

Added

Fixed

  • mlf-core sync does now correctly find attributes

Dependencies

Deprecated

1.7.0 (2020-11-06)

Added

  • fix-artifact-paths which replaces the artifact paths with the paths of the current system

  • More structured documentation

Fixed

  • Now using GPUs by default only when GPUs are available for XGBoost templates

Dependencies

Deprecated

1.6.1 (2020-11-06)

Added

  • Workflows for package-prediction

  • Documentation for package-prediction

Fixed

Dependencies

Deprecated

1.6.0 (2020-11-02)

Added

  • New package templates (package-prediction) for Pytorch, Tensorflow and XGBoost

Fixed

Dependencies

Deprecated

1.5.0 (2020-10-29)

Added

  • Check for non-deterministic functions for mlflow-tensorflow linter

  • Check for all_reduce for mlflow-xgboost templates

  • Check for OS for system-intelligence runs. If not Linux -> don’t run system-intelligence

  • .gitattributes to templates, which ignores mlruns

  • Documentation on creating releases

Fixed

  • Sync now operates correctly with the correct PR URL

Dependencies

Deprecated

1.4.4 (2020-10-22)

Added

Fixed

  • Conda report generation

Dependencies

Deprecated

1.4.3 (2020-09-17)

Added

Fixed

  • Internal Github workflows

  • Docker documentation

Dependencies

Deprecated

1.4.2 (2020-09-11)

Added

Fixed

  • Accidentally left a - in the train_cpu.yml of mlflow-pytorch

  • mlflow-pytorch and mlflow-tensorflow now only train for 2 epochs on train_cpu.yml

Dependencies

Deprecated

1.4.1 (2020-09-10)

Added

Fixed

  • Github username must now always be lowercase, since Docker does not like uppercase letters

  • Fixed train_cpu workflows to use the correct containers

Dependencies

Deprecated

1.4.0 (2020-08-28)

Added

  • model.rst documentation for all templates

  • added support for verbose output

Fixed

  • Publish Docker workflows now use the new Github registry

  • Default Docker container names are now `image: ghcr.io/{{ cookiecutter.github_username }}/{{ cookiecutter.project_slug_no_hyphen }}:{{ cookiecutter.version }}`

Dependencies

Deprecated

1.3.0 (2020-08-27)

Added

  • automatically mounting /data now in all mlflow templates (#56)

  • mlflow-xgboost xgboost from 1.1.1 to 1.2.0

Fixed

  • mlf_core.py now uses project_slug; adapted linter accordingly (#55)

  • Removed dask-cuda from mlflow-xgboost

Dependencies

Deprecated

1.2.2 (2020-08-21)

Added

Fixed

  • A couple of parameters were not with hyphen -> now default behavior

Dependencies

Deprecated

1.2.1 (2020-08-21)

Added

Fixed

  • flake8 for mlflow-pytorch

Dependencies

Deprecated

1.2.0 (2020-08-21)

Added

  • Option –view to config to view the current configuration

  • Option –set_token to sync to set the sync token again

Fixed

Dependencies

Deprecated

1.1.0 (2020-08-19)

Added

  • Publish Docker workflow. Publishes to Github Packages per default, but can be configured.

  • Linting function, which checks mlflow-pytorch for any used atomic_add functions.

  • system-intelligence 1.2.2 -> 1.2.3

  • Support for both, MLF-CORE TODO: and TODO MLF-CORE: statements

Fixed

  • Default project version from 0.1.0 to 0.1.0-SNAPSHOT.

  • Outdated screenshots

  • Nightly versions now warn instead of wrongly complaining about outdated versions.

  • Sync actor, but not yet completely for organizations

  • A LOT of documentation

  • Now using project_slug_no_hyphen to facilitate the creation of repositories with - characters.

  • Removed boston dataset from XGBoost and XGBoost_dask

  • Renamed all parameters to use hyphens instead of underscores

Dependencies

Deprecated

1.0.1 (2020-08-11)

Added

Fixed

  • Sync workflow now uses the correct secret

Dependencies

Deprecated

1.0.0 (2020-08-11)

Added

  • Created the project using cookietemple

  • Added all major commands: create, list, info, lint, sync, bump-version, config, upgrade

  • Added mlflow-pytorch, mlflow-tensorflow, mlflow-xgboost, mlflow-xgboost_dask templates

Fixed

Dependencies

Deprecated

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mlf-core-1.7.6.tar.gz (1.5 MB view details)

Uploaded Source

Built Distribution

mlf_core-1.7.6-py2.py3-none-any.whl (6.0 MB view details)

Uploaded Python 2 Python 3

File details

Details for the file mlf-core-1.7.6.tar.gz.

File metadata

  • Download URL: mlf-core-1.7.6.tar.gz
  • Upload date:
  • Size: 1.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.53.0 CPython/3.8.6

File hashes

Hashes for mlf-core-1.7.6.tar.gz
Algorithm Hash digest
SHA256 645116514eb08c6f03283c95e1647a35f73e71d95354de786356a79f44304cfd
MD5 05cb07a739aee9ab2f7876b8acf9307e
BLAKE2b-256 4a1ff0dcd2b164ad03f8323db8f4180fbd746b2c2f1d73e1722ce5bcee4da8ac

See more details on using hashes here.

File details

Details for the file mlf_core-1.7.6-py2.py3-none-any.whl.

File metadata

  • Download URL: mlf_core-1.7.6-py2.py3-none-any.whl
  • Upload date:
  • Size: 6.0 MB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.53.0 CPython/3.8.6

File hashes

Hashes for mlf_core-1.7.6-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 8c75c4f85e747322941c287181ac24138f127dce92ab52edc271493855f48704
MD5 b5ef414633def3742a5a95aaa0c83260
BLAKE2b-256 49dc955ca405a4ccea6ee74e56c5df3bc440f5753776eb0c574431041ff48875

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page