Backend implementation for running MLFlow projects on Slurm
Project description
MLFlow-Slurm
Backend for executing MLFlow projects on Slurm batch system
Usage
Install this package in the environment from which you will be submitting jobs. If you are submitting jobs from inside jobs, make sure you have this package listed in your conda or pip environment.
Just list this as your --backend in the job run. You should include a json
config file to control how the batch script is constructed:
mlflow run --backend slurm \
--backend-config slurm_config.json \
examples/sklearn_elasticnet_wine
It will generate a batch script named after the job id and submit it via the
Slurm sbatch command. It will tag the run with the Slurm JobID
Configure Jobs
You can set values in a json file to control job submission. The supported properties in this file are:
| Config File Setting | Use |
|---|---|
| partition | Which Slurm partition should the job run in? |
| account | What account name to run under |
| gpus_per_node | On GPU partitions how many GPUs to allocate per node |
| gres | SLURM Generic RESources requests |
| mem | Amount of memory to allocate to CPU jobs |
| modules | List of modules to load before starting job |
| time | Max CPU time job may run |
| sbatch-script-file | Name of batch file to be produced. Leave blank to have service generate a script file name based on the run ID |
Development
The slurm docker deployment is handy for testing and development. You can start up a slurm environment with the included docker-compose file
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file mlflow_slurm-1.0.1.tar.gz.
File metadata
- Download URL: mlflow_slurm-1.0.1.tar.gz
- Upload date:
- Size: 6.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.18
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f153524a4b1e3775e6b2ec8677beaad252f9645701d63c0c1b0253481cc7986a |
|
| MD5 | 1dba9ae38707072fe87ec278c4f53cfc |
|
| BLAKE2b-256 | 1a5a04196c9113e5acc7ac975ffc57e7e274de1b46185b0fdb6e511e85eb2416 |
Provenance
File details
Details for the file mlflow_slurm-1.0.1-py3-none-any.whl.
File metadata
- Download URL: mlflow_slurm-1.0.1-py3-none-any.whl
- Upload date:
- Size: 8.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.18
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 779ae48c1fbc3c870551ce9ef8f73d4197bed416caaa322ff0da52c80b7317b4 |
|
| MD5 | 1b1bdacbf326641f4cdd9580a19f4590 |
|
| BLAKE2b-256 | c3c0e2c5f846d7105263c653a2b38cb4ac12ffbf5a38ba5f25bc929ba44b93f7 |
