Deploy Dask on job queuing systems like PBS or SLURM
Project description
This helps to deploy Dask on batch-style job schedulers like PBS and SLURM.
Example
from dask_jobqueue import PBSCluster
cluster = PBSCluster(processes=6, threads=4, memory="16GB")
cluster.start_workers(10)
from dask.distributed import Client
client = Client(cluster)Adaptivity
This can also adapt the cluster size dynamically based on current load. This helps to scale up the cluster when necessary but scale it down and save resources when not actively computing.
cluster.adapt()History
This package came out of the Pangeo collaboration and was copy-pasted from a live repository at this commit. Unfortunately, development history was not preserved.
Original developers include the following:
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dask-jobqueue-0.1.0.tar.gz
(23.9 kB
view details)
Built Distribution
File details
Details for the file dask-jobqueue-0.1.0.tar.gz.
File metadata
- Download URL: dask-jobqueue-0.1.0.tar.gz
- Upload date:
- Size: 23.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e11d4d8baf72a31864b9745467f19d83e2c0cda6386e79003de0d221f0283c76 |
|
| MD5 | 40f51db7ec56ad037466c7fa7d179b0f |
|
| BLAKE2b-256 | 010bdaa1abb1b3244e788acb94586b0b9a35953f62d6f2b1ccfb041a97a53c99 |
Provenance
File details
Details for the file dask_jobqueue-0.1.0-py3-none-any.whl.
File metadata
- Download URL: dask_jobqueue-0.1.0-py3-none-any.whl
- Upload date:
- Size: 11.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 939ea92a8f43c628a5cfeef866b3537c8524014ea3e6568454fa1c84ba8f6638 |
|
| MD5 | bc8eb7339e130c17854b73d8eae5dc63 |
|
| BLAKE2b-256 | 83d1acfa164d8747f68b8281c9f96851d2dbab0b847333fac8ccca0368cd0d20 |
