Configuration for various dask clusters
Project description
dask-hpcconfig
To install, use
python -m pip install git+https://github.com/umr-lops/dask-hpcconfig.git#egg=dask-hpcconfig
or clone the source:
git clone https://github.com/umr-lops/dask-hpcconfig.git
cd dask-hpcconfig
and then install from there:
python -m pip install .
or as "editable":
python -m pip install -e .
Usage
import dask_hpcconfig
To list the available cluster definitions:
dask_hpcconfig.print_clusters()
or, as a mapping of name to type:
clusters = dask_hpcconfig.available_clusters()
To create a cluster, use:
cluster = dask_hpcconfig.cluster(name)
where name
is the name of one of the available clusters.
To override any particular setting: For example on 'datarmor-local' to use only 7 workers for increasing memory size of each worker:
overrides = {"cluster.n_workers": 7}
cluster = dask_hpcconfig.cluster("datarmor-local", **overrides)
For example on 'datarmor' to use only 7 workers for increasing memory size of each worker, and use 49 workers (i.e. 7 mpi_1 nodes) :
overrides = {"cluster.cores": 7}
cluster = dask_hpcconfig.cluster("datarmor", **overrides)
cluster.scale(49)
cluster
can then be used to create a Client
:
from distributed import Client
client = Client(cluster)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dask-hpcconfig-2022.3.0.tar.gz
.
File metadata
- Download URL: dask-hpcconfig-2022.3.0.tar.gz
- Upload date:
- Size: 11.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/0.0.0 pkginfo/1.8.2 readme-renderer/27.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.4.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | af5db3151f481bbcf06dd644a965a6387ec5a933da3f92735fa3bc5de33d13ca |
|
MD5 | 005183c73164da4b0f3401a415a7d501 |
|
BLAKE2b-256 | dd85290258d05620d59cd68c85324bc85916dd671dd806465454d41778044b85 |
File details
Details for the file dask_hpcconfig-2022.3.0-py3-none-any.whl
.
File metadata
- Download URL: dask_hpcconfig-2022.3.0-py3-none-any.whl
- Upload date:
- Size: 7.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/0.0.0 pkginfo/1.8.2 readme-renderer/27.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.4.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e80e123b364c5aad5c55abb5ea4a3f504525bb0b7f436df84be4a62d28c6d416 |
|
MD5 | feb2faeb26742380d479372a06aacd95 |
|
BLAKE2b-256 | b98adde3bdf98ab948b8614e155cbe491ddb87b69675538b1ebe2fc0f94bf2dd |