Skip to main content

Configuration for various dask clusters

Project description

dask-hpcconfig

To install, use

python -m pip install git+https://github.com/umr-lops/dask-hpcconfig.git#egg=dask-hpcconfig

or clone the source:

git clone https://github.com/umr-lops/dask-hpcconfig.git
cd dask-hpcconfig

and then install from there:

python -m pip install .

or as "editable":

python -m pip install -e .

Usage

import dask_hpcconfig

To list the available cluster definitions:

dask_hpcconfig.print_clusters()

or, as a mapping of name to type:

clusters = dask_hpcconfig.available_clusters()

To create a cluster, use:

cluster = dask_hpcconfig.cluster(name)

where name is the name of one of the available clusters.

To override any particular setting: For example on 'datarmor-local' to use only 7 workers for increasing memory size of each worker:

overrides = {"cluster.n_workers": 7}
cluster = dask_hpcconfig.cluster("datarmor-local", **overrides)

For example on 'datarmor' to use only 7 workers for increasing memory size of each worker, and use 49 workers (i.e. 7 mpi_1 nodes) :

overrides = {"cluster.cores": 7}
cluster = dask_hpcconfig.cluster("datarmor", **overrides)
cluster.scale(49)

cluster can then be used to create a Client:

from distributed import Client

client = Client(cluster)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask-hpcconfig-2022.3.0.tar.gz (11.2 kB view details)

Uploaded Source

Built Distribution

dask_hpcconfig-2022.3.0-py3-none-any.whl (7.3 kB view details)

Uploaded Python 3

File details

Details for the file dask-hpcconfig-2022.3.0.tar.gz.

File metadata

  • Download URL: dask-hpcconfig-2022.3.0.tar.gz
  • Upload date:
  • Size: 11.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/0.0.0 pkginfo/1.8.2 readme-renderer/27.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.4.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.9

File hashes

Hashes for dask-hpcconfig-2022.3.0.tar.gz
Algorithm Hash digest
SHA256 af5db3151f481bbcf06dd644a965a6387ec5a933da3f92735fa3bc5de33d13ca
MD5 005183c73164da4b0f3401a415a7d501
BLAKE2b-256 dd85290258d05620d59cd68c85324bc85916dd671dd806465454d41778044b85

See more details on using hashes here.

File details

Details for the file dask_hpcconfig-2022.3.0-py3-none-any.whl.

File metadata

  • Download URL: dask_hpcconfig-2022.3.0-py3-none-any.whl
  • Upload date:
  • Size: 7.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/0.0.0 pkginfo/1.8.2 readme-renderer/27.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.4.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.9

File hashes

Hashes for dask_hpcconfig-2022.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e80e123b364c5aad5c55abb5ea4a3f504525bb0b7f436df84be4a62d28c6d416
MD5 feb2faeb26742380d479372a06aacd95
BLAKE2b-256 b98adde3bdf98ab948b8614e155cbe491ddb87b69675538b1ebe2fc0f94bf2dd

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page