Utilities for expanding dask-jobqueue with appropriate settings for NCAR's clusters
Project description
ncar-jobqueue
Badges
CI | |
---|---|
Docs | |
Package | |
License |
Utilities for expanding dask-jobqueue with appropriate settings for NCAR's clusters.
Supported clusters:
- Cheyenne
- Casper (DAV)
- CGD's Hobart
- CGD's Izumi
Installation
NCAR-jobqueue can be installed from PyPI with pip:
python -m pip install ncar-jobqueue
(COMING SOON) NCAR-jobqueue will also be available from conda-forge for conda installations:
conda install -c conda-forge ncar-jobqueue
Usage
Casper
>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=2, memory=50.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)
Cheyenne
>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=72, memory=218.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)
Hobart
>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=96, memory=192.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)
Izumi
>>> from ncar_jobqueue import NCARCluster
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster(cores=0, memory=0 B, workers=0/0, jobs=0/0)
>>> cluster.scale(jobs=2)
>>> cluster
NCARCluster(cores=96, memory=192.00 GB, workers=2/2, jobs=2/2)
>>> client = Client(cluster)
Non-NCAR machines
On non-NCAR machines, ncar-jobqueue
will warn the user, and it will use distributed.LocalCluster
:
>>> from ncar_jobqueue import NCARCluster
.../ncar_jobqueue/cluster.py:42: UserWarning: Unable to determine which NCAR cluster you are running on... Returning a `distributed.LocalCluster` class.
warn(message)
>>> from dask.distributed import Client
>>> cluster = NCARCluster()
>>> cluster
NCARCluster('tcp://127.0.0.1:49334', workers=4, threads=8, memory=17.18 GB)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
ncar-jobqueue-2020.11.18.tar.gz
(14.7 kB
view hashes)
Built Distribution
Close
Hashes for ncar_jobqueue-2020.11.18-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8cbe35894eab4c96cb1d565ac7b72a71276f9688074b37727205a191af550508 |
|
MD5 | 1a836bf5e4372a78e9e27cdc2bf78146 |
|
BLAKE2b-256 | 4159f1c52c314d7b1ea3c043dd7fed60c4ba0ed33d2bbb80483ac25839be26a1 |