Run code on a dask worker via a context manager
Project description
Afar
One man's magic is another man's engineering
Robert A. Heinlein
afar allows you to run code on a remote Dask worker using context managers. For example:
import afar
with afar.run, remotely:
import dask_cudf
df = dask_cudf.read_parquet("s3://...")
result = df.sum().compute()
Outside the context, result is a Dask Future whose data resides on a worker. result.result() is necessary to copy the data locally.
By default, only the last assignment is saved. One can specify which variables to save:
with afar.run("a", "b"), remotely:
a = 1
b = a + 1
a and b are now both Futures. They can be used directly in other afar.run contexts:
with afar.run as data, remotely:
c = a + b
assert c.result() == 3
assert data["c"].result() == 3
data is a dictionary of variable names to Futures. It may be necessary at times to get the data from here.
For motivation, see https://github.com/dask/distributed/issues/4003
This code is highly experimental and magical!
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file afar-0.1.0.tar.gz.
File metadata
- Download URL: afar-0.1.0.tar.gz
- Upload date:
- Size: 22.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | c433a26015dd724f142b89a7ba1066e16086220567bd4c34a08ce2954d48dda4 |
|
| MD5 | fb1cd9e55318e05ed2123b033ad73fa0 |
|
| BLAKE2b-256 | 5cd366f4afae0ca68b08f54158e250f4e993dfa77758632fb4b16f79037b7d02 |
File details
Details for the file afar-0.1.0-py3-none-any.whl.
File metadata
- Download URL: afar-0.1.0-py3-none-any.whl
- Upload date:
- Size: 6.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.6.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.61.2 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 83dd3c141ccc84f689f8bbbbb34f469642ec858df76665864b29986e2accf16b |
|
| MD5 | 913c6a8e7968fb430e307a7f5b9c31f4 |
|
| BLAKE2b-256 | 2b48dba467b340b36af9d1bd15fb2a11f78bcccec6ba38dcc05bfb1681249d7d |
