experimental Dask array that opens/closes a resource when computing
Project description
resource-backed-dask-array
ResourceBackedDaskArray
is an experimental Dask array subclass
that opens/closes a resource when computing (but only once per compute call).
installation
pip install resource-backed-dask-array
motivation for this package
Consider the following class that simulates a file reader capable of returning a dask array (using dask.array.map_blocks) The file handle must be in an open state in order to read a chunk, otherwise it segfaults (or otherwise errors)
import dask.array as da
import numpy as np
class FileReader:
def __init__(self):
self._closed = False
def close(self):
"""close the imaginary file"""
self._closed = True
@property
def closed(self):
return self._closed
def __enter__(self):
if self.closed:
self._closed = False # open
return self
def __exit__(self, *_):
self.close()
def to_dask(self) -> da.Array:
"""Method that returns a dask array for this file."""
return da.map_blocks(
self._dask_block,
chunks=((1,) * 4, 4, 4),
dtype=float,
)
def _dask_block(self):
"""simulate getting a single chunk of the file."""
if self.closed:
raise RuntimeError("Segfault!")
return np.random.rand(1, 4, 4)
As long as the file stays open, everything works fine:
>>> fr = FileReader()
>>> dsk_ary = fr.to_dask()
>>> dsk_ary.compute().shape
(4, 4, 4)
However, if one closes the file, the dask array returned
from to_dask
will now fail:
>>> fr.close()
>>> dsk_ary.compute() # RuntimeError: Segfault!
A "quick-and-dirty" solution here might be to force the _dask_block
method to
temporarily reopen the file if it finds the file in the closed state, but if the
file-open process takes any amount of time, this could incur significant
overhead as it opens-and-closes for every chunk in the array.
usage
ResourceBackedDaskArray.from_array
This library attempts to provide a solution to the above problem with a
ResourceBackedDaskArray
object. This manages the opening/closing of
an underlying resource whenever .compute()
is called – and does so only once for all chunks in a single compute task graph.
>>> from resource_backed_dask_array import resource_backed_dask_array
>>> safe_dsk_ary = resource_backed_dask_array(dsk_ary, fr)
>>> safe_dsk_ary.compute().shape
(4, 4, 4)
>>> fr.closed # leave it as we found it
True
The second argument passed to from_array
must be a resuable context manager
that additionally provides a closed
attribute (like io.IOBase). In other words,
it must implement the following protocol:
- it must have an
__enter__
method that opens the underlying resource - it must have an
__exit__
method that closes the resource and optionally handles exceptions - it must have a
closed
attribute that reports whether or not the resource is closed.
In the example above, the FileReader
class itself implemented this protocol, and so was suitable as the second argument to ResourceBackedDaskArray.from_array
above.
Important Caveats
This was created for single-process (and maybe just single-threaded?)
use cases where dask's out-of-core lazy loading is still very desireable. Usage
with dask.distributed
is untested and may very well fail. Using stateful objects (such as the reusable context manager used here) in multi-threaded/processed tasks is error prone.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file resource_backed_dask_array-0.1.0.tar.gz
.
File metadata
- Download URL: resource_backed_dask_array-0.1.0.tar.gz
- Upload date:
- Size: 10.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8fabcccf5c7e29059b5badd6786dd7675a258a203c58babf10077d9c90ada54f |
|
MD5 | dea76680ec59e13b0bc9b3df93bbf65c |
|
BLAKE2b-256 | 6280b8952048ae1772d33b95dbf7d7107cf364c037cc229a2690fc8fa9ee8e48 |
Provenance
File details
Details for the file resource_backed_dask_array-0.1.0-py2.py3-none-any.whl
.
File metadata
- Download URL: resource_backed_dask_array-0.1.0-py2.py3-none-any.whl
- Upload date:
- Size: 8.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec457fa72d81f0340a67ea6557a5a5919323a11cccc978a950df29fa69fe5679 |
|
MD5 | 3da22ac0ac7f3d70d1d87639aa399a46 |
|
BLAKE2b-256 | 0db5852f619e53fa7fb70d8915fcae66632df3958cac7e926c4ac38458958674 |