GroupBy operations for dask.array
Project description
flox
This project explores strategies for fast GroupBy reductions with dask.array. It used to be called dask_groupby
It was motivated by
(See a presentation about this package, from the Pangeo Showcase).
Acknowledgements
This work was funded in part by NASA-ACCESS 80NSSC18M0156 "Community tools for analysis of NASA Earth Observing System Data in the Cloud" (PI J. Hamman), and NCAR's Earth System Data Science Initiative. It was motivated by very very many discussions in the Pangeo community.
API
There are two main functions
flox.groupby_reduce(dask_array, by_dask_array, "mean")"pure" dask array interfaceflox.xarray.xarray_reduce(xarray_object, by_dataarray, "mean")"pure" xarray interface; though work is ongoing to integrate this package in xarray.
Implementation
See the documentation for details on the implementation.
Custom reductions
flox implements all common reductions provided by numpy_groupies in aggregations.py.
It also allows you to specify a custom Aggregation (again inspired by dask.dataframe),
though this might not be fully functional at the moment. See aggregations.py for examples.
mean = Aggregation(
# name used for dask tasks
name="mean",
# operation to use for pure-numpy inputs
numpy="mean",
# blockwise reduction
chunk=("sum", "count"),
# combine intermediate results: sum the sums, sum the counts
combine=("sum", "sum"),
# generate final result as sum / count
finalize=lambda sum_, count: sum_ / count,
# Used when "reindexing" at combine-time
fill_value=0,
# Used when any member of `expected_groups` is not found
final_fill_value=np.nan,
)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file flox-0.4.0.tar.gz.
File metadata
- Download URL: flox-0.4.0.tar.gz
- Upload date:
- Size: 200.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6672a706a63825e4322cdbfd3e9887bf9bb9ebe9ebd6528692d44a4187c661af |
|
| MD5 | 5c1d0bc7fc210337955ce2e5898a1e9d |
|
| BLAKE2b-256 | fc564508d3639c577c2c54a8296fced4af2c1177b82d9b319c491915c1fdb7e3 |
File details
Details for the file flox-0.4.0-py3-none-any.whl.
File metadata
- Download URL: flox-0.4.0-py3-none-any.whl
- Upload date:
- Size: 53.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 08d4e66062cb8c7b5c5617f05139e6b2c0ec946231bc168dc2f8313e7f2bbe41 |
|
| MD5 | d20549d7af275ae62c00b39fb63e215d |
|
| BLAKE2b-256 | ba37162136fed85989702566a07cbd97c8d653a6005e6a2358c153038ad6750c |
