High Level Expressions for Dask
Project description
Dask Expressions
Dask DataFrames with query optimization.
This is a proof-of-concept rewrite of Dask DataFrame that includes query optimization and generally improved organization.
More in our blog posts:
Example
import dask_expr as dx
df = dx.datasets.timeseries()
df.head()
df.groupby("name").x.mean().compute()
Query Representation
Dask-expr encodes user code in an expression tree:
>>> df.x.mean().pprint()
Mean:
Projection: columns='x'
Timeseries: seed=1896674884
This expression tree will be optimized and modified before execution:
>>> df.x.mean().optimize().pprint()
Div:
Sum:
Fused(375f9):
| Projection: columns='x'
| Timeseries: dtypes={'x': <class 'float'>} seed=1896674884
Count:
Fused(375f9):
| Projection: columns='x'
| Timeseries: dtypes={'x': <class 'float'>} seed=1896674884
Stability
This project is a work in progress and will be changed without notice or deprecation warning. Please provide feedback, but it's best to avoid use in production settings.
API Coverage
dask_expr.DataFrame
absadd_prefixadd_sufixalignallanyapplyassignastypeclipcombine_firstcopycountdaskdropdrop_duplicatesdropnaevalexplodefillnagroupbyheadidxmaxidxminindexisinisnajoinmapmap_partitionsmaxmemory_usagemergeminminmodenlargestnsmallestnunique_approxpartitionspivot_tableprodrenamerename_axisrepartitionreplacereset_indexroundsamplesort_valuesselect_dtypesset_indexshufflestdsumtailto_parquetto_timestampvarvisualize
dask_expr.Series
absalignallanyapplyastypebetweenclipcombine_firstcopycountdaskdrop_duplicatesdropnaexplodefillnagroupbyheadidxmaxidxminindexisinisnamapmap_partitionsmaxmemory_usageminminmodenlargestnsmallestnunique_approxpartitionsprodrename_axisrepartitionreplacereset_indexroundshufflestdsumtailto_frameto_timestampuniquevalue_countsvarvisualize
dask_expr.Index
absalignallanyapplyastypeclipcombine_firstcopycountdaskfillnagroupbyheadidxmaxidxminindexisinisnamap_partitionsmaxmemory_usageminminmodenunique_approxpartitionsprodrename_axisrepartitionreplacereset_indexroundshufflestdsumtailto_frameto_timestampvarvisualize
dask_expr._groupby.GroupBy
aggaggregateapplycountfirstlastmaxmeanminprodsizestdsumtransformvalue_countsvar
Binary operators (DataFrame, Series, and Index):
__add____radd____sub____rsub____mul____rmul____truediv____rtruediv____lt____rlt____gt____rgt____le____rle____ge____rge____eq____ne____and____rand____or____ror____xor____rxor__
Unary operators (DataFrame, Series, and Index):
__invert____neg____pos__
Accessors:
CategoricalAccessorDatetimeAccessorStringAccessor
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dask-expr-0.2.0.tar.gz.
File metadata
- Download URL: dask-expr-0.2.0.tar.gz
- Upload date:
- Size: 98.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a31bbd15eb57c899ec9b9495c229a21ba58f518f38f46774f0a74d33d2ea211e |
|
| MD5 | 0355ed57e31fe3c7660f57de925a073f |
|
| BLAKE2b-256 | c4c6b4be6b6efebd315294c284b24a41ea10c75d753abb3c00f7a8f9b4410f16 |
Provenance
File details
Details for the file dask_expr-0.2.0-py3-none-any.whl.
File metadata
- Download URL: dask_expr-0.2.0-py3-none-any.whl
- Upload date:
- Size: 87.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7b703cf94c0743f596ac450758f02340ef59307b78d29da56132038e4b9fdf7c |
|
| MD5 | 364211ffcb55e6f7c3baf22375782b61 |
|
| BLAKE2b-256 | 8ef5437878dc42dbfa0319cf6bfe42c94fb3cf7dddd1e7f36b150f84313f8c97 |
