High Level Expressions for Dask
Project description
Dask Expressions
Dask DataFrames with query optimization.
This is a proof-of-concept rewrite of Dask DataFrame that includes query optimization and generally improved organization.
More in our blog posts:
Example
import dask_expr as dx
df = dx.datasets.timeseries()
df.head()
df.groupby("name").x.mean().compute()
Query Representation
Dask-expr encodes user code in an expression tree:
>>> df.x.mean().pprint()
Mean:
Projection: columns='x'
Timeseries: seed=1896674884
This expression tree will be optimized and modified before execution:
>>> df.x.mean().optimize().pprint()
Div:
Sum:
Fused(375f9):
| Projection: columns='x'
| Timeseries: dtypes={'x': <class 'float'>} seed=1896674884
Count:
Fused(375f9):
| Projection: columns='x'
| Timeseries: dtypes={'x': <class 'float'>} seed=1896674884
Stability
This project is a work in progress and will be changed without notice or deprecation warning. Please provide feedback, but it's best to avoid use in production settings.
API Coverage
dask_expr.DataFrame
absaddadd_prefixadd_sufixalignallanyapplyassignastypebfillclipcombine_firstcopycountcummaxcummincumprodcumsumdaskdivdividedropdrop_duplicatesdropnadtypesevalexplodeffillfillnafloordivgroupbyheadidxmaxidxminìlocindexisinisnajoinmapmap_overlapmap_partitionsmaskmaxmeanmemory_usagememory_usage_per_partitionmergeminminmodmodemulnlargestnsmallestnunique_approxpartitionspivot_tablepowprodqueryraddrdivrenamerename_axisrepartitionreplacereset_indexrfloordivrmodrmulroundrpowrsubrtruedivsampleselect_dtypesset_indexshiftshufflesort_valuesstdsubsumtailto_parquetto_timestamptruedivvarvisualizewhere
dask_expr.Series
absaddalignallanyapplyastypebetweenbfillclipcombine_firstcopycountcummaxcummincumprodcumsumdaskdivdividedrop_duplicatesdropnadtypeexplodeffillfillnafloordivgroupbyheadidxmaxidxminindexisinisnamapmap_partitionsmaskmaxmeanmemory_usagememory_usage_per_partitionminminmodmodemulnlargestnsmallestnunique_approxpartitionspowprodproductraddrdivrenamerename_axisrepartitionreplacereset_indexrfloordivrmodrmulroundrpowrsubrtruedivshiftshufflestdsubsumtailto_frameto_timestamptruedivuniquevalue_countsvarvisualizewhere
dask_expr.Index
absalignallanyapplyastypeclipcombine_firstcopycountdaskdtypefillnagroupbyheadidxmaxidxminindexisinisnamap_partitionsmaxmemory_usageminminmodenunique_approxpartitionsprodrenamerename_axisrepartitionreplacereset_indexroundshufflestdsumtailto_frameto_timestampvarvisualize
dask_expr._groupby.GroupBy
aggaggregateapply- `bfill
countffillfirstlastmaxmeanmedianminnuniqueprodshiftsizestdsumtransformvalue_countsvar
Support for SeriesGroupBy and DataFrameGroupBy.
dask_expr._resample.Resampler
aggcountfirstlastmaxmeanmedianminnuniqueohlcprodquantilesemsizestdsumvar
dask_expr._rolling.Rolling
aggapplycountmaxmeanmedianminquantilestdsumvarskewkurt
Binary operators (DataFrame, Series, and Index):
__add____radd____sub____rsub____mul____pow____rmul____truediv____rtruediv____lt____rlt____gt____rgt____le____rle____ge____rge____eq____ne____and____rand____or____ror____xor____rxor__
Unary operators (DataFrame, Series, and Index):
__invert____neg____pos__
Accessors:
CategoricalAccessorDatetimeAccessorStringAccessor
Function
concatfrom_pandasmergepivot_tableread_csvread_parquetrepartitionto_datetimeto_numericto_timedeltato_parquet
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file dask-expr-0.3.0.tar.gz.
File metadata
- Download URL: dask-expr-0.3.0.tar.gz
- Upload date:
- Size: 116.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fe5d62b076d92d6741ba3532fa9199ef1f815946573423171f36c0f02ea9e8cf |
|
| MD5 | 6521f6a130390de12520f843337120c9 |
|
| BLAKE2b-256 | 297d1eccf099448a722d054d3714c13e72a9fe846801a4cf617446221739db5c |
Provenance
File details
Details for the file dask_expr-0.3.0-py3-none-any.whl.
File metadata
- Download URL: dask_expr-0.3.0-py3-none-any.whl
- Upload date:
- Size: 108.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.12.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ad8ff3260b623246cd44bc12c05f025fbe3bf68543bf442cc1da95785293dd06 |
|
| MD5 | 066e60d80b44b6169de01eb03dc79fdc |
|
| BLAKE2b-256 | 73fe92dd2d5dfb99e235003709e87c6cbffc6c4ea954a8b2486242d9bb918356 |
