Skip to main content

High Level Expressions for Dask

Project description

Dask Expressions

Dask DataFrames with query optimization.

This is a proof-of-concept rewrite of Dask DataFrame that includes query optimization and generally improved organization.

More in our blog posts:

Example

import dask_expr as dx

df = dx.datasets.timeseries()
df.head()

df.groupby("name").x.mean().compute()

Query Representation

Dask-expr encodes user code in an expression tree:

>>> df.x.mean().pprint()

Mean:
  Projection: columns='x'
    Timeseries: seed=1896674884

This expression tree will be optimized and modified before execution:

>>> df.x.mean().optimize().pprint()

Div:
  Sum:
    Fused(375f9):
    | Projection: columns='x'
    |   Timeseries: dtypes={'x': <class 'float'>} seed=1896674884
  Count:
    Fused(375f9):
    | Projection: columns='x'
    |   Timeseries: dtypes={'x': <class 'float'>} seed=1896674884

Stability

This project is a work in progress and will be changed without notice or deprecation warning. Please provide feedback, but it's best to avoid use in production settings.

API Coverage

dask_expr.DataFrame

  • abs
  • add_prefix
  • add_sufix
  • align
  • all
  • any
  • apply
  • assign
  • astype
  • clip
  • combine_first
  • copy
  • count
  • dask
  • drop
  • drop_duplicates
  • dropna
  • eval
  • explode
  • fillna
  • groupby
  • head
  • idxmax
  • idxmin
  • index
  • isin
  • isna
  • join
  • map
  • map_partitions
  • max
  • memory_usage
  • merge
  • min
  • min
  • mode
  • nlargest
  • nsmallest
  • nunique_approx
  • partitions
  • pivot_table
  • prod
  • rename
  • rename_axis
  • repartition
  • replace
  • reset_index
  • round
  • sample
  • sort_values
  • select_dtypes
  • set_index
  • shuffle
  • std
  • sum
  • tail
  • to_parquet
  • to_timestamp
  • var
  • visualize

dask_expr.Series

  • abs
  • align
  • all
  • any
  • apply
  • astype
  • between
  • clip
  • combine_first
  • copy
  • count
  • dask
  • drop_duplicates
  • dropna
  • explode
  • fillna
  • groupby
  • head
  • idxmax
  • idxmin
  • index
  • isin
  • isna
  • map
  • map_partitions
  • max
  • memory_usage
  • min
  • min
  • mode
  • nlargest
  • nsmallest
  • nunique_approx
  • partitions
  • prod
  • rename_axis
  • repartition
  • replace
  • reset_index
  • round
  • shuffle
  • std
  • sum
  • tail
  • to_frame
  • to_timestamp
  • unique
  • value_counts
  • var
  • visualize

dask_expr.Index

  • abs
  • align
  • all
  • any
  • apply
  • astype
  • clip
  • combine_first
  • copy
  • count
  • dask
  • fillna
  • groupby
  • head
  • idxmax
  • idxmin
  • index
  • isin
  • isna
  • map_partitions
  • max
  • memory_usage
  • min
  • min
  • mode
  • nunique_approx
  • partitions
  • prod
  • rename_axis
  • repartition
  • replace
  • reset_index
  • round
  • shuffle
  • std
  • sum
  • tail
  • to_frame
  • to_timestamp
  • var
  • visualize

dask_expr._groupby.GroupBy

  • agg
  • aggregate
  • apply
  • count
  • first
  • last
  • max
  • mean
  • min
  • prod
  • shift
  • size
  • std
  • sum
  • transform
  • value_counts
  • var

dask_expr._resample.Resampler

  • agg
  • count
  • first
  • last
  • max
  • mean
  • median
  • min
  • nunique
  • ohlc
  • prod
  • quantile
  • sem
  • size
  • std
  • sum
  • var

Binary operators (DataFrame, Series, and Index):

  • __add__
  • __radd__
  • __sub__
  • __rsub__
  • __mul__
  • __rmul__
  • __truediv__
  • __rtruediv__
  • __lt__
  • __rlt__
  • __gt__
  • __rgt__
  • __le__
  • __rle__
  • __ge__
  • __rge__
  • __eq__
  • __ne__
  • __and__
  • __rand__
  • __or__
  • __ror__
  • __xor__
  • __rxor__

Unary operators (DataFrame, Series, and Index):

  • __invert__
  • __neg__
  • __pos__

Accessors:

  • CategoricalAccessor
  • DatetimeAccessor
  • StringAccessor

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask-expr-0.2.2.tar.gz (99.7 kB view details)

Uploaded Source

Built Distribution

dask_expr-0.2.2-py3-none-any.whl (88.6 kB view details)

Uploaded Python 3

File details

Details for the file dask-expr-0.2.2.tar.gz.

File metadata

  • Download URL: dask-expr-0.2.2.tar.gz
  • Upload date:
  • Size: 99.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for dask-expr-0.2.2.tar.gz
Algorithm Hash digest
SHA256 27aef52a90fc3ecc40995f5109ad069f230aaa9dd9751fae36dcd10b7e919ffa
MD5 ad61c97ffa7ec02f6a05369fe7f2b599
BLAKE2b-256 c2fa69c0dbba923180dae77e7f5eb361a2a622c804d401fe23b015de16c16ab4

See more details on using hashes here.

Provenance

File details

Details for the file dask_expr-0.2.2-py3-none-any.whl.

File metadata

  • Download URL: dask_expr-0.2.2-py3-none-any.whl
  • Upload date:
  • Size: 88.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for dask_expr-0.2.2-py3-none-any.whl
Algorithm Hash digest
SHA256 c6c83f3cf6118b41abb32d2680d8e58407cdb687f0e31deb8c92154bd0d9d54f
MD5 c4c94e16abea8b5c60ca4e69c0dd1c31
BLAKE2b-256 9df9345de55f54dac8119deb40e7ca8c6511692fe82bbeee6e03cc5e2afc136a

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page