Skip to main content

High Level Expressions for Dask

Project description

Dask Expressions

Dask Dataframes with query optimization.

This is a proof-of-concept rewrite of Dask dataframe that includes query optimization and generally improved organization.

Example

import dask_expr as dx

df = dx.datasets.timeseries()
df.head()

df.groupby("name").x.mean().compute()

Query Representation

Dask-expr encodes user code in an expression tree:

>>> df.x.mean().pprint()

Mean:
  Projection: columns='x'
    Timeseries: seed=1896674884

This expression tree will be optimized and modified before execution:

>>> df.x.mean().optimize().pprint()

Div:
  Sum:
    Fused(375f9):
    | Projection: columns='x'
    |   Timeseries: dtypes={'x': <class 'float'>} seed=1896674884
  Count:
    Fused(375f9):
    | Projection: columns='x'
    |   Timeseries: dtypes={'x': <class 'float'>} seed=1896674884

Stability

This project is a work in progress and will be changed without notice or deprecation warning. Please provide feedback, but it's best to avoid use in production settings.

API Coverage

dask_expr.DataFrame

  • abs
  • add_prefix
  • add_sufix
  • align
  • all
  • any
  • apply
  • assign
  • astype
  • clip
  • combine_first
  • copy
  • count
  • dask
  • drop
  • drop_duplicates
  • dropna
  • eval
  • explode
  • fillna
  • groupby
  • head
  • idxmax
  • idxmin
  • index
  • isin
  • isna
  • join
  • map
  • map_partitions
  • max
  • memory_usage
  • merge
  • min
  • min
  • mode
  • nlargest
  • nsmallest
  • nunique_approx
  • partitions
  • pivot_table
  • prod
  • rename
  • rename_axis
  • repartition
  • replace
  • reset_index
  • round
  • sample
  • sort_values
  • select_dtypes
  • set_index
  • shuffle
  • std
  • sum
  • tail
  • to_parquet
  • to_timestamp
  • var
  • visualize

dask_expr.Series

  • abs
  • align
  • all
  • any
  • apply
  • astype
  • between
  • clip
  • combine_first
  • copy
  • count
  • dask
  • drop_duplicates
  • dropna
  • explode
  • fillna
  • groupby
  • head
  • idxmax
  • idxmin
  • index
  • isin
  • isna
  • map
  • map_partitions
  • max
  • memory_usage
  • min
  • min
  • mode
  • nlargest
  • nsmallest
  • nunique_approx
  • partitions
  • prod
  • rename_axis
  • repartition
  • replace
  • reset_index
  • round
  • shuffle
  • std
  • sum
  • tail
  • to_frame
  • to_timestamp
  • unique
  • value_counts
  • var
  • visualize

dask_expr.Index

  • abs
  • align
  • all
  • any
  • apply
  • astype
  • clip
  • combine_first
  • copy
  • count
  • dask
  • fillna
  • groupby
  • head
  • idxmax
  • idxmin
  • index
  • isin
  • isna
  • map_partitions
  • max
  • memory_usage
  • min
  • min
  • mode
  • nunique_approx
  • partitions
  • prod
  • rename_axis
  • repartition
  • replace
  • reset_index
  • round
  • shuffle
  • std
  • sum
  • tail
  • to_frame
  • to_timestamp
  • var
  • visualize

dask_expr._groupby.GroupBy

  • agg
  • aggregate
  • count
  • first
  • last
  • max
  • mean
  • min
  • prod
  • size
  • std
  • sum
  • value_counts
  • var

Binary operators (DataFrame, Series, and Index):

  • __add__
  • __radd__
  • __sub__
  • __rsub__
  • __mul__
  • __rmul__
  • __truediv__
  • __rtruediv__
  • __lt__
  • __rlt__
  • __gt__
  • __rgt__
  • __le__
  • __rle__
  • __ge__
  • __rge__
  • __eq__
  • __ne__
  • __and__
  • __rand__
  • __or__
  • __ror__
  • __xor__
  • __rxor__

Unary operators (DataFrame, Series, and Index):

  • __invert__
  • __neg__
  • __pos__

Accessors:

  • CategoricalAccessor
  • DatetimeAccessor
  • StringAccessor

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask-expr-0.1.10.tar.gz (93.4 kB view details)

Uploaded Source

Built Distribution

dask_expr-0.1.10-py3-none-any.whl (82.1 kB view details)

Uploaded Python 3

File details

Details for the file dask-expr-0.1.10.tar.gz.

File metadata

  • Download URL: dask-expr-0.1.10.tar.gz
  • Upload date:
  • Size: 93.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for dask-expr-0.1.10.tar.gz
Algorithm Hash digest
SHA256 986bba01b7e0140d5f74b27604b03d58ddf0cd49a909bd67b7c19ef7f89a5ba3
MD5 596abaf560323159028f74b5c1d281f7
BLAKE2b-256 b0bc82ca7153084f7601c380c9994fac16370441f87ad8ef3cb2c42987973a8a

See more details on using hashes here.

Provenance

File details

Details for the file dask_expr-0.1.10-py3-none-any.whl.

File metadata

  • Download URL: dask_expr-0.1.10-py3-none-any.whl
  • Upload date:
  • Size: 82.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for dask_expr-0.1.10-py3-none-any.whl
Algorithm Hash digest
SHA256 18e27771056b49d9ad1df08c5d86c5f3260d8500f4418a8274d0faef3fb0722c
MD5 18c006ceedde5dd471b90e259b3ed04c
BLAKE2b-256 226009c8d94280d665b7134cafa5cc64e816c975fe13f7ba87d59f1c4c7eb949

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page