Skip to main content

Utilities for Dask and cuDF interactions

Project description

 Dask cuDF - A GPU Backend for Dask DataFrame

Dask cuDF (a.k.a. dask-cudf or dask_cudf) is an extension library for Dask DataFrame that provides a Pandas-like API for parallel and larger-than-memory DataFrame computing on GPUs. When installed, Dask cuDF is automatically registered as the "cudf" dataframe backend for Dask DataFrame.

[!IMPORTANT] Dask cuDF does not provide support for multi-GPU or multi-node execution on its own. You must also deploy a distributed cluster (ideally with Dask-CUDA) to leverage multiple GPUs efficiently.

Using Dask cuDF

Please visit the official documentation page for detailed information about using Dask cuDF.

Installation

See the RAPIDS install page for the most up-to-date information and commands for installing Dask cuDF and other RAPIDS packages.

Resources

Quick-start example

A very common Dask cuDF use case is single-node multi-GPU data processing. These workflows typically use the following pattern:

import dask
import dask.dataframe as dd
from dask_cuda import LocalCUDACluster
from distributed import Client

if __name__ == "__main__":

  # Define a GPU-aware cluster to leverage multiple GPUs
  client = Client(
    LocalCUDACluster(
      CUDA_VISIBLE_DEVICES="0,1",  # Use two workers (on devices 0 and 1)
      rmm_pool_size=0.9,  # Use 90% of GPU memory as a pool for faster allocations
      enable_cudf_spill=True,  # Improve device memory stability
      local_directory="/fast/scratch/",  # Use fast local storage for spilling
    )
  )

  # Set the default dataframe backend to "cudf"
  dask.config.set({"dataframe.backend": "cudf"})

  # Create your DataFrame collection from on-disk
  # or in-memory data
  df = dd.read_parquet("/my/parquet/dataset/")

  # Use cudf-like syntax to transform and/or query your data
  query = df.groupby('item')['price'].mean()

  # Compute, persist, or write out the result
  query.head()

If you do not have multiple GPUs available, using LocalCUDACluster is optional. However, it is still a good idea to enable cuDF spilling.

If you wish to scale across multiple nodes, you will need to use a different mechanism to deploy your Dask-CUDA workers. Please see the RAPIDS deployment documentation for more instructions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_cudf_cu12-24.10.1.tar.gz (2.3 kB view details)

Uploaded Source

File details

Details for the file dask_cudf_cu12-24.10.1.tar.gz.

File metadata

  • Download URL: dask_cudf_cu12-24.10.1.tar.gz
  • Upload date:
  • Size: 2.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.12

File hashes

Hashes for dask_cudf_cu12-24.10.1.tar.gz
Algorithm Hash digest
SHA256 75c47358d90feb19aac8dccd9ee9f79130a65fc15911dc281968c69960464305
MD5 6b97e299b0b323052ebf5747ada3a9ac
BLAKE2b-256 da49c20c9e5f6cf1145234d15ff58ebdcbc877841bce914a1075796d1c349241

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page