Skip to main content

Native Delta Lake Python binding based on delta-rs with Pandas integration

Project description

Deltalake-python

PyPI userdoc apidoc

Native Delta Lake Python binding based on delta-rs with Pandas integration.

Example

from deltalake import DeltaTable
dt = DeltaTable("../rust/tests/data/delta-0.2.0")
dt.version()
3
dt.files()
['part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet',
 'part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet',
 'part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet']

See the user guide for more examples.

Installation

pip install deltalake

NOTE: official binary wheels are linked against openssl statically for remote objection store communication. Please file Github issue to request for critical openssl upgrade.

Build custom wheels

Sometimes you may wish to build custom wheels. Maybe you want to try out some unreleased features. Or maybe you want to tweak the optimization of the Rust code.

To compile the package, you will need the Rust compiler and maturin:

curl https://sh.rustup.rs -sSf | sh -s
pip install maturin

Then you can build wheels for your own platform like so:

maturin build --release --out wheels

For a build that is optimized for the system you are on (but sacrificing portability):

RUSTFLAGS="-C target-cpu=native" maturin build --release --out wheels

Cross compilation

The above command only works for your current platform. To create wheels for other platforms, you'll need to cross compile. Cross compilation requires installing two additional components: to cross compile Rust code, you will need to install the target with rustup; to cross compile the Python bindings, you will need to install ziglang.

The following example is for manylinux2014. Other targets will require different Rust target and Python compatibility tags.

rustup target add x86_64-unknown-linux-gnu
pip install ziglang

Then you can build the wheel with:

maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux2014 \
    --out wheels

If you expect to only run on more modern system, you can set a newer target-cpu flag to Rust and use a newer compatibility tag for Linux. For example, here we set compatibility with CPUs newer than Haswell (2013) and Linux OS with glibc version of at least 2.24:

RUSTFLAGS="-C target-cpu=haswell" maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux_2_24 \
    --out wheels

See note about RUSTFLAGS from the arrow-rs readme.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

deltalake-0.20.0.tar.gz (4.9 MB view details)

Uploaded Source

Built Distributions

deltalake-0.20.0-cp38-abi3-win_amd64.whl (28.5 MB view details)

Uploaded CPython 3.8+ Windows x86-64

deltalake-0.20.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35.9 MB view details)

Uploaded CPython 3.8+ manylinux: glibc 2.17+ x86-64

deltalake-0.20.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (35.7 MB view details)

Uploaded CPython 3.8+ manylinux: glibc 2.17+ ARM64

deltalake-0.20.0-cp38-abi3-macosx_11_0_arm64.whl (29.0 MB view details)

Uploaded CPython 3.8+ macOS 11.0+ ARM64

deltalake-0.20.0-cp38-abi3-macosx_10_12_x86_64.whl (30.7 MB view details)

Uploaded CPython 3.8+ macOS 10.12+ x86-64

File details

Details for the file deltalake-0.20.0.tar.gz.

File metadata

  • Download URL: deltalake-0.20.0.tar.gz
  • Upload date:
  • Size: 4.9 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.6.0

File hashes

Hashes for deltalake-0.20.0.tar.gz
Algorithm Hash digest
SHA256 edb2d440ac5f8df6ae1514812f53edb027c70b316aa1753767620fa0d1fe25f1
MD5 9d81af851a8cc0127051d3c372d079d7
BLAKE2b-256 ff395ccdbcb39476517e6ff9ecfcfae3a079b04efc55eb9c4cb9e333e3d450e5

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.20.0-cp38-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for deltalake-0.20.0-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 c5a008330071c891946b9f344e8243dd657ce14c091653cbd78b1d4c94beb3e6
MD5 521851f7a58c31c1cb3311e303fda8da
BLAKE2b-256 f2bb3297403b37e2b2d42d76aa37525cf85e56e2955446cd651d23ba6264ef19

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.20.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.20.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 70ae5fabf56e91a0eae2f1995a6f79f7c37865f4ca3d058a031294cb0e051ab8
MD5 a9228576ae640bd034400c4d9d32707a
BLAKE2b-256 8963b21dca553e3f9a390c0e6528d586765c809455b22cd4b9326dfe208892c7

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.20.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deltalake-0.20.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 dabafe00c4fba0fb10e46d5df064b8b6cd0d770217dd358bc5ac41e0cb3c416f
MD5 4b4e988d1e28d778313c9cbd95e63838
BLAKE2b-256 7bcbf698eb58c5fe5b6826d39ed1de5238c4cd1716ff7defff1990f8bda9a36b

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.20.0-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deltalake-0.20.0-cp38-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 06160050b6472c3c5125369738e6af8f31cc476682d4a3a60a2998acdbc999e6
MD5 2babe7fa5b3bb1fb60933a171dbaed5b
BLAKE2b-256 0f022ba5900da835b53779f86da9bd0ddde863fc3e5caab7b8bdaad69442a6d7

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.20.0-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.20.0-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 34302a239e2268d666dff1f6809d07116fe0a327d47fd260a2bfd6e1b43e273b
MD5 ad945ab62e98e80842520e6bf056c399
BLAKE2b-256 53ec8b1ad53538b333ee7445ffab8b2bfca640464e54ee423f9891ceedd2bc47

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page