Skip to main content

Native Delta Lake Python binding based on delta-rs with Pandas integration

Project description

Deltalake-python

PyPI userdoc apidoc

Native Delta Lake Python binding based on delta-rs with Pandas integration.

Example

from deltalake import DeltaTable
dt = DeltaTable("../rust/tests/data/delta-0.2.0")
dt.version()
3
dt.files()
['part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet',
 'part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet',
 'part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet']

See the user guide for more examples.

Installation

pip install deltalake

NOTE: official binary wheels are linked against openssl statically for remote objection store communication. Please file Github issue to request for critical openssl upgrade.

Build custom wheels

Sometimes you may wish to build custom wheels. Maybe you want to try out some unreleased features. Or maybe you want to tweak the optimization of the Rust code.

To compile the package, you will need the Rust compiler and maturin:

curl https://sh.rustup.rs -sSf | sh -s
pip install maturin

Then you can build wheels for your own platform like so:

maturin build --release --out wheels

For a build that is optimized for the system you are on (but sacrificing portability):

RUSTFLAGS="-C target-cpu=native" maturin build --release --out wheels

Cross compilation

The above command only works for your current platform. To create wheels for other platforms, you'll need to cross compile. Cross compilation requires installing two additional components: to cross compile Rust code, you will need to install the target with rustup; to cross compile the Python bindings, you will need to install ziglang.

The following example is for manylinux2014. Other targets will require different Rust target and Python compatibility tags.

rustup target add x86_64-unknown-linux-gnu
pip install ziglang

Then you can build the wheel with:

maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux2014 \
    --out wheels

If you expect to only run on more modern system, you can set a newer target-cpu flag to Rust and use a newer compatibility tag for Linux. For example, here we set compatibility with CPUs newer than Haswell (2013) and Linux OS with glibc version of at least 2.24:

RUSTFLAGS="-C target-cpu=haswell" maturin build --release --zig \
    --target x86_64-unknown-linux-gnu \
    --compatibility manylinux_2_24 \
    --out wheels

See note about RUSTFLAGS from the arrow-rs readme.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

deltalake-0.8.1.tar.gz (4.4 MB view details)

Uploaded Source

Built Distributions

deltalake-0.8.1-cp37-abi3-win_amd64.whl (17.6 MB view details)

Uploaded CPython 3.7+ Windows x86-64

deltalake-0.8.1-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ x86-64

deltalake-0.8.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (17.3 MB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARM64

deltalake-0.8.1-cp37-abi3-macosx_11_0_arm64.whl (15.8 MB view details)

Uploaded CPython 3.7+ macOS 11.0+ ARM64

deltalake-0.8.1-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (33.4 MB view details)

Uploaded CPython 3.7+ macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

deltalake-0.8.1-cp37-abi3-macosx_10_7_x86_64.whl (17.6 MB view details)

Uploaded CPython 3.7+ macOS 10.7+ x86-64

File details

Details for the file deltalake-0.8.1.tar.gz.

File metadata

  • Download URL: deltalake-0.8.1.tar.gz
  • Upload date:
  • Size: 4.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/0.14.15

File hashes

Hashes for deltalake-0.8.1.tar.gz
Algorithm Hash digest
SHA256 b5483604170ffcbb7d48db3c919a386abfeea1a4169e98a7badde96f551a6e74
MD5 51d787186cba8a2ddd189c1bc689ab55
BLAKE2b-256 fd0eb5472564df2041caea5b9d85de4be27ba13a03d845250b370f01319fa857

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.8.1-cp37-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for deltalake-0.8.1-cp37-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 2373e4b6504105a5d0cfb83ab8476d1b1e0278556b519f0e7f3683a8495bbd49
MD5 2a69453108baf71cc2b0aa26388796e8
BLAKE2b-256 9261d89af2a2ba8a6e4a2400e2a0d12df78cf122761ee1062ced539769f19f2f

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.8.1-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.8.1-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9112fc2824c5a1ed155a1f323c37132812d9a657bfb03122bc0fce81f3c57973
MD5 5aa7c14a2844a1425828a8f4ef740bbd
BLAKE2b-256 f387a90dc7849167e216c238069d9e3a9eb7c905943936fc09db567c1dc899fc

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.8.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deltalake-0.8.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 d3ce98189fd1e85b53c4f8adb052afdd7df26d9275b0cc2ac96a6cfa854744a9
MD5 f718e8c111b8a3fa0108c3f034d07823
BLAKE2b-256 cc5ca4a7100ba74cd3c736f60b1fe24cb68a2a2dcbf71b3d29d2c7b08c3a0738

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.8.1-cp37-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deltalake-0.8.1-cp37-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f1628d059ef05786f385c00c2edc7ec9c530e5df3916b7bd53353189912043d9
MD5 ba4f39c5d53a915a508d40f47ace7204
BLAKE2b-256 7abec4089903e6457c1f481e8ce2d11afd6a7c06d4de094187210f2bd9482bdf

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.8.1-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for deltalake-0.8.1-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 a783d51a6064ca951caee6bdb241918be7fd366b8d61b0eb93aa50129e5757a7
MD5 b81665e20a2899f9917927bdb453396a
BLAKE2b-256 0263a6dc1f7924621612dd6ad93e5c75cd9a4e5327a7abde7d78ad368f9f9335

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.8.1-cp37-abi3-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.8.1-cp37-abi3-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 277b64dfdf46d56816ca1ff75a75d5b51cfaba2d5e61819cb264945074f57aff
MD5 144f6b8fd4778200e502838e29be49f0
BLAKE2b-256 9e95dbbf8263291a31aa529f32d861a965d595dbef2710e3a577764801f57e6a

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page