Native Delta Lake Python binding based on delta-rs with Pandas integration
Project description
Deltalake-python
Native Delta Lake Python binding based on delta-rs with Pandas integration.
Example
from deltalake import DeltaTable
dt = DeltaTable("../rust/tests/data/delta-0.2.0")
dt.version()
3
dt.files()
['part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet',
'part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet',
'part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet']
See the user guide for more examples.
Installation
pip install deltalake
NOTE: official binary wheels are linked against openssl statically for remote objection store communication. Please file Github issue to request for critical openssl upgrade.
Build custom wheels
Sometimes you may wish to build custom wheels. Maybe you want to try out some unreleased features. Or maybe you want to tweak the optimization of the Rust code.
To compile the package, you will need the Rust compiler and maturin:
curl https://sh.rustup.rs -sSf | sh -s
pip install maturin
Then you can build wheels for your own platform like so:
maturin build --release --out wheels
For a build that is optimized for the system you are on (but sacrificing portability):
RUSTFLAGS="-C target-cpu=native" maturin build --release --out wheels
Cross compilation
The above command only works for your current platform. To create wheels for other
platforms, you'll need to cross compile. Cross compilation requires installing
two additional components: to cross compile Rust code, you will need to install
the target with rustup; to cross compile the Python bindings, you will need
to install ziglang.
The following example is for manylinux2014. Other targets will require different
Rust target and Python compatibility tags.
rustup target add x86_64-unknown-linux-gnu
pip install ziglang
Then you can build the wheel with:
maturin build --release --zig \
--target x86_64-unknown-linux-gnu \
--compatibility manylinux2014 \
--out wheels
If you expect to only run on more modern system, you can set a newer target-cpu
flag to Rust and use a newer compatibility tag for Linux. For example, here
we set compatibility with CPUs newer than Haswell (2013) and Linux OS with
glibc version of at least 2.24:
RUSTFLAGS="-C target-cpu=haswell" maturin build --release --zig \
--target x86_64-unknown-linux-gnu \
--compatibility manylinux_2_24 \
--out wheels
See note about RUSTFLAGS from the arrow-rs readme.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file deltalake-0.16.4.tar.gz.
File metadata
- Download URL: deltalake-0.16.4.tar.gz
- Upload date:
- Size: 4.8 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.5.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3cbfe1c894df1f62097ec64278292212793127f676a9d3014762b37b59ca55ee |
|
| MD5 | f597f8589bc1a89381954923ce8d5b13 |
|
| BLAKE2b-256 | aafaa8291ec91291bd8696979a8c62a95da1ee9ef72b946bb7192caa253b48f2 |
Provenance
File details
Details for the file deltalake-0.16.4-cp38-abi3-win_amd64.whl.
File metadata
- Download URL: deltalake-0.16.4-cp38-abi3-win_amd64.whl
- Upload date:
- Size: 23.4 MB
- Tags: CPython 3.8+, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.5.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 93d494d608cb95093a49ab2ef5ea177c00950b345fe2c1d960592a37187eabd8 |
|
| MD5 | b3dfdfceb134a5460207db62524f4683 |
|
| BLAKE2b-256 | 9c8bf280e54c989ca9b6a017ba280aa4a7c51638e9ef1734320cf6825d3ce4d8 |
Provenance
File details
Details for the file deltalake-0.16.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: deltalake-0.16.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 25.6 MB
- Tags: CPython 3.8+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.5.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8bccdf198481ab7f115702e25349d6c7f5a43aee60d8cfdfa7f0de1c59abdd12 |
|
| MD5 | 2eb7753e1859e273f8e3a4ad6d422bbd |
|
| BLAKE2b-256 | a6603d430fc0270e4b1a83557a94722baed1b66de6d83b7ab7683503b17140e1 |
Provenance
File details
Details for the file deltalake-0.16.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.
File metadata
- Download URL: deltalake-0.16.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 24.2 MB
- Tags: CPython 3.8+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.5.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b59360556c921fdf29307aa2ecff2c0bccff49aaeb9f0168842f4e45e3aca051 |
|
| MD5 | 2806ecc5d189cf8f00c3da0f0958445e |
|
| BLAKE2b-256 | 2b8b1212271e9936dcd43cb3902398aef18270e28a90acd828415a277b28ae36 |
Provenance
File details
Details for the file deltalake-0.16.4-cp38-abi3-macosx_11_0_arm64.whl.
File metadata
- Download URL: deltalake-0.16.4-cp38-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 20.9 MB
- Tags: CPython 3.8+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.5.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 40b2dc128c4b30620fd4bfc5ef2aec9694070fc5a82cca8552e7a5c7c2c98633 |
|
| MD5 | 51facdf36ef87c8ecfbb54336f3d4b60 |
|
| BLAKE2b-256 | dd39702203672bba58a79743fe7be408c176d14edae092d1ddc7f1aaa1858f88 |
Provenance
File details
Details for the file deltalake-0.16.4-cp38-abi3-macosx_10_12_x86_64.whl.
File metadata
- Download URL: deltalake-0.16.4-cp38-abi3-macosx_10_12_x86_64.whl
- Upload date:
- Size: 22.1 MB
- Tags: CPython 3.8+, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.5.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b5957258358083e63cccad5bca6e52527b8b2733e92c69600d4cad9fd85b7fa0 |
|
| MD5 | 5161fa14a8bb1946ea586dcdf05450e9 |
|
| BLAKE2b-256 | 208ab317472e4c68403a6eea129d59dba820ef70f32de6ec14e19f5fe161985f |
