Skip to main content

Python binding for delta-rs

Project description

Deltalake-python

PyPI

Native Delta Lake binding for Python based on delta-rs.

Installation

pip install deltalake

NOTE: official binary wheels are linked against openssl statically for remote objection store communication. Please file Github issue to request for critical openssl upgrade.

Usage

Resolve partitions for current version of the DeltaTable:

>>> from deltalake import DeltaTable
>>> dt = DeltaTable("../rust/tests/data/delta-0.2.0")
>>> dt.version()
3
>>> dt.files()
['part-00000-cb6b150b-30b8-4662-ad28-ff32ddab96d2-c000.snappy.parquet', 'part-00000-7c2deba3-1994-4fb8-bc07-d46c948aa415-c000.snappy.parquet', 'part-00001-c373a5bd-85f0-4758-815e-7eb62007a15c-c000.snappy.parquet']

Convert DeltaTable into PyArrow Table and Pandas Dataframe:

>>> from deltalake import DeltaTable
>>> dt = DeltaTable("../rust/tests/data/simple_table")
>>> df = dt.to_pyarrow_table().to_pandas()
>>> df
   id
0   5
1   7
2   9
>>> df[df['id'] > 5]
   id
1   7
2   9

Time travel:

>>> from deltalake import DeltaTable
>>> dt = DeltaTable("../rust/tests/data/simple_table")
>>> dt.load_version(2)
>>> dt.to_pyarrow_table().to_pandas()
   id
0   5
1   7
2   9
3   5
4   6
5   7
6   8
7   9

Develop

maturin is used to build the python package.

To install development version of the package into your current Python environment:

$ maturin develop

Code are formatted with https://github.com/psf/black.

Build manylinux wheels

docker run -e PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig -it -v `pwd`:/io apache/arrow-dev:amd64-centos-6.10-python-manylinux2010 bash
curl https://sh.rustup.rs -sSf | sh -s -- -y
source $HOME/.cargo/env
rustup default stable
cargo install --git https://github.com/PyO3/maturin.git --rev 98636cea89c328b3eba4ebb548124f75c8018200 maturin
cd /io/python
export PATH=/opt/python/cp37-cp37m/bin:/opt/python/cp38-cp38/bin:$PATH
maturin publish -b pyo3 --target x86_64-unknown-linux-gnu --no-sdist

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

deltalake-0.2.1-cp36-abi3-win_amd64.whl (4.8 MB view details)

Uploaded CPython 3.6+ Windows x86-64

deltalake-0.2.1-cp36-abi3-manylinux2010_x86_64.whl (6.4 MB view details)

Uploaded CPython 3.6+ manylinux: glibc 2.12+ x86-64

deltalake-0.2.1-cp36-abi3-macosx_10_7_x86_64.whl (4.8 MB view details)

Uploaded CPython 3.6+ macOS 10.7+ x86-64

File details

Details for the file deltalake-0.2.1-cp36-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for deltalake-0.2.1-cp36-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 f35ddf1955ebd60173f64be23986f0dfed3af34da4f1d4884926f47a70a566cb
MD5 5dd10183ef3530f6881d0e68d7a60474
BLAKE2b-256 d33b8abaf486fcb72b16178067415bc0437307222ec4147c51bf78c99d63c254

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.2.1-cp36-abi3-manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.2.1-cp36-abi3-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 9807dd564cab83d8585fe3a0513959afe505544d9307efcbdec4bf01497c4ef3
MD5 7a7945300499b6b7fb3581273eb6f1e9
BLAKE2b-256 101556e3c68d3aadce880a40a811c25887027bf5f5cc3af868c1490b1be173ff

See more details on using hashes here.

Provenance

File details

Details for the file deltalake-0.2.1-cp36-abi3-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for deltalake-0.2.1-cp36-abi3-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 b662d97664aae3f14d656e32a568b7f0979c0a440eaeae96ed930a98307c1c96
MD5 2061af5164b1336ec2647dcff4e44267
BLAKE2b-256 2708b0fff1a6327789c5d8277717da396fec8f23405524c42716c0d1e03a39cf

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page