Skip to main content

python wrapper for Lance columnar format

Project description

Python bindings for Lance Data Format

:warning: Under heavy development

Lance Logo

Lance is a new columnar data format for data science and machine learning

Why you should use Lance

  1. Is order of magnitude faster than parquet for point queries and nested data structures common to DS/ML
  2. Comes with a fast vector index that delivers sub-millisecond nearest neighbors search performance
  3. Is automatically versioned and supports lineage and time-travel for full reproducibility
  4. Integrated with duckdb/pandas/polars already. Easily convert from/to parquet in 2 lines of code

Quick start

Installation

pip install pylance

Make sure you have a recent version of pandas (1.5+), pyarrow (10.0+), and DuckDB (0.7.0+)

Converting to Lance

import lance

import pandas as pd
import pyarrow as pa
import pyarrow.dataset

df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')

parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")

Reading Lance data

dataset = lance.dataset("/tmp/test.lance")
assert isinstance(dataset, pa.dataset.Dataset)

Pandas

df = dataset.to_table().to_pandas()

DuckDB

import duckdb

# If this segfaults, make sure you have duckdb v0.7+ installed
duckdb.query("SELECT * FROM dataset LIMIT 10").to_df()

Vector search

Download the sift1m subset

wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
tar -xzf sift.tar.gz

Convert it to Lance

import lance
from lance.vector import vec_to_table
import numpy as np
import struct

nvecs = 1000000
ndims = 128
with open("sift/sift_base.fvecs", mode="rb") as fobj:
    buf = fobj.read()
    data = np.array(struct.unpack("<128000000f", buf[4 : 4 + 4 * nvecs * ndims])).reshape((nvecs, ndims))
    dd = dict(zip(range(nvecs), data))

table = vec_to_table(dd)
uri = "vec_data.lance"
sift1m = lance.write_dataset(table, uri, max_rows_per_group=8192, max_rows_per_file=1024*1024)

Build the index

sift1m.create_index("vector",
                    index_type="IVF_PQ", 
                    num_partitions=256,  # IVF
                    num_sub_vectors=16)  # PQ

Search the dataset

# Get top 10 similar vectors
import duckdb

dataset = lance.dataset(uri)

# Sample 100 query vectors. If this segfaults, make sure you have duckdb v0.7+ installed
sample = duckdb.query("SELECT vector FROM dataset USING SAMPLE 100").to_df()
query_vectors = np.array([np.array(x) for x in sample.vector])

# Get nearest neighbors for all of them
rs = [dataset.to_table(nearest={"column": "vector", "k": 10, "q": q})      
      for q in query_vectors]

*More distance metrics, HNSW, and distributed support is on the roadmap

Python package details

Install from PyPI: pip install pylance # >=0.3.0 is the new rust-based implementation Install from source: maturin develop (under the /python directory) Run unit tests: make test Run integration tests: make integtest

Import via: import lance

The python integration is done via pyo3 + custom python code:

  1. We make wrapper classes in Rust for Dataset/Scanner/RecordBatchReader that's exposed to python.
  2. These are then used by LanceDataset / LanceScanner implementations that extend pyarrow Dataset/Scanner for duckdb compat.
  3. Data is delivered via the Arrow C Data Interface

Motivation

Why do we need a new format for data science and machine learning?

1. Reproducibility is a must-have

Versioning and experimentation support should be built into the dataset instead of requiring multiple tools.
It should also be efficient and not require expensive copying everytime you want to create a new version.
We call this "Zero copy versioning" in Lance. It makes versioning data easy without increasing storage costs.

2. Cloud storage is now the default

Remote object storage is the default now for data science and machine learning and the performance characteristics of cloud are fundamentally different.
Lance format is optimized to be cloud native. Common operations like filter-then-take can be order of magnitude faster using Lance than Parquet, especially for ML data.

3. Vectors must be a first class citizen, not a separate thing

The majority of reasonable scale workflows should not require the added complexity and cost of a specialized database just to compute vector similarity. Lance integrates optimized vector indices into a columnar format so no additional infrastructure is required to get low latency top-K similarity search.

4. Open standards is a requirement

The DS/ML ecosystem is incredibly rich and data must be easily accessible across different languages, tools, and environments. Lance makes Apache Arrow integration its primary interface, which means conversions to/from is 2 lines of code, your code does not need to change after conversion, and nothing is locked-up to force you to pay for vendor compute. We need open-source not fauxpen-source.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

pylance-0.19.1-cp39-abi3-win_amd64.whl (28.4 MB view details)

Uploaded CPython 3.9+ Windows x86-64

pylance-0.19.1-cp39-abi3-manylinux_2_28_x86_64.whl (30.4 MB view details)

Uploaded CPython 3.9+ manylinux: glibc 2.28+ x86-64

pylance-0.19.1-cp39-abi3-manylinux_2_24_aarch64.whl (29.0 MB view details)

Uploaded CPython 3.9+ manylinux: glibc 2.24+ ARM64

pylance-0.19.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30.4 MB view details)

Uploaded CPython 3.9+ manylinux: glibc 2.17+ x86-64

pylance-0.19.1-cp39-abi3-macosx_11_0_arm64.whl (26.5 MB view details)

Uploaded CPython 3.9+ macOS 11.0+ ARM64

pylance-0.19.1-cp39-abi3-macosx_10_15_x86_64.whl (28.6 MB view details)

Uploaded CPython 3.9+ macOS 10.15+ x86-64

File details

Details for the file pylance-0.19.1-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: pylance-0.19.1-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 28.4 MB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.13

File hashes

Hashes for pylance-0.19.1-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 b341e547c995b5d6b32eb63e1e015d31b608de49a9ad03f8981453f4c667e8e1
MD5 4b24eb739342941fb118dd7a03f9ea86
BLAKE2b-256 d9730641124631a4f5f03d988c4a8000046d4c182340719ba4cf228cb45c5dca

See more details on using hashes here.

Provenance

File details

Details for the file pylance-0.19.1-cp39-abi3-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for pylance-0.19.1-cp39-abi3-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 e26ce273840912c45dd2b8f6f8fb9082c1c788d696e11b78ddad3949e3892d50
MD5 683260b3d89d560d2aae4e4e102c478a
BLAKE2b-256 3baf3bf6d0c9dc52e2ae048c575249527f3c2cc8a4df85c94905900c719b42e0

See more details on using hashes here.

Provenance

File details

Details for the file pylance-0.19.1-cp39-abi3-manylinux_2_24_aarch64.whl.

File metadata

File hashes

Hashes for pylance-0.19.1-cp39-abi3-manylinux_2_24_aarch64.whl
Algorithm Hash digest
SHA256 7c2e0e00b40214edae576075dbfa432cadaf5ba21354b0c46f307daf4e77403f
MD5 25165cc5c804cdda9e0c6b111be032e4
BLAKE2b-256 ee33eee18d4d8996e0db79c71231b653b5c9ae86a9002e3e3d708e1461717f5c

See more details on using hashes here.

Provenance

File details

Details for the file pylance-0.19.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pylance-0.19.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8315152f57329e7668ff5c82c252591ea0e3d2aed702dd19a42d645945e7a07e
MD5 00efb54bbb6cc95398ab29d1368d63e3
BLAKE2b-256 478a2a8a07af0983577b8fc68eeb85fbb3ea0e774d84125a6395d748fde1b580

See more details on using hashes here.

Provenance

File details

Details for the file pylance-0.19.1-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pylance-0.19.1-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 9859c372b2d7fe443b6218f62e9d77caf94961cac73b274c85b724f20dd6b690
MD5 fe3f3e9f18df4e72bb7941c98807edf2
BLAKE2b-256 6547390622a3f7b2873966c0e197979afb72eb3cb0352342bdff6581c8600fa4

See more details on using hashes here.

Provenance

File details

Details for the file pylance-0.19.1-cp39-abi3-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for pylance-0.19.1-cp39-abi3-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 a254d09690a5e09cadc5fecc7b43b2bfc20b477e0f0ba31497e1d6abb36b524a
MD5 241ca6ff6950d1ba26a9719a3058c3af
BLAKE2b-256 6ffb5e269c0c310646099fea4cc47872e21172d3d917fe68a036bf2f3e42b374

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page