python wrapper for Lance columnar format
Project description
Python bindings for Lance Data Format
:warning: Under heavy development
Lance is a new columnar data format for data science and machine learning
Why you should use Lance
- Is order of magnitude faster than parquet for point queries and nested data structures common to DS/ML
- Comes with a fast vector index that delivers sub-millisecond nearest neighbors search performance
- Is automatically versioned and supports lineage and time-travel for full reproducibility
- Integrated with duckdb/pandas/polars already. Easily convert from/to parquet in 2 lines of code
Quick start
Installation
pip install pylance
Make sure you have a recent version of pandas (1.5+), pyarrow (10.0+), and DuckDB (0.7.0+)
Converting to Lance
import lance
import pandas as pd
import pyarrow as pa
import pyarrow.dataset
df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')
parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")
Reading Lance data
dataset = lance.dataset("/tmp/test.lance")
assert isinstance(dataset, pa.dataset.Dataset)
Pandas
df = dataset.to_table().to_pandas()
DuckDB
import duckdb
# If this segfaults, make sure you have duckdb v0.7+ installed
duckdb.query("SELECT * FROM dataset LIMIT 10").to_df()
Vector search
Download the sift1m subset
wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
tar -xzf sift.tar.gz
Convert it to Lance
import lance
from lance.vector import vec_to_table
import numpy as np
import struct
nvecs = 1000000
ndims = 128
with open("sift/sift_base.fvecs", mode="rb") as fobj:
buf = fobj.read()
data = np.array(struct.unpack("<128000000f", buf[4 : 4 + 4 * nvecs * ndims])).reshape((nvecs, ndims))
dd = dict(zip(range(nvecs), data))
table = vec_to_table(dd)
uri = "vec_data.lance"
sift1m = lance.write_dataset(table, uri, max_rows_per_group=8192, max_rows_per_file=1024*1024)
Build the index
sift1m.create_index("vector",
index_type="IVF_PQ",
num_partitions=256, # IVF
num_sub_vectors=16) # PQ
Search the dataset
# Get top 10 similar vectors
import duckdb
dataset = lance.dataset(uri)
# Sample 100 query vectors. If this segfaults, make sure you have duckdb v0.7+ installed
sample = duckdb.query("SELECT vector FROM dataset USING SAMPLE 100").to_df()
query_vectors = np.array([np.array(x) for x in sample.vector])
# Get nearest neighbors for all of them
rs = [dataset.to_table(nearest={"column": "vector", "k": 10, "q": q})
for q in query_vectors]
*More distance metrics, HNSW, and distributed support is on the roadmap
Python package details
Install from PyPI: pip install pylance # >=0.3.0 is the new rust-based implementation
Install from source: maturin develop (under the /python directory)
Run unit tests: make test
Run integration tests: make integtest
Import via: import lance
The python integration is done via pyo3 + custom python code:
- We make wrapper classes in Rust for Dataset/Scanner/RecordBatchReader that's exposed to python.
- These are then used by LanceDataset / LanceScanner implementations that extend pyarrow Dataset/Scanner for duckdb compat.
- Data is delivered via the Arrow C Data Interface
Motivation
Why do we need a new format for data science and machine learning?
1. Reproducibility is a must-have
Versioning and experimentation support should be built into the dataset instead of requiring multiple tools.
It should also be efficient and not require expensive copying everytime you want to create a new version.
We call this "Zero copy versioning" in Lance. It makes versioning data easy without increasing storage costs.
2. Cloud storage is now the default
Remote object storage is the default now for data science and machine learning and the performance characteristics of cloud are fundamentally different.
Lance format is optimized to be cloud native. Common operations like filter-then-take can be order of magnitude faster
using Lance than Parquet, especially for ML data.
3. Vectors must be a first class citizen, not a separate thing
The majority of reasonable scale workflows should not require the added complexity and cost of a specialized database just to compute vector similarity. Lance integrates optimized vector indices into a columnar format so no additional infrastructure is required to get low latency top-K similarity search.
4. Open standards is a requirement
The DS/ML ecosystem is incredibly rich and data must be easily accessible across different languages, tools, and environments. Lance makes Apache Arrow integration its primary interface, which means conversions to/from is 2 lines of code, your code does not need to change after conversion, and nothing is locked-up to force you to pay for vendor compute. We need open-source not fauxpen-source.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pylance-0.8.21-cp38-abi3-win_amd64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f7028d3944868efd1a00c23507952d043c663b3e37b2bea9d582422a8e06a613 |
|
| MD5 | 805fc6a8b4d12adac63e8067cd1be219 |
|
| BLAKE2b-256 | 3f54a2761b7a5ef6ecd844a1b6e844d0b793601f324be661e7395a26dc541914 |
Hashes for pylance-0.8.21-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | be428e4a808b55681f2352fb69dc426a2e6abcf17267c51131b9e24679401feb |
|
| MD5 | 4e28d93f2708fbb1e3d079bbe9b86c0c |
|
| BLAKE2b-256 | 51010f9922dd6fe7bdc80067a1e3ac4a3e8e089b07ef6041f38cb41f623bef35 |
Hashes for pylance-0.8.21-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 028bfabc074e295d9f805543382a5d64be9d94f25d1cfee9f524a19511270faf |
|
| MD5 | ef53151c06f0e77e910e7c87f6032562 |
|
| BLAKE2b-256 | 438a2dd59201cf027997ba1487cdc813f4eb2c94c1add8306d9a91078457954d |
Hashes for pylance-0.8.21-cp38-abi3-macosx_11_0_arm64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | c083100ca7c9c89cf257c6d0d812115b3e35fc87028d5125f295e0c87544b252 |
|
| MD5 | 22e7296e36e84d765864386cd92fc730 |
|
| BLAKE2b-256 | 5fda2af40ed19053c6660a194116638300e49a1c2959b6fce821df51e1e2345d |
Hashes for pylance-0.8.21-cp38-abi3-macosx_10_15_x86_64.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 72aaf3c4399367b21132a8169a8f461061983da21067ff849c01689af79e3298 |
|
| MD5 | 14a002fa4edbb4d652c5dfa2170679fe |
|
| BLAKE2b-256 | 37667be65416e5a63d160b0ee38b40eafb5e6437a802882a0ba80c9bd2bf6e6b |
