Arrow, pydantic style

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for jorgecarleitao from gravatar.com jorgecarleitao

Unverified details

These details have not been verified by PyPI
Project links
Meta

Project description

Welcome to arrowdantic

Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow that can interoperate with

For simple (but data-heavy) data engineering tasks, this package essentially replaces pyarrow: it supports reading from and writing to Parquet, Arrow at the same or higher performance and higher safety (e.g. no segfaults).

Furthermore, it supports reading from and writing to ODBC compliant databases at the same or higher performance than turbodbc.

This package is also suitable for environments such as AWS Lambda functions. It takes 13M of disk space, compared to 82M taken by pyarrow.

Features

  • declare and access Arrow-backed arrays (integers, floats, boolean, string, binary)
  • read from and write to Apache Arrow IPC file
  • read from and write to Apache Parquet
  • read from and write to ODBC-compliant databases (e.g. postgres, mongoDB)

Examples

Use parquet

import io
import arrowdantic as ad

original_arrays = [ad.UInt32Array([1, None])]

schema = ad.Schema(
    [ad.Field(f"c{i}", array.type, True) for i, array in enumerate(original_arrays)]
)

data = io.BytesIO()
with ad.ParquetFileWriter(data, schema) as writer:
    writer.write(ad.Chunk(original_arrays))
data.seek(0)

reader = ad.ParquetFileReader(data)
chunk = next(reader)
assert chunk.arrays() == original_arrays

Use Arrow files

import arrowdantic as ad

original_arrays = [ad.UInt32Array([1, None])]

schema = ad.Schema(
    [ad.Field(f"c{i}", array.type, True) for i, array in enumerate(original_arrays)]
)

import io

data = io.BytesIO()
with ad.ArrowFileWriter(data, schema) as writer:
    writer.write(ad.Chunk(original_arrays))
data.seek(0)

reader = ad.ArrowFileReader(data)
chunk = next(reader)
assert chunk.arrays() == original_arrays

Use ODBC

import arrowdantic as ad


arrays = [ad.Int32Array([1, None]), ad.StringArray(["aa", None])]


with ad.ODBCConnector(r"Driver={SQLite3};Database=sqlite-test.db") as con:
    # create an empty table with a schema
    con.execute("DROP TABLE IF EXISTS example;")
    con.execute("CREATE TABLE example (c1 INT, c2 TEXT);")

    # and insert the arrays
    con.write("INSERT INTO example (c1, c2) VALUES (?, ?)", ad.Chunk(arrays))

    chunks = con.execute("SELECT c1 FROM example", 1024)
    assert chunks.fields() == [
        ad.Field("c1", ad.DataType.int32(), True),
        ad.Field("c2", ad.DataType.string(), True),
    ]
    chunk = next(chunks)
assert chunk.arrays() == arrays

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for jorgecarleitao from gravatar.com jorgecarleitao

Unverified details

These details have not been verified by PyPI
Project links
Meta

Release history Release notifications | RSS feed

0.2.3

0.2.2

0.2.1

0.2.0

0.1.0

This version

0.0.3

0.0.2

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

arrowdantic-0.0.3.tar.gz (13.4 kB view details)

Uploaded Source

Built Distributions

arrowdantic-0.0.3-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded PyPy manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.3-cp310-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.10 Windows x86-64

arrowdantic-0.0.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.3-cp310-cp310-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 macOS 10.7+ x86-64

arrowdantic-0.0.3-cp39-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.9 Windows x86-64

arrowdantic-0.0.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.3-cp39-cp39-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.9 macOS 10.7+ x86-64

arrowdantic-0.0.3-cp38-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.8 Windows x86-64

arrowdantic-0.0.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.3-cp38-cp38-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 macOS 10.7+ x86-64

arrowdantic-0.0.3-cp37-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.7 Windows x86-64

arrowdantic-0.0.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.3-cp37-cp37m-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.7m macOS 10.7+ x86-64

File details

Details for the file arrowdantic-0.0.3.tar.gz.

File metadata

  • Download URL: arrowdantic-0.0.3.tar.gz
  • Upload date:
  • Size: 13.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.12

File hashes

Hashes for arrowdantic-0.0.3.tar.gz
Algorithm Hash digest
SHA256 941e524d630e08ab49fa9de95e50056703e07138a382d6a71705831450fdda3f
MD5 748b672862d92aa78e50fb11b137aa0c
BLAKE2b-256 4a512921e5f89ed369cd46ca4d7db8d19cd3d18bd073dc7a7007bcabfeee5ab8

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 590ce3ae47277cbd31eaa7f49f5fe163c1105b3373c5daab4b2cfb0ee0784bd8
MD5 e7892844787fe2118ea53de0942c455f
BLAKE2b-256 b959dc2e7b74435ad4b1dca9e5e7b877b25397521ac7afa332acc5290608ac85

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp310-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp310-none-win_amd64.whl
Algorithm Hash digest
SHA256 7b7ae85e1c2a77428f94ad7efc2322e5b2175dfb12ce037f30f6288c19684d6e
MD5 637d52d98e48026dd86e8f50b8f18312
BLAKE2b-256 4c3bfb1717876daa99d7827323856944502725562c4744a93e3fdd124222c254

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b6b177195417f95b6da3a169f50a72d227fa54ed6d19fda006292ef4c17e2708
MD5 ac7a1bb3654f376e185d2bdd9a21782d
BLAKE2b-256 29da57d94585bbaee3c77c7a9fdcdaf1f8108139f09847b7d6e533c1b20b180d

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp310-cp310-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 ffaf9e1056471e261fc354600b3b424506d6b91459f98e17b9cf81c1a047fdc3
MD5 0d2512ff54693ca12ad8254a3eb3cd38
BLAKE2b-256 ac34e566e1c0ddfd0e9e72c34562bcc0ba496904c7701ea9f7ca0c0ad3446989

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp39-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp39-none-win_amd64.whl
Algorithm Hash digest
SHA256 c28da64eec3fa12e59fb845259d1985f3c04838b2309afc9724f179617e19617
MD5 475edaf4b958d74d1e5f2e989f5efa17
BLAKE2b-256 cf13b49e0d4a58d65157330fbff4f88d18f17cef34f4179a282fc4725eb47023

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 07fc111263026999e8eaa524ddcabf734aca554f82cf3ab030cc03045c8c3cc5
MD5 6b0c507184f3afa2d76172a40a4f4481
BLAKE2b-256 0f67ed68b2c7bb9306e41d0d5345196367e37d0fbc8bf5b14c96748a8017f2fd

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp39-cp39-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 04af79e3019f3d6611720f1dafdb78120de89a0492c90e9684d4d42f5a445318
MD5 82a7f83134ff60d171227feab7fab73d
BLAKE2b-256 bf86ea41d185a1a7713c5049559d29db9a9f80b7b2db401ccc79fcefcf937fe6

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp38-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp38-none-win_amd64.whl
Algorithm Hash digest
SHA256 cc9374f003aff9db6f3c8f8068e7385fa402b38124a073061193b92ab2734634
MD5 ee7db36db1857617c6f0e1bf3701d99e
BLAKE2b-256 d6bac4e3a865eb706e1b0310750bf3b8bca3e80bb1c34c8d16df0743b4ceb9a8

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 698beba542cd77a37f06cd0a3450d29c3cd43ab1a702763f61e1721a09454f2d
MD5 90f4965008364c37de5fe4436a8b16be
BLAKE2b-256 5f9f0f251df4b949dcd613f56a7e7b22c6611ac4c6e947df187cd82626db6051

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp38-cp38-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 3611ebfa9cbf91057bca27f39f89fe0bd0982897570bde988e9e5859851875a7
MD5 a915afbdf82ce6c67669817bea0d9aa9
BLAKE2b-256 dda60b9151afdfbe14f6207310d78a7696767f55ce3330f3b40ab0b12e95ed29

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp37-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp37-none-win_amd64.whl
Algorithm Hash digest
SHA256 42ea5a98a98b055d3299ca3355db3fb2a23e44c55649fe99d6c61a06ecde7ff8
MD5 ab7cb110412fc091258f1476fab65366
BLAKE2b-256 8dff865c994448cb78e89c24fa01e922a10768f794b7b5f4f1f57f1440c39a0c

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 c60d40ac99b0fb7cd8011d614df0e94ac1206889f2fd7dcd932c3c812f9ea536
MD5 46678bcbef84f921ac933be7d86ec556
BLAKE2b-256 4a59b5e2cb1d5a93db7b7c8aab30d0d92f623c63176cee3d660016bb99b31273

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.0.3-cp37-cp37m-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.0.3-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 65bcdc363ba1821a7446de11fc3b66cd2456563cd974ed8847274e72db818293
MD5 5256357e202bf190aec85b10e78acd11
BLAKE2b-256 296b92d8f1e84b3c26d84794adafdedbce1ce36eccb9e2c7c8393705def7067f

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page