Arrow, pydantic style

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for jorgecarleitao from gravatar.com jorgecarleitao

Unverified details

These details have not been verified by PyPI
Project links
Meta

Project description

Welcome to arrowdantic

Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow that can interoperate with

For simple (but data-heavy) data engineering tasks, this package essentially replaces pyarrow: it supports reading from and writing to Parquet, Arrow at the same or higher performance and higher safety (e.g. no segfaults).

Furthermore, it supports reading from and writing to ODBC compliant databases at the same or higher performance than turbodbc.

This package is particularly suitable for environments such as AWS Lambda - it takes 8M of disk space, compared to 82M taken by pyarrow.

Features

  • declare and access Arrow-backed arrays (integers, floats, boolean, string, binary)
  • read from and write to Apache Arrow IPC file
  • read from and write to Apache Parquet
  • read from and write to ODBC-compliant databases (e.g. postgres, mongoDB)

Examples

Use parquet

import io
import arrowdantic as ad

original_arrays = [ad.UInt32Array([1, None])]

schema = ad.Schema(
    [ad.Field(f"c{i}", array.type, True) for i, array in enumerate(original_arrays)]
)

data = io.BytesIO()
with ad.ParquetFileWriter(data, schema) as writer:
    writer.write(ad.Chunk(original_arrays))
data.seek(0)

reader = ad.ParquetFileReader(data)
chunk = next(reader)
assert chunk.arrays() == original_arrays

Use Arrow files

import arrowdantic as ad

original_arrays = [ad.UInt32Array([1, None])]

schema = ad.Schema(
    [ad.Field(f"c{i}", array.type, True) for i, array in enumerate(original_arrays)]
)

import io

data = io.BytesIO()
with ad.ArrowFileWriter(data, schema) as writer:
    writer.write(ad.Chunk(original_arrays))
data.seek(0)

reader = ad.ArrowFileReader(data)
chunk = next(reader)
assert chunk.arrays() == original_arrays

Use ODBC

import arrowdantic as ad


arrays = [ad.Int32Array([1, None]), ad.StringArray(["aa", None])]

with ad.ODBCConnector(r"Driver={SQLite3};Database=sqlite-test.db") as con:
    # create an empty table with a schema
    con.execute("DROP TABLE IF EXISTS example;")
    con.execute("CREATE TABLE example (c1 INT, c2 TEXT);")

    # insert the arrays
    con.write("INSERT INTO example (c1, c2) VALUES (?, ?)", ad.Chunk(arrays))

    # read the arrays
    with con.execute("SELECT c1, c2 FROM example", 1024) as chunks:
        assert chunks.fields() == [
            ad.Field("c1", ad.DataType.int32(), True),
            ad.Field("c2", ad.DataType.string(), True),
        ]
        chunk = next(chunks)
assert chunk.arrays() == arrays

Use timezones

This package fully supports datetime and conversions between them and arrow:

import arrowdantic as ad


dt = datetime.datetime(
    year=2021,
    month=1,
    day=1,
    hour=1,
    minute=1,
    second=1,
    microsecond=1,
    tzinfo=datetime.timezone.utc,
)
a = ad.TimestampArray([dt, None])
assert (
    str(a)
    == 'Timestamp(Microsecond, Some("+00:00"))[2021-01-01 01:01:01.000001 +00:00, None]'
)
assert list(a) == [dt, None]
assert a.type == ad.DataType.timestamp(datetime.timezone.utc)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for jorgecarleitao from gravatar.com jorgecarleitao

Unverified details

These details have not been verified by PyPI
Project links
Meta

Release history Release notifications | RSS feed

0.2.3

0.2.2

0.2.1

This version

0.2.0

0.1.0

0.0.3

0.0.2

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

arrowdantic-0.2.0.tar.gz (15.7 kB view details)

Uploaded Source

Built Distributions

arrowdantic-0.2.0-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.9 MB view details)

Uploaded PyPy manylinux: glibc 2.12+ x86-64

arrowdantic-0.2.0-cp310-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.10 Windows x86-64

arrowdantic-0.2.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.12+ x86-64

arrowdantic-0.2.0-cp310-cp310-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 macOS 10.7+ x86-64

arrowdantic-0.2.0-cp39-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.9 Windows x86-64

arrowdantic-0.2.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

arrowdantic-0.2.0-cp39-cp39-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.9 macOS 10.7+ x86-64

arrowdantic-0.2.0-cp38-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.8 Windows x86-64

arrowdantic-0.2.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

arrowdantic-0.2.0-cp38-cp38-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 macOS 10.7+ x86-64

arrowdantic-0.2.0-cp37-none-win_amd64.whl (2.4 MB view details)

Uploaded CPython 3.7 Windows x86-64

arrowdantic-0.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

arrowdantic-0.2.0-cp37-cp37m-macosx_10_7_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.7m macOS 10.7+ x86-64

File details

Details for the file arrowdantic-0.2.0.tar.gz.

File metadata

  • Download URL: arrowdantic-0.2.0.tar.gz
  • Upload date:
  • Size: 15.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.12

File hashes

Hashes for arrowdantic-0.2.0.tar.gz
Algorithm Hash digest
SHA256 3a16f0b512fc4f94b6b47226cdc3e447f521d432b9c808a048b547901e855298
MD5 37c66748aaf4374943c7806f6826889b
BLAKE2b-256 1aa7baf9e665e99974901ac53ffee3c5dca6ce5bd17dcb8d1c8bffb91146b09a

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 0467f975355b275dad4fe63bbe0dc91909e5f7fe61a1dfe961bf57a188b10120
MD5 d44125dd23520b646d6922c220d26f8d
BLAKE2b-256 98ca1ce174f3fc46b8e79f485d76260e1769623fb37d7cb1b0627fdb2b072095

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp310-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp310-none-win_amd64.whl
Algorithm Hash digest
SHA256 8a2c7b9caec79610deeb204eeb7e95e29ad96c667d9de77a4e41ddeb4e56f6a6
MD5 9dd7dfd0797d33ce084bc8c1d95f2700
BLAKE2b-256 81e0027793b925e173c047447c124160a91b02a28afc6f83cd5a05b5dcdb32c0

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 0d07f04d18d6eba941f9d7e45cdb750cd828b0955afd5aeb1634c8923189ccf5
MD5 5df3ea9316613cd44d7d4c57a70b0bd4
BLAKE2b-256 0881bb93f1c5d12cb49ddfa5bbeb30959c6082112aabbb714c51095ce683cdc1

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp310-cp310-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 33bf6fe699a21eab8c4253b8184cbcda8eab1b8252850a9dc054b14b8c900f37
MD5 17bb6c335bc68ff1d1f8bca6addee100
BLAKE2b-256 9eaecd9e69875e66df0c417d94c31380c5434438a5a8520cfa988ca14041ef23

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp39-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp39-none-win_amd64.whl
Algorithm Hash digest
SHA256 92c64d897e69a22d3e4a69e0feea02f301d5d606bef89459177c59ea2dcb8ac3
MD5 41e8d821fcdddf4b70432c70c3edce8b
BLAKE2b-256 5c705a635f232f3785a02a1e78c37a9ce3dc33428f36a44dc149c872b201b055

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 7e855b0f98206d5ea95fcf3b8eec953b072f18eedbb1211b91bd99431264fa8e
MD5 4eaab85b91dab8094d53edb41e0d27a5
BLAKE2b-256 1af855f3fab3ffb55fc11c8f321e79a3319337d7fda416ec417df2f260593c00

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp39-cp39-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 6fc4860310f2d57e7ad66068ffc2c4e88da27a22b763cd19e27360f45a560477
MD5 f84bd6cdc197fda531a4a5fbb2b3fcbc
BLAKE2b-256 40c0380ad795bf72dfae7ac9b778c6ef3332befcd5ad4a09ddbd085df667f2ea

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp38-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp38-none-win_amd64.whl
Algorithm Hash digest
SHA256 8f232ca88351253ae441dd3920ecd2f8fc7aedab4b8e98f6fa89afcc9755d09b
MD5 e2dac7aba99ec998d7f62a9e93b6ecc8
BLAKE2b-256 c7d6a4fd5306ad18f70f54d901f7b87de2219de978e0393d25f9defe077cda3d

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b3996c27b945bac11c61edef0501837ccfdbc7e294396c829bd18e27f263088b
MD5 ef506e35b326561b3b4534bfd12c0d01
BLAKE2b-256 34b4cf5db95d3f9f1133c321f753b7e36b39337fb764ad44fdfe4a6567a1ffbb

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp38-cp38-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 961786231d8c9be51332cd23d490dae37697585ba9344fcb6db4a3ce6cd1aac5
MD5 7e8e55058a0285f3c629f64684f21ef0
BLAKE2b-256 e4c4a4d51b364bec3331da692e189885759e9164fcb05b257f8ffae1344bdda8

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp37-none-win_amd64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp37-none-win_amd64.whl
Algorithm Hash digest
SHA256 a244a20c98986ba5616ebea8ec15a5004b4383c6aa6dbe31d5d8a69bbbd56700
MD5 1e3d5445497577ed8349422c8348e49e
BLAKE2b-256 375abcf53633e9660e85f18b8867e1f553f11bf7b2a2718d44a9c72269c2c8be

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 a5f58a1871016d0d704afec0807edefefe4b8ef94bc320d72801bfd16b472f76
MD5 75503b033b2aeffd4810e1fa1134b029
BLAKE2b-256 e417e39380cf880ea29b4d7c09e8aa31fa9852f4d8571d338b8010f0532f73ec

See more details on using hashes here.

Provenance

File details

Details for the file arrowdantic-0.2.0-cp37-cp37m-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for arrowdantic-0.2.0-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 b507ff32283bc481b413aba38924bb5a548fba185e0a3a73861d760ce1fcbab3
MD5 42329e037481349645864e4054f7d0d3
BLAKE2b-256 5a2c37e9ba006d79f2f370eb1557980607dd2d18c8386e1b8ca51654b483af20

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page