Arrow -> PostgreSQL encoder

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for adriangb from gravatar.com adriangb

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Adrian Garcia Badaracco
  • Requires: Python >=3.7
Classifiers

Project description

pgpq

Convert PyArrow RecordBatches to Postgres' native binary format.

Usage

import psycopg
import pyarrow.dataset as ds
from pgpq import ArrowToPostgresBinaryEncoder


dataset = ds.dataset("tests/testdata/")

encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
# get the expected postgres destination schema
# note that this is _not_ the same as the incoming arrow schema
# and not necessarily the schema of your permanent table
# instead it's the schema of the data that will be sent over the wire
# which for example does not have timezones on any timestamps
pg_schema = encoder.schema()
# assemble ddl for a temporary table
# it's often a good idea to bulk load into a temp table to:
# (1) Avoid indexes
# (2) Stay in-memory as long as possible
# (3) Be more flexible with types (you can't load a SMALLINT into a BIGINT column without casting)
cols = [f"\"{col['name']}\" {col['data_type']['ddl']}" for col in pg_schema["columns"]]
ddl = f"CREATE TEMP TABLE data ({','.join(cols)})"

with psycopg.connect("postgres://postgres:postgres@localhost:5432/postgres") as conn:
    with conn.cursor() as cursor:
        cursor.execute(ddl)  # type: ignore
        with cursor.copy("COPY data FROM STDIN WITH (FORMAT BINARY)") as copy:
            copy.write(encoder.write_header())
            for batch in dataset.to_batches():
                copy.write(encoder.write_batch(batch))
            copy.write(encoder.finish())
        # load into your actual table, possibly doing type casts
        # cursor.execute("INSERT INTO \"table\" SELECT * FROM data")

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for adriangb from gravatar.com adriangb

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Adrian Garcia Badaracco
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

0.9.0

0.8.0

0.7.3

0.7.2

This version

0.7.1

0.7.0

0.6.1

0.5.0

0.4.2

0.4.1

0.2.0

0.1.2

0.1.1

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pgpq-0.7.1.tar.gz (23.8 kB view hashes)

Uploaded Source

Built Distributions

pgpq-0.7.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (584.0 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pgpq-0.7.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (534.8 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

pgpq-0.7.1-pp39-pypy39_pp73-macosx_10_7_x86_64.whl (559.3 kB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

pgpq-0.7.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (583.6 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pgpq-0.7.1-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (534.5 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

pgpq-0.7.1-pp38-pypy38_pp73-macosx_10_7_x86_64.whl (559.2 kB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

pgpq-0.7.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (585.8 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pgpq-0.7.1-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (536.9 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

pgpq-0.7.1-pp37-pypy37_pp73-macosx_10_7_x86_64.whl (561.1 kB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

pgpq-0.7.1-cp37-abi3-win_amd64.whl (502.0 kB view hashes)

Uploaded CPython 3.7+ Windows x86-64

pgpq-0.7.1-cp37-abi3-win32.whl (455.7 kB view hashes)

Uploaded CPython 3.7+ Windows x86

pgpq-0.7.1-cp37-abi3-musllinux_1_2_x86_64.whl (755.9 kB view hashes)

Uploaded CPython 3.7+ musllinux: musl 1.2+ x86-64

pgpq-0.7.1-cp37-abi3-musllinux_1_2_i686.whl (772.1 kB view hashes)

Uploaded CPython 3.7+ musllinux: musl 1.2+ i686

pgpq-0.7.1-cp37-abi3-musllinux_1_2_armv7l.whl (778.8 kB view hashes)

Uploaded CPython 3.7+ musllinux: musl 1.2+ ARMv7l

pgpq-0.7.1-cp37-abi3-musllinux_1_2_aarch64.whl (713.7 kB view hashes)

Uploaded CPython 3.7+ musllinux: musl 1.2+ ARM64

pgpq-0.7.1-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (583.3 kB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ x86-64

pgpq-0.7.1-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (732.3 kB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ s390x

pgpq-0.7.1-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (823.7 kB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ppc64le

pgpq-0.7.1-cp37-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (887.6 kB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ppc64

pgpq-0.7.1-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (515.8 kB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARMv7l

pgpq-0.7.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (534.0 kB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARM64

pgpq-0.7.1-cp37-abi3-manylinux_2_5_i686.manylinux1_i686.whl (619.5 kB view hashes)

Uploaded CPython 3.7+ manylinux: glibc 2.5+ i686

pgpq-0.7.1-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (1.1 MB view hashes)

Uploaded CPython 3.7+ macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

pgpq-0.7.1-cp37-abi3-macosx_10_7_x86_64.whl (559.3 kB view hashes)

Uploaded CPython 3.7+ macOS 10.7+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page