Skip to main content

Arrow -> PostgreSQL encoder

Project description

pgpq

Convert PyArrow RecordBatches to Postgres' native binary format.

Usage

"""Example for README.md"""
from tempfile import mkdtemp
import psycopg
import pyarrow.dataset as ds
import requests
from pgpq import ArrowToPostgresBinaryEncoder

# let's get some example data
tmpdir = mkdtemp()
with open(f"{tmpdir}/yellow_tripdata_2023-01.parquet", mode="wb") as f:
    resp = requests.get(
        "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2023-01.parquet"
    )
    resp.raise_for_status()
    f.write(resp.content)

# load an arrow dataset
# arrow can load datasets from partitioned parquet files locally or in S3/GCS
# it handles buffering, matching globs, etc.
dataset = ds.dataset(tmpdir)

# create an encoder object which will do the encoding
# and give us the expected Postgres table schema
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
# get the expected Postgres destination schema
# note that this is _not_ the same as the incoming arrow schema
# and not necessarily the schema of your permanent table
# instead it's the schema of the data that will be sent over the wire
# which for example does not have timezones on any timestamps
pg_schema = encoder.schema()
# assemble ddl for a temporary table
# it's often a good idea to bulk load into a temp table to:
# (1) Avoid indexes
# (2) Stay in-memory as long as possible
# (3) Be more flexible with types
#     (you can't load a SMALLINT into a BIGINT column without casting)
cols = [f'"{col_name}" {col.data_type.ddl()}' for col_name, col in pg_schema.columns]
ddl = f"CREATE TEMP TABLE data ({','.join(cols)})"

with psycopg.connect("postgres://postgres:postgres@localhost:5432/postgres") as conn:
    with conn.cursor() as cursor:
        cursor.execute(ddl)  # type: ignore
        with cursor.copy("COPY data FROM STDIN WITH (FORMAT BINARY)") as copy:
            copy.write(encoder.write_header())
            for batch in dataset.to_batches():
                copy.write(encoder.write_batch(batch))
            copy.write(encoder.finish())
        # load into your actual table, possibly doing type casts
        # cursor.execute("INSERT INTO \"table\" SELECT * FROM data")

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pgpq-0.7.3.tar.gz (24.7 kB view details)

Uploaded Source

Built Distributions

pgpq-0.7.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (576.2 kB view details)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pgpq-0.7.3-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (534.1 kB view details)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

pgpq-0.7.3-pp39-pypy39_pp73-macosx_10_7_x86_64.whl (553.5 kB view details)

Uploaded PyPy macOS 10.7+ x86-64

pgpq-0.7.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (575.5 kB view details)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pgpq-0.7.3-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (533.9 kB view details)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

pgpq-0.7.3-pp38-pypy38_pp73-macosx_10_7_x86_64.whl (551.0 kB view details)

Uploaded PyPy macOS 10.7+ x86-64

pgpq-0.7.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (577.6 kB view details)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pgpq-0.7.3-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (535.7 kB view details)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

pgpq-0.7.3-pp37-pypy37_pp73-macosx_10_7_x86_64.whl (555.7 kB view details)

Uploaded PyPy macOS 10.7+ x86-64

pgpq-0.7.3-cp37-abi3-win_amd64.whl (492.2 kB view details)

Uploaded CPython 3.7+ Windows x86-64

pgpq-0.7.3-cp37-abi3-win32.whl (454.5 kB view details)

Uploaded CPython 3.7+ Windows x86

pgpq-0.7.3-cp37-abi3-musllinux_1_2_x86_64.whl (748.3 kB view details)

Uploaded CPython 3.7+ musllinux: musl 1.2+ x86-64

pgpq-0.7.3-cp37-abi3-musllinux_1_2_i686.whl (770.2 kB view details)

Uploaded CPython 3.7+ musllinux: musl 1.2+ i686

pgpq-0.7.3-cp37-abi3-musllinux_1_2_armv7l.whl (774.8 kB view details)

Uploaded CPython 3.7+ musllinux: musl 1.2+ ARMv7l

pgpq-0.7.3-cp37-abi3-musllinux_1_2_aarch64.whl (714.8 kB view details)

Uploaded CPython 3.7+ musllinux: musl 1.2+ ARM64

pgpq-0.7.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (576.3 kB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ x86-64

pgpq-0.7.3-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (718.5 kB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ s390x

pgpq-0.7.3-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (812.6 kB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ppc64le

pgpq-0.7.3-cp37-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (882.1 kB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ppc64

pgpq-0.7.3-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (510.5 kB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARMv7l

pgpq-0.7.3-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (534.6 kB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.17+ ARM64

pgpq-0.7.3-cp37-abi3-manylinux_2_5_i686.manylinux1_i686.whl (619.4 kB view details)

Uploaded CPython 3.7+ manylinux: glibc 2.5+ i686

pgpq-0.7.3-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (1.1 MB view details)

Uploaded CPython 3.7+ macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

pgpq-0.7.3-cp37-abi3-macosx_10_7_x86_64.whl (551.1 kB view details)

Uploaded CPython 3.7+ macOS 10.7+ x86-64

File details

Details for the file pgpq-0.7.3.tar.gz.

File metadata

  • Download URL: pgpq-0.7.3.tar.gz
  • Upload date:
  • Size: 24.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for pgpq-0.7.3.tar.gz
Algorithm Hash digest
SHA256 6e6dbd3fdeeb9f922495f44f400e40a033a1fd72190946b4a31184fb2e79cec3
MD5 d9fade6b4e803f12caf187f08396e66f
BLAKE2b-256 ee930c03e4de100405e2402b21a29fd416afd57428afef89f45b314d724d6deb

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1c7c457015a23cd7c4992583ce9c96d7c011d74d83fb1bed0ff0b5efc7e2e2c7
MD5 d5d1f4acec78927569bbd1f74f4f2fb2
BLAKE2b-256 a00dd792c8b157645c335c45227c92f4e244f4892ae96e189f9f876fb3267726

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1b349a4acc1e0fd42a7acf701e3fa36d1d21e9b8fecd83ec810bfa0cc599af68
MD5 eec466757dd2ddeac38e478e9f4c1d22
BLAKE2b-256 f90c309107a18a43770dc12bf6859b0e66f362f4f65e3db953c2f6f31b24e5a0

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp39-pypy39_pp73-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp39-pypy39_pp73-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 cfcd04649fe2af63d244e42dfd76b423b29c2e2687219ca6943517189ef8a260
MD5 de7438ec7fd918eb9f6bcfb9f4fa8274
BLAKE2b-256 4b412fe749f68155f1c2a5372481a9e29287223a7f0cd6de2fc177e8fa44f188

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 253c85134f4791bb95fb07b45584213e2fb87b07bc757470e3de5450abc782c3
MD5 c2300cfe508292981c0eefee3c7b4c37
BLAKE2b-256 33f366b29ffa41e28d15b9a656f593dc9b398d6b2df08dbcda6d117f5548fb6b

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 d9ba9a60d84c92881be489163918b78847a592771240dccaf55c63bcd7ae0f79
MD5 112b7f269e14ee1c79bd472a229c2b35
BLAKE2b-256 4838f7db99f81129a01069be87e14f16a18b64f2692ec0bfc23d2a42b0cbbece

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp38-pypy38_pp73-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp38-pypy38_pp73-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 001a81f80e969a04a9c39f9d5e71b9604a26c0471f77ecdd22fd10f638c238ef
MD5 eceb209aa8b81ac39c5c53071271fdd9
BLAKE2b-256 8221283c8435f8353ffb6cad30daf1a0538e965a34deca543f3e34d39fe12bda

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5610471873c25007be2669e2eaa824e4103a195fcd989ab8bc238f6a7b697500
MD5 056b90205e0d2325df6d5263d40b93a9
BLAKE2b-256 331efe28f227765c6565f0dba8f00ed667e25890f12514e33c23c123a63e6326

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 450c0dc7908db08ec6f827b44d48c8bb4afcd7e6eda710f1dcf5b99c27aaef18
MD5 4883850de0a2b374d8498a145208bb71
BLAKE2b-256 891457323fad1fcab206d57f468075e35da4c2385e06834bf997ff3a92892623

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-pp37-pypy37_pp73-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-pp37-pypy37_pp73-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 a31f6efabc3e25ec34f7001922dad09cbca0eb8d4b18432683a8a600441b9678
MD5 ac47d527da236bbb1737dbd8bdab8479
BLAKE2b-256 724ddbeaa5101c8d2cc29de64363d0dc078ce22e30226fcb5cd8e31ff2d43d29

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-win_amd64.whl.

File metadata

  • Download URL: pgpq-0.7.3-cp37-abi3-win_amd64.whl
  • Upload date:
  • Size: 492.2 kB
  • Tags: CPython 3.7+, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 b36ee72dfbfc90bf524385d950083d03087e3e401e501768395e21cef7f0df53
MD5 81e80788dd4a464c269eb214f6c529b2
BLAKE2b-256 5bbe2b958372bbfeebd0da80d2c0e6dc998cfe9837438116267db9cd69b21e5e

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-win32.whl.

File metadata

  • Download URL: pgpq-0.7.3-cp37-abi3-win32.whl
  • Upload date:
  • Size: 454.5 kB
  • Tags: CPython 3.7+, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-win32.whl
Algorithm Hash digest
SHA256 0c108fd756e985d573c17478dfb2f90248c3b833220c78cecdeabfee210c2be1
MD5 8472a719a1a3104ae4f16a8ad724b087
BLAKE2b-256 aaa2cc7a57539266d7d1ae4b8b8b677a98f79b392704b0f55751a045e4507792

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 6f4cbb11a4b0fe0596f550bd4e9a610b929946dadd3717b4c07a5a4419d141ae
MD5 093cc89f3dece663ea8f4c9b2c22940b
BLAKE2b-256 6e1136d65237347fcb0dbd438fd3633e55d7c2bc0655e80dbceccd2837394492

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-musllinux_1_2_i686.whl.

File metadata

  • Download URL: pgpq-0.7.3-cp37-abi3-musllinux_1_2_i686.whl
  • Upload date:
  • Size: 770.2 kB
  • Tags: CPython 3.7+, musllinux: musl 1.2+ i686
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 67bf32ab64e58d3ac25b764396151b589d98a3b53271f02e5a0c3d5f935298a2
MD5 3d8dbdc7ef95b87dac51f08523567443
BLAKE2b-256 0edeb26a49d7dada5083695e957572ad3614c34f4c9792bc1bc66de28a7eabd5

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 1bdc7c168b0bb22314a56d2dd75d79b32d6fc8495c676d5aa452df49a08f5b1d
MD5 bfbebadcd2846db6db92dbcbdf7d0fd0
BLAKE2b-256 a5561c8b4abb329667117a94c2ecfa60e63d1a788f91362c49c4684bafbd15c5

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 5ab2b6e9292c151d02ba59d9ae509f8324ea527dae5c86274a6cf7c4c9d1ae10
MD5 2438b57117f7a70a768534a9a71945af
BLAKE2b-256 c452051d71e2ccf473084781beb9b6b08acb7fd417cfea946279d1f6e2932890

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 afe971abdd14310bd1ec334b6dfc1a8261ecb57dcc5e3a5a71150226bf2922d0
MD5 b2c0c36db75594d4a6386d7f9e98e5d9
BLAKE2b-256 d0f93680d2a592ee7e4acaa662b15009d770047e2eb79b031f1fae41c6552867

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 a40866582c20040633cacdc295c752ccc6c3409dee480fc70fe7744ba8d4751a
MD5 5483bbf23adbe1c87bd0184932e48fad
BLAKE2b-256 32fdbca556477ed91db4ece5e96437ba531e100bb72479bec807ac3811ccbd2a

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 182a811c7b2cf28c55f6b3be68ffc3e1ded634026d6c38d2e72f1d0d29258471
MD5 3c14eb9ac3583054e814444fc7debb03
BLAKE2b-256 1aaabf66c6f15a65d1664deb4269dfa1132b96fd248b7973e8ae1ac5fd9840d4

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
Algorithm Hash digest
SHA256 27910c9d4e00d862a411f555efde10d99a8e8747f843bfa94bad280e0ebb285e
MD5 08efe44a715d0bd62eb4454f596c92a3
BLAKE2b-256 ee5a50c62e290e3a0abe798eeb0ac4294504744e911cbc78494776e16835b07d

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm Hash digest
SHA256 973e258c91d037fb339e00bfde55d41652fcbdcb404aee25c8190da243ffd325
MD5 c26c6f2c98c960419752e531c99fbfa8
BLAKE2b-256 d61cf27d458edb1b8e5d5dc9a3086fb63e824413236f588b2e5f33477790941f

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f16b6912eee6ff184c86933e9eaeac414798d3a20b19dd4fb8755cbfd0551274
MD5 0c6925db4e0542c31fddc03a9aca82a2
BLAKE2b-256 3a46004c625fad9545d2e99405e566d8dbc656b337eae02df4d51b0fbd3d00d4

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-manylinux_2_5_i686.manylinux1_i686.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm Hash digest
SHA256 3fa09019f156b54148a48897a1eb246f8a57b9121fff815e13d7c306d2c2d775
MD5 bed1235260c566f70c8edfaef92d2351
BLAKE2b-256 110adc5951f3d78322d7c1649c867f1e025e2639b4aebc50d7f493757000f2ac

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 d64aea430825a33e1dc31631de818dd8ef44f770e4034f43dceff1885cd9655c
MD5 d2bb7b49c9dd7b59d3524f62353fa08c
BLAKE2b-256 7a304029ac82e09e72f6c3f3516236ee7656d603e401d72ad5561320ba21638d

See more details on using hashes here.

Provenance

File details

Details for the file pgpq-0.7.3-cp37-abi3-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for pgpq-0.7.3-cp37-abi3-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 d6948098ff5187e9a25d9720e143b1fefb8d8df2a892399a550baafe3cef53e9
MD5 922a6d279b87214742c5205bcc34cd75
BLAKE2b-256 ac4818c564ae33ea1d5ce65a3cf3a0448ba06394124b8a80d1c707616d7034b2

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page