Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

This version

1.2.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.2.1.tar.gz (662.1 kB view details)

Uploaded Source

Built Distributions

fastavro-1.2.1-cp39-cp39-win_amd64.whl (373.6 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.2.1-cp39-cp39-manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.9

fastavro-1.2.1-cp39-cp39-macosx_10_14_x86_64.whl (431.9 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.2.1-cp38-cp38-win_amd64.whl (373.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.2.1-cp38-cp38-manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.8

fastavro-1.2.1-cp38-cp38-manylinux2014_aarch64.whl (1.7 MB view details)

Uploaded CPython 3.8

fastavro-1.2.1-cp38-cp38-macosx_10_14_x86_64.whl (426.6 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.2.1-cp37-cp37m-win_amd64.whl (359.0 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.2.1-cp37-cp37m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.1-cp37-cp37m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.1-cp37-cp37m-macosx_10_14_x86_64.whl (419.3 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.2.1-cp36-cp36m-win_amd64.whl (358.2 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.2.1-cp36-cp36m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.1-cp36-cp36m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.1-cp36-cp36m-macosx_10_14_x86_64.whl (434.0 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.2.1.tar.gz.

File metadata

  • Download URL: fastavro-1.2.1.tar.gz
  • Upload date:
  • Size: 662.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1.tar.gz
Algorithm Hash digest
SHA256 49a88f1007617eecec8d9d00584de9c17e3eb806b5ecce779c8e882832bf786f
MD5 ea069b61cac8ff8580679a2f1d63dbec
BLAKE2b-256 04ca08174950b1f8e998c57c3959f418e93a25f5b9a53e310f9a971ee11ce2ea

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 373.6 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 c67606b25820dc4616b51948d34d54d75c2c1a8ac61fd1b881591599db7a9127
MD5 80f8267c279bb9345713abca2144012b
BLAKE2b-256 c56881265bfb9909e099560dfc605daae98b404d6dbfad360d647a3fa62d95dc

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 af04098dadc28d370732d7909a60aeaf15fd4aebbbcdaf68fff288dafcd3b7b5
MD5 80df73be643877010fa735b8570c7112
BLAKE2b-256 579be12d9226a16713e8b3fa9681f744d53fb993bb99e6e88e9aa39fd47d6e63

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 431.9 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 86d69475e8980e4837f9a8f1970d1c8988b01d801757c8a2eb7490be34e9b6e1
MD5 4a521ced953bd4cfc7aad9cb3014d814
BLAKE2b-256 8a1f077a1b29ff63bc635ea50311da7f4ba2a3d3d494c06d60c1b78355d9422d

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 373.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 0a95bdf62b274417dfd1ed072e31059b598407b9401e56d70bca02c25724364e
MD5 9d3184435674616173d9c7beae61f31a
BLAKE2b-256 ca827f2f7647e81a97938e53eaf8399bbeee2d0ea91f61ea9adc0bb8000f3789

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 143e0f85872d82ee1195d7b7a69c0ca9bb9eebf8e0b31440b831b7003b3ae5d0
MD5 0b5adb8a71581e871a215756e0f2c6a8
BLAKE2b-256 3033c10dd0608e03e19ebd9bc201f4232b7a66b8d35620b8e9b4385a6d457966

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c4fc6ccf4c9af37c43d8d166fd3726ad85375e7e0362581ccbf87d7b193dc3b6
MD5 7e3d6c05d4d3accf240a408ab0311404
BLAKE2b-256 9de7eef58d8bcfdecd6383190fa18cbb1a0d9c2695c762f521a83b67a07446e9

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 426.6 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 745feb862db81ed6998389709b6561bf95f4f99170ef2e6a8de06fa4c4de030a
MD5 bb79822a696b8b1f67a709b32abaa361
BLAKE2b-256 5139e93e178dba2bebabd5d4e9ffc81ba7b3ece736c72de87b38c1ae0920324a

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 359.0 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 db150a7c86a06c8fd822d47497c20228435f30dbc8e9101a748772df69c3c3dd
MD5 2da08718460d2123f8ff438ab11f2156
BLAKE2b-256 08016dfcbcd2d7249ea4f8ed33ba2b658d2b8740ef6ea6f0106bc7f7effaa5a9

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 210f2212beee2ce0b1deb24e9d6eccac3b82132bd47418d65f9e72ffbbec3cba
MD5 a1657a9b6c01ea04c4901783114f0b53
BLAKE2b-256 1d48d5616d5afc43732ed96dd9d1226525b1fe6ac470d5c288d9937a5277d166

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp37-cp37m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp37-cp37m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8be29b2e32f60f2083403d68c6ec60b441f1f048592434e3fe297a3e26afb2c7
MD5 530d794139a92a4e276a75b617b4f433
BLAKE2b-256 001c3029535ac2fd469541c76cdb4789ac71d75736ec75dd433b6a07d3e82ae9

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 419.3 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 2198503c8a93328e7d23a919d4bb71dec8397a0988f757e606949535f9a1935c
MD5 4e6e12003e208b759b27ddfae10c9b6e
BLAKE2b-256 90216759c7fd75bd46c8f2da612c0625123436b0f6fddd310e9528db98debf30

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 358.2 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 eb7daf7d1cd61358a17076b2b69f4af0fed9212d61dac10b83e4747e88e8d0fc
MD5 39dd368dcaac5975b7e9d8b05e075c05
BLAKE2b-256 4a483223f0daa1d0e1a30ed8623f1df3be376e107410faf5be693f66f96761ad

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9309c83289288816c5b6675220fed1420e75bb64f44ac8ed41e4ac747227ed61
MD5 4afe5e6bc80c862d7e53ca5dccadf07a
BLAKE2b-256 5ea27dfb27ff9526bb5ac0de00b9d33846c9a7becc9fea5e9564a2080274ad6a

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 54966b195fffd7131104028b7bc82f3825515e5d60da5d123e483ca558ecae7d
MD5 c4bd6304dc6aa239629475501cc37b76
BLAKE2b-256 a1214463d2fc60caef7f76c912b2f4dbcc306c33c741b1792b5f44b4f460b279

See more details on using hashes here.

File details

Details for the file fastavro-1.2.1-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.1-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 434.0 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.1-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 e1604e682a5ba676ffb9b10e6a798c01c3cd7cdfa4dde863f7b95eff25eab7b7
MD5 dce356f2fd1f7c544539a9c6775f8050
BLAKE2b-256 5bbc3459b4254f1b20bde85f3927fe66389e155ba1247c6eb3716699b3b669c9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page