Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.3.2.tar.gz (701.1 kB view details)

Uploaded Source

Built Distributions

fastavro-1.3.2-cp39-cp39-win_amd64.whl (398.4 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.3.2-cp39-cp39-manylinux2014_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.9

fastavro-1.3.2-cp39-cp39-manylinux2014_aarch64.whl (1.7 MB view details)

Uploaded CPython 3.9

fastavro-1.3.2-cp39-cp39-macosx_10_14_x86_64.whl (467.5 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.3.2-cp38-cp38-win_amd64.whl (397.8 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.3.2-cp38-cp38-manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8

fastavro-1.3.2-cp38-cp38-manylinux2014_aarch64.whl (1.9 MB view details)

Uploaded CPython 3.8

fastavro-1.3.2-cp38-cp38-macosx_10_14_x86_64.whl (462.3 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.3.2-cp37-cp37m-win_amd64.whl (382.9 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.3.2-cp37-cp37m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m

fastavro-1.3.2-cp37-cp37m-macosx_10_14_x86_64.whl (454.7 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.3.2-cp36-cp36m-win_amd64.whl (382.2 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.3.2-cp36-cp36m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.2-cp36-cp36m-manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.2-cp36-cp36m-macosx_10_14_x86_64.whl (472.3 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.3.2.tar.gz.

File metadata

  • Download URL: fastavro-1.3.2.tar.gz
  • Upload date:
  • Size: 701.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2.tar.gz
Algorithm Hash digest
SHA256 665bec52628f3b9e0f5efaa2519493a3e9947a67ac00700ef06a703cf4d10ab8
MD5 da0f6d195a0bacf248e359d544c28dca
BLAKE2b-256 33caccc7c9d9cec8a999ff4dbfaa1355c94e1151d2420fb0df5de3cac76442e2

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 398.4 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 5410c1d71a17c7006b084665d99b35c30fe5df318a1db9e1a7d917a08a0c8859
MD5 6b800b1b3a7f23d276209fc670ca362e
BLAKE2b-256 f7e3a0a41ceac4543e18efcf05ffb81bff2433d49a77dac7ad03680bd7593737

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d1a7e1f11420e99a788cb783c794cc3d9476f735cf49ee86f22084d86ae0fa1d
MD5 06bd41c089b3ed42a0a5efa36978129f
BLAKE2b-256 b6e81e42fd46fe85c41ea5f11eec54c5f48ab58d5f1dd6468b7e54983b689c5d

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp39-cp39-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp39-cp39-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp39-cp39-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 16552128560b22d7a64ac5a1d10e85290c00591540e4e9b8b962e9f201170311
MD5 b4b1d967aac3ab8682751262844a0f29
BLAKE2b-256 095b061bc812ad7d3f5eda51b18d6156ce196e3888ed724cf334e2b293a1b028

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 467.5 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 57a5a281eb62001030116ee80f060b3bb5a24827d764ed068e8c0aac56096053
MD5 70a98345f7d7f85f15a872f32eed58c1
BLAKE2b-256 68065de81da898bc734650388cee862e9908c82c6a846ccbf05844e73f40228c

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 397.8 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 683c1dea95a514290190c2ca5407130427620f98183e3f3ed2f19b02150fb1f2
MD5 a70c0c531cbd13e10e291cd3bb60f739
BLAKE2b-256 1c707622e94d7cc8fcfd561d8df1c2d11216237625e6379914979ee9726f6421

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b2b16c4c3b3ed380997b07b1c2a706415a4fa905a6c900d72ed1a402d06d3f47
MD5 ec38e04441e681529d8d9f43f989424a
BLAKE2b-256 e1c4e5647b152b8a71385862ee9d780e08b21a7bc08a0e98df742a612e7ac20c

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a908a4ad237089ad298d1c43eba24cc288cb6fb7d9530ad154cf58110c6ace7e
MD5 d1eb34475e6f5d4badb9f7f54cdcd4e2
BLAKE2b-256 c5882f97fd4a1b18346fafc244613bd8a747bcabc7667def3843caf4cc77b600

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 462.3 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 37bc3d4fd086fbb09d21c54961f0e6a5953d3c200d97da34cdbb4f50fc2f4d52
MD5 300032c549d5f50b88b6a0f9a195b8e6
BLAKE2b-256 94cc893c02c84c89836a5efacaa1f67fb914db62d1b3a12f7e7a9b948c144e28

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 382.9 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 ca974d3fab7c2174bc45081191c8e283558a3643cba39d5edaab89f8d42bdf80
MD5 fc156288072bea6656966577a3949559
BLAKE2b-256 9a2bcd051202b85b8203c96bf27163e4669ac69a6cd0fde4ebfe86a3eabb3e75

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 31fd897609b9f96bd88dd563876142da5ca9c17b9992946e1a6c27cf8dc3f99d
MD5 275e3c7dbc935f0878f11d821249d565
BLAKE2b-256 ceab95c416abe14608c3dfb3bf51478232520a8e450ddbac7ea348d614357db4

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 454.7 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 6275f86a3cb8bd4f10382ed28c0263a8679ff888d416a129abec045b21997302
MD5 372f5eb10152276898c22315d3384c1b
BLAKE2b-256 7a5d399f064c8cf2b9d35d0fc6c3e5cf44fbca75e6eb5ded377eb17b549beaf6

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 382.2 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 dc720c2bea98e1fcf7a984b3acfd49c322b0aa2cc45fd8e219804e1e0ff166b3
MD5 0a96a9bd54934ab38d2bfae8aa6e2a17
BLAKE2b-256 76ba1ba19672ca0273f345b633b554f9034494f90faee05149e80dbafd9a8e1e

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 85abc44d5f16a050c49b2d76f31d0e6e5895e7b0b13a6e968857adf2c92b5fd6
MD5 e1935879a85fba52f5dd17dcc5821a67
BLAKE2b-256 b8deb97ba22868817c021705fd9e3376530fbc0e4a575fa3d0a78492d6f980e3

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 dd07f628d1cb9897bb9e028f9cd90ea8e41feb453f564654419d60ba0df3d9fe
MD5 60e11622e7135dc41ab1be81dc732e0e
BLAKE2b-256 c7abaa2462162d679b609a7cf678048c26390047a145933b4986d086b380cc03

See more details on using hashes here.

File details

Details for the file fastavro-1.3.2-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.2-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 472.3 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.2-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 8f7bf53dadaa9b1bd818d3e9bf032e3ac3a6a4960b6c06ae5ae504e35fe0b669
MD5 fcc20a3f8e3fabc883e3324f7a2b58a9
BLAKE2b-256 2e25026acaa85935fdf6a4a16cf649e445e9cc0a05720de23feb51a544088c05

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page