Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.5
  • Python 3.6
  • Python 3.7
  • Python 3.8
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.0.0.post1.tar.gz (658.7 kB view details)

Uploaded Source

Built Distributions

fastavro-1.0.0.post1-cp38-cp38-win_amd64.whl (353.3 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.0.0.post1-cp38-cp38-win32.whl (310.4 kB view details)

Uploaded CPython 3.8 Windows x86

fastavro-1.0.0.post1-cp38-cp38-macosx_10_14_x86_64.whl (435.3 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.0.0.post1-cp37-cp37m-win_amd64.whl (341.9 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.0.0.post1-cp37-cp37m-win32.whl (301.1 kB view details)

Uploaded CPython 3.7m Windows x86

fastavro-1.0.0.post1-cp37-cp37m-macosx_10_14_x86_64.whl (426.5 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.0.0.post1-cp36-cp36m-win_amd64.whl (340.7 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.0.0.post1-cp36-cp36m-win32.whl (300.6 kB view details)

Uploaded CPython 3.6m Windows x86

fastavro-1.0.0.post1-cp36-cp36m-macosx_10_14_x86_64.whl (445.9 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

fastavro-1.0.0.post1-cp35-cp35m-win_amd64.whl (323.1 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-1.0.0.post1-cp35-cp35m-win32.whl (284.2 kB view details)

Uploaded CPython 3.5m Windows x86

fastavro-1.0.0.post1-cp35-cp35m-macosx_10_14_x86_64.whl (405.8 kB view details)

Uploaded CPython 3.5m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.0.0.post1.tar.gz.

File metadata

  • Download URL: fastavro-1.0.0.post1.tar.gz
  • Upload date:
  • Size: 658.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1.tar.gz
Algorithm Hash digest
SHA256 77e21bdcefc278df1d4c12e7bf787dcf767870478870a6e9e334a2cd486f1820
MD5 94e2dc3a4342791c4a4e844e63bdb2cc
BLAKE2b-256 bdbbbe327ed421ec063418ca2e328871f6b28f4efaeeef1bed5e3215c0140110

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 353.3 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 e713127a9faec87767137c93d01e7c60ac000b440a7f331b718f34e32c451797
MD5 8bd3a59b595e54b0f70dbbd5efa70108
BLAKE2b-256 680c9bc9844432cb8fb4861a985195892684552bd584f7fc69ff657b6fef16a4

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp38-cp38-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp38-cp38-win32.whl
  • Upload date:
  • Size: 310.4 kB
  • Tags: CPython 3.8, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp38-cp38-win32.whl
Algorithm Hash digest
SHA256 f69ae43180e89ffbf6feb8569fe5573f8f37214a9e4705ef2f00e279d3641971
MD5 5b7caacb0dc3ea634a750cf0a3f0b62e
BLAKE2b-256 a0dd0c8ae540a1edce89936f967723a5a2bf785f3ca3a2f7dc537b712111bed9

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2f4f521e31a4fe1da1807da696181e3dcb984b9ca8a64bad4816724974d11bf8
MD5 f6059e9f39a650d35f4b45638ccb412e
BLAKE2b-256 46ec6d42434527cc4fc1f24398f85ad49b206e8bb02c06d22aeda55afc0b299b

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 435.3 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 40ea33e099da96abeee3ab0110c5865b220e71b03aa1c1b90135bd40c21e6f23
MD5 93aceac27e71e2476d69ba5566fbcd95
BLAKE2b-256 1f7cd617fb710e1449c6c6a0bf07797b10b52c4cc6f62d7589885fd4d57a936b

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 341.9 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 b2cfe2ee4c4c9ee4d782ab5b8e3f6c427f7774d9061e31209e29d9d7071127ff
MD5 5b605fe92d41530f3395a765d3367d07
BLAKE2b-256 8b5875635884f2fce379f77a5b4ccfa83149f74cf4000a607e85dedf6e161193

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp37-cp37m-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 301.1 kB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 844985b80e331013ce5d341d41a21fe876ed8aa2badeeac6c1cf0592a5db26c5
MD5 d0b056721982d0bb7698e43adc99e642
BLAKE2b-256 9d23245c2a61547a95f9bf477aea27d2512a62edd221b5732f7f7c0ef4611f86

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 07a4fa3ea767dbdaeb827b144c8620212cf2e31bd573e31a5508e2a6757a1f17
MD5 4e718bfcf888368e91d14f9d00b5a447
BLAKE2b-256 6215e8388ace0ce6b253a531d5232ca8f13af3ea17ddab5e82e46321a4085a38

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 426.5 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 22f3501913116e81d6f323dc70ec9ac117728a168d54d7b9efd5256217e49c2a
MD5 7e78e12625d6b630bcba2d56b2caa6f2
BLAKE2b-256 7faca1b0f0f59f1e7310e694a316bb9a1ea17cd03e5bacc00775289708150105

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 340.7 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 7871df79df6349f1c4de55b57dfb1f663f2f9736ce915604a1667d8655fa076c
MD5 53f697e0f9c974fd7309f00bd9d97884
BLAKE2b-256 48053ecc9b8d17fb77e208a8fad1cad6465731fb7264f29b18ee6a73f00b673d

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp36-cp36m-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp36-cp36m-win32.whl
  • Upload date:
  • Size: 300.6 kB
  • Tags: CPython 3.6m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp36-cp36m-win32.whl
Algorithm Hash digest
SHA256 d389f54f7086c8e8400b9a0a7db1d2f9a94c144f3747e43bc3bf3a8b3eb6603d
MD5 9ea5d16416ab87e9f92100608e5aa78d
BLAKE2b-256 dc37512114f0bd61de2ebb79729b9038b5f751608d2b58f4c651620d7920143c

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 34384e2409a44791bab60730378cd82a2e6f2e60fc8373a802fc5004793d6f0a
MD5 d7d0fd930ad4b2391f1fa3ec46138638
BLAKE2b-256 3c471ce5e783fd7ecabcda82c6cb35c79779c747e77d23e459d46ece529392db

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 445.9 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 6e2e3688597c8556ca0c60a130c4fabbb4d603806c6b5f95f3744d5dbb9e51b6
MD5 a7399da1d199af58cbd3c454c8222cab
BLAKE2b-256 59aa81a3eb4183f2ad0dedb0b75d57f752263cae5464659b75b3681a8cdbaab3

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 323.1 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 d419396c4fde2696449f333d14381a43efaa98ba6e4a0c2ccafd560e6d8ed26f
MD5 54342571bfd8b887aeef074b6e4ca1a6
BLAKE2b-256 1610a36c6c1838b56b99011128ece1ab4faa757f0fd258e6bb6ca6e47af11130

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp35-cp35m-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp35-cp35m-win32.whl
  • Upload date:
  • Size: 284.2 kB
  • Tags: CPython 3.5m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp35-cp35m-win32.whl
Algorithm Hash digest
SHA256 7c9ebc6d030d99ec5213291b4caf94071aa1f47f1ec4356dfec878921333111a
MD5 43f77bd2ae9a6b2304c4e3c303dda8d5
BLAKE2b-256 952ec33272e9c74262fda6883e12ef82eaf532ce7112fa103f9932ac587dd1ab

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp35-cp35m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp35-cp35m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp35-cp35m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4316fbcab83d745313210e5df3b033392c7ce0f33731a7da3708a135e08065c5
MD5 d12e0006d185d1d0d966d9464830dce4
BLAKE2b-256 913a4e43312031107cea8d0f60939737732b46c51ac938e28b2d31a017c5a8df

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0.post1-cp35-cp35m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0.post1-cp35-cp35m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 405.8 kB
  • Tags: CPython 3.5m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.post1-cp35-cp35m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 904c897857550325a9a88802ee3fdeaf77f1dc15ab77072da803f0d69f2667f4
MD5 e241e2175f29de48fb4a4cc9d133e3f8
BLAKE2b-256 4b037839a4328b87eb8bda5f549ee3956038e56936e7997e8436c17088d93e89

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page