Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.2.2.tar.gz (663.0 kB view details)

Uploaded Source

Built Distributions

fastavro-1.2.2-cp39-cp39-win_amd64.whl (374.1 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.2.2-cp39-cp39-manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.9

fastavro-1.2.2-cp39-cp39-macosx_10_14_x86_64.whl (432.8 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.2.2-cp38-cp38-win_amd64.whl (373.7 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.2.2-cp38-cp38-manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.8

fastavro-1.2.2-cp38-cp38-manylinux2014_aarch64.whl (1.7 MB view details)

Uploaded CPython 3.8

fastavro-1.2.2-cp38-cp38-macosx_10_14_x86_64.whl (427.8 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.2.2-cp37-cp37m-win_amd64.whl (359.6 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.2.2-cp37-cp37m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.2-cp37-cp37m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.2-cp37-cp37m-macosx_10_14_x86_64.whl (420.3 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.2.2-cp36-cp36m-win_amd64.whl (358.8 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.2.2-cp36-cp36m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.2-cp36-cp36m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.2-cp36-cp36m-macosx_10_14_x86_64.whl (435.1 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.2.2.tar.gz.

File metadata

  • Download URL: fastavro-1.2.2.tar.gz
  • Upload date:
  • Size: 663.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2.tar.gz
Algorithm Hash digest
SHA256 78b298e722ea6da1daaf287d3e1d964658ccc9603939e03a412a85a6d26c8f0a
MD5 b069ec5a5779b0274a1c45d175cbc0b6
BLAKE2b-256 8b81934d339978587e6b7bb66435da0b5b745724784880cc84396644971cb66e

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 374.1 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 61c87e7bc1ae4ac2b9c22439fa48eb68a551c35c9d61e32675e81bca6d42d35d
MD5 915b5e716d420ebb2c9f3d46611c2d6a
BLAKE2b-256 522b968c5b123e7684b0741b91e8f8184292c554e77e919f4b148ea7d910fdeb

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7ed80b089bb4dc927f77d9447b3dc07241850af713c322f78e3c412062554878
MD5 d17cbbe24249ddc424769c837b87464e
BLAKE2b-256 e4ffeeb3c38888e1e2fb01a7369d157d9fb8ec527928bf82c7d3a856d899fcbf

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 432.8 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 7dd11fe5da9db3b0d83f184e77a4db1267a114a138767702e4d6a34b5cc8a4e6
MD5 6c69237bb2f6cf519be60d5bf66b4a05
BLAKE2b-256 2ff8bc42b620e76e7f06be8aea632dd1dca261802e7ff52654f61d128cb6ea78

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 373.7 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 da18fb1c9100ed2d16f32af5fdaa0bef5e48b5988550e600d64040d5a9790bb3
MD5 ffd38522f7327a86ef2d3ae46c99abb6
BLAKE2b-256 b5e8ab6e7ff6402f8f087c49a5e3bb071bf2c685d265667075371ef612eee6f1

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9bf0193a5f0b824bd667f99345246c6d688becd5dac1ae1b5b2f37b5df7e9fb9
MD5 462295bd5c48f4d696dc02cddc1f57ce
BLAKE2b-256 0cf357e39afff3dd1c0ac4ae3329f1d227fc13f9b58c42dec43f5ac66f672247

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9b53a2573c0107da9eb0e32c2ab8e29dc8b262755bcfde775367549830b7e8c7
MD5 3da1e55a34fca59a6ccd7199288c69bf
BLAKE2b-256 94f606b09caad023594243e1fe5c1fb36aca6ca69fc6e48d13ba4efe78b48461

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 427.8 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 6c78e317ab306b536df02449b6497944f081f7a9d21433a7ea32d59b37e2b43c
MD5 7a5852ece075f8780cd090beb8e57899
BLAKE2b-256 df37b90af68619708605b0e889d7a39152453cf39181a6e6060cb0b75a9cbff1

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 359.6 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 656e8517b79570b2e0362df7883d98131a3797103411fc73df88d52b18249278
MD5 b418ef756deee8f270b04cb78c55d699
BLAKE2b-256 82d4496312f4b41cb219b86e4c83a82074b796377aa162659a97aaa75d69c891

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 581ad645b10504e9f97ae2d94c084de7ecc53df76e0db876605ac11e9c8be14d
MD5 1598cfbdde17a82b6c0170bf5620e51a
BLAKE2b-256 79a122518a90eab33059f57379b6ced33ba56b332bacb81609a1b76611e3a538

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp37-cp37m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp37-cp37m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 63763e65b1c35f25521c0f771972b857b7c718101ebe9a77575dc72cb7b3589e
MD5 3fd6477fc657d88e52339c9741de1233
BLAKE2b-256 d55362e29461b0a5419ab529cb06fe7e96fc9097148685cc38a42c8f612599c8

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 420.3 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 de19f3a67cee75e5d6603d51539bfd1adb99a6caf0831707803dc77a11c071bf
MD5 a9d414aabe437b68a09040010f4d5f59
BLAKE2b-256 6dff612c7ff775e9950644bf1b2c2108f124c2e7d2a5a921e79f4b9e470ff031

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 358.8 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 45af2d79b3be00f8a059d51e606d9b2bdab5ad694f5c06a62c5a5f1eff6ab6d1
MD5 f036fcb8a009cd60252de0c3ae09ddbc
BLAKE2b-256 3a012dc4a826df824d176aa19a63434a280ca388ade3a5ab2cb7c8e9f4b3ee95

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ee54fd3e88d5969e5f8316dca980d9c64803bed0209053b56ba4f60174ff45b9
MD5 b9a8db776c1b972f53554193fe37706c
BLAKE2b-256 d7cb6cdfb1528717558b50647ec33d049192a050975cd6fa42e1f969a34a92ab

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 de2f735d97ea08e23deb20924fe46d3b336a1dc5c3b1f9bf2ed4d1439c3ff14f
MD5 07b412d9214f0b2524acc529c4aee7da
BLAKE2b-256 527c7089c94457bcbdf5761e8d84fba9be07bb2a6d0d0cf37a7bc318eac606cc

See more details on using hashes here.

File details

Details for the file fastavro-1.2.2-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.2-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 435.1 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.2-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 75008cb1e52aa893dec9eb0370fe624f854993879ce4de719429dd86f2a675a0
MD5 7afc7fe7b8f58680c8ce283a25f3aea0
BLAKE2b-256 b0d8136b38d463cccc506d25c233f2424878cdca81aec248609d4fb57c5224a0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page