Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.8.2.tar.gz (795.4 kB view details)

Uploaded Source

Built Distributions

fastavro-1.8.2-cp311-cp311-win_amd64.whl (409.4 kB view details)

Uploaded CPython 3.11 Windows x86-64

fastavro-1.8.2-cp311-cp311-musllinux_1_1_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

fastavro-1.8.2-cp311-cp311-musllinux_1_1_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ ARM64

fastavro-1.8.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

fastavro-1.8.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

fastavro-1.8.2-cp311-cp311-macosx_10_9_universal2.whl (922.1 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.8.2-cp310-cp310-win_amd64.whl (411.9 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.8.2-cp310-cp310-musllinux_1_1_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

fastavro-1.8.2-cp310-cp310-musllinux_1_1_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ ARM64

fastavro-1.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.8.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.8.2-cp310-cp310-macosx_11_0_x86_64.whl (521.7 kB view details)

Uploaded CPython 3.10 macOS 11.0+ x86-64

fastavro-1.8.2-cp39-cp39-win_amd64.whl (459.7 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.8.2-cp39-cp39-musllinux_1_1_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

fastavro-1.8.2-cp39-cp39-musllinux_1_1_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 musllinux: musl 1.1+ ARM64

fastavro-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.8.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.8.2-cp39-cp39-macosx_11_0_x86_64.whl (533.8 kB view details)

Uploaded CPython 3.9 macOS 11.0+ x86-64

fastavro-1.8.2-cp38-cp38-win_amd64.whl (459.9 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.8.2-cp38-cp38-musllinux_1_1_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

fastavro-1.8.2-cp38-cp38-musllinux_1_1_aarch64.whl (2.9 MB view details)

Uploaded CPython 3.8 musllinux: musl 1.1+ ARM64

fastavro-1.8.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.8.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.8.2-cp38-cp38-macosx_11_0_x86_64.whl (528.3 kB view details)

Uploaded CPython 3.8 macOS 11.0+ x86-64

File details

Details for the file fastavro-1.8.2.tar.gz.

File metadata

  • Download URL: fastavro-1.8.2.tar.gz
  • Upload date:
  • Size: 795.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.2.tar.gz
Algorithm Hash digest
SHA256 ab9d9226d4b66b6b3d0661a57cd45259b0868fed1c0cd4fac95249b9e0973320
MD5 e7761a1a6c7376cba0bda8465fda6747
BLAKE2b-256 cceaf60f22c91e4b43e6e4feed52de575c1bef0f3bc611490b2d7c0d9d78fc59

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.2-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 409.4 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.2-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 fea75cf53a93c56dd56e68abce8d314ef877b27451c870cd7ede7582d34c08a7
MD5 ea1dce0cc8d11173a136d2c20bf67dc3
BLAKE2b-256 9c04fc6c6f7376a61f10944853261dbbf4e890285530bcb2375a39da9be0916c

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 d9e2662f57e6453e9a2c9fb4f54b2a9e62e3e46f5a412ac00558112336d23883
MD5 6ba9ebca063fb14c0b6a64b9adc5480e
BLAKE2b-256 e899a6ad9bad35fdb468bee7c620ff9c0ebf1224b9820fd701378025feadbae4

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp311-cp311-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 add51c70d0ab1175601c75cd687bbe9d16ae312cd8899b907aafe0d79ee2bc1d
MD5 6338a01f308ddfa01200b8fc4075cd19
BLAKE2b-256 8fe2059bf5f21b868e4a9789db2868f366d0bfcb40f2459d14e301a3ec9f6114

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ccc0e74f2c2ab357f39bb73d67fcdb6dc10e23fdbbd399326139f72ec0fb99a3
MD5 fa46deae0690ac49fca7ecefef075514
BLAKE2b-256 5b74da6fd6b4a818b9ffa4912988c47e432716ab03a4b5015524aabf5a9aa519

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 d3510e96c0a47e4e914bd1a29c954eb662bfa24849ad92e597cb97cc79f21af7
MD5 b4cdfa559c63728a03e3ce7a76272d95
BLAKE2b-256 a8b1a9e559a0f794095d515224a50da13970af80b5a6e793ea3dead430e6c5ad

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 2b837d3038c651046252bc92c1b9899bf21c7927a148a1ff89599c36c2a331ca
MD5 d7c01b409e76d201a21b5c976c3ec5b8
BLAKE2b-256 692c7b8b29bd49b3e76897afe269992c423978165e5ddeb6fe38a6fb0ab74a9e

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.2-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 411.9 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.2-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 bb545eb9d876bc7b785e27e98e7720ada7eee7d7a1729798d2ed51517f13500a
MD5 e77243639070f65a3b7fdb82d86330f9
BLAKE2b-256 e0dcae03e0b3a1644ed5abb8184a9ffbeab1a74c7251d5c96284ceb3b9168fa6

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 ffbf8bae1edb50fe7beeffc3afa8e684686550c2e5d31bf01c25cfa213f581e1
MD5 e2dcb5bd1368c3ac7f90f160d2963d54
BLAKE2b-256 92456f8f175acabc42d2511f77bc400f54d48a70e59599cb67e3577293a1f75d

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp310-cp310-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 543185a672ff6306beb329b57a7b8a3a2dd1eb21a5ccc530150623d58d48bb98
MD5 a4bed5402cb2c8fef29fa1d921976cf3
BLAKE2b-256 6c7ae2b50f3b4c0a3c742d96030a68343518d3e386a0489a20d890ed9f8985e1

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 eef1ed3eaa4240c05698d02d8d0c010b9a03780eda37b492da6cd4c9d37e04ec
MD5 50172da2ca8c225b40c3897dd0a7272e
BLAKE2b-256 43968cecc293becb041c0a6106d9b4cd8295f9eff30f883731204c3ca43813f2

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 401a70b1e5c7161420c6019e0c8afa88f7c8a373468591f5ec37639a903c2509
MD5 3cb36a7caa9d7dd34a057c88dbea03f0
BLAKE2b-256 cfb2e060e42e41b1893ac11ce4b95a6560d7bb11bb7ce39c6945cebe9c8d4fe4

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp310-cp310-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 0e08964b2e9a455d831f2557402a683d4c4d45206f2ab9ade7c69d3dc14e0e58
MD5 e0fe734c4aea3f55918be16bc4d2607a
BLAKE2b-256 5a52ceae6f26a32f7745001e7534431235f45be178316cba6151001588cf6d9d

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 459.7 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 650b22766259f7dd7519dfa4e4658f0e233c319efa130b9cf0c36a500e09cc57
MD5 ecdebbe96c2d55a904536a5b8b09145d
BLAKE2b-256 55124231c1c20207759fd72b274f8567436d2ad8a12518a2fe62848ed813090a

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp39-cp39-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 792adfc0c80c7f1109e0ab4b0decef20691fdf0a45091d397a0563872eb56d42
MD5 cd85908d28cef189f2c60cf63768b8a3
BLAKE2b-256 251e69817e55f24a2b19545f53531ae820a8e679a398b678ef8682c5dd868673

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp39-cp39-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 e9d5027cf7d9968f8f819958b41bfedb933323ea6d6a0485eefacaa1afd91f54
MD5 3fa08a64f736010c9a8a4dbdc559cefc
BLAKE2b-256 630ce1146e708a7a1a69ca45a44599d65404a63479df170540f99f226bb5e409

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 536cb448bc83811056be02749fd9df37a69621678f02597d272970a769e9b40c
MD5 fad3a47d2324bbce4f9581c12c19c73b
BLAKE2b-256 d13212202e93d0cac2fe6a38066a1264391fdbfd8fcfea446846d9dff4f68c53

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9787835f6449ee94713e7993a700432fce3763024791ffa8a58dc91ef9d1f950
MD5 df0132cab02603635674a811de5ea3aa
BLAKE2b-256 81f839e93de909556e722ad99565982b959865d16eb0f3f2260896e364ed4c43

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp39-cp39-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 38aca63ce604039bcdf2edd14912d00287bdbf8b76f9aa42b28e6ca0bf950092
MD5 56e79627a6e2f9c8a546fa0bc302ad0c
BLAKE2b-256 4cf05232af54f790d6ae5ea4d19cb9a63f53a08276d9e764157eb1abd4669a43

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 459.9 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 66e132c710663230292bc63e2cb79cf95b16ccb94a5fc99bb63694b24e312fc5
MD5 b241e5599c0830f2606b0504cb410738
BLAKE2b-256 5a96af952078d61b9d23852fe11f3e4310e14b70fc97874e92d70ae920da70bf

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp38-cp38-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 d29e9baee0b2f37ecd09bde3b487cf900431fd548c85be3e4fe1b9a0b2a917f1
MD5 d6b908df6839f9de2702a50cb12e28da
BLAKE2b-256 ccf83db3b44880059521c84a2d968623901050302c26da5e722ac43bfe41177a

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp38-cp38-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 7b1b2cbd2dd851452306beed0ab9bdaeeab1cc8ad46f84b47cd81eeaff6dd6b8
MD5 5487ad5aae5add4e460c5119d21b1b2e
BLAKE2b-256 77f39bdec760c907f00b127df4cfd30920b2a028c68b8fd022effef3156796ca

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 53beb458f30c9ad4aa7bff4a42243ff990ffb713b6ce0cd9b360cbc3d648fe52
MD5 4a0a1324ccd9da13f9cab6b2ede73595
BLAKE2b-256 bc24a0e07113b1f26e7707f0c0a3923b6e2bfda743ad65da2104e9ef8d985aa6

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a547625c138efd5e61300119241041906ee8cb426fc7aa789900f87af7ed330d
MD5 fd3f471ef9a57e8cbaba88edfd785ee8
BLAKE2b-256 90538cca2e098695b5753766cde0af5773bd4067cc1098e1b5c1aec3c5fcace6

See more details on using hashes here.

File details

Details for the file fastavro-1.8.2-cp38-cp38-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.2-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 f489020bb8664c2737c03457ad5dbd490579ddab6f0a7b5c17fecfe982715a89
MD5 40bad846e7c65fc1e66cf880250d7451
BLAKE2b-256 62bf00d5f6a64c00c1aacba48057bce48485a950e1b1dc8e0e25ff896f8f0b88

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page