Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.7.2.tar.gz (791.3 kB view details)

Uploaded Source

Built Distributions

fastavro-1.7.2-cp311-cp311-win_amd64.whl (406.7 kB view details)

Uploaded CPython 3.11 Windows x86-64

fastavro-1.7.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

fastavro-1.7.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

fastavro-1.7.2-cp311-cp311-macosx_10_9_universal2.whl (916.5 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.7.2-cp310-cp310-win_amd64.whl (409.5 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.7.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.7.2-cp310-cp310-macosx_11_0_x86_64.whl (518.5 kB view details)

Uploaded CPython 3.10 macOS 11.0+ x86-64

fastavro-1.7.2-cp39-cp39-win_amd64.whl (457.6 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.7.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.7.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.7.2-cp39-cp39-macosx_11_0_x86_64.whl (530.3 kB view details)

Uploaded CPython 3.9 macOS 11.0+ x86-64

fastavro-1.7.2-cp38-cp38-win_amd64.whl (457.9 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.7.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.7.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.7.2-cp38-cp38-macosx_10_15_x86_64.whl (526.1 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.7.2-cp37-cp37m-win_amd64.whl (443.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.7.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.7.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.7.2-cp37-cp37m-macosx_10_15_x86_64.whl (514.8 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.7.2.tar.gz.

File metadata

  • Download URL: fastavro-1.7.2.tar.gz
  • Upload date:
  • Size: 791.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.2.tar.gz
Algorithm Hash digest
SHA256 3c524c65a5d578c020997800fce992cd90cc88ef13148b70f01a3429de34c8a4
MD5 1cffc697e9b6b5fa630deface4ea6e41
BLAKE2b-256 9c224dbd4e88640a7fd083d97c7d1de2e8d4f3cccc5d52d8f61a930363bcdb41

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.2-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 406.7 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.2-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 4c2523c5e2279debb5fa788bf665a5abbd16a1e889338b973e3b318d50fcce77
MD5 5a7abc66ca81b47fe4e11b12caf9c2a2
BLAKE2b-256 b066b2ded91879c111945a3a52f7a7a139b71ccb70bcaabcc95c90cf0cda388e

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a5c5d466631f455213b8bfd54888d2d8691cfc240cd6dbb40eb3694bf05e562e
MD5 511e497250a2fcac66ba8059196ee686
BLAKE2b-256 ec97ab916331e522e7135427bf932cab04457a14e0091234ecb1b5e5fad5097b

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 88005d5c6fe436e80f756399a52aaa6d0cdc506c1ee195277287a09c7617aec3
MD5 36fec4a4169c1cf3ecc0c4a71f4b32ef
BLAKE2b-256 bf18cf77abc9a21dd811e973fcc55ed88173cf98a380a21fa20006987775c46a

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 4c44f72a1a7b2d28cc21c95686916b782354dcc1f1aa1c8ca044c66d5ada3e14
MD5 ce7a77289ba49372da37b3b15a20c02f
BLAKE2b-256 4cd35cd974631f2d2377a999533451b3fc97828fff9ae4c9bddb7185224f81f4

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.2-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 409.5 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.2-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 e83069f971f307175ab806e9929a15826e586ef05a620fcfc7768ceaa3ec19c8
MD5 0b97da24163a8b37b29e64d3c21b8ca2
BLAKE2b-256 48590d85718e990bedd1cb997cb3910a67f6f0311261738d38e550c1ee4c479d

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8993e2f34e9ae195f417105d495fdd3d7f6d1532bacf57dc5dea600082f9b1a0
MD5 ad999af5272317a0bb2430e53e9d2ed2
BLAKE2b-256 1a6feaf68cb048df22114a5b0924ab03354c5c169861a6d487704fd9af6f8794

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b34e9cd755e4d9d12377fe151ad95b9bff08215f0a3756601b21ed31f4d127ab
MD5 bf6dad7c38e51518c6cd05e927453c78
BLAKE2b-256 41938e4637e76586c7b4d09f89eb3f8a7c0fd04d195b18d65c0a57d88e1a5291

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp310-cp310-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 20a94269bc0910c41f06106424429289e5c8cdce65b006e48d600c9bb6c442a5
MD5 10dbf6afa47a6bbeebc0c50a15bd5d15
BLAKE2b-256 a22e1c186f80346cad1dcf037646ac0c2976fed290aa6149913220285777f16b

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 457.6 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 d787e7739d93db59f378c0e44a4eca8e56ff979e5b43dea6378e87c48548818c
MD5 13eb1a34bd992ee633c2e53c4ab4485e
BLAKE2b-256 548d0ce3593abb0c5acbdd17478e685e37850400d98bbce49f28993e9ab7af6b

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 06bd51962df9b3f28c7081c0a3c8663d840df05d964b7817d85bd37c1d101236
MD5 97a12a286c2cb9040adc3b2bdc34db16
BLAKE2b-256 166feef2430bd35cbfb9b831c066332ba0ff9f0968c8196909966a4df863a2a0

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 22cb5d2bf87c1ed9a1dcc97b6b13dc6b7ee4395dbf1fd5903dac01cdde0291c2
MD5 b12e20d3ac4f6fcd29042d178ba10b6b
BLAKE2b-256 da13e712a70cfa2aca6414ca96e024eb27b81e2c06429f9aabee147f699dd4d6

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp39-cp39-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 c1d5805c50a9507c99b814cb014da2812e6e0ee2770f4b793ce56b82a5b0af21
MD5 49c896a7b950e19871539a0a59ecdc3c
BLAKE2b-256 04f03530dddd633d52c36bd408dcbc587d80e1106e82f1a9c325030639ca78e2

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 457.9 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 e842beff23b0b207c45f99fa79d50cfc0d84ac137d84a7e283e9e59a8368b115
MD5 114bc742531e02b19761ea77935e221f
BLAKE2b-256 8ef0202b5dacb69c84da1b08cecaadf782015b68be8a27214a4f65fc34207b98

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8da1a9d4f0b871e7c968c8413caef19e19002100036b2714e037543c36550b8d
MD5 c17695ae90a91d9ff540ce33d8715d3e
BLAKE2b-256 d61ee0cef294653eb6c4c797ff1a1c54ac7d71511e4f82efe3f306e8fdbeb290

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9919d866319b1e123374fced84afb4728802872bd5938ba7e12a691c54593542
MD5 d4acf6e4876f3841dbb24ec2dd1f0eb1
BLAKE2b-256 b7e2f0dbd3fedd1de962200b81be5952961a77ab863618fe849f3e84244d9b07

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 f07d1f2ebdcc18938908e710a79ab3731061936fbea27534e436f9e12342450b
MD5 32f9315b7f53cc9ead3ec0f6dad0fb02
BLAKE2b-256 e2b342b100f56d6b74ec4d05735435d6a8fffebc4ad76d63a2721c52ca9eeac0

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 443.2 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 b8cd8bd9a8ad4b687bd8dbf12effc2996bdf41ec04960875c19a7a90c949e8a8
MD5 90a3c516880a241cb5784903b055beb4
BLAKE2b-256 ec414c7c7e4eaa6042fe242f158830f544b0fe603c3679a16a515dd0674685a6

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5417dfbf99ac5bf963a929f2541c643d35e1a5a22876ba5ae0dca7b86326d9c2
MD5 5ad9389d74f220556917a66add704216
BLAKE2b-256 89063d5722a6db57fe6ddfaed22bfd7e98fda071b14aa6cca414b592ef95e3cc

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e82862cb7d3bf99fdca65048b8b09db295795d7fb1eb7c5dae29c0a4d7dcf1f2
MD5 70ed414f16531342f3f891eb28862656
BLAKE2b-256 466de6dcdf40bbf2f74fa552c73e79b0af8a0a4b927fce9a62cd2cf4884f1b7e

See more details on using hashes here.

File details

Details for the file fastavro-1.7.2-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.2-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 db3d9b6237934be5001fc18c07d25c09d3008ce6ab3448daf4b30608b84488e0
MD5 1ebed3d4d6685cf0d93742687ad19baa
BLAKE2b-256 c7a7106d72f08026f24dcc565d9977224807e5de64579d5780cf04c63b67e3c2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page