Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.8.1.tar.gz (793.5 kB view details)

Uploaded Source

Built Distributions

fastavro-1.8.1-cp311-cp311-win_amd64.whl (408.7 kB view details)

Uploaded CPython 3.11 Windows x86-64

fastavro-1.8.1-cp311-cp311-musllinux_1_1_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

fastavro-1.8.1-cp311-cp311-musllinux_1_1_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ ARM64

fastavro-1.8.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

fastavro-1.8.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

fastavro-1.8.1-cp311-cp311-macosx_10_9_universal2.whl (920.9 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.8.1-cp310-cp310-win_amd64.whl (411.4 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.8.1-cp310-cp310-musllinux_1_1_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

fastavro-1.8.1-cp310-cp310-musllinux_1_1_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ ARM64

fastavro-1.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.8.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.8.1-cp310-cp310-macosx_11_0_x86_64.whl (521.2 kB view details)

Uploaded CPython 3.10 macOS 11.0+ x86-64

fastavro-1.8.1-cp39-cp39-win_amd64.whl (459.2 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.8.1-cp39-cp39-musllinux_1_1_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

fastavro-1.8.1-cp39-cp39-musllinux_1_1_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 musllinux: musl 1.1+ ARM64

fastavro-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.8.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.8.1-cp39-cp39-macosx_11_0_x86_64.whl (533.3 kB view details)

Uploaded CPython 3.9 macOS 11.0+ x86-64

fastavro-1.8.1-cp38-cp38-win_amd64.whl (459.4 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.8.1-cp38-cp38-musllinux_1_1_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

fastavro-1.8.1-cp38-cp38-musllinux_1_1_aarch64.whl (2.9 MB view details)

Uploaded CPython 3.8 musllinux: musl 1.1+ ARM64

fastavro-1.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.8.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.8.1-cp38-cp38-macosx_11_0_x86_64.whl (528.0 kB view details)

Uploaded CPython 3.8 macOS 11.0+ x86-64

File details

Details for the file fastavro-1.8.1.tar.gz.

File metadata

  • Download URL: fastavro-1.8.1.tar.gz
  • Upload date:
  • Size: 793.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.1.tar.gz
Algorithm Hash digest
SHA256 85bb4f129ea947dd18432cd63fd3d9ca30d75921dd79202c1258cbcfe7928644
MD5 b3da77a213e9c5356b7eba80918324d1
BLAKE2b-256 be6d9f0b8839b4b2916554420826214001a464fc8de04ab5564777c5d4018ef5

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.1-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 408.7 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 e4bb62111717d1d9545547eae7d1b56e51cca59233cb99dad71d2aa6da21694b
MD5 56f97ecfcd97e41a0ad2e207ab0d817c
BLAKE2b-256 9ab89fb88ffc916ba80cfcf98035807166a30c453e5b3fa31110515e4601b5c9

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 44636e43b56eb9011b8b7118522abdb6f01ad40e31a6ecc808a5a6e28411f541
MD5 e4b5bd700c373d9c15953be0a4fdb8a2
BLAKE2b-256 36048fe08fd4c0aed66a6f4ade4a683d65579365d8ddc1ab8a52c6c19e5430ca

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp311-cp311-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 418c94c663c044c563050dd58948a41ad78b20264946b13d2143ac6ee3c2f4c0
MD5 5bf6625fc280b4be64e3720df212928b
BLAKE2b-256 0efc982bd812ca1e637c80391a8652a38613cd369691d04bd2f9832c05c6737f

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 730b7f39370525c5939746b353b366bc7189091ae1d040f00dd8c2e7186ff248
MD5 fd3f54a237801f51c220df8cf0194e3b
BLAKE2b-256 e843bb433dbbd4f42e92fcabea60563aa3267937b28372754bc8bec5fd5d21b3

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 df038246337e588d2590fdf063578d51d1ff3ab93defaf5197a9f0333d5e24f8
MD5 1f7590b7895f57583e7dfdfdee2b2745
BLAKE2b-256 a570cc9dfb45a6feee29cfe5b4442a846c8de898ab0ca980975ffaf52ffdb9e2

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 07aabb0317257cd98e55e1d7461bec4230eaa671e2a46df3d15940a006f069e4
MD5 69482a4891e59e0e8d240eac2e217950
BLAKE2b-256 5d08c3d7a2b21ec7b6bd1efa56482a067c05c7dc7fbff406e628b37241030cf4

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.1-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 411.4 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 4ddd3658ddcfe878098effc936804fcf8b39204ad32ceb9a151a9c83b1aea2a2
MD5 1bc06c42a6c414a0d3dae9e781120cfd
BLAKE2b-256 adb891933e47f090204305f4f395ebfa522e6e0ad6ef34b8a6d53952af814fb2

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 247c164c21e9ee398a3a4d6d9722fb638f58aa453cc5da9e17d9e5e660b33212
MD5 e58c200a4e1c4bcecb5b0250ca7aadef
BLAKE2b-256 d809dfb6e2dda23a88bc90417022401cb2ba33ce0c1faf8ea2e52014f8db64e6

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp310-cp310-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 b46383b58ccd7b4ce08ff4b58b6cf76fa6ae6404d1794735ba90737094d2569e
MD5 2aa8a27b7f426594d592e524a8a2a905
BLAKE2b-256 230ccc08620ce12c75214702a9b2808909001b2747f9c83dd96d88084b344409

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8f9ff18fe8045a2122d804e50ae66545b2cba2520c4694da49c58ed1b98579fc
MD5 47d29e8018fcfb5defbc832a9fa0755e
BLAKE2b-256 e0b25842419ae3f2c8334b70188c41de54b9e4e5cab490fde71c050236637062

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2c9d5b630429665aa0f3779522aeec81389a7e7b587ccbb67c03acb5bbb182b8
MD5 30db9593aa2cd8edca7f141acff5a2a4
BLAKE2b-256 66f55f31d61891ecc56fa463081f0a074bf514aa5666554512e0fa8077ffa56c

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp310-cp310-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 26f137660594a377a7f913fd9dd453845ee242ecb78bae62646778ab6b519002
MD5 43dea7f613d00ae726d73ffb4fb2fcb2
BLAKE2b-256 8f4923dcfebef777b053bd0fca14e71cb4f1c1e4e0639ce47f653519a7c1aca3

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 459.2 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 840c052f2db02a7ec366a26920716161e5753c111f34095a8ef9b8664e036725
MD5 715c0b718c0728d59b187208396ff057
BLAKE2b-256 f30ca273f8f91926c5e7fd6b88bd9ba89cd065b2792bdd649f6560277f1190a3

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp39-cp39-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 0b71669d4fc4a67ef6c10d2b66fb504e8512d4658268c34b4cac0f1ada548eee
MD5 3afec488d192a34309e51b9adbaa0927
BLAKE2b-256 f8331f8770dbbc68ec32432f842d280bcbef9f7a465aea596022b9cec3770aa4

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp39-cp39-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 abd8b396b35c7422fea4d17cac2c79adc8c3fc858c7a3ff400ec06b7a7a86091
MD5 42428e56a98b72b78fd9bcbc8b171359
BLAKE2b-256 485f92050c830e184b9a3169c9385dc3856c8629a5d44fb791e93f04f23546f4

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 00ada78cf9d899e03867f6398aadadc89ced1bfb92f0c471caeff2101b3d16ee
MD5 c327c032bc2f4d9f7fc2c06365801bfe
BLAKE2b-256 5198e464c0d23d3f8943f69a4c30b7400a93dae5ae9ddfad2df1fc6a27b273ee

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 5ca2a4e96b4e5da56d67a12d79b4dbdee8050735a4d75d49f36d2ffb00f1dc6c
MD5 1587ed16a95488ec92866a3b582a48eb
BLAKE2b-256 ce84169362b8b4dc55cca1f778f84cdd64ea0d9a79572fe88e3fedbb73061fe9

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp39-cp39-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 494035c8d547d80d37d841ab9a68726280c9bb4fc0b42c6ec10e6a11cadc1fd8
MD5 7cfae9b7f1c3779b4846ea7ab6383923
BLAKE2b-256 45cc55a3a36de78366b456b1029d4424e83f739ef493ce8cab983a37356bbd29

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 459.4 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 09641a75b8051e5235640b4abe7fb204f63b4df99cf9f5826913fdc8e980e9be
MD5 c0b014d77eeb58ccbc080388fe74a1de
BLAKE2b-256 9a9eb1e0740b79ccbdfce7c16cb8273f1fe502bc2c998fbb00f3f861f8f038d1

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp38-cp38-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 d4245bc7a12b6e2ef5f42313ced0e4db1beb076558cde76f38bee48ae6d2f9ac
MD5 8b8884bdbf3d40efa63177a1b3eab21c
BLAKE2b-256 6fd0e0e707979154615c7409ef455117a64fa1e46696266ffc2490c18e911fc3

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp38-cp38-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 ad467a5f4787710044c58450eb36719de3181fd90ed1f99ab325ec0aa80e7a39
MD5 f96390e29e541fc842f5a2c57701f7e2
BLAKE2b-256 aacc3aa1ace417cffaeeb98649d3870bf9c7561192c2852c8b83c44a86631769

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ea0ef0104518deea8cd5e1c38c205ff31ffaf5db8eabf587605d7665d1e4b872
MD5 ec95d4cc9eb87b281e096b0a625fe051
BLAKE2b-256 e94bde62e8ac60dd3b4746545910f0380ae380538f91bad1467afd156ddc90af

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 0e8516db9ed76436ff0d5da46e15adf59cd30f68c68af5aecf13c062594fa673
MD5 244d83c5e3c026fb7f2925663733519b
BLAKE2b-256 488a3d95823e9083bd5abd8218e7006d5faafb566c1448e627916a3aff4872af

See more details on using hashes here.

File details

Details for the file fastavro-1.8.1-cp38-cp38-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.1-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 2344b8bd3e666ab516d3f9deb6812b65d2a9f61886c2bdb7e78fa09b5b171da4
MD5 e7ca0b4daed6a044674823ff9f86675e
BLAKE2b-256 fd3664c60a00c39f2d35a31b107b5b8ab06f11c51e58b644b25ec5cf5fd157ca

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page