Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.7.1.tar.gz (787.9 kB view details)

Uploaded Source

Built Distributions

fastavro-1.7.1-cp311-cp311-win_amd64.whl (405.7 kB view details)

Uploaded CPython 3.11 Windows x86-64

fastavro-1.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

fastavro-1.7.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

fastavro-1.7.1-cp311-cp311-macosx_10_9_universal2.whl (915.1 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.7.1-cp310-cp310-win_amd64.whl (408.6 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.7.1-cp310-cp310-macosx_11_0_x86_64.whl (517.5 kB view details)

Uploaded CPython 3.10 macOS 11.0+ x86-64

fastavro-1.7.1-cp39-cp39-win_amd64.whl (456.8 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.7.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.7.1-cp39-cp39-macosx_11_0_x86_64.whl (528.9 kB view details)

Uploaded CPython 3.9 macOS 11.0+ x86-64

fastavro-1.7.1-cp38-cp38-win_amd64.whl (456.8 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.7.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.7.1-cp38-cp38-macosx_10_15_x86_64.whl (525.2 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.7.1-cp37-cp37m-win_amd64.whl (442.0 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.7.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.7.1-cp37-cp37m-macosx_10_15_x86_64.whl (514.0 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.7.1.tar.gz.

File metadata

  • Download URL: fastavro-1.7.1.tar.gz
  • Upload date:
  • Size: 787.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.1.tar.gz
Algorithm Hash digest
SHA256 4b8bcae4ed6343af186e638061cdfbc5331cdb5e026d055099c91d4f07be838c
MD5 6a46902528d5fae5bea602c2372e5a55
BLAKE2b-256 fb4ee0383cf8ebae9ad613d22e21f85e438c8b34c73f9d3db01e3c5118248107

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.1-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 405.7 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 31c924ed2fdad56182b543941cdec9cc384443cc3ca9462e0580afeb4dc64f4b
MD5 9cad4e78d66a3c4d14c3b94fcd1ab2ac
BLAKE2b-256 63bbce0308c597205dc298a55d0c3cefd817a85aa194663f4f8d117f44de2571

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6f233a1d0d95265f81a25c274f55017bd39d9b8f7f1387a4235bf8e01841a9ff
MD5 b6786f5e002216f57de610a6059966f4
BLAKE2b-256 2cea20e6967c79d44644a3ab22efaf0857fae30dbec101c0d5a203e3d8b5a171

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e9be910ca1645e0e5c4192d667cfdfa58dff4f8690db5af1ae33602643d41a78
MD5 b5bdd3aede8bd41f347913d7c0bb44bf
BLAKE2b-256 b88d57aab0b45ec833ec988849cc3255435aefd5642e9eb3273745b539947ed2

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 8b671a97e864e4c024061c0d6f93f2768ba0595f917317ca4fe3e99dca6fcf3d
MD5 59993e8e693688f50a48bfa2672233a1
BLAKE2b-256 56896783876e9de2bcaa5316462d796785acd9e8488b7de870910ff0a0b1d716

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.1-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 408.6 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 2e3f5ae42e033dbb6e0efa788c4b8a4e5a59bc2b9cb83f717b6d85176727faed
MD5 9a7ffef2168e310367632fc8dc24f383
BLAKE2b-256 a105f11aa9d22a410ec56c867d4b403e69e8cbdcff42ea40da3994d78145c8fb

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 86ce89743711355eee7d3fec34847de1ab574f4567aa4a667e966711bb2e0cd9
MD5 8168b45237cc9dadaab59b43d3f07a12
BLAKE2b-256 461caff517b62e87a9363f11bd4d71e66a340c4f7abbdf5b96068e6fed5026c6

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 11159cf23f5ff4b752b028a77bd2a7941599932527e8a6512779e25b0503f037
MD5 4ba993e2623d30287f8fb66a43521b93
BLAKE2b-256 0113655a08c6ad303037e14a2ab54bff8866380d1d928d605d109fb647be0c4b

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp310-cp310-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 f8f7f11af8c2c074341217d6247b7ba09cadcd55f899e046c14e3a44afa5fc95
MD5 36a48bbede781e17fe82c740c972a3d8
BLAKE2b-256 4435032cd3341aec2138f8671dcac635efea6dce9d4af6e0b7741fb3cf8790af

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 456.8 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 4d56cc05ceb1c9f4de19c1f330a4f140af3b944834d63cd0e11517deaea21ee1
MD5 d8836beefbd5254886c999641cfb819d
BLAKE2b-256 8ea00c88ce093849cb0e753b4e2fc9007501697e0b5ed1ed01683c1e4ab6bf42

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e15e8163581983f07a3156902761cf746bbe1e646abd9553cc9a1cede6e23ae9
MD5 5480841925aefe68cf8d308665d9f3f5
BLAKE2b-256 3c2511ed2051ce5b9f9924ebe0f72d53e822e348818a1836ef82b069ad3e56bb

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 5cb2d8ea7b21493bb18ff7c68401edbc663bbd8a57016d6cf2c4b0a2dc4464e7
MD5 72661134d9199e6287a2394ad58ea5d0
BLAKE2b-256 995bfaf43a073b210ce12cd20273e514fea74627487c995c06855d7ebde73dd9

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp39-cp39-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 5ff701d6b228218a3d9c09b392205dd0899afa501a4f14724bef0ce13a719700
MD5 5bdff5bb330bddb3acbb3a073dfdfff3
BLAKE2b-256 40d56b99841e89de22a3dde576b69c01e06aa385cd194e77e13a3915a0dec0b6

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 456.8 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 a5f104bc5b4986bbbcab170918c4d8ffa4f8efa3ebe8ec190954178630074d5a
MD5 6c291eedb588dea81ddf7613f348eb8b
BLAKE2b-256 ac1768e71e40ec98b9b753f52159574549a9be1c0fe084aa9fa9655cac647b8f

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 980ce7fdccf328d287e8b789f4e3b422f64c84ed9cd81c05dd7c560c3d8076b1
MD5 d7c4f1b9a48d353f16002541cb952388
BLAKE2b-256 76d031fe2068b0b3edd3714eeaadec8de667894cada20d3834fba99c68f5dfb5

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 182671595683054ec234beec03f485b5c889c21c08e429577ae7929480703409
MD5 d157284d071ad03637aa31c723fda925
BLAKE2b-256 c557a0c82084bb105265d3807342614b7549535574911bdc2c3bde974523774d

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 a107a8b333628c0f8a2ece5d5a57c69923719a139b106ace4050206250df4b13
MD5 ce3585f5af53bbca3f93f4a5aab6785d
BLAKE2b-256 7f45e6a71cd73a3670b387c0281ad27f93ce855d776c669dcf72cc055d26afa1

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 442.0 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 b793252a6325341890bbbbdca2804d4db3d5d29ff7f15a58fcf84dda440808fa
MD5 1d2460b283b4a0ee91fd1c12ded60bf1
BLAKE2b-256 c31d3ca21d7ec943de6a8a55ed5463d0e444164a77835411f6f7a31f7ad91774

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4f7a744569b4b44ea1f4f87e7ea8298e1e2bf23779aa6ef255b95f9f38faad48
MD5 34eda59481b9b7260ac5f7536d24c82f
BLAKE2b-256 6787d8ccce14a9c85065562fb51e89fe92e8f767ed45ea43554d467e79fb094c

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 74fe0fb24a489e2fd567775935eb55a461fc6c4da8b5e3467245752ac2098284
MD5 b7f993917d93beb344f8892c7a76aeba
BLAKE2b-256 c67899bdb63ba0b399b9a7f890173f0a9360aa7263c22d81569106838a5f6651

See more details on using hashes here.

File details

Details for the file fastavro-1.7.1-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.1-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 8ad522c7f2c7791cfe54a7e87af8ac09634872f4fdeef28deec97fab8de38b24
MD5 f6511dffaeebea4893c4b4f205916bcf
BLAKE2b-256 0b92e3644667f340e21620257edd6a39149584bfc881724f45443d1f3fb3fa71

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page