Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.2.0.tar.gz (661.6 kB view details)

Uploaded Source

Built Distributions

fastavro-1.2.0-cp39-cp39-win_amd64.whl (372.5 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.2.0-cp39-cp39-manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.9

fastavro-1.2.0-cp39-cp39-macosx_10_14_x86_64.whl (431.0 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.2.0-cp38-cp38-win_amd64.whl (372.1 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.2.0-cp38-cp38-manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.8

fastavro-1.2.0-cp38-cp38-manylinux2014_aarch64.whl (1.7 MB view details)

Uploaded CPython 3.8

fastavro-1.2.0-cp38-cp38-macosx_10_14_x86_64.whl (426.0 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.2.0-cp37-cp37m-win_amd64.whl (357.7 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.2.0-cp37-cp37m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.0-cp37-cp37m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.0-cp37-cp37m-macosx_10_14_x86_64.whl (418.1 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.2.0-cp36-cp36m-win_amd64.whl (357.1 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.2.0-cp36-cp36m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.0-cp36-cp36m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.0-cp36-cp36m-macosx_10_14_x86_64.whl (433.8 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.2.0.tar.gz.

File metadata

  • Download URL: fastavro-1.2.0.tar.gz
  • Upload date:
  • Size: 661.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0.tar.gz
Algorithm Hash digest
SHA256 9f80c8d87b6d5ec4294caaa15ea8381724e71638c5bdf33af9a0b4d739e8860f
MD5 5d409c9c4d54d461e09d23b8c4ce9328
BLAKE2b-256 e6a2ecaa275a3917dc5fb1aa498ac50a368dd43e8b76d07e293c422e15931e27

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 372.5 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 bc65d74c35d2ae53af0647e9ffa008c6fdb41e0a9c37e34c5681d915437db8ee
MD5 a7f39617cbd17ded8a8ea7768f78f8fa
BLAKE2b-256 dd6f1aacd7fae6af4ee4d3583c013b499bfd5a5ae2b2a6aba75e4b8263ec7b59

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c787d793e9b2e4e12d03ae2b0abb52625717d3f4d3161c841bd09dc2520b67a1
MD5 b7cd349aeb65a664b77cce5cb8f3204f
BLAKE2b-256 fcbb5cb966250521a98ac72ab5dddf659831448bf5be3b3d133a7ec5e049e295

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 431.0 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 a0fc504908da59bd3610d0d544bb8845746f82aa169c4c1fb1547e31bc783c51
MD5 550fd6ce3e8106d09ef756ffb98fe400
BLAKE2b-256 92bde1b441b7e4749b0e0c157d67592f32c76e5dca157d6be4bdcfc2fce9dcc8

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 372.1 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 9223f4af6e14d985c59fb2e85788927725a6e2b37530ae65f4cce33f9af2b113
MD5 d2146ad9af52c694d9b161509b410fce
BLAKE2b-256 e89550c943c26725aaf9d763183036596355875e4be5e0682830f33eadfb6a14

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a8abe3347b322925da8b5a97961ed87111db97e18e03195a6aedad8709a47136
MD5 86c92d484ee012e13634b3f5133b9c2f
BLAKE2b-256 303b26283ccd47eff8a8fc6fe2aa9a4847f58cf871b769acd1629342fafe9c74

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9964eb776d705dee923c55a488d118995637b0e3535f94e84d04285ba5a27d36
MD5 9d35cffa3707da0f4c87886cb7388539
BLAKE2b-256 d11ac98c8a2eeb94231042a0e7a6b7ad71ef2cbc80d7d9e0a6f93e87ff8e512a

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 426.0 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 5a5dd6defe7b4e9e07fce85a863414e1b898257a889de0bb3240f9c91ab25695
MD5 558817811579579c092911daadbb551e
BLAKE2b-256 fccb3f124ed62b5d30d107d2cb936a81391faa3de6346bedd508c5c3efce0503

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 357.7 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 1ff45a051798feee8fd521e700eb6315afe741b7014644bb50454e32a35819a9
MD5 333a6030e19334d9084a4881c145d7b0
BLAKE2b-256 c2ee996aeadd0ec7e46bd80cbe022364099839f8c20de16ce4806627895b8ec6

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4b207b11cee694f30875eb9bc05271ef165e2a4a509b1f116b3b10dcbef1de6b
MD5 e1c2625e81e66f46942572b49c235fde
BLAKE2b-256 52e0d85b4f1aa06e2153e1af1b3f8ca75e88d8fee8020ddda09fcd4b0234d2a1

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp37-cp37m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp37-cp37m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2a4073fc831fbe0ea8ecb6c5d12cd5c7c947aff609c35a3020f99204cfed10d4
MD5 e23532f422c92ad17ea9440287820d2d
BLAKE2b-256 db72fda7a73b52c0555141e226644a8e4f33cdad39072dccece3f2498e1852e5

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 418.1 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 315e1d6da9be7512246977d1e89b77dedd72ab914c6f4f6af5409d24a16eae51
MD5 cbc69f24f566a6ae0ade1736d1f12ff3
BLAKE2b-256 aa5c7e79c3a0e549e8ba0a451fefbd708fedb93bb1180a70500959ac6b475302

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 357.1 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 213038249aa6ee83a2ae6da22a55008cc08bf15f8089f86da38d375a4092bec0
MD5 5a14f0e2b05442a060ecca6a4923e76d
BLAKE2b-256 9a15cb6bcc319ea31d28515935ca9dddfb493e77f7f7620aa080f1901f760365

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5f5b13e9eb0ba6a1351e3d218763c2514e53a95135eb20e3e1d8eb4e6160a5a1
MD5 90b6fc63d7ec7944eac90f481caa7f3a
BLAKE2b-256 85a9473ef678c8862d74c63e11d14afbdbeabe67f92fedd82405de5337d7e6de

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7b7e9872e8ff37d9ee274ba3a3bfc672fcbfa65f231071a7d53df6fe42a72f69
MD5 64e3ea5f7124254ce15d0700358b66d4
BLAKE2b-256 f3f2e280933df3422268f2c857b4a41ba6ebfae229371b209dd1081694af98e5

See more details on using hashes here.

File details

Details for the file fastavro-1.2.0-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.0-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 433.8 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 fac9ecf8babd90f9d1da080180fdde6b4c1f1c63ed03adb17aa1b0caa09c2bdc
MD5 aee490aeb362451872205abc30020082
BLAKE2b-256 da674c79514fe73d1b08b8b349e63e066121b62c889a0af3186e97bd96c5ae31

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page