Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.6.tar.gz (728.4 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.6-cp39-cp39-win_amd64.whl (415.8 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.6-cp39-cp39-macosx_10_14_x86_64.whl (492.2 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.6-cp38-cp38-win_amd64.whl (414.9 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.4.6-cp38-cp38-macosx_10_14_x86_64.whl (485.7 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.6-cp37-cp37m-win_amd64.whl (399.6 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.6-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.6-cp37-cp37m-macosx_10_14_x86_64.whl (477.4 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.4.6-cp36-cp36m-win_amd64.whl (398.7 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.4.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

fastavro-1.4.6-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

fastavro-1.4.6-cp36-cp36m-macosx_10_14_x86_64.whl (474.6 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.6.tar.gz.

File metadata

  • Download URL: fastavro-1.4.6.tar.gz
  • Upload date:
  • Size: 728.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6.tar.gz
Algorithm Hash digest
SHA256 208c5738b91dd0e1a877228133642f2a8229979cca6923dd78dc00d9e9afadda
MD5 94550389bca63ec97e7566d4484dd46c
BLAKE2b-256 99cd8935ace0837aba172a9257ba13cf034089073297ae2fc07d11f245cfdb77

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 415.8 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 b03674d80173151ad4a9048f8c719bac809b9cb7a7c3d505643f0e5087551147
MD5 09033a1df8b2068ab2277f836ece548f
BLAKE2b-256 15dcb286fa7595f7b3d4601400b39a296eab42f86c52dcf3d60d93e110cedf57

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 bd0391d8a3b7d7518c47abbab8eb678138d890a0701c6db9ce050b71b5201a51
MD5 012b48d13cccc2e84c0884402d67b264
BLAKE2b-256 6110261847142601fcf5ffce30fac5a072f545dc6462a3fbfdf980e82685ea44

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f3da4edf62e2d4ab406998294668aa375a1777342ef05a8287a40fa3aeb1d4d6
MD5 df271e055954b26b62c6ffec257b5247
BLAKE2b-256 fa7878e86aea09aedbbfd24969d5a531b8fddb61532ec3bfb10bc5242f83c040

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 492.2 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 de360af285b0083a9c8aa19c778513c08473a9c947c6d086cde068eee3119d93
MD5 5530a4b749af1d90857fb2a407b31de1
BLAKE2b-256 be45f7bd92f558fee04d5b3cca0e7c8b6ca92097e62d1688aede10940fa9d5af

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 414.9 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 1dc97f23c7aa18d0f19766abd06f1e8deb00f238f6ba3eb4eee39c613b031581
MD5 825a9418d2795f7d5a279b7d4d728735
BLAKE2b-256 fe8d52b8c4a557f61c43361dda18bd75144d307b02bf3b5e5104021fcab467e8

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f2b89fa5fc17acccf608e49cf0ab1edec217ca08783c39b627a6eb29acd5d0c8
MD5 981417ca59194a7fe011ba0581e7c184
BLAKE2b-256 7f471ba18e3850bfaa79f095ad91d9ce5a1fe9bb825748309f2ae6e44885a9e3

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b4aef13da04503416b42273b2fee338a098f544754e0432b1b909416b1a3d758
MD5 867bcf5ad1f2dda6a8de3775ac1e3b55
BLAKE2b-256 a8d4b96f0c392dad9ee425d2278555510d1b0530f879063725f1818834e13a94

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 485.7 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 8050ea31f71ce8c86048f20ce1b1ae317d15696d1b1b206b081ab4d86b8c8a42
MD5 23f43833f6fb352146048977f6e45676
BLAKE2b-256 81d52c726a9dcee8b78e658f6652d8bd9a7d8cdcb955c6798dbae2d46b913363

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 399.6 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 2574fa92b8f85fcba92d7241c56156c14b362fdb80b0852841d4d9884708da21
MD5 dceb9d0902e1f12d00dfa11c199d3399
BLAKE2b-256 322efc9d35e862e30aa617fda7b63817edeb93054b56f9a43b08974c327fd405

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7b6cc98ff091ed5cd461439cde8e6987a86d39f468ac2d82878536f66a2d2099
MD5 58b51bc6e4b160152bf2e250c0bc0f49
BLAKE2b-256 879814d8ab9b40df07d075c2938e781574bb24ba749a34a4cc263cc79b315290

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 d1cf2ef77e458b1ab52632dd8fbb888ae5fc59b459ff8c2d7c1879c9b3f419a3
MD5 9255e7e483706bc464fcccf20ee7da2c
BLAKE2b-256 d3282bdc91a053b7f2947a8885852b9a2a93b3c9f71bd667765ce7367f58e5e9

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 477.4 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 aacdd87144aa3b15a061c0135c496c2ce59484ea469436d846f0662aec943e99
MD5 5167852f89d43ae2cd599cd2a5a1407c
BLAKE2b-256 f554cd19ba108eb000ee06a8690b037dffbb9ea4da6539d96ab22e5dd91947f3

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 398.7 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 7aa5fcd2956d0303161274973a7fe91a6e371fb0211187e00536762e0ab499fe
MD5 d159297fb4e10ef808a03201a2a6268f
BLAKE2b-256 850ac0557346f0668b246dab2cb227ad06a28bfc8a0d233eed6bdbe039cc1960

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 50398de4901fce1106ffbcc82e65c6779ae2c60e49e3f418b7ef6c54ead49487
MD5 b3804418f26b91c31d2cf364eb315f5b
BLAKE2b-256 c2079cf438cfe5ead7d7426ba02ccf6edd03c764801e33ad595a77b6a562800a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.6-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3707a72d766790a317e89f050977f63febc329c37161f10916a2fccf75d54a63
MD5 33942cba8c2824f604b18961a8b81f11
BLAKE2b-256 049bd0dbdd034017f46924deae6184ddd3465181e6962d0c6c0db756ec10ca6e

See more details on using hashes here.

File details

Details for the file fastavro-1.4.6-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.6-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 474.6 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.6-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 480454ca2702424aee9181767b6daeca17fd99cc77b0de70dbecced0c12cbed9
MD5 f527a0fa1892d29914d078c4e34febb3
BLAKE2b-256 d104aecb9b7e53174656fb102447575e8b75e4ce7beef75d94d933682e89b8c1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page