Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.2.3.tar.gz (663.7 kB view details)

Uploaded Source

Built Distributions

fastavro-1.2.3-cp39-cp39-win_amd64.whl (374.2 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.2.3-cp39-cp39-manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.9

fastavro-1.2.3-cp39-cp39-macosx_10_14_x86_64.whl (432.9 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.2.3-cp38-cp38-win_amd64.whl (373.8 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.2.3-cp38-cp38-manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.8

fastavro-1.2.3-cp38-cp38-manylinux2014_aarch64.whl (1.7 MB view details)

Uploaded CPython 3.8

fastavro-1.2.3-cp38-cp38-macosx_10_14_x86_64.whl (427.7 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.2.3-cp37-cp37m-win_amd64.whl (359.7 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.2.3-cp37-cp37m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.3-cp37-cp37m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.7m

fastavro-1.2.3-cp37-cp37m-macosx_10_14_x86_64.whl (420.4 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.2.3-cp36-cp36m-win_amd64.whl (358.9 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.2.3-cp36-cp36m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.3-cp36-cp36m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.6m

fastavro-1.2.3-cp36-cp36m-macosx_10_14_x86_64.whl (435.3 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.2.3.tar.gz.

File metadata

  • Download URL: fastavro-1.2.3.tar.gz
  • Upload date:
  • Size: 663.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3.tar.gz
Algorithm Hash digest
SHA256 357cd99a21efad2f37311beb65dede3d2d7dd90ddc02aaf53525ae93c9456014
MD5 4d848ed92cb9e73d781837629c0d2e28
BLAKE2b-256 26916b7b00a5858872936b6a6712e33e07bd26fcdc8fdc5dfeced58f386afa7c

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 374.2 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 ef6be5d75334946e92614eb6fd32a7cef1955f60a1cacd5e7695feef4cf561e5
MD5 5903b2aad9509dc3a638a0c65aa2838d
BLAKE2b-256 f00459e97e5eea4c3a8b9a739305623df2e43c9d91dc86f707acdaba3d534e09

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9638c88284898ea972088f064031b8f290a41b4a37248c2038addf8e403c458b
MD5 38493ebce8f3d3382e1cf364030c35f3
BLAKE2b-256 6a508b984f603a76cc63ff2f9fa7f5a29905080fae82ebbff65b51ea7bce90c7

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 432.9 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 e942fb72702b7644218018b375539ea34d65dd60e710d92af6b29f2cc01da470
MD5 fb3bb2d9c20cb63d5c5933486d4f7d77
BLAKE2b-256 1b8c7c1895afa9ef6285c97e2fc996338ceda2880c0ef989e504a323f6e74406

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 373.8 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 fea2f0178fe7088029a8fc41336938db7c4b323295b7d591107581291587ee99
MD5 18fd412e9a2dadedd0ad44ecea1b17ce
BLAKE2b-256 25f0ae80bc78a50a339b20f8e2d5d76360dfcb1b68819b3f45ebc7df27371317

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 50211544bfc026a555fd43d52cdb505482ea30b1370d573a4d206936d82f0c15
MD5 6af29cc3d1ea172df2f990d9d15cfa38
BLAKE2b-256 41f4ecbd23934dfaa83c273d8a95ac43c60a56fa68c6e6a4cbbf24cb9579b2ba

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 97e4e07dec477f6876a73a5592067375358be777fc8a0d0124b448f21afa1f2a
MD5 913211d218dad66b83501ea6ce4649ed
BLAKE2b-256 f32656275899cf44aa3663c0083e7fd1bcb14682affd088e38aae1a486ffd056

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 427.7 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 fa89a87977590e6138a2206c3563bd2550cfa37378ab9fb4b130197705cc2313
MD5 b87f07ea297c8868c082f45b803c1f77
BLAKE2b-256 b6498c169dd5bedc90938cd14d5758b97d5ef83669ba728432afed263076e2a3

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 359.7 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 9f6a2280ccd584db927053c7464677147c47871efc1881db4d1285ac515c3b3d
MD5 0d12bc6210d343036f885a03ecd252e0
BLAKE2b-256 ac8558ba3887ec988c8fec3b550038918532c6036541f2a6011162fe1676846d

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0b084562a7864947b11b85b659e98f0a01d31890082f88a83b8714e5c9829651
MD5 500b6eb9c139b22ef26979c6928141d4
BLAKE2b-256 b7bdbbd1de859255726c2aa47b553567780335a65dc2cf690b174332f39fa904

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp37-cp37m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp37-cp37m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 24cc280af4c70c60d490a4f2090e6e7866a39555587ebca146359c89b10ff1bd
MD5 3d72dc7a2ed7bb7d201771b7397cf763
BLAKE2b-256 70490e5bfed62dcc05be82c8d2f55f39e3ed1fd33b5133d6d876dd2a3f7694d8

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 420.4 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 a7397c57d46e00543658b5611744346c4066a33771537966f655a5cdd0689a71
MD5 a6e7eb5eca085b377f90137a8ae7b4e6
BLAKE2b-256 a92f0631cdcde78927af4f1b9bfcdef3a9f0ec4ad77761307c2623f1e5b5db0c

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 358.9 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 3cb87049827f431f1070672dda03acfd611292fe7cbc0a537c557be5348f682a
MD5 82aed1c673d225e1b5a1d8f11139198a
BLAKE2b-256 569f9e9bdcac1cb7595c95d712110dc3a34d5ed1c7e954079406e7cfe279918c

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ef8ea2d6d1bf3e92a07dc525bd30dc0a9308d61b71b324956debdd5339ea449e
MD5 025af799b857b0d881a5baf073147fcd
BLAKE2b-256 21def3f8eb2406ba426c92330c1e02abfc74adc10d23eda1953871b087313b56

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c300de95395516ac5bf6ea3cdc118415d6323a6230198e3718d617e018046050
MD5 fc91a709bc921760ef87cb984fc69c97
BLAKE2b-256 7ce174a31b02e7a5ab66f6a4c113e38cd922f434d46bea3c2188c18761b28ef5

See more details on using hashes here.

File details

Details for the file fastavro-1.2.3-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.2.3-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 435.3 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.2.3-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 471541e5c3549e5ddb63fe1e9bd770a502496c08bc54e986f7d4fde25b657b2e
MD5 8df6865e04babc95af6299b428c7b6fb
BLAKE2b-256 5739b7548bd5942f25a6693ceefbb84aa249d1d1244a20e7a33057e4374a8d50

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page