Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.6.tar.gz (456.8 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.6-cp37-cp37m-win_amd64.whl (285.7 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.6-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.6-cp37-cp37m-macosx_10_13_x86_64.whl (336.0 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.6-cp36-cp36m-win_amd64.whl (285.6 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.6-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.6-cp36-cp36m-macosx_10_13_x86_64.whl (346.8 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.6-cp35-cp35m-win_amd64.whl (278.3 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.6-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.6-cp35-cp35m-macosx_10_13_x86_64.whl (332.1 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.6-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.6-cp34-cp34m-macosx_10_13_x86_64.whl (340.4 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.6-cp27-cp27m-win_amd64.whl (280.3 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.6-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.6-cp27-cp27m-macosx_10_13_x86_64.whl (345.8 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.6.tar.gz.

File metadata

  • Download URL: fastavro-0.21.6.tar.gz
  • Upload date:
  • Size: 456.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6.tar.gz
Algorithm Hash digest
SHA256 e9771bb712ad12054f75bbe27d8008ff18598f29e90e1ac879f4c204915ae9b5
MD5 5c567c1a33945421625eaa1917878021
BLAKE2b-256 e6b08bba3c71b913e6d576c661bb54041818e09ce891304ca552acfc53d357d9

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 285.7 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 03aa7cf4eb1670ecc546d1d98e4c5bce58da7a7b405d8c81544a876c6f369f91
MD5 e24d331331dff29cfa270add81deb519
BLAKE2b-256 219c6e9bcdb7eb552e98384dccfded716fd5f79a42404c7f06d167117a5d3ae9

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 a348a082588d185447858358756e69d88c482b4d1ebdb1d2a08a5cf0110fc635
MD5 19cabb7f81361e271683234ce87a65f3
BLAKE2b-256 ac1c7f804ec5b09d67ca3ee02697682b30181be0db4399742bb965bae950700c

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 336.0 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 a71a7709694b05bfcf7999e46d002b527a7c576d75d8a7b13502e4674f08f69c
MD5 87cc4e5aeb2d24e07482d5628cc97cf7
BLAKE2b-256 63091dbaa4917c562c649e888774d9665c3e057ec1f8ac40870c4934a7969572

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 285.6 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 a2556106ea6b4399790fc7508142207077359031f3e3f66e627ee88e4b17c395
MD5 2d51e85f658b7e09408a1c16da67aa6b
BLAKE2b-256 8cd760043a66d22aece85206d47745b19a1ce56329c5d123c582984cb17e8efd

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2d3dddd1802f7b13c954ed33c3bc63f14c41ddddd2cb01e726171baf9a703bf7
MD5 9dcf7943ec977fc2605ef8ee1b93afbd
BLAKE2b-256 aead35d2496dbbe892b257f2bad0648923f8af83d97c4ea905150587112cfb71

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 346.8 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 e2d8d220fe8749425aebc6d73169e4d7651fd13bbf56b4216e98cd632cfa938a
MD5 8f7f4630091bec432fe2478b45e686b2
BLAKE2b-256 d2ccacb0d2a098ff2452f5f8bebe6107d27167818a9f17f65c97919599d9997a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 278.3 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 6d3e545d65e49da339236c9ff94c49b7099f7ab767e0c12722290121ba726e78
MD5 d5c33134d450e6edf7492dc90ae25d47
BLAKE2b-256 db3b1a9670febae6d36d3317d26b3fd4118d554e0ef7dafe18117e5d18bf9a02

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 e8aed026d1fed755e2a92cb8e1feeac0b805fba8bcea60395ebb725ef945aa64
MD5 a25fe0150a6c5e03d780fa00b3e3cc4e
BLAKE2b-256 24e258025eb0740ab8961aeaf24fecb0375dd851f74fbde4d072c501535285fb

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 332.1 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 db2c4a1e3ac65eabcb20a48b55d56884cdc8893e658df554a4f932c66666327c
MD5 2e7aff1aab9535d339c978c86af559c4
BLAKE2b-256 10b181db39dfaebbd2bfd0eff12994cb20ec7a4b6c52e17a97167104a4d18ba1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 7e7e7f53b13a0f409ad20bf5ae98d676f4bbe8d6263681874ad3a7e56d93567a
MD5 6e5e0ae5b6d7a4fd5fe10cc3c04c64f1
BLAKE2b-256 b46a717d0523e1c22e7a6a719734367fd73c39eb62e01335d20296bff0f33b3a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 340.4 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 b0784b2780b651f6efe19a980a6c843f8563bbc786a3da6588095bc8078d19af
MD5 3de8970f564d926bb562eae1ee1a4d90
BLAKE2b-256 7c94d563e723c5ab80c412cc0dfc1ed8217f799473cf50959927b35415c07e9d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 280.3 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 13cf4c83109d2ac0a8698b0984a2b1c41e9fd5a46d8b150feb87e7408eaf1f46
MD5 28215e3e0321f6ee809f622a2813c31d
BLAKE2b-256 6aab0f5e12cd94eb59ce8dd404b087bd032cd086cd7e5af56588a873f26040f3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d11d8900ea1101cf730244053ed7fd976cb57a1b966bb80db46adc71864ac331
MD5 d8c2bf82a18dff102b6203723d553b6c
BLAKE2b-256 83406440d07acaf2d2a241d03ae26ff06a65b02eaf71b7f6fc1f5fcf699b1387

See more details on using hashes here.

File details

Details for the file fastavro-0.21.6-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.6-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 345.8 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.6-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 5131755a381e07b1cbf4275f9d4882349658d9ef5462f27206b9e13a530bb7ae
MD5 45346eb61f02b3b8a7115cc3c3bdab25
BLAKE2b-256 68b669e9262c72749022945cdb99c09a8e9c069aa0c54b7209bca9ceb4c1d0f7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page