Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.12.tar.gz (465.9 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.12-cp37-cp37m-win_amd64.whl (293.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.12-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.12-cp37-cp37m-macosx_10_13_x86_64.whl (351.0 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.12-cp36-cp36m-win_amd64.whl (292.5 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.12-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.12-cp36-cp36m-macosx_10_13_x86_64.whl (365.5 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.12-cp35-cp35m-win_amd64.whl (275.9 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.12-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.12-cp35-cp35m-macosx_10_13_x86_64.whl (329.9 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.12-cp34-cp34m-win_amd64.whl (269.6 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.12-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.12-cp34-cp34m-macosx_10_13_x86_64.whl (320.9 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.12-cp27-cp27m-win_amd64.whl (278.3 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.12-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.12-cp27-cp27m-macosx_10_13_x86_64.whl (331.6 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.12.tar.gz.

File metadata

  • Download URL: fastavro-0.21.12.tar.gz
  • Upload date:
  • Size: 465.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12.tar.gz
Algorithm Hash digest
SHA256 1c236ac44c779126cb46803b18d6c6b82850a8447355bbcbaaf0f6e322218c91
MD5 39a5ec44f0ab5791c29e2b4b91eb9770
BLAKE2b-256 257a3b094c001ce1b64eda0f9f4472fb8ff97e40391a38f2d6054c118c927b2f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 293.2 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 c366c36cdb89bf09e4bf7ea53ca16f21a06eaf49f75bf1d75f1f2d964e4cee4c
MD5 8a8613f24e36e883698749f0662264c8
BLAKE2b-256 5f2d2cc051b4f89319570f42442b31c00d55fa88e503af7aef17416b47cf3b93

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 a3bd403a9085f9d59cdff6c5359d3b8aa30b50754fa695a7b6f20ec220646bb5
MD5 78d68813693c29e31e1f040ccb6d855e
BLAKE2b-256 8e745570e52ae75a6af20c2a0ecb18289ecc04c462fca81a00c76ae54169ed61

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 351.0 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 085ae538838d2d7aec0bf331956724ba42e27d4a664b4a019d8c2f629a10ca1e
MD5 5a3d705682803bee672ba8d356f33df8
BLAKE2b-256 f6a435531492cfcb0d88c6959070e49dfc9d921cbdec4101014fc4e46cd1a78e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 292.5 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 e355b25846e75ffded0c60c7f04bd082e6bb0fd83b33a7745248ff7205e30130
MD5 6bdc3e2deb95fa8fed8d469f53cea948
BLAKE2b-256 ed732bc895fee68cab80089840ab4d5979d19367473cb12b52928c71190b2dd3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 b3dc4e7f49bf55e1da1b77f5d0fa3ea4c38ef0599db411624eb8d96a2799a239
MD5 ae1c02f4ad0883f14835e66bc5d2bed5
BLAKE2b-256 3392ebb40b316b982b1de636a32c283aa260a8553c068575ffd43853df57a02f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 365.5 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 b71dd275e0bc959de91e27d90ffd999a4e8509066851dcb376d373b4f4cecc8a
MD5 d3ef34af4b60af894d6a5fc3d295fdd1
BLAKE2b-256 6310433e7eebe6487b7dc7381adde527569bf7a90e8eb08fd980608790959dad

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 275.9 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 a38405de9acb9f45bd3647f1fb2049bd613815a1d387ca34c2c2cfabbccea5f9
MD5 d9c2478724ac6032436f58fef184817d
BLAKE2b-256 2fd53f31fe48be238948758919024985cc2784463827f8b6777f5ec0885e59bd

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 cb98874363ddeaae860edf4270e52501704927956c04a7ed7a2978c99f1a6dd5
MD5 cb35189e927367ff162625c1cb056e8c
BLAKE2b-256 270b17cb7096794f53d5f4245ecea902ba25069618934ae32e85e85efef1ae61

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 329.9 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 77d1a92916d32805e98e90e0bd8c1c182f472201bb6083c6f8063dad522789df
MD5 afaa19847e9b75bf59c31ff81aac12f9
BLAKE2b-256 5b6a6275db7d8e674e4f3d14c388c710ce2356d85adbf226dd611a3f0f806bd4

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 269.6 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 84807a06830ae2af736afb122f243afb36e414855f0b0e3014973446e0433473
MD5 0db5eed200698649f94456f0d8d0dc5a
BLAKE2b-256 2d707d6379ed7a5b10ad7e4516c703e5ec569649fdf638807a18cc712e2eab7d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 876d1f20558336dd0f8988ea4b2a49fc8c39988b3e2e4d7ae92db0f2610c1bbf
MD5 d4435498ceaa9349f1ef0a06967d8f78
BLAKE2b-256 ac9eb464cc511b63a89d317589fad2fc0e0011aec76ca542d6a3ef527b2e561e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 320.9 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 1fe6e4a4021c07134e7e86eed48f8ea0b217116cea325de3665b8c95bcda4246
MD5 1aab9ac3104d1de8034dda3b50840393
BLAKE2b-256 a2be6d793d362d8a3ff788e68b85b89f95f74e50cc1743d86e663634ad1a8d64

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 278.3 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 e52475a3f3446bb6fdf44291fe32b2a4491be22c53cc846e56c0764cab013932
MD5 879e6b7e9b8aec55809ea084e55519ab
BLAKE2b-256 427a206aea1ad54541d59b2cdf0fd7b7014c6fe114dd5f73c52cbb22e0e038ab

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 3a2f293fa5b325f03a093b5514791d11beca3af3cc64dfeb3391a03f06a7563b
MD5 22890b86d7e21d4e1cacd27ff7f2c3f5
BLAKE2b-256 f3a1359a850d0fd4328747466f638640e3b7bdf7e3aecb42e7defd52d841cf87

See more details on using hashes here.

File details

Details for the file fastavro-0.21.12-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.12-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 331.6 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.12-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 6e48ae15fc5c2752bc84e08315c059541a53ec370a74a2b309cab3059c6af86c
MD5 6b6cc861912cc321ac8b1980dd080df6
BLAKE2b-256 dc636287780b9d351c4c601dd7d5db7d4c580e81067bfa6493d635e71f00d04a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page