Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.9.tar.gz (466.9 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.9-cp37-cp37m-win_amd64.whl (293.3 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.9-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.9-cp37-cp37m-macosx_10_13_x86_64.whl (346.1 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.9-cp36-cp36m-win_amd64.whl (293.1 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.9-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.9-cp36-cp36m-macosx_10_13_x86_64.whl (358.2 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.9-cp35-cp35m-win_amd64.whl (285.6 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.9-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.9-cp35-cp35m-macosx_10_13_x86_64.whl (342.8 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.9-cp34-cp34m-win_amd64.whl (279.7 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.9-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.9-cp34-cp34m-macosx_10_13_x86_64.whl (350.4 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.9-cp27-cp27m-win_amd64.whl (287.7 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.9-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.9-cp27-cp27m-macosx_10_13_x86_64.whl (355.9 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.9.tar.gz.

File metadata

  • Download URL: fastavro-0.21.9.tar.gz
  • Upload date:
  • Size: 466.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9.tar.gz
Algorithm Hash digest
SHA256 f472f7b8791a0e307b43a1bc8def19cdc82af600f1150a26924b18e24556c145
MD5 fd4216d9c58ba8dd2c587d7544d77506
BLAKE2b-256 4604429ac64be4af21c17e23a5107edaa3939d591a5dabe6aefa7d4e69a9091b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 293.3 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 91cab51560063de9561e9aab61f42718eb138f233b81eb4e960aff9792c6e4a8
MD5 63d63e21b52f7b960be04569b8a58b90
BLAKE2b-256 afc382a86027e7c038885fd71a82e0876142be48128a3fafe4481c13121da516

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 00abb90c80551f8f6d91675adf8bd8b4f200a29eaccfd41d1775f7fcaca0cba7
MD5 8fc8e8b1e5ebc6e0ebe026b5a8a59482
BLAKE2b-256 26dc3ea65699c0b9b1af5f7060c9c27669f8e5aad1ce7d4639e868ac1faafcc0

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 346.1 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 8180df1e578192f46dcc758f5996f8809759e457d83407ca46908d02be7e8f3c
MD5 67759f31581a7dc7efd5b20f8309a24c
BLAKE2b-256 eb8d23e3ddd7e04fde41cc415739127ebe5278cc05e211b3df8db2f79ba3227d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 293.1 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 8815520e6e4abf5d85b56e2ee7f8179fe6f4c286a2331d082e9a7cdc820b758e
MD5 7b7daf2c9ffc0cd86a343ffa34a2be06
BLAKE2b-256 202c4ad9bf2c3efc96ca5684fd0ca8bf6de72a7f7893a8f88a41967be90c9188

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.3 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 a65853e57d28be7edf8bc713732eae094e71b98250b532e657ebf9b364cd3569
MD5 198608639f2781d102a6da133f0bbf19
BLAKE2b-256 332a38b12d0464c673ce4a12c1bb72e98b5e74741ac427a3df9d10c5ca600376

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 358.2 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 fb9838b8d20945569134a53aa3b070d3c6d79f9ee6139795fb1fbc279121dd61
MD5 085941b7ec62f68b73e9bafcc12fab43
BLAKE2b-256 1ff15c0de5aeeeae391b48aaf8867d239376e04214c50b6f31ec778f7648fb0a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 285.6 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 7dcbca88c9baacf96d7dd33ce1d39d82bb4a5228951fba7c5d56fb9cc372306c
MD5 6d7083669bce986b2c62c71b0d204e2d
BLAKE2b-256 5c7a5f5a5760759bf4039bb7abb7a58ff440851d89a6cd83a457ec24e1e22d50

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 539470b83d2bcd05cb97c78a448b5e0e8a3c9cd5ca9dd1f5822d341fe8c606d4
MD5 618a56a0f261f7814d1c96bfae0f0fd0
BLAKE2b-256 542a3adabfc9953326874e51c28cfdcdf99012e0e7e7dfc0860f9d7c63e110ad

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 342.8 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 c4333c8f586188de0fe0bcb35cd206857d14486e48252fc592c5796067806aa4
MD5 74e45a1aa30152d2fc124ec0fe883bc6
BLAKE2b-256 078b07df2903e3acf34cb5426a8504789b03c84b0a3b36f3e107f4ae87a51e18

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 279.7 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 2bf9c5beb6cee9f03c364aa841f5fea87067c6d329bde5af20babf1c66eb8bf6
MD5 87d4515b02090be53d0c92b729dab358
BLAKE2b-256 7602e04fb3625d49d3ed6802c83e301e3c46ee1cf5d6f5926b5b9054f95aa164

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 4224842597af61c58b30b8efd71769f60867823feb30c918008c140e94655036
MD5 74a5eae2c4b5ab7ebbaea7da555120b7
BLAKE2b-256 8861f84eea5021f7d51be622049c2e28a763513223c48d2666fa37fddd818065

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 350.4 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 6c0b998dca82592e9292c660e7e27befd2685e46f7658082acaa315c6f9110ca
MD5 07b77ba41516d808b777ec52b7d57c5a
BLAKE2b-256 f29ba5d2e2558caa744706a5687132110832651c3a1c9979fa9ed25c63f66135

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 287.7 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 67dab7a4aa13fd34eb8ba32e280938b4fde213672207462732ca74bceee05608
MD5 f5e95024f3e19c8d0ce584de5c31b60b
BLAKE2b-256 74bf31fd48768a83f5900d1cdb35da8f5ae579e7cd489baa9d10c0c2cbc297af

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 1ad891ebb9c962481c21d1930c1cbaa2264d80fe61a599d42fccc46e84a90c2d
MD5 6cd6117efdf28f524dfd385693e714a2
BLAKE2b-256 8d87c1e4bcc409f4b543f0412f0195a824e797f400ff5428fa77ceb14e62e2a3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.9-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.9-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 355.9 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.9-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 7e111f3fdeffab58e99aefd90e6d1f13e69fb1f47d490803e89e3c6f35de6f41
MD5 712b91d22134cb4b7cb73a7b1c72324c
BLAKE2b-256 a74b1b2e1f10867d25516ffa6fe4a9833b0678a3fe1b09d1077b07eceb615a36

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page