Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.13.tar.gz (468.5 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.13-cp37-cp37m-win_amd64.whl (293.8 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.13-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.13-cp37-cp37m-macosx_10_13_x86_64.whl (351.4 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.13-cp36-cp36m-win_amd64.whl (293.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.13-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.13-cp36-cp36m-macosx_10_13_x86_64.whl (367.2 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.13-cp35-cp35m-win_amd64.whl (276.6 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.13-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.13-cp35-cp35m-macosx_10_13_x86_64.whl (332.2 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.13-cp34-cp34m-win_amd64.whl (270.2 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.13-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.13-cp34-cp34m-macosx_10_13_x86_64.whl (322.8 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.13-cp27-cp27m-win_amd64.whl (277.6 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.13-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.13-cp27-cp27m-macosx_10_13_x86_64.whl (333.9 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.13.tar.gz.

File metadata

  • Download URL: fastavro-0.21.13.tar.gz
  • Upload date:
  • Size: 468.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13.tar.gz
Algorithm Hash digest
SHA256 bdeb54d5af2a7a080a8c251d04d2d15e0f22c1560e617cc50b5197b7eeee3951
MD5 5dd005f80204c3835bcdfd3a452ab1f0
BLAKE2b-256 641549ba9347ed7900dea631d2030972f35dbd5db09e5a47f44ae38816b4f686

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 293.8 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 2ed5408fd086fd76b13dc763c4dad73d3ae219319ef662432ff9080cad05bc35
MD5 d92a7a6ee40913fbf5efecb985942c96
BLAKE2b-256 d71b778f4a01ea23d07be83fcc9255c16a6e49fee1744370267a5ad2231afefc

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 557f8e8f3f40798717500f794bc5ca1d199fef10be39606458195fee6d3a29b3
MD5 b02086a2a42c37ea0fd0fa1dd879d178
BLAKE2b-256 108b4d54352e1654138a4a0196550d5804fa6869065e4222925f7724fb977eda

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 351.4 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 0567ffb19bb79947d61d5f42e541757421f2fe4458a5752ca8ef68693423c6d0
MD5 0c29c193c0df5e59967969868b9c3723
BLAKE2b-256 b8fc04d049da428ab36ac9d3a78f1ba98af2a582e9cf94a4854b7256e9e6e19b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 293.0 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 acb3a7a0a1e6e1b0ff412ece4e85c79db9ecad306cdd6a0b98957121224ce562
MD5 bfb962046bf69b53085f9cd1b6ca2013
BLAKE2b-256 c9e77923cdda93924cadd069bfef47c7f753ef6b8a78a6a1d97dbee0324effa3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 5b9548214bb6cea837d19b77c32f2783bb571889109b082044353c83805d201e
MD5 4e82378150e656acd33176d1487debe4
BLAKE2b-256 4d7188d302745e3a2c76952b84800ad84e5e9f2c55d26685400c7a66dedd6974

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 367.2 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 0219892e34d821cbad8bc5d119484c832dfbd649a7374db95d42bda58f727f7e
MD5 ca23a5f99dfaceaec146e4c23ac95e30
BLAKE2b-256 5f5a3ac7beaeffeefeba6efdec77d634d488269ef4730bf65d3f0ec78ad691a2

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 276.6 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 d8516cc820674e0ddf5b6f4aded4bf5be67e9db9427ca6f0561d4dc10bb70cfd
MD5 4f4dd3cb821a90605c070b982e1bbfe5
BLAKE2b-256 bf8944f82f6eef85cb7ab3114e2762a4945199494eeaa4884a4a4b1b57c5f298

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d3201a05bb2187bfacb1104c619fb7608ab101a1c588225e2da92c3ad9756fb4
MD5 604bf71615202e7a0d2682446386040b
BLAKE2b-256 2f027c9ab07d8e66aa8c7d7ab68c3acd1a87b8fa0ef2e8ecc6c50ce50104f53b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 332.2 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 fca83fe1fda35e8629d9527bda7216a4577bc21766a8cc9cd93bcad0e1514569
MD5 32fc5532e4981f60c3c9eca9bfe9ed91
BLAKE2b-256 b8aa08c5812c121a4952a0d9b72ca2d14e307c2fa5d4f7862b8b52e4fda6a152

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 270.2 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 f6205448741139023540b12a407da4e7d1075aaaf67dc4a348cb45baa777df42
MD5 28949b6e16aafa1cfe45280cfe7d2f4b
BLAKE2b-256 8be309d8e78146aa7d4846af348b007624f79845f8e581a0613d9cd176e8f0dc

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 5b1731db21b63113d78419fdbfabac7895323fa529a9e3c291c139559106f630
MD5 f1ddbf0cd7be9e0f2a7e3809acff5425
BLAKE2b-256 a03b559e0e249df296d638812ca69b54ca40749f332b1c5ee82cf4da9fbcc303

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 322.8 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 6788740d23bc1d12223bcceeb84c8c35b233abcba0843a33f70f0fceafd9a98f
MD5 79fefd21028de877eda268d5030a7067
BLAKE2b-256 d37c256cc3bd1db94ba84c487ce522be9052076263c3a160674c7b5c773569da

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 277.6 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 93da94ae3cbb67350586afd203382f859498a9801c9f6e7050c3f46e205404f1
MD5 a50a115cfc34464aa7793f2d4d9d710e
BLAKE2b-256 eef9cda285c7a3042d4e0a498515f485b6367da85aaf0cc1d09c9c4ac52cc137

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 c038b0959e19a6436249b08d1f5e5a0e38a3711dfee49f700ab524841376f422
MD5 5c226991752595545b901474ac89d735
BLAKE2b-256 c770aeb25df558be19c9201bc9fe2b61e5844f67e3e17facaa1980908df298d5

See more details on using hashes here.

File details

Details for the file fastavro-0.21.13-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.13-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 333.9 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.13-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 42fa4a142b7d3de6465a0fc71b689b4fa764ae88b644939474270fad44ca7229
MD5 381c1282ea8b693f350cf39421ea55a1
BLAKE2b-256 6ec9def543d995b49a8707614a1911c68d155938d1da8ba67aed4331195785c9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page