Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.10.tar.gz (465.3 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.10-cp37-cp37m-win_amd64.whl (293.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.10-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.10-cp37-cp37m-macosx_10_13_x86_64.whl (350.3 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.10-cp36-cp36m-win_amd64.whl (292.4 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.10-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.10-cp36-cp36m-macosx_10_13_x86_64.whl (364.5 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.10-cp35-cp35m-win_amd64.whl (275.7 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.10-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.10-cp35-cp35m-macosx_10_13_x86_64.whl (328.8 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.10-cp34-cp34m-win_amd64.whl (269.4 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.10-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.10-cp34-cp34m-macosx_10_13_x86_64.whl (320.0 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.10-cp27-cp27m-win_amd64.whl (278.1 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.10-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.10-cp27-cp27m-macosx_10_13_x86_64.whl (331.0 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.10.tar.gz.

File metadata

  • Download URL: fastavro-0.21.10.tar.gz
  • Upload date:
  • Size: 465.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10.tar.gz
Algorithm Hash digest
SHA256 691b7f7217fdd4dad7fb3a8ad1384e30b9ed60a04d050fdf3ebd440c9f9601a2
MD5 c99687a421b0df8b1ccc9a6c4c4d191f
BLAKE2b-256 c3a694a33be49e85c1d0bd0a86c7b91d0798f6f9c730cb22e106f6c4fecff01b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 293.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 8c38347ce9093e49ee79a14e42dadb3960f42d2488f31a2d9886056add415c20
MD5 3719baad897650f21b7bc0880cc286a4
BLAKE2b-256 69e58d1236f6e504e19ec2843deeecf18bcfce42a749dd71b7472e7ab2b288e6

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 5060fe9c92e5b54a9d1414de4cb4085ea4acb65fc9c59c25d35658b7decaf696
MD5 7cc197c252b31558df7e1dcb4b512575
BLAKE2b-256 81ffed2eb746a859e1ca62549bb91301834cf59feafc5259856a3f7f83081c41

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 350.3 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 15cc51f74e5abd9e88b03ab331f6d7bff6cd01480bb6d2f9109f597ba2161f05
MD5 9a6f6bd8e86a817566b933ab7c5b25e3
BLAKE2b-256 f646f27a92e2547f86d277e39efb67450d428205cae922b5451707c73a897247

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 292.4 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 35372cafa32b1a94d3e5f0ffc78edd0f6e4404ae3b5145948b08bdd3ab0f935d
MD5 c363e0c3d0b2d22f497411edbf50eaf4
BLAKE2b-256 7d7bd1efd3fee7e734c64684dcfabeeb6c13779543edee22cf25970d565191f2

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 acb01c1dff3ea739b6f9e3cc70d7ae83c42444c0ddbacb74fa02a6166ad9e484
MD5 758555e1447f60e259409807188abb47
BLAKE2b-256 b7b6514461932048e12bc8f0567cb23d4ecc8fd927d694a13eef52fef31bfb1a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 364.5 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 75fc0b7dcd91c5641911f66b9948b15fc6399814a34b8a9023ae08a227e96c56
MD5 9e4be68cd8988f992a7d9100254adc83
BLAKE2b-256 31d0c8b35eb76d85babd2dc445e37fb533db69b88dd862131aec18a3b8a1ede4

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 275.7 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 363c716fbd4e645bc1208f1e7a4e38afeca4dd6d5773272b8a7fb0a4e79d329e
MD5 146b600e5ecec703e9a182a544d27b53
BLAKE2b-256 1a1e944b62e44c09b4be615dff8c7110ead1f1f5982a0f92a76742e203d5990a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 0d7563d5efaa20e2811fa4992f669228bcd31229ce4107c2c1144964de8cf581
MD5 b58be69b595757d25e2b32b91205a207
BLAKE2b-256 53ccfcf1ef8b3727305a0f0818dbdd775273974527aabfa63870d3882f69571f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 328.8 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 1dd096e1579e878ce19e70eaf693614826323998097f4a6d2b34e175779d2006
MD5 de265f95fe842146143a9a444a678ea5
BLAKE2b-256 34ae7fc43c861f72ba343d09e68e9d457983fe7ec6fdd786d16cef21e4f8cd7f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 269.4 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 7ae9ced48f7ad0f96d53d7be6be4634284beac7642975cb6b12cc8717794deaf
MD5 2d248945a8f3886899872899f201d1f4
BLAKE2b-256 481214af5361df6b78273bc4687cab0c99c6680308ffb21794e5427b52e06b80

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 96e81bb9fada8df51977055b8d9a3de21b3b56200e0aa028168bfdb3940493f2
MD5 779dc89c2b3d2bab5929e12bbc5a0287
BLAKE2b-256 ba13a8387c3b8093fd4606f70eaab5fcb852e53a70303b783a32985f7017bb46

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 320.0 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 2c634c0580c5b1c9c9b5e4f54b7fa93aaea4c241f2673915f123043f6b170e0e
MD5 17bd3e81e381bc7ed9df0fa54eaf2bf0
BLAKE2b-256 0b63a3beedfcfb168f77416b21de5572b8da6bd38f8f8111125332442afb9992

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 278.1 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 bc62e65175c77c5fadbdea0e77c3e811f6f6c4e90c4494c05716e42fbc980ea1
MD5 481c1f364241fe700813c7b7aa7173c7
BLAKE2b-256 90b5424e39af9a90871560a16136cb7897611899d49160f55645f5d8cffe5ed9

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 be500b6414619ba7b3071286357e0cca8500f82083d81226797119c655934260
MD5 e01334b226b96fbe2718c1ba7a9a458f
BLAKE2b-256 906c86c803264a8fb073a9b373ef225aceaea59ccc12b7a5b3376aa62e391895

See more details on using hashes here.

File details

Details for the file fastavro-0.21.10-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.10-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 331.0 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.10-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 5937021f74f125b7530754a5633c382bb86769bb1b1ab0682933c7ec650e7505
MD5 0fe3632f9b2be93c69e7e8a2ec4e3942
BLAKE2b-256 22b8e5209245c2672fe6540ba4f7ef52bcb27d33cd43384ca9f98bd06a7a783a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page