Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.8.tar.gz (466.0 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.8-cp37-cp37m-win_amd64.whl (292.0 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.8-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.8-cp37-cp37m-macosx_10_13_x86_64.whl (344.6 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.8-cp36-cp36m-win_amd64.whl (291.8 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.8-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.8-cp36-cp36m-macosx_10_13_x86_64.whl (356.8 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.8-cp35-cp35m-win_amd64.whl (284.3 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.8-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.8-cp35-cp35m-macosx_10_13_x86_64.whl (341.6 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.8-cp34-cp34m-win_amd64.whl (278.4 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.8-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.8-cp34-cp34m-macosx_10_13_x86_64.whl (348.6 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.8-cp27-cp27m-win_amd64.whl (286.1 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.8-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.8-cp27-cp27m-macosx_10_13_x86_64.whl (354.7 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.8.tar.gz.

File metadata

  • Download URL: fastavro-0.21.8.tar.gz
  • Upload date:
  • Size: 466.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8.tar.gz
Algorithm Hash digest
SHA256 06b7c4ba99f4194725e7832fb6641ef24d6b428dcb8c1f1fb452525df3b45ef4
MD5 6ccd24b9322da747c7d9427a2032496b
BLAKE2b-256 cbaad1713521b19763135c27d98e6f92db0bff4c09d4c66d0869f4f53a0b0fee

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 292.0 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 d1972c034056f49c1fea44494474ec456deb38481a4022dd75f70d987cd66187
MD5 0cb14731e8425066ea556b36d93e24e8
BLAKE2b-256 1051d55bfcda3deca3cf250f9c53d24c2e9bc5b23541b00631c28d97f8cc3bde

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 6d8b9a74f53d578b70cacc1a0130e694862bdde0bc380776e0b1657dffcffe8f
MD5 c691dc25818a0c75f1c973d06b9ac11a
BLAKE2b-256 9eefc8ba6a998fc130df619fb63e6f1f381b5057e63fdb5999c7dab9701f5d3c

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 344.6 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 3a30333f32479048a9185c744695bea1071cca8c3f6a0d1de8c8257599758436
MD5 7b1d83179b363bcb280676c0ff9415e7
BLAKE2b-256 eeeef3f9764f43d4f80aa8d0c8b30128f1fc45e4e958fe39596b42d4a81f7a4b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 291.8 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 d7787b87d249f91574ba9ee280d90e94fcae2b53df3b9d5f960e09f27381d0f2
MD5 0bb15b2cb5481f4cb9203263bdcbb4f6
BLAKE2b-256 99651058eaaec53b6a6683de197a33ae8dc46f1511a4a3367e91b8bc738139dd

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.3 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 670d00bc7b3ece1c16116b88a4b9cafb8da8f645add84b3b514b51a62c2ccf3e
MD5 eccdaab29172bce82b804e9b7e664c69
BLAKE2b-256 94dd2c37b9e75d599a23f6bd4c5bdd82fe947547fa64433993f03638a2479b5a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 356.8 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 8f2e67731df72e218282d847d2d682659dbf67a0b2f586f765718ec0bcb18d2d
MD5 916c2da19b55480adef8d33049437b4c
BLAKE2b-256 e524bb5d7490623ccf11840153acfbbab356fe32e9c63024cd0ac3bb7862de59

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 284.3 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 65e11d723c5d5356dad619449bd80736935da875579adbc484252a3564ad3b70
MD5 081bad7a3ad295e4a9bb1477e0422c72
BLAKE2b-256 5430e973271bce820c22b8602b7a148f9be8d4ca52df3cab5a20a2003ee144f6

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8082e9fabd88ccf729f2718f77f4ddb57cc509677d7860febdef7fb3b027e832
MD5 5992eb07cbc52f58dbd676dc8a030f7c
BLAKE2b-256 c163ffdc040088771fef4d2ee0887ee9f10eec96b50dc72bd963a8b19cdaafad

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 341.6 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 9dffb67744dc18b60c971892a9e3f239f0a85ff0a5295fccd6c1d7ad14c7075e
MD5 000467c17920430d6fe464293841fcb1
BLAKE2b-256 0ee47419fb2b8b1e031a92da6edb8cc6cf50693828b343972e60057657f5f9b8

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 278.4 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 f2277c591bfef8a19aef43fb2911be2072d7cbd2a1a1ed43e439df7201e7d969
MD5 d825f4033f49a96892855d2c08eef62e
BLAKE2b-256 dcd7cf95647ffb96c267af909f4e459230c69dcc65ed374380ce7fefe86d5d10

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2c657c7bfd1594309e2f0ab30cea9d98e3c457376378a713a98084987dacfd69
MD5 3263be83e3c6523a68f5984aadcacf57
BLAKE2b-256 f195ba2c0e8c1d4171340371e71f39f48de048580df2f654d865800503bd0a1b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 348.6 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 dc897bb46947c6993e5159cc43ecc761aebfd3c53db3010828971a79ffdb0470
MD5 113347636681ec00e15405806b59a73d
BLAKE2b-256 19a0e9bd36453226a3f804a0291bd8c3906ce121ad917a349a26d9a444283cd3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 286.1 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 fcf564c3bff82c3410afb570b773a1ba6706883d19cf143c9a32296852498499
MD5 83e6278dedbe753668b5a4591c8cda92
BLAKE2b-256 85af1a92322365d9a5ead6b4c95d36390eb2a5a59d1aeb309ebf4a83e1bab8c3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2d45a82def873720443054c46f2e5ccc876b693fe057aaef746aac22bc2a32d3
MD5 133dfe3c464f326efc71d4258671355d
BLAKE2b-256 8732ff819aa6b869179d81a1082c399d4a2342fa6345322e5c99ae755b6fb62c

See more details on using hashes here.

File details

Details for the file fastavro-0.21.8-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.8-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 354.7 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.8-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 62c9d18241d44ba7bb706df39c012c2b42e20db731f849734565019a2ef3047f
MD5 f0e5f3aa79dcb9c8fa96b58017b96296
BLAKE2b-256 1eac7b8c446b95fa4d1aea7bb8964e48079d4760d95b2b97e6335adb4acbb74b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page