Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.11.tar.gz (466.4 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.11-cp37-cp37m-win_amd64.whl (293.4 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.11-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.11-cp37-cp37m-macosx_10_13_x86_64.whl (351.3 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.11-cp36-cp36m-win_amd64.whl (292.7 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.11-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.11-cp36-cp36m-macosx_10_13_x86_64.whl (365.6 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.11-cp35-cp35m-win_amd64.whl (276.0 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.11-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.11-cp35-cp35m-macosx_10_13_x86_64.whl (329.9 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.11-cp34-cp34m-win_amd64.whl (269.7 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.11-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.11-cp34-cp34m-macosx_10_13_x86_64.whl (321.1 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.11-cp27-cp27m-win_amd64.whl (278.5 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.11-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.11-cp27-cp27m-macosx_10_13_x86_64.whl (332.1 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.11.tar.gz.

File metadata

  • Download URL: fastavro-0.21.11.tar.gz
  • Upload date:
  • Size: 466.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11.tar.gz
Algorithm Hash digest
SHA256 4303381579869c3feb99a32cc75c10a9d2aa48f64c6d998346b3dfd8808a3622
MD5 ea52523eca47ffef2e78d023b35cdab5
BLAKE2b-256 f4b00b2cbd209c4911bbc0941f4845014720f099828ce42d2f410d9bd0e9dd57

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 293.4 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 f1229d14b66c0d9cc82a59da579651f7d9accfad66db8f0074c8c7ffb2a8a05c
MD5 818a8a000a97b1e658fddc58e2e4652c
BLAKE2b-256 2654755ade66db808bd26b8b09725a21a29e38e44b638feb60a4e9210c7f796f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 01613e74235e220f50f97db869638d9970383e89e319eb327bd86c7dfe9b19af
MD5 93f3c521ea2c2f430393e963f94c97a1
BLAKE2b-256 697fc9afc1f4ad1344196038375018913c070407c0c2bc39837598f6fde8e597

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 351.3 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 db1ffa72673eafd4493d5316198998bf489aa13fc80432fd04f4373377f63a15
MD5 c9622b5568b2793ac9009dcbe9a294e6
BLAKE2b-256 282b85de2169368dc7bfa5c5c0eb9bbe0d9262b21a8640edbd44086518f39eaa

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 292.7 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 d67e1f2c378a855e9d19f885e0ff2a618a17cb633cd66d903a48dbc09662554e
MD5 eb69c71361c76fb7d5bb59ad613f275d
BLAKE2b-256 6c4a9dbe69c379fc9eaa0b895827959fae3cc80f13f9ec7ad32effc7129c9762

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 7459cf7c9432b2607cd9dc6b5c55bc09db7295fadbb3174768e872c8420b0ca4
MD5 a53501ac6d6000475260c8d8cf46de4f
BLAKE2b-256 58503a8c0a58090046115c36db1036a74246c614a3a39e31c7471e7c20f24368

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 365.6 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 9dbc22fdb961c703f458fef451d7887ec2e004fe2c9f2e87dae9255485bcad66
MD5 cbfee39a6797ae73d689bc8523ea1c80
BLAKE2b-256 1ba5f09a9699fd54ffe023edcc22808fbd9ae4286a9df18c843c7f100e18540a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 276.0 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 d91b9327941bc7c55135c7f72981bab6e49d5e95b370711f2ed0b676d751c929
MD5 48a5a960c41a642d6bc0532ff14d8a39
BLAKE2b-256 ba0d6d4f40c46d7383f06489c4edd7125434ba3421bc71955bc79d164d551da1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 e29ed7aca737f7f9e7d86c645416e794a6ca5d0b9e03999b1fc81a75d62b40c0
MD5 369a80525bc2ba329575861b873e7926
BLAKE2b-256 4af48f58c6056e89b6c37b7a25a264777081a86ceaa5bfd49f039e5ac35264aa

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 329.9 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 a16bb98a5457bd3ed4c7581f5a30830fed88ff3ec09f1fcce0ec3f93c2c06745
MD5 dff8242d7ed9dcda9f8e23053d76fe2b
BLAKE2b-256 adc3cf9947fa7940526824366918db6fb0fd7199da7f1070332d5fb90adc6ab1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 269.7 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 4f9be3b56895123b014773fc341e8edaf706b7192ea66bf57410ab8ebc2a5358
MD5 49e544f6fe95355f7ec2fb11a08ca3f4
BLAKE2b-256 bf16be2cbc2fddbe1cebf86d4923bd4e8fb5d662f501156a386ba2d653db6dcb

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 6d2b899858ddf061e7084b64adf2a1f41a2ba5264c03132ba8461fccc3ffb561
MD5 302a8bb036cdaf42cf29b97c3479a850
BLAKE2b-256 40d401d8e065e7f2ec7dbf74680e78619166ba4ab85b325457080e5c2870e99f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 321.1 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 2b8e0ed99ecfb386b3a1ae6a1ee0b50c0044d8345dc92be1b79b36c840907f9a
MD5 de8b4879d13c1e2aa879b79794bb36a0
BLAKE2b-256 be37cae4e624e645026cc9a7fc57093fdf6fbf72e0bc3dc3abfa0c987cdfdb25

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 278.5 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 5f949c32e8a2a461484b857a1ae6e850074812fe848b82c9d9f51cbaf0d20e1f
MD5 f66bd42058b9f0cad58cd102ab1bd52a
BLAKE2b-256 7dac40750571b7ca027023c8a2e18d9db7899e218ae220302350cc3f22c187bc

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 547ff99b1f7c9cf4901700b77c6a7683c81cbe6adb56e6957fc05a1cc05ca9fc
MD5 db3db972b888ee270fcaf0c70cd7819f
BLAKE2b-256 59ca919c90f20ef84f3be94a61f8c52cfaaabbbe023a2b72fc7f87b8d9ef9baa

See more details on using hashes here.

File details

Details for the file fastavro-0.21.11-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.11-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 332.1 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.11-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 7c5a53d70ebf7d085484b9b49d08d1d7196af917c14a01684f59310c2d9beaec
MD5 9733246346223f8d01530d1dfeba7190
BLAKE2b-256 02dafdb4c03e93d69c427326bd08caf5624d7e9f624e6fdf38c7b6133b5bde2d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page