Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.14.tar.gz (468.6 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.14-cp37-cp37m-win_amd64.whl (293.8 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.14-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.14-cp37-cp37m-macosx_10_13_x86_64.whl (351.4 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.14-cp36-cp36m-win_amd64.whl (293.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.14-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.14-cp36-cp36m-macosx_10_13_x86_64.whl (367.2 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.14-cp35-cp35m-win_amd64.whl (276.6 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.14-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.14-cp35-cp35m-macosx_10_13_x86_64.whl (332.2 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.14-cp34-cp34m-win_amd64.whl (270.2 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.14-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.14-cp34-cp34m-macosx_10_13_x86_64.whl (322.8 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.14-cp27-cp27m-win_amd64.whl (277.6 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.14-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.14-cp27-cp27m-macosx_10_13_x86_64.whl (333.9 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.14.tar.gz.

File metadata

  • Download URL: fastavro-0.21.14.tar.gz
  • Upload date:
  • Size: 468.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14.tar.gz
Algorithm Hash digest
SHA256 f433dd00ce49edbd93030741f58d415d22cd1093835dde375cb037b15da58c02
MD5 bf4a64d78cbdbb96ad76c641a8252f32
BLAKE2b-256 4d9f0cc925af89b0762925a2ec681bb82f2a1d5105554850086744c3f4ba5136

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 293.8 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 7db7e27f0d4e471435de1d55baf224721d474ed5df8ba70eff5dd706e6466701
MD5 13863cfc1e74d0149b1213032c71576e
BLAKE2b-256 3b0afd9555c2d83576eba2c739b1787466b5db8b7d65034b8dbe04242f0d7eba

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 6c6d0ba880d3ea22c90a918b04956292beecf5ddaaee7b73f00efb6dba6933ed
MD5 02672a324dbff491a1dd44cc28bac0da
BLAKE2b-256 d876b2a890b94523f57df5e2e50fb8d4fec0b11dfe45b112c6ad7dafd04107ff

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 351.4 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 6e6b2820584b13c965804231f1204ecc66e6e704596d33809266d4aa36746f5e
MD5 27842fd4f269207ab9e072ac24ee5dd4
BLAKE2b-256 b5350b5382570c1c215fc57a0780243df0b62ce769cd9f10a5851cf510dfe218

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 293.0 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 3b40102e4ca513a85649713cb25a5eb29c40349880e333b18f2de7586eb70987
MD5 bfb19567b27cd2ecae9ac676d09665aa
BLAKE2b-256 a9c404046aff62333168d32c3bd8760380b9d6f9b6f4c057979e901539a1891d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 3dfcc848529e59f2e3f83fdfe060eb446e54a69a6b3c9513cbcfd9f936127e72
MD5 3288be2dcdf491322bbaabde63d18262
BLAKE2b-256 b7f9513877c4b7cfc1ad6d16b59e1a8612946be9e4d0019154ae97ea2463bfe5

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 367.2 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 6c385e90c6ee215eb747fed9fc70dfb909a5f796f55feaaada8a054adecf703f
MD5 fb543cf9289b22923097c493282d01d3
BLAKE2b-256 15d096ef343db028aadb4f6ec98811b983aec1c521ba9517444ace50e54f9bea

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 276.6 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 a3c3caebaf5ee96dd400cdb6e948b7d42aaee41e18e45f791f85069d6fb07d2a
MD5 2f9fcf0b47220b828b540c8bca1278b4
BLAKE2b-256 e48b4329722827f29a1d0aba830be0d90f866a6888a13f81476fd520849ecc2f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 10824c5ed63348a3b1b7693ce33e6b8d619e652d96471cd696925741b9381fd4
MD5 43c9c59489458acbcb5df4f80824e33e
BLAKE2b-256 2df46ef21a8abb5d378bb49de963ac7d2d8ddc8dd61f47039358c10335104e78

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 332.2 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 07d262e20543eb01fe6685315f35b8e8441e829bcc70e424f7b17fbd4d22553c
MD5 cbbb2a015f2d403f8b36e44531de576b
BLAKE2b-256 e6b429a174719738a7e1a8c3734519980e8b5b7f152240a92fd38d3bd1dbbd23

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 270.2 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 567156ab98c6c1192dc602b2c2804d5c62ce9019fd3c05aadae31f664b916694
MD5 1275d2626764764072834b91a401ef89
BLAKE2b-256 8fad1e171472cbbec4023e9c5d81e545560a8239945b297eb2ec11f34c8627d2

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2118758a63ee474929a063b6985dd5770f474c78b906ba7b71a6bc0ae03c8ae0
MD5 e4b0df0a193a9cc62be3ba5e63cd279e
BLAKE2b-256 0f4a906f023970b560fcdc54ce56edb640330d7f401eb5ec379584f76f7a794d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 322.8 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 dc4e2b03f3131ce3441e61166949d860180dcd9bc1793e2d15408bb4374f19cf
MD5 e1466386cc7f118de0197709b9a870a0
BLAKE2b-256 4f0f88a8f283cf35332ee0f7770bc056807ad2a1ff7e201380122c5a6f884ebf

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 277.6 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 8b89c03476d3e06ee276c822b4ae98c6d3a97c95e0bffa8fe1e88cfb89909577
MD5 1f5b6044df5566cd0a3f851120173858
BLAKE2b-256 4061f1214e1e0d8bf0bf988bde56bd814e76bd991e46aea5ed1dca3502b4c2cd

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 f467bc44b14a20b84ef9d124299ce08c3fdd7ae59f28a22ab2a5082dc58416a2
MD5 83d1c20f935a8e3e085e12230823026b
BLAKE2b-256 083181a98c65f4ff9a439910000b6fd46ae86a1020b32c83646cda4535808d51

See more details on using hashes here.

File details

Details for the file fastavro-0.21.14-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.14-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 333.9 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.14-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 3dd4fa7004afcc8391f7a97c2b955c8657f9b2bf748aff275ac34ff176a28bfb
MD5 c1009c79cc01e3ae5a467b7d286fcb94
BLAKE2b-256 1063abb27b8c8b6884de27190ae099594b2fa72094ffd057bd29c7e831607119

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page