Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.16.tar.gz (473.3 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.16-cp37-cp37m-win_amd64.whl (296.7 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.16-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.16-cp37-cp37m-macosx_10_13_x86_64.whl (355.4 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.16-cp36-cp36m-win_amd64.whl (295.9 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.16-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.16-cp36-cp36m-macosx_10_13_x86_64.whl (370.9 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.16-cp35-cp35m-win_amd64.whl (279.2 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.16-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.16-cp35-cp35m-macosx_10_13_x86_64.whl (335.9 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.16-cp34-cp34m-win_amd64.whl (272.8 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.16-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.16-cp34-cp34m-macosx_10_13_x86_64.whl (326.1 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.16-cp27-cp27mu-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7mu

fastavro-0.21.16-cp27-cp27m-win_amd64.whl (282.0 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.16-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.16-cp27-cp27m-macosx_10_13_x86_64.whl (338.6 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.16.tar.gz.

File metadata

  • Download URL: fastavro-0.21.16.tar.gz
  • Upload date:
  • Size: 473.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16.tar.gz
Algorithm Hash digest
SHA256 c8f9d1a07cdb2da4826973aa80ce53744fe42515ad6cbedc44ef699642bf6934
MD5 f586ef62a1820ae5b3630db49eb6bb09
BLAKE2b-256 ff2542ccc4121b5e633d355835d13bff5c96a80a2e3477285351b0ceb44f2575

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 296.7 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 dfab9107d64210dd676a2a1534141dd245403addfcc9022279df985cfb7a3dba
MD5 5aeb7fbd23202ef00aeaa46e076aa195
BLAKE2b-256 ef6cc229bc49369911f517ad0161acf0a5bcaf818086e4203c87513e59d97914

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 0fd671dd2c180cf3f3378ff4bc05fb69d2b8af9840e2c8ee113333bfd7f56ffa
MD5 b6c1c1403cc155835d526389261c013b
BLAKE2b-256 182d4b91f7c0e46068261d07a1c72f810e28aa961f7c2449908729dca97f2944

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 355.4 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 11e041ee9019917fea2d410ed36a9db421052466e16c7a3d2757bf3953be3dc8
MD5 93194c2df886301f9e0ba479d2bfb10b
BLAKE2b-256 0ead7e70985f45bea6f1639b8a49596aec27c4b28db491787049ddeb903375e0

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 295.9 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 3ab295896750dbe948cbd2d1d6c081e4695a8e401488d45dc0a90c8c414fcbad
MD5 12ddce7e685ee00df8e36f41fd51477d
BLAKE2b-256 8c0045a114cfcaa0776fa687ae8fb105009a8f4393c0f2a9f34b4de74da70c00

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.3 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 4cf8b659235d0b24907571179ab5a3d11e7480c5311b1e16f3ecbe75ec6e4dbd
MD5 f7e56e7d95d5072c4c939849c5aabfe5
BLAKE2b-256 6fce2f9a8a25903c8f01c11cd24f3084b8f49833bbd32ca882b29ba8548230e7

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 370.9 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 2c48419ae0600ef5c6425883d7149d4827671fbd4f82e552beb8012b7c1710c9
MD5 823e8a57e84fa3b1b35f4bccdd5cbef1
BLAKE2b-256 1233cbee015bfe792ad3be5a9819dcf148ae7d8148c2bb88fe9165e4c758774e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 279.2 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 de40d461774034222f1b2ec0d3ac64226c56051f2a1ca85301bef6ca13a7edaa
MD5 1e7c03e8e4af0e54548f88c86ca80528
BLAKE2b-256 9302cb6b69197ebd5b646a047e746068c3ffdb56d3c80fd386e1822713ab88fa

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 57f9e2387ab093c65dcbac811362a47f5c7d7245604d1795da7899f70f4746f8
MD5 5f3a5ac2f589b5d9a9db6288d404231c
BLAKE2b-256 f735a543c9f06f45650a6f6ffc05afaad9890256d9c14ebb6e9202dc2f261d51

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 335.9 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 0d5d90bd88959a04596a7320043d9c29adf7d6651c6fcde4ec9692daff0c3a7a
MD5 53176bef2c948dd59885aaceb9c59305
BLAKE2b-256 eb87e643b2e28c59d5c308518fe55f3320e1bc2322777c3023db9a909a4eba3d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 272.8 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 9289ed369a2d060e896e59ec7367abd4ad22e9219e454a534d10a55d9171ea75
MD5 8e6e554b2967b7d1c778eb5b674a038b
BLAKE2b-256 b04f21fc700b698bdb2d88c4f00298104cc0ee51015297ab2025f62e8abd12a6

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 22b9cb23e965370861b657463036bfb16f74a3ed35fd06d71857618ac76528b9
MD5 999e9882045ab06cd4c670200d39f0e0
BLAKE2b-256 0c8be3fa3cb806491920d4801de7cbcb96d939478671f03ad1d07eb066ae00c5

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 326.1 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 b846be6f1b1993ccca80850be6cf9dc5086d78e89f66b64ef08cba7b68d9250b
MD5 01efc6bb18f2257f80754c742ad836bf
BLAKE2b-256 ad5a2749c5908cd3412103287ad3a33e130652c160456d7083c4276a56c4bc9e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp27-cp27mu-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp27-cp27mu-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7mu
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 def9ae6046cd727909062e715addddf194b3c6243a757bcf0dd4479bf3d7ab85
MD5 6a6577ff58df14e5aaacc62e3fe54dbd
BLAKE2b-256 e6819ecee4b61514da16e6e11b5bfd3cc35112bc9c2846d846ec7cc1cda0b106

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 282.0 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 0d9a757f511f0c083d40a90fe0d3731bb11bf0db2af82e7da54c254c0034b1b4
MD5 54f00b84055608ba8169c6d22af85a2f
BLAKE2b-256 905683da0c6c9c00a8d3d05ed053bc0d3ecfbbfa3596e0e5ba35842aebdcd60d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 06b416aaea3ffea68d27987c9c9659fcaf0334993535c82a38abd71b1406e87f
MD5 4da197a49d6d2c2a3dcc062b3867c925
BLAKE2b-256 c918879ac007d54eaec0cd255beecfb5f3528a2e023d911d04a37d7ef0f8fbb7

See more details on using hashes here.

File details

Details for the file fastavro-0.21.16-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.16-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 338.6 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.16-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 93e9a5845691d5d05e3fff9bfaebe58f2775147e2285cab26962d7c6d69e48d3
MD5 e18c5f0b76d42a0a84d44e5d3767e94e
BLAKE2b-256 1c348285a48e90d50e04100ea1c54171dee324a885b288cfd322b41462b9484a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page