Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.5.tar.gz (455.5 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.5-cp37-cp37m-win_amd64.whl (284.5 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.5-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.5-cp37-cp37m-macosx_10_13_x86_64.whl (334.3 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.5-cp36-cp36m-win_amd64.whl (284.4 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.5-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.5-cp36-cp36m-macosx_10_13_x86_64.whl (345.1 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.5-cp35-cp35m-win_amd64.whl (277.2 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.5-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.5-cp35-cp35m-macosx_10_13_x86_64.whl (330.5 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.5-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.5-cp34-cp34m-macosx_10_13_x86_64.whl (338.8 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.5-cp27-cp27m-win_amd64.whl (279.1 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.5-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.5-cp27-cp27m-macosx_10_13_x86_64.whl (344.2 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.5.tar.gz.

File metadata

  • Download URL: fastavro-0.21.5.tar.gz
  • Upload date:
  • Size: 455.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5.tar.gz
Algorithm Hash digest
SHA256 807cb59c7d9e5f7da8ae6567fa134a17ea2007c81605d432ced7f079bb1ba1d8
MD5 a66a1d5130c2deba34b1735120a1ca2c
BLAKE2b-256 3865e6ea7fc5ab74aa4072e5d7c463efc316d43a3bc7fd793241a142104a9be6

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 284.5 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 71f521affa45dd895402fd2ee448773beb93af2cb11e1df77e6f4ca713a34eeb
MD5 962c8dda02578058041700d4c564e575
BLAKE2b-256 1a1c1fd2fa3a3003e6d2fcba086effaa074a4cf8035ee42e9c880ec83e67ad62

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 545d136db18cb51cc72ebb3350ac6a3311d67a1ee136e6c30066f3c0b104e42d
MD5 4365970d6aae911b652899b2e9a8aa3f
BLAKE2b-256 9647775c0e9aaa27cb753c6613eeb4edf71e85729673c5d193b31967ed4d2fe9

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 334.3 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 3ba1b9465407412382ccd6db31399a4488f2a05b3cacdd78aa9db10b3ec080b7
MD5 81506ea569fc636987b9c5a654f67f59
BLAKE2b-256 ed04cc12b6a9bb731f684aa1405c88556ec3ff36fc7adf937d59f3c803c8060a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 284.4 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 81e5793a05e17006c8b6e8fce3b90ba9463e523a4164b1bc0408d20c8b3dbdff
MD5 2f7da8497e18ced8bf1d09c5d95549d5
BLAKE2b-256 7a8fce55a58c4271becc06758731049194686bf5a6e9997bc50dfbe2d9f16815

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 615b37b826480384db2d875b4aedd9efe3a81bf5b16756eeaef6f386c9ee1923
MD5 02dc35934a5491b6c99eb618a41a34d0
BLAKE2b-256 4b5dc7b461764a78ffb3416e7fac3a7c7193a6801d5ff9052f4eaad594eb092e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 345.1 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 07b9279530fde83c9a3d6207e35b5863f947a14951a7fd0c116b04ebe1fb7edc
MD5 0b76d15bd19af856eb35ddec62ebbaf3
BLAKE2b-256 e52b39c644e5c8199dea27acb588f1e106c8dcb413a066853d51745bd2adb238

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 277.2 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 5c563d83aaf4030e5bbab276d9f4f2b4e51ffa4f2fc13823ad5303cdcc15c268
MD5 ee11304636ff6e6353259c8cea53f9cf
BLAKE2b-256 a1d4a4dbc41ccbe48d5dfbad8d39747b56b87be27f11b484a1fff38ed1585b1e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 65607d48904d0866a9f21dd4e286cf5c4f097bb10624c11dba3117131a7e12b2
MD5 59c9dd2a32c7fa53a57457788372aa17
BLAKE2b-256 2506ba4b1ce9914ee6976de27e4337cdae2944bbb67807bc5b09c4fe0a88b5ec

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 330.5 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 1fe8cee734f954bb440ad803c87d61f7eb543910741d5f389d4ff7f21e00d894
MD5 3271432b458755f196b44f43349fcbfc
BLAKE2b-256 66e45a47114910a2f46b7dbde5e3292e15fef77d8124cfd8c639d1d2eb48b5bb

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 f365010d394c3df0cd905b7d5fd1c7bbe48981187a60a8c0f976aab818f7f10d
MD5 063e810d6248eea9ea94020e5cc4bb27
BLAKE2b-256 3337219e148730b625fec50007f48f788ee3889024bb5be14480fa495c20a1d3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 338.8 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 3dc1958f10493881c4e78a8526c889254fd79ff3262a6ad23085ae1f2283882b
MD5 fec749984c11ea6132fd2ae003cca11c
BLAKE2b-256 02fd58509f133315f85fe23e06f3f814fc6c6425415992f9202aa9789cef97b4

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 279.1 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 5e8afb8a6977efb3931a9d45b4a671108e6e0db29b16fa8dd93c7ad54c697665
MD5 03f5b38e8853398a447c69b0b3aa1fd7
BLAKE2b-256 0f8900350459c08ae081f744567537c721bfa0bfe46882b4fb3f4ae719f0df53

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 146bf3cfae8e0b90390f74e952d36c20251674accde0fdad6abaf700bdced756
MD5 cc603b89c3395a3155b4d9c099b86104
BLAKE2b-256 d1604a74ada3da2ce9e0610325377d792290a9c57ea8f3d14dec17c904c02f79

See more details on using hashes here.

File details

Details for the file fastavro-0.21.5-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.5-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 344.2 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.5-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 9f8ec56a4d2dbef4512efa210e15be468ccf478c2b5c9cace9584eeaa86f2538
MD5 f5028f8b703e7b7dc650d8907e1414d0
BLAKE2b-256 0c730c7abc0db7108712d9e809c4dc66c4b988f9c3bbe9d534fe47834a5c810e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page