Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.18.tar.gz (473.5 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.18-cp37-cp37m-win_amd64.whl (297.3 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.18-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.18-cp37-cp37m-macosx_10_13_x86_64.whl (356.2 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.18-cp36-cp36m-win_amd64.whl (296.5 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.18-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.18-cp36-cp36m-macosx_10_13_x86_64.whl (371.8 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.18-cp35-cp35m-win_amd64.whl (279.6 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.18-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.18-cp35-cp35m-macosx_10_13_x86_64.whl (336.4 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.18-cp34-cp34m-win_amd64.whl (273.3 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.18-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.18-cp34-cp34m-macosx_10_13_x86_64.whl (327.1 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.18-cp27-cp27mu-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7mu

fastavro-0.21.18-cp27-cp27m-win_amd64.whl (282.9 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.18-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.18-cp27-cp27m-macosx_10_13_x86_64.whl (339.1 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.18.tar.gz.

File metadata

  • Download URL: fastavro-0.21.18.tar.gz
  • Upload date:
  • Size: 473.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18.tar.gz
Algorithm Hash digest
SHA256 ec5d4d3d18710d65e5b3d32a0ea7bd49842ba896685934010d5175a9f16f077f
MD5 8efb67917319ebf7c2054b3f0de58ed2
BLAKE2b-256 5a41946257f5c4df1dc6e84c0ee2303a20ffdc2df9eb2a107d5e58fd495c6994

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 297.3 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 c79cf3e00bb91eb0a0622ddb8b8facd7bd01c2a9a7f4daa77f7c95d0e09ef1a1
MD5 62c449e87275623f633f7a90b569577a
BLAKE2b-256 2b54266d22a49b2b90087b7a71dc3a537000f2cafb50dd207d117264f061ffda

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 9f9961beaa50825c3c4e8fbcc423e4aabdb495971f616ca5a2a7d170f7a8c313
MD5 c8e0ce40641fbd8d52c6ac24c1f1165e
BLAKE2b-256 57440c791ae237d2d69abd0fa91df8896bd0cd06d5c3d9e48969ae9ceee664c6

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 356.2 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 022067e166ef8810ce327d07620e9327b15ebc5965b987a847e1d63047e819b2
MD5 42248fed746ae198ba379077904bc49a
BLAKE2b-256 f6fcd8a241ddd6e5e83d95e32e38fae5ab2fdd2f384349db1c83acb071c678d1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 296.5 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 4dc1dd0db53decf9613eb9bd19eb788a8ccacae038b8b0be769524eea7e0581e
MD5 2be46bffd84f5796026304f04979cbd1
BLAKE2b-256 7a3493e61d21b14ce6a6e61a37ae687ba0991f670cb928adeabb64f813359212

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.3 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 40cb287fcf1cd31141cd4c074ce13c126d420ba727c27d1be38b154a97863868
MD5 bbb65be62a86b2274fb5abbb03928a8f
BLAKE2b-256 a5466a6496ebb6f4a47b82c4a0a5e6af27a08110e57ee604e8a655d670850495

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 371.8 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 781f6d89144f5e14c9f24a536b7ea8c7dd6befc7e52945a22b6559e6bdde6d91
MD5 6e43e2892945db68f49a867aceb7e14a
BLAKE2b-256 f6f2c431012c7dbf99954a81a7c402454249c0fde159a71db583da2cc9924824

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 279.6 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 c2575da966b4619af79f7c0bc61d3c35082a3710a4f0855ea00916b00941a9ac
MD5 405ebac5459ef201d2065e0263e37814
BLAKE2b-256 cfbf300965d92c528bdc1022579a2a566de376ca679504d2094a4928f39f6f04

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 3664661a799c5666fa4e0abd2913e174b14b591b12588b13fe805bf3756f1aed
MD5 9e8e3c4191f679f6837b75de561fc483
BLAKE2b-256 d706f92d643c8e48bf931b93db61757780c176dd9427baa57c40a9c1507b19cb

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 336.4 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 541b91c4f24c480970a48e5dc689d0be7ade9a035469c4cdad69b201cc9394a9
MD5 5e1f07ff2aa95b66d3846896cb874ca7
BLAKE2b-256 830ddee260075995211ab0f889b03af646f7db606d32f64637ba9fe0e61ffa4f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 273.3 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 99deb201b8d89a265866293a55657641edf66ffe20a482d8ea7bb73b0ae3e9fb
MD5 72dfb5bd7ca2428b8d2afa40d2e66d9d
BLAKE2b-256 a54722e35b22de290caf67d80430eadf28cf9d95c570e3e3b3038cbd08164668

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 878ac416301f01ec2f73c1d506a86ed0b636e56a3ecb097388d5aceb5ba7a3eb
MD5 f33720c09e471b77ae69f3b353dbf0fa
BLAKE2b-256 405317cd8f41d95be8b8c6caef5773f06377542637f3c6c5c676166ecabbfba9

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 327.1 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 55756868a591989ccb9a15d1d1de44159d4814dd8ca2b95a9463fa844eed5aee
MD5 68c344fe1d07964bfc721a7cc765e441
BLAKE2b-256 9ce83fadca40656b1bc9ad61f379bb8730b4dd749b58b03b4c3ade794a4c5af3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp27-cp27mu-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp27-cp27mu-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7mu
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2614f652b8e233f182407692c6a4459204cf18d1639ccac02614e0de1449984d
MD5 e8e439df8cbc847352c7fcf01d3d834e
BLAKE2b-256 52f5eb52fd7e7b261334f0d9340f55069b8d85cd0bfde8da33a2f6a3929ac911

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 282.9 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 078fb31592c85545f6394fbfc97df5da38504caf54b4c2cf9ea64d13ba60f923
MD5 196be32ed38148020844bc5ce8147c96
BLAKE2b-256 6edb97fffff2072b09fdb6a3dd6ab2ad029c368ad835cfd09e90a8fd78bbfd9b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 87f3844118439146cd988affb91d9e7371a940d16cdde510f8646f12f46f19af
MD5 961fbc5f63e2c352d9dd74566fa76e4e
BLAKE2b-256 e308d764cb9fc3b33434c04bf38bedd19b213306abe349bfe9b62c5fa2df9dc4

See more details on using hashes here.

File details

Details for the file fastavro-0.21.18-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.18-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 339.1 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.18-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 0defb3297172e0f8118f14432c7aeaedf8492134226b587f5d00d37124409659
MD5 8ad2414b1eb4bf196e2c0740d34e5228
BLAKE2b-256 5a73d08cf2c36654501b2e4766f6d7a7f223ea1c6ad275d8728f85f859ff02bf

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page