Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.17.tar.gz (473.6 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.17-cp37-cp37m-win_amd64.whl (297.0 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.17-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.17-cp37-cp37m-macosx_10_13_x86_64.whl (355.9 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.17-cp36-cp36m-win_amd64.whl (296.2 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.17-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.17-cp36-cp36m-macosx_10_13_x86_64.whl (371.5 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.17-cp35-cp35m-win_amd64.whl (279.3 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.17-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.17-cp35-cp35m-macosx_10_13_x86_64.whl (336.3 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.17-cp34-cp34m-win_amd64.whl (272.9 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.17-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.17-cp34-cp34m-macosx_10_13_x86_64.whl (326.7 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.17-cp27-cp27mu-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7mu

fastavro-0.21.17-cp27-cp27m-win_amd64.whl (282.2 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.17-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.17-cp27-cp27m-macosx_10_13_x86_64.whl (339.0 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.17.tar.gz.

File metadata

  • Download URL: fastavro-0.21.17.tar.gz
  • Upload date:
  • Size: 473.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17.tar.gz
Algorithm Hash digest
SHA256 1121d91426325c166212886298371f14cb8e8f9d6d3072e047d38e3927d5480a
MD5 cd0f95554c91a3d243ebea2ed055c3e2
BLAKE2b-256 336b0a2ce53fd404170fc296a2956bff759324c3ae39545b1dcdec81bc875afa

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 297.0 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 d0f36178236b508073fdcc1bd679f5572a02ec5361ef8f6fb10c6aa988746258
MD5 be501a97a2c5f270b7270b8e80286b95
BLAKE2b-256 0a9e60f3c8bee3ea14400ecb824626fac32b6903d0f99b577b70e1932c3983fc

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 1ad25ebb7fb62995dc58fd6099468e64887126d6b7097f95f380577fa85ba087
MD5 9a85217b949f952d1dbf445fb4e6cd73
BLAKE2b-256 947b376f3d5f3e23c870dfafecdebf883e229771702e334e05372262150a8856

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 355.9 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 5c8a2731d7eb448f804740b627663d389af14d697936f679e4180b0f6c59e9c0
MD5 0ba5057ae32927d271e14d6fd817a8cc
BLAKE2b-256 7ce42642b3d3872d850a98faffa2ddc2ed05ea68f7cb4d5dac84837cbcab40e3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 296.2 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 21f4998d9478d632cbc7cf7d60e043b1f185b44d2c0117c25991a6faebc33451
MD5 fe0bab4c00b4ba9ee2de1d7b3a8eef37
BLAKE2b-256 a3f6c5cd4ab70d163b5041d55485a7a659073ef6d84cc14ebb1958e2e389bd82

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.3 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8ed610b040d85cb4710444b918857ef1c47b6427d5320a2d4fe1704a8adbf285
MD5 cfedbc42dda0a9f66b3bf31f91d8608d
BLAKE2b-256 d643df488f3511b3da2f3a18749f308d97ac0d97523dbb2adac2fa1858f18ada

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 371.5 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 c6c9dc6ccbf4a468c050f80c37217ea79672f8fb02e2172d84889fd50e6633b2
MD5 80796dfbdaa980e8b2ad0945537d3aeb
BLAKE2b-256 943fd6f6d307539d84dd2f541d159e9d58a76ebaa470cc83999950294abd5335

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 279.3 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 568e92b4763f61c0072cebe1ae29f4f224349b894c9381b5995a4258ee6bbf38
MD5 2b17fbc2fef68fae754db36818d68924
BLAKE2b-256 c16f903696f03012177ffa7cf29e317685f27eabe751f1de5be356ad57fc311b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 401fdf8048c52034e148e3bf6a67ae96bf22916d41bd5a3646b1f436f963e21d
MD5 62bcefddd069381f988322e603f3ae90
BLAKE2b-256 bd6098df68f3702b84325abb64ffadc5fd6db07734ca8303f79305386b74159e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 336.3 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 d703ad726fd57339fdb493ca34debc99088f51e664888ef292564966ca5e0f36
MD5 2acbd36413c6bffc86dcbc458b6e5c73
BLAKE2b-256 72d409ca735beefafddb79d591ebc5830fee8665e88cd51cf55a668cbe231371

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 272.9 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 1b1011e48d001441d77e6c83c7ec9d309d537da65c8fac2eb42738c483a38758
MD5 510a9b040e5b47e2aa7a7c521ba924b7
BLAKE2b-256 8932dac6eef9ebd90d33bb594ed81cbdff0aa84eddaffc5b45269e1320c45195

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 103a6abf64073f6a6bf3401027d8b70d5ceb03ca58e6b9bf88b7140bea1a901f
MD5 19bf074fff9dd358e2732358097bad30
BLAKE2b-256 e9060f55cfc336ea6c84ba527bda81ec38fef2aa0e4e31cddd6b6a7961cd6a8f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 326.7 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 08719e77e595beecc7baf6396769ab8dbf1eb3d9e405920055142deddb70ea43
MD5 0cfcdef5a574e41f34f359e3c7c0cfc0
BLAKE2b-256 33921cdc80d95748ea30d9667210d7e7be6e08697c57c5d204a027ea7b5def0d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp27-cp27mu-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp27-cp27mu-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7mu
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 98075c87422c5f94a5a7dbe7a6a5c20667832322d7ddb5efb0818792cb08a2a8
MD5 c5d2e071adb052ad8d47939f7ffedca9
BLAKE2b-256 dabea98f9a81f4f31813ba4714e0bf26fca4bcac7e25f76a515bd86aa45dcfe2

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 282.2 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 7445bd691b032a00d29d7087b3aa8d338f32f9cae4fd075f132bec8f66755fc2
MD5 8ea255b2c4d5b7c3de508c3029e7aec1
BLAKE2b-256 b4f43e631170d02daadb1c3451681b613a84ee0871a416ddba4cbd7abea6145f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 52d54a2ca79dae1806a85f41b73f8cc15a94c90d7730e510c3f30625eb97f911
MD5 3a69c5bea65189af02ab901026b71dea
BLAKE2b-256 b08735e328414a5fc02e243e245e3d420eab671eafcce29bef7231a77c46ba76

See more details on using hashes here.

File details

Details for the file fastavro-0.21.17-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.17-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 339.0 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.17-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 4455c40d72fbc7e869b964b2e5c04ec791fbc09aa18b0ad31a811535f2204c74
MD5 98a9d00be7b5b7f63f7d01a86a8a310e
BLAKE2b-256 b8d756f95ca3b7cafdcf33612d46086800bd77c0502f83a29f0ae698ea2454cd

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page