Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.7.tar.gz (456.4 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.7-cp37-cp37m-win_amd64.whl (285.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.7-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.7-cp37-cp37m-macosx_10_13_x86_64.whl (335.1 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.7-cp36-cp36m-win_amd64.whl (285.1 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.7-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.7-cp36-cp36m-macosx_10_13_x86_64.whl (346.8 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.7-cp35-cp35m-win_amd64.whl (277.8 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.7-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.7-cp35-cp35m-macosx_10_13_x86_64.whl (331.7 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.7-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.7-cp34-cp34m-macosx_10_13_x86_64.whl (339.6 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.7-cp27-cp27m-win_amd64.whl (279.9 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.7-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.7-cp27-cp27m-macosx_10_13_x86_64.whl (345.1 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.7.tar.gz.

File metadata

  • Download URL: fastavro-0.21.7.tar.gz
  • Upload date:
  • Size: 456.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7.tar.gz
Algorithm Hash digest
SHA256 9741856e497282120fd30b3692050a9f3eb3731cfbe20f8d4f8a1ae300d39673
MD5 a58ba42fd1df3f7bc144b9c4e6cb7b96
BLAKE2b-256 a7c3c4bbca0eb46e8fe0515834b6c2fb2e2538893d9b209dfdae7b151b5cab07

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 285.2 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 5fb504d4f590ddaf5adb3c781e539e198f7b3d7e718f36ea5c2de06b8c10071d
MD5 728217ac231a49fb5cdb37b963455c3f
BLAKE2b-256 3ed0fc12144141c0f94cac99d969ea3895590969a5d47395124a3d6f0dc4d31d

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 f764184a462913070a4d40574f4992c563eb2cf079885317162e79fb80f8cd31
MD5 8bb389d93e2115bd92b0f1f129606fe1
BLAKE2b-256 50043d063f1b7be6a7cc57a40163a4da40a2b5fbfa43fc48bc5dfba7262e63e8

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 335.1 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 933c32bcdd2fb1351d2151c18b4f4243dee46e10505f857a0ae39071b9294bb8
MD5 32dc455f9b222090954e8769f5978aac
BLAKE2b-256 cc262a6bc3deae18af242c98c7d24a2c480683e745dffa6072a92b55abcdd5c4

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 285.1 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 206b884055d75c033f112f07e3c6df2b964e50a09664762d00aa98c5f042d48c
MD5 b42a60111826d65118f315be3c18beea
BLAKE2b-256 607ad89a94145eef3be90bbd601f4e34b1caec850be43fbb111d2acf15bcfa7b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d675b4ac9b0a36b0df39e9e9704301cbc12ea5fd37568f1fa095436a806d8f91
MD5 ac43e67fcd063c9ac92e12525dca9017
BLAKE2b-256 c05fee791072ce30b29b9111abae2fa4f4cbb4d8257a0fc2b524333b80d584b3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 346.8 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 f912368684865ac5203aa75ca1e0964a0d74340ab35aee23124f52858311335c
MD5 b5e2ec0eb477573d294d35f0b1576470
BLAKE2b-256 27b8778a952cdadfb8081b23508c46a5bb011b0c7dba35df54329afe503c6724

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 277.8 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 82d8688a3b8a962e48c785f52cad0d1759c9812bbf0b60d5a8528a1068c69a5e
MD5 4ea06762a45a13d2ccd1f35547348347
BLAKE2b-256 59c114e1cacbe4eaa161ee6481793842d102e2d7131f01a180d3519cb735c9d5

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 e7536180f226955a556a2584501c0ddeee5ea7d1c7d369cf0c71f9f47b200aca
MD5 7ace5c364d244853f530e5bc6cb6a95d
BLAKE2b-256 267d95dc1a905f64984273fe12bafa98679e185d5c52a8f8771c1de68ed6d967

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 331.7 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 6247fb0db2682b593eedfd54cf37dd25785510c83343efb23f437fd47133a240
MD5 cdfbae448239a25b76462d88102380c6
BLAKE2b-256 00dc446b04ccb8d6dfdd95cfa20a4578b0bdd0470046247692b8e84ae89c9df0

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 aeb765ac5bf5143419e4d7154f3f4b7547e2fc8ceb4355b4d7212040fd6e1a0a
MD5 52dfeeb44941030b73864e860ff6958e
BLAKE2b-256 18fa13d0fa11e6e36b5b26b0d773eb191a63114ecce4e8e0bc0ddb536a21f91c

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 339.6 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 2828be2b1ad59f2e51fd1729508b1fdf75b724f47bad556198b2e5c59cede8f6
MD5 8869126d0d3e1bfe5fd79679e9dd9af4
BLAKE2b-256 f7f7f00315a3e5ca1faba46cccdbe6833b117bf8077e7097c0bb1c100cbafbe4

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 279.9 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 31222e2907b90341884f08c5cde426d4d2a3a4c48bfc88bb6db07d7c310e8d0d
MD5 d6cac5a8a8442756816e9200628ce453
BLAKE2b-256 4252754e8e984782739074dda309d22704128c09220352e3d88381b30f09b7e3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 625b1f5ff9979c32a6d4d267ec30d62aeeeb7d8fbb38c86108f20e265b437602
MD5 75804ab54c637dccca3cdef4e83d5f0e
BLAKE2b-256 d7d5a0ac917a1501da4bfc7f36a6fed3c2109bf7264089be70d777a1497df314

See more details on using hashes here.

File details

Details for the file fastavro-0.21.7-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.7-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 345.1 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.7-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 430870af10441fd4129de01fe93803a34f4d78f85058476bacd8a8961fbeb69f
MD5 05111ec801e7ac03df853311c823c2d0
BLAKE2b-256 ca8d3b4d7b6ac3cc71ae8d290706b66214ff58ab9410b57d26fa33dff8f3d1bb

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page