Skip to main content

Fast read/write of AVRO files

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml
    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.15.tar.gz (470.8 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.15-cp37-cp37m-win_amd64.whl (294.8 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.15-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.15-cp37-cp37m-macosx_10_13_x86_64.whl (352.5 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.15-cp36-cp36m-win_amd64.whl (294.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.15-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.15-cp36-cp36m-macosx_10_13_x86_64.whl (368.4 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.15-cp35-cp35m-win_amd64.whl (277.5 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.15-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.15-cp35-cp35m-macosx_10_13_x86_64.whl (333.6 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.15-cp34-cp34m-win_amd64.whl (271.0 kB view details)

Uploaded CPython 3.4m Windows x86-64

fastavro-0.21.15-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.15-cp34-cp34m-macosx_10_13_x86_64.whl (324.0 kB view details)

Uploaded CPython 3.4m macOS 10.13+ x86-64

fastavro-0.21.15-cp27-cp27mu-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7mu

fastavro-0.21.15-cp27-cp27m-win_amd64.whl (278.5 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.15-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.15-cp27-cp27m-macosx_10_13_x86_64.whl (335.1 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.15.tar.gz.

File metadata

  • Download URL: fastavro-0.21.15.tar.gz
  • Upload date:
  • Size: 470.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15.tar.gz
Algorithm Hash digest
SHA256 e84e238d75336cb7e636caf4b2772672d2a6f0b15a0e36d2a7f379071563feaf
MD5 82517f7b38858408d25695be10108eaa
BLAKE2b-256 eeb25ff83d39a549b6fb35c846b5939371f7c30ee5e9e5a7451ca56e26554e58

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 294.8 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 79da736f71a6dab7f979bbd4cccdcdcd62d92da113b0efc4cbba99ef91e27718
MD5 11156c14aed47fb9695622136599e441
BLAKE2b-256 bbc3bcd4178716480f30878ca351d8bc42c6d2f0d14fefaffb12db14187dbf22

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 0f802bb700a11cd69b54ab55b53c8d5c08351e3cf651b1c7ee0a1099ba7f1f55
MD5 8fb565236ffc7c2a5df7f04f2b9d6043
BLAKE2b-256 6cd49db0a494ab1c17e00deb0015eb25442852bf923bc745cc2237977b89468c

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 352.5 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 efc07e4fe4ca4e8e8b8b0b2fc5b7d065582073fa014e50511517dd4edbae7645
MD5 f7b3ebe437a4143c4d194b62013ccfc1
BLAKE2b-256 9170c90844cc4cc78433adf57aa0df202ef07a16ba61ad41c011aec7a4096a37

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 294.0 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 c6b22e91643a374bac4e5719ff69c55493239e475a3324a5b3708d39ae93668b
MD5 5db568d55be37294870391cebb46dc1d
BLAKE2b-256 0d9fa83d13aaf718f121d49d551ca0ec53211ee86e87c56d1603f3bad6146f76

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8a898f60303f103e625ccb08c0b210a755531b045d75f88319889af63dac9763
MD5 c0de38c643548c7395abaf9e4d9d5696
BLAKE2b-256 b7dbc0f0e7d5176c35985ec25b652e78a4072c34b353070b2acadccd8637c712

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 368.4 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 67e7a13848efa65135a47b618ad4e91d68de50f648d12823c8be965b3617bdc7
MD5 fa0b3355d5fc6fd59a98fb16bd2179be
BLAKE2b-256 134cecb6fedca63f03d7bd039f7707794d90d4471e8cc01aa5461b5883333cf1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 277.5 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 5e475d30afce75da4515a2ec596e480cbb4165ea31ae420208749775daa65273
MD5 47933043a74cc01bba011f25fd371eb3
BLAKE2b-256 af09a1d6d527967f36414755f8662ff515d79415e95e59655efefcb98e24ea9f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 9d78743b0b02703140dee6bb3ffba6dc76c92a945601c9c2bb5060b9e7470cbf
MD5 8c8b8068c24e0277e5132ccb66f942d9
BLAKE2b-256 8f1e9b4b42e7f482c4d5edb1d017c38ec1a29d0602b4f795dac7828d750a53e1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 333.6 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 63e6953e47d30249c54816cf8147a159a78e31370585eefd4535df0ad5ff6050
MD5 06ce93a70a70eedbd532993b7a4f5466
BLAKE2b-256 d81c990acc33dac2d0e00097b0d1b11a8c71e00de658e6b4ebd4e611fa78e27b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp34-cp34m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp34-cp34m-win_amd64.whl
  • Upload date:
  • Size: 271.0 kB
  • Tags: CPython 3.4m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp34-cp34m-win_amd64.whl
Algorithm Hash digest
SHA256 452a061aea8ee057d43b2c14ac84fc0eeb0f3e048035f0fae723a7d884c7ea2d
MD5 decac4b7b04ba47732fe1267a3d1b2ba
BLAKE2b-256 6fb611875da77872ca178ed8ded4b5257fe3932d671d5b29f09e6f1c852edb95

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp34-cp34m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.4m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 192f9709c61bcda03d794f04d67e93976b0353ab2dbbfab23469936ee0b95877
MD5 6cb1a47b3c9f690136c0d9f19f0125f5
BLAKE2b-256 e96a44b5447410727a8ff9ce5649b387f7ee0293b6934ed0ef1778ecb08b2b23

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp34-cp34m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp34-cp34m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 324.0 kB
  • Tags: CPython 3.4m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp34-cp34m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 6c2158bdba5f079047e5b163f8eac2093cc96f6acf1620a1b7b1828a3563b551
MD5 b43a86c5435d2f6d53a100b3c629aa7b
BLAKE2b-256 620ee6574a2d586016e81b1868f95706a3d4fc2ba3c5c9e248cfdabbe6ee1269

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp27-cp27mu-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp27-cp27mu-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7mu
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 30dff5aa9f8d764e0a9be6dfca0937a1cb4f8ade3633535cc64402525363cb44
MD5 80fa6ac8dcb7c1629054288108756172
BLAKE2b-256 160eaf63a29f559506341e31764b5716b410162d28871271f2379e1511a64d82

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 278.5 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 0c46e14135579016192b45bdbf2e567b09adccf13c5aacbe5d005d96164a8b0c
MD5 330d2bb46eb3292ec15822e0eef2e5a8
BLAKE2b-256 167cf3c4ae8a0b0840dacf589fd5cfedec0cfcab307dce1fc40937ca698e9a8a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 14bf9d23b05956cbb90dfb46c5494c55aa3f4c7ff07176eec3b691f9d68479e3
MD5 192c0e9846c2bebade14f899d31ff726
BLAKE2b-256 db16efd60e41c238a7767350bdbe486e64294408cb87912e371ea63f8ff17719

See more details on using hashes here.

File details

Details for the file fastavro-0.21.15-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.15-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 335.1 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.15-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 68a6dd6533d6af0b0293a73eb318fa96bd3f6916d623276df0b02446810e849b
MD5 2dd471ec47e6d5b939d257ccc2e9d487
BLAKE2b-256 ef3e601c23c1d4ec7f23fcd7d23a8bd4240b60678c5a0d7b33520ec228641e12

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page