Fast read/write of AVRO files
Project description
# fastavro [](https://travis-ci.org/fastavro/fastavro) [](http://fastavro.readthedocs.io/en/latest/?badge=latest) [](https://codecov.io/gh/fastavro/fastavro)
Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.
The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).
fastavro supports the following Python versions:
Python 2.7
Python 3.4
Python 3.5
Python 3.6
Python 3.7
PyPy
PyPy3
## Supported Features
File Writer
File Reader (iterating via records or blocks)
Schemaless Writer
Schemaless Reader
Snappy and Deflate codecs
Schema resolution
Aliases
Logical Types
## Missing Features
Anything involving Avro’s RPC features
Parsing schemas into the canonical form
Schema fingerprinting
[Cython]: http://cython.org/
# Documentation
Documentation is available at http://fastavro.readthedocs.io/en/latest/
# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)
pip install fastavro
and on [conda-forge](https://conda-forge.github.io) conda channel.
conda install -c conda-forge fastavro
# Contributing
Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
Get approval from discussing on the github issue before opening the pull request
Tests must be passing for pull request to be considered
Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.
### Releasing
We release both to [pypi][pypi] and to [conda-forge][conda-forge].
We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].
Make sure the tests pass
Run make tag
Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder
Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder
Run make publish
Note the sha signature emitted at the above
- Switch to feedstock directory and edit recipe/meta.yaml
Update version and sha256 variables at the top of the file
Run python recipe/test_recipe.py
Submit a [PR][pr]
[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine
# Changes
See the [ChangeLog]
[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog
# Contact
[Project Home](https://github.com/fastavro/fastavro)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file fastavro-0.21.19.tar.gz.
File metadata
- Download URL: fastavro-0.21.19.tar.gz
- Upload date:
- Size: 476.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d332dc914de7e6bd6bdb9c246b916b62397c30be9f7ae5a0494ac5e9e079458c |
|
| MD5 | e6d171b11da2cacc2ee0b87e02a1f92f |
|
| BLAKE2b-256 | 0b974538c5e85ecc6d076d937075a4647d015263338a695e6638ebcd596a9ec4 |
File details
Details for the file fastavro-0.21.19-cp37-cp37m-win_amd64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp37-cp37m-win_amd64.whl
- Upload date:
- Size: 297.5 kB
- Tags: CPython 3.7m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e5361d01c39507dc194558e0f8029d13e3b76fe02ee854c02368cd4147fd306c |
|
| MD5 | ecefb89279fd0c65931f298a189f37b1 |
|
| BLAKE2b-256 | c1871d0adecefdd7808e8f2740e22bb62f2355fee91a0397ea6630add35ca2cf |
File details
Details for the file fastavro-0.21.19-cp37-cp37m-manylinux1_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp37-cp37m-manylinux1_x86_64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 25136b0e42081092cd1d11634dae443c8c802c029daf1365eec63026e359f077 |
|
| MD5 | 7ce7ab320c7baac174a2429443e820d1 |
|
| BLAKE2b-256 | 7fd75882de7273bfebedf671ae19b5770a2970aa3c10c5bdad46833e9f54e67d |
File details
Details for the file fastavro-0.21.19-cp37-cp37m-macosx_10_13_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp37-cp37m-macosx_10_13_x86_64.whl
- Upload date:
- Size: 356.5 kB
- Tags: CPython 3.7m, macOS 10.13+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0710dd1374132a62adbf5dacc41a2d69a6975aab9ebb50ce6c2e98dac075aa4e |
|
| MD5 | 56be1b1e1d33a83aeb5769f96df4f557 |
|
| BLAKE2b-256 | 78ae2b889d19040f11b547f550e6319e6f2602a1764e19e9c84551ece163d51e |
File details
Details for the file fastavro-0.21.19-cp36-cp36m-win_amd64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp36-cp36m-win_amd64.whl
- Upload date:
- Size: 296.7 kB
- Tags: CPython 3.6m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 64477b899cb94efeb1b16baf5b42d8003dc025a3dd05fd12ff9516c9cf9f3bf0 |
|
| MD5 | b3f12a311fc49d524c7e65c07aed905d |
|
| BLAKE2b-256 | 247d9bf54d2d85c759b779775c219038181e06a2541f17ce43e7ab1b95cf021f |
File details
Details for the file fastavro-0.21.19-cp36-cp36m-manylinux1_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp36-cp36m-manylinux1_x86_64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.6m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a1cc7255cc6b7ba729eef3b4699b29fe373bcb34a1a72481f847b6edd7c7c9e6 |
|
| MD5 | 1ac572826bbb0d301a19cf53f7e2d192 |
|
| BLAKE2b-256 | 99cbd6139a3090290d8f9d3b8f866a646bbe33a829bb11e749fb6623759f13de |
File details
Details for the file fastavro-0.21.19-cp36-cp36m-macosx_10_13_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp36-cp36m-macosx_10_13_x86_64.whl
- Upload date:
- Size: 372.3 kB
- Tags: CPython 3.6m, macOS 10.13+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 365999f69bd064dfff30a5bfc2b479763de3fcbc2aa7e19cbf8c5ee9e38cd02d |
|
| MD5 | d6282673c7cfc75d1727c82035406654 |
|
| BLAKE2b-256 | ab2bc4a6242f0f5bff0f3fb413d753b8ee5c00d7ba55facfddf9ce83075b662f |
File details
Details for the file fastavro-0.21.19-cp35-cp35m-win_amd64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp35-cp35m-win_amd64.whl
- Upload date:
- Size: 279.8 kB
- Tags: CPython 3.5m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | aeaa164ea02678e2e663020601824336409886bba937f92693b2716056dcd671 |
|
| MD5 | 07a1b88eb215a838f184db1b77f0c229 |
|
| BLAKE2b-256 | 8803b76780b9021861d4232be528dd97f30ead7a7057fbcefa8f6ba8c3f82525 |
File details
Details for the file fastavro-0.21.19-cp35-cp35m-manylinux1_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp35-cp35m-manylinux1_x86_64.whl
- Upload date:
- Size: 1.1 MB
- Tags: CPython 3.5m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fcf12523e920db618a00e431b788d8c314ffe0d4a726b63d4644db991c3afef9 |
|
| MD5 | 753f80fbfede9d6411cb5f7bf595965b |
|
| BLAKE2b-256 | 71fe40b991f2b3c1b4597ad27bf7ce0757371d5d800f7cea89561ef25c1ab01a |
File details
Details for the file fastavro-0.21.19-cp35-cp35m-macosx_10_13_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp35-cp35m-macosx_10_13_x86_64.whl
- Upload date:
- Size: 336.8 kB
- Tags: CPython 3.5m, macOS 10.13+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3db62e93f6c66827056de2ad4bf4dbccf07975b640aab7ac8739ead255fe7233 |
|
| MD5 | 47e00d5382ac29dec85365f46d14fb5d |
|
| BLAKE2b-256 | 4cc74e066b44d3177d7a945c4a8080af45c9f0e4220f498be54014e3d4034d7e |
File details
Details for the file fastavro-0.21.19-cp34-cp34m-win_amd64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp34-cp34m-win_amd64.whl
- Upload date:
- Size: 273.6 kB
- Tags: CPython 3.4m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 213530c8aa8dca55c5d9d92f7de8b9a72770830653d6e49bdcee17e4310b1e13 |
|
| MD5 | daaae8bdbda8a578e1ec4eed5835ee4d |
|
| BLAKE2b-256 | 06d885ddb9b20157600021f8672fec91784054f3b28d71069a7c0be64b7c6a34 |
File details
Details for the file fastavro-0.21.19-cp34-cp34m-manylinux1_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp34-cp34m-manylinux1_x86_64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.4m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fb314b17e39ff8d056f92ddd6e3c73741b3841fbbbe69370cfc1a9b4887b6c2b |
|
| MD5 | 4cee8255e3dae817336f464879146631 |
|
| BLAKE2b-256 | ea3d99f42bc340b98cdf3b62379ba90a53fa88d132af0993266ea4bc0d8f5122 |
File details
Details for the file fastavro-0.21.19-cp34-cp34m-macosx_10_13_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp34-cp34m-macosx_10_13_x86_64.whl
- Upload date:
- Size: 327.5 kB
- Tags: CPython 3.4m, macOS 10.13+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2e4ed3c72d138f7f7011f85a60fb8b481c9af408fcbee21e7f4baf072e4880d8 |
|
| MD5 | abf6ec97c437e4506e56958d4afe1f01 |
|
| BLAKE2b-256 | 6e3e174714037b97f8eefa01178ed019ef65eff6bd25859e169214b8b2808024 |
File details
Details for the file fastavro-0.21.19-cp27-cp27mu-manylinux1_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp27-cp27mu-manylinux1_x86_64.whl
- Upload date:
- Size: 1.0 MB
- Tags: CPython 2.7mu
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8e50ead0ab4855d61748b974c48f7d093fee32d6682c97e023ad6f6aaa86a36d |
|
| MD5 | 552adf0aff2378d5b81ae817adb78b3e |
|
| BLAKE2b-256 | 6769be1402c904506a5ec79ba65c7f0035342d506d5305cfd9ae8bc6f1c8db47 |
File details
Details for the file fastavro-0.21.19-cp27-cp27m-win_amd64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp27-cp27m-win_amd64.whl
- Upload date:
- Size: 283.2 kB
- Tags: CPython 2.7m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 805be0e232f9f10d3dd9f62fad4cead02753d55fcd4d7fdbc2d548485650fc1a |
|
| MD5 | 8095483b1ad944b3f04446a338815698 |
|
| BLAKE2b-256 | 28abc435df8a59a4e8bf3a05f3fdbe1ca2f5beb2f9a79b8670f542ef4e0de34e |
File details
Details for the file fastavro-0.21.19-cp27-cp27m-manylinux1_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp27-cp27m-manylinux1_x86_64.whl
- Upload date:
- Size: 1.0 MB
- Tags: CPython 2.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f366c10e6485b6b5452dd05ec8e235b72ae8485abbea1dd7616ca5ed42794ad5 |
|
| MD5 | b57613728b40ec4581e1113777c1c093 |
|
| BLAKE2b-256 | d3959ae7f0ba9e1a8fe0004d65d0dd08b509fd2c00881045114d4cf3c73f544f |
File details
Details for the file fastavro-0.21.19-cp27-cp27m-macosx_10_13_x86_64.whl.
File metadata
- Download URL: fastavro-0.21.19-cp27-cp27m-macosx_10_13_x86_64.whl
- Upload date:
- Size: 339.4 kB
- Tags: CPython 2.7m, macOS 10.13+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 635531aaeccd439a328561db9c4b2924f06b94204248e3c24e203d3446d793bf |
|
| MD5 | 813fb390eada6a1ce18171d531854ce7 |
|
| BLAKE2b-256 | d1213c1fdd40445567c23df52460ed16dfd3b5ff8ce39802e7385580cecc4a48 |
](https://travis-ci.org/fastavro/fastavro){kind=link}
](https://codecov.io/gh/fastavro/fastavro){kind=link}