Fast read/write of AVRO files
Project description
fastavro
Because the Apache Python avro
package is written in pure Python, it is
relatively slow. In one test case, it takes about 14 seconds to iterate through
a file of 10,000 records. By comparison, the JAVA avro
SDK reads the same file in
1.9 seconds.
The fastavro
library was written to offer performance comparable to the Java
library. With regular CPython, fastavro
uses C extensions which allow it to
iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5
seconds (to be fair, the JAVA benchmark is doing some extra JSON
encoding/decoding).
fastavro
supports the following Python versions:
- Python 3.6
- Python 3.7
- Python 3.8
- Python 3.9
- PyPy3
Supported Features
- File Writer
- File Reader (iterating via records or blocks)
- Schemaless Writer
- Schemaless Reader
- JSON Writer
- JSON Reader
- Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
- Schema resolution
- Aliases
- Logical Types
Missing Features
- Anything involving Avro's RPC features
- Parsing schemas into the canonical form
- Schema fingerprinting
Documentation
Documentation is available at http://fastavro.readthedocs.io/en/latest/
Installing
fastavro
is available both on PyPi
pip install fastavro
and on conda-forge conda
channel.
conda install -c conda-forge fastavro
Contributing
- Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
- Get approval from discussing on the github issue before opening the pull request
- Tests must be passing for pull request to be considered
Developer requirements can be installed with pip install -r developer_requirements.txt
.
If those are installed, you can run the tests with ./run-tests.sh
. If you have trouble
installing those dependencies, you can run docker build .
to run the tests inside
a docker container. This won't test on all versions of python or on pypy, so it's possible
to still get CI failures after making a pull request, but we can work through those errors
if/when they happen. .run-tests.sh
only covers the Cython tests. In order to test the
pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace
and re-run.
NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.
Releasing
We release both to pypi and to conda-forge.
We assume you have twine installed and that you've created your own fork of fastavro-feedstock.
- Make sure the tests pass
- Run
make tag
- Copy the windows build artifacts for the new version from
https://ci.appveyor.com/project/scottbelden/fastavro to the
dist
folder - Copy the linux build artifacts for the new version from
https://github.com/fastavro/fastavro/releases/tag/ to the
dist
folder - Run
make publish
- Note the sha signature emitted at the above
- Switch to feedstock directory and edit
recipe/meta.yaml
- Update
version
andsha256
variables at the top of the file - Run
python recipe/test_recipe.py
- Submit a PR
- Update
Changes
See the ChangeLog
Contact
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file fastavro-1.2.0.tar.gz
.
File metadata
- Download URL: fastavro-1.2.0.tar.gz
- Upload date:
- Size: 661.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f80c8d87b6d5ec4294caaa15ea8381724e71638c5bdf33af9a0b4d739e8860f |
|
MD5 | 5d409c9c4d54d461e09d23b8c4ce9328 |
|
BLAKE2b-256 | e6a2ecaa275a3917dc5fb1aa498ac50a368dd43e8b76d07e293c422e15931e27 |
File details
Details for the file fastavro-1.2.0-cp39-cp39-win_amd64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp39-cp39-win_amd64.whl
- Upload date:
- Size: 372.5 kB
- Tags: CPython 3.9, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc65d74c35d2ae53af0647e9ffa008c6fdb41e0a9c37e34c5681d915437db8ee |
|
MD5 | a7f39617cbd17ded8a8ea7768f78f8fa |
|
BLAKE2b-256 | dd6f1aacd7fae6af4ee4d3583c013b499bfd5a5ae2b2a6aba75e4b8263ec7b59 |
File details
Details for the file fastavro-1.2.0-cp39-cp39-manylinux2014_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp39-cp39-manylinux2014_x86_64.whl
- Upload date:
- Size: 1.6 MB
- Tags: CPython 3.9
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c787d793e9b2e4e12d03ae2b0abb52625717d3f4d3161c841bd09dc2520b67a1 |
|
MD5 | b7cd349aeb65a664b77cce5cb8f3204f |
|
BLAKE2b-256 | fcbb5cb966250521a98ac72ab5dddf659831448bf5be3b3d133a7ec5e049e295 |
File details
Details for the file fastavro-1.2.0-cp39-cp39-macosx_10_14_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp39-cp39-macosx_10_14_x86_64.whl
- Upload date:
- Size: 431.0 kB
- Tags: CPython 3.9, macOS 10.14+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0fc504908da59bd3610d0d544bb8845746f82aa169c4c1fb1547e31bc783c51 |
|
MD5 | 550fd6ce3e8106d09ef756ffb98fe400 |
|
BLAKE2b-256 | 92bde1b441b7e4749b0e0c157d67592f32c76e5dca157d6be4bdcfc2fce9dcc8 |
File details
Details for the file fastavro-1.2.0-cp38-cp38-win_amd64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp38-cp38-win_amd64.whl
- Upload date:
- Size: 372.1 kB
- Tags: CPython 3.8, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9223f4af6e14d985c59fb2e85788927725a6e2b37530ae65f4cce33f9af2b113 |
|
MD5 | d2146ad9af52c694d9b161509b410fce |
|
BLAKE2b-256 | e89550c943c26725aaf9d763183036596355875e4be5e0682830f33eadfb6a14 |
File details
Details for the file fastavro-1.2.0-cp38-cp38-manylinux2014_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp38-cp38-manylinux2014_x86_64.whl
- Upload date:
- Size: 2.4 MB
- Tags: CPython 3.8
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8abe3347b322925da8b5a97961ed87111db97e18e03195a6aedad8709a47136 |
|
MD5 | 86c92d484ee012e13634b3f5133b9c2f |
|
BLAKE2b-256 | 303b26283ccd47eff8a8fc6fe2aa9a4847f58cf871b769acd1629342fafe9c74 |
File details
Details for the file fastavro-1.2.0-cp38-cp38-manylinux2014_aarch64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp38-cp38-manylinux2014_aarch64.whl
- Upload date:
- Size: 1.7 MB
- Tags: CPython 3.8
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9964eb776d705dee923c55a488d118995637b0e3535f94e84d04285ba5a27d36 |
|
MD5 | 9d35cffa3707da0f4c87886cb7388539 |
|
BLAKE2b-256 | d11ac98c8a2eeb94231042a0e7a6b7ad71ef2cbc80d7d9e0a6f93e87ff8e512a |
File details
Details for the file fastavro-1.2.0-cp38-cp38-macosx_10_14_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp38-cp38-macosx_10_14_x86_64.whl
- Upload date:
- Size: 426.0 kB
- Tags: CPython 3.8, macOS 10.14+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a5dd6defe7b4e9e07fce85a863414e1b898257a889de0bb3240f9c91ab25695 |
|
MD5 | 558817811579579c092911daadbb551e |
|
BLAKE2b-256 | fccb3f124ed62b5d30d107d2cb936a81391faa3de6346bedd508c5c3efce0503 |
File details
Details for the file fastavro-1.2.0-cp37-cp37m-win_amd64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp37-cp37m-win_amd64.whl
- Upload date:
- Size: 357.7 kB
- Tags: CPython 3.7m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ff45a051798feee8fd521e700eb6315afe741b7014644bb50454e32a35819a9 |
|
MD5 | 333a6030e19334d9084a4881c145d7b0 |
|
BLAKE2b-256 | c2ee996aeadd0ec7e46bd80cbe022364099839f8c20de16ce4806627895b8ec6 |
File details
Details for the file fastavro-1.2.0-cp37-cp37m-manylinux2014_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp37-cp37m-manylinux2014_x86_64.whl
- Upload date:
- Size: 2.0 MB
- Tags: CPython 3.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4b207b11cee694f30875eb9bc05271ef165e2a4a509b1f116b3b10dcbef1de6b |
|
MD5 | e1c2625e81e66f46942572b49c235fde |
|
BLAKE2b-256 | 52e0d85b4f1aa06e2153e1af1b3f8ca75e88d8fee8020ddda09fcd4b0234d2a1 |
File details
Details for the file fastavro-1.2.0-cp37-cp37m-manylinux2014_aarch64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp37-cp37m-manylinux2014_aarch64.whl
- Upload date:
- Size: 1.4 MB
- Tags: CPython 3.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a4073fc831fbe0ea8ecb6c5d12cd5c7c947aff609c35a3020f99204cfed10d4 |
|
MD5 | e23532f422c92ad17ea9440287820d2d |
|
BLAKE2b-256 | db72fda7a73b52c0555141e226644a8e4f33cdad39072dccece3f2498e1852e5 |
File details
Details for the file fastavro-1.2.0-cp37-cp37m-macosx_10_14_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp37-cp37m-macosx_10_14_x86_64.whl
- Upload date:
- Size: 418.1 kB
- Tags: CPython 3.7m, macOS 10.14+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 315e1d6da9be7512246977d1e89b77dedd72ab914c6f4f6af5409d24a16eae51 |
|
MD5 | cbc69f24f566a6ae0ade1736d1f12ff3 |
|
BLAKE2b-256 | aa5c7e79c3a0e549e8ba0a451fefbd708fedb93bb1180a70500959ac6b475302 |
File details
Details for the file fastavro-1.2.0-cp36-cp36m-win_amd64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp36-cp36m-win_amd64.whl
- Upload date:
- Size: 357.1 kB
- Tags: CPython 3.6m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 213038249aa6ee83a2ae6da22a55008cc08bf15f8089f86da38d375a4092bec0 |
|
MD5 | 5a14f0e2b05442a060ecca6a4923e76d |
|
BLAKE2b-256 | 9a15cb6bcc319ea31d28515935ca9dddfb493e77f7f7620aa080f1901f760365 |
File details
Details for the file fastavro-1.2.0-cp36-cp36m-manylinux2014_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp36-cp36m-manylinux2014_x86_64.whl
- Upload date:
- Size: 2.0 MB
- Tags: CPython 3.6m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f5b13e9eb0ba6a1351e3d218763c2514e53a95135eb20e3e1d8eb4e6160a5a1 |
|
MD5 | 90b6fc63d7ec7944eac90f481caa7f3a |
|
BLAKE2b-256 | 85a9473ef678c8862d74c63e11d14afbdbeabe67f92fedd82405de5337d7e6de |
File details
Details for the file fastavro-1.2.0-cp36-cp36m-manylinux2014_aarch64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp36-cp36m-manylinux2014_aarch64.whl
- Upload date:
- Size: 1.4 MB
- Tags: CPython 3.6m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b7e9872e8ff37d9ee274ba3a3bfc672fcbfa65f231071a7d53df6fe42a72f69 |
|
MD5 | 64e3ea5f7124254ce15d0700358b66d4 |
|
BLAKE2b-256 | f3f2e280933df3422268f2c857b4a41ba6ebfae229371b209dd1081694af98e5 |
File details
Details for the file fastavro-1.2.0-cp36-cp36m-macosx_10_14_x86_64.whl
.
File metadata
- Download URL: fastavro-1.2.0-cp36-cp36m-macosx_10_14_x86_64.whl
- Upload date:
- Size: 433.8 kB
- Tags: CPython 3.6m, macOS 10.14+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fac9ecf8babd90f9d1da080180fdde6b4c1f1c63ed03adb17aa1b0caa09c2bdc |
|
MD5 | aee490aeb362451872205abc30020082 |
|
BLAKE2b-256 | da674c79514fe73d1b08b8b349e63e066121b62c889a0af3186e97bd96c5ae31 |