Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.1.1.tar.gz (656.4 kB view details)

Uploaded Source

Built Distributions

fastavro-1.1.1-cp39-cp39-win_amd64.whl (369.1 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.1.1-cp39-cp39-manylinux2014_x86_64.whl (1.5 MB view details)

Uploaded CPython 3.9

fastavro-1.1.1-cp39-cp39-macosx_10_14_x86_64.whl (425.1 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.1.1-cp38-cp38-win_amd64.whl (368.7 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.1.1-cp38-cp38-manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.8

fastavro-1.1.1-cp38-cp38-manylinux2014_aarch64.whl (1.7 MB view details)

Uploaded CPython 3.8

fastavro-1.1.1-cp38-cp38-macosx_10_14_x86_64.whl (420.0 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.1.1-cp37-cp37m-win_amd64.whl (354.6 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.1.1-cp37-cp37m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.7m

fastavro-1.1.1-cp37-cp37m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.7m

fastavro-1.1.1-cp37-cp37m-macosx_10_14_x86_64.whl (412.1 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.1.1-cp36-cp36m-win_amd64.whl (354.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.1.1-cp36-cp36m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.6m

fastavro-1.1.1-cp36-cp36m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.6m

fastavro-1.1.1-cp36-cp36m-macosx_10_14_x86_64.whl (428.2 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.1.1.tar.gz.

File metadata

  • Download URL: fastavro-1.1.1.tar.gz
  • Upload date:
  • Size: 656.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1.tar.gz
Algorithm Hash digest
SHA256 17e745d0ebd26ed4372b382ae159fa678f7a216cd1589e95515408e630840338
MD5 74b9d480390db56a7cf4f5cf27467b6e
BLAKE2b-256 43f938034b768f2c08246ed3450cad247fdc3e017af02e8f638ea16447839ede

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 369.1 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 6e4449ff8265c0de18c7bcffe0f62d47a247dbdd05e415e90ab855cc889c0a03
MD5 92a87bd98c81b1a19517c50d9114cbbb
BLAKE2b-256 5a4df6fef9e959df91021947aba65c6b2cad9450f14ae7e164c3f67342511dad

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.5 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 aacb3fa0df2465d7b560497ef4647dd298821b05ac4754bc58d8b8c090c9a43c
MD5 03b91e651524efdb26769f41d22c84bc
BLAKE2b-256 74167b9a3ff1457f58140ecd719fe3673ee5feebb8b2f7177e1328c0a8923a78

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 425.1 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 4eea2b12c9e8b7b0498c87f4e65fe971a9d8c76a8d4daa11c5d403c6c78dc2a9
MD5 8d52ac820c224a16b5c8f7049545e8ec
BLAKE2b-256 242f41b68139c14be066521dc64d668310967fd0fbd264801c514115fda1fea9

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 368.7 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 79d3f1a5a194d3727002660726a0aeff833418ae41cf546bd211944182329a70
MD5 bb1efec65739243b99b0a0268d5caaa4
BLAKE2b-256 bccfba618d86a1291dbb9af3ed5367b79574fe50119961ada079bc6056d10832

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2e84633d3ac32677d3f2ebb64b4b00b4dfb0674aae28ae85f6959fcf92ae9d2f
MD5 d8ed6656a6fefd9de57321aedf2ca27f
BLAKE2b-256 8074d32484afcafd0cd05935704c636c1625df97a9812c68358443855ca28447

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 86e7dee32e305e0b58cb2d650f1370dd5fd3c42740788928030902b71958b01d
MD5 65700d5a63e2443695532d1f666f4dc4
BLAKE2b-256 8851b2ee3c2148de782533ee1d92a491c0ac61164351bc14717bee2badacb565

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 420.0 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 3d8f041883cda9baf309bcbfb3a399fb7ed89b98964ade20bc5f3a23406158a4
MD5 214826d6b8c1de27dd4e368f8d9d79a4
BLAKE2b-256 c906cc04e12cd3f9166cba4aa746cbfa58ca9d6e1c16fe054d0711758f9fd3ba

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 354.6 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 d630516ded357bbad5a61289068f7208b3a7cb6f2ff90ea592bbc4c58be93862
MD5 baef25939a0ab8effa1c4339737c7ee6
BLAKE2b-256 4c3f48d6ecbc3e7db956a4dce9763b8de376b64d30e66a4edf67b1cb79446a47

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 aefaf3e09fd2d4d390de78a13556050af18aa92470c7820bda8e04233c59c745
MD5 d2299f2321327b0dbb00637e519de04e
BLAKE2b-256 1b98b37210aeb4cd4dda3efd2cbb9840c515e63db7bc9a80c09fcedfbfcd9461

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp37-cp37m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp37-cp37m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 920d7846d925d0d445f29329e64244ba0ef865e26450c24c810fc17ff659fb36
MD5 20c6ca883b36aa41847b67b09c0fa100
BLAKE2b-256 f83888c9e6755029dc94b4da341f056c719e9b5ab0f7cfd6cd10ced7179f0922

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 412.1 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 de59a607ed7c096e76c93627537a4c5572a974170f37ebdc29a5005f3f3dae9b
MD5 d48783479408039c692bb2776667e480
BLAKE2b-256 8470d8a51efe35de6ecb981efd98ef7f75111acecbb59ee134828031b9319087

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 354.0 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 46e193513b21ff5e22411d95410ec78f8051caa598048be3bfc28f8a4ebfc812
MD5 6f7f4529bc2b12cc5ce4db6c0454e3b1
BLAKE2b-256 61c313865bfe8ba5c4c1699cbfdea9f0060e093bfeacd67d7ecb0057c5053b46

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9313abaf8e47abfc6af0d91b0dfee385e7cdefbe163705181019fa3dfc42ae94
MD5 950deb708aa093d2ba9b03bc34aed77f
BLAKE2b-256 0dd52e0f7455e9a6aeb35913dbe322957ee24346948728052c7850eee8e8fb51

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1f5232de18d021ea15b533d0dca33babb8c76b365a61d9796ec5cb6e71e8836f
MD5 f7492e8ed3de7bc010e65abb381b59ea
BLAKE2b-256 65e30ead42f6b13ad1b6c85144c3e944ea6419f5e579fbb0fbf3d139c5515efa

See more details on using hashes here.

File details

Details for the file fastavro-1.1.1-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.1.1-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 428.2 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.1.1-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 6e1f13780922d47335f22f3960c7221138c8a48d27294ff75dcd9cb15048a116
MD5 dfa749ff447560dd1897ec9044684bff
BLAKE2b-256 52524de830da38f11aa9fb8cf9377e63b2c5fd4d3b7e21ae4d821085127da9b8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page