Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.2.tar.gz (725.4 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.2-cp39-cp39-win_amd64.whl (414.0 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.2-cp39-cp39-macosx_10_14_x86_64.whl (489.4 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.2-cp38-cp38-win_amd64.whl (413.0 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.2-cp38-cp38-macosx_10_14_x86_64.whl (483.3 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.2-cp37-cp37m-win_amd64.whl (397.8 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.2-cp37-cp37m-macosx_10_14_x86_64.whl (475.2 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.4.2-cp36-cp36m-win_amd64.whl (397.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.4.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

fastavro-1.4.2-cp36-cp36m-macosx_10_14_x86_64.whl (472.4 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.2.tar.gz.

File metadata

  • Download URL: fastavro-1.4.2.tar.gz
  • Upload date:
  • Size: 725.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2.tar.gz
Algorithm Hash digest
SHA256 41a15918344723748ca2f3f3887d72d22aaeed4578420d054170aaf9f9c0ec9f
MD5 aecd2c01f9ced8dd29afcf627ecd7b09
BLAKE2b-256 08d584de2589021fee4fe69b6a9b51368771a47787949f08a9b704bb33df0686

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 414.0 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 ce316ef3ce250e1b2e36cd38ca66dcf2a3f8da8a47fb15e9af674514fa23d8a7
MD5 2ea92a8b7d186e55f4e1da17c217461a
BLAKE2b-256 50f130089a73091f5090b3557dc9eff5f06bd545b34363b8894bcb20ccb55876

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 070cb4050df5a71f4892ce58826bb5773c4dd276209a0213c48d06b771355c6c
MD5 f00077462198837add09a8e8a4741438
BLAKE2b-256 fa622bfe16608b7d31559a544ba0c0d64112df8cfa92b0a5e0f2433343eea1fa

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 489.4 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 70dc160d0390a9a4a035d25525d884e0c2ddee5d0a1e0cb278f4357bed65c832
MD5 75588c6698f5c2abc3936436ba3cad56
BLAKE2b-256 a2cf3f59994fcdf3dafa103e857aae1ead2b498955a5d9b34327e1ae7ba95088

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 413.0 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 cdcae144dfd3e9edfcd514ccff50648a826a3dcccb1427e19bcaf103a84024ea
MD5 72eb78e6142d564753a924691e56921b
BLAKE2b-256 72916b206a52947071d2bd9d92d6e821370918e4d130b57facd5f89ff6faea58

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7e5e5c0325c773721d40f069736bf7de018c89e8ce295df7155b4f401b78dcc0
MD5 d3de67d51f0c0533b499c32a11c3dfd6
BLAKE2b-256 0c58527127c2203d89a251e57b0f50eaf804e1334006908ddd30a6321ce2b1e0

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 483.3 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 bb73d874b9db7056473d4fcd1bd91c90ca8f3e828c13dfc3ca62771f86874b5b
MD5 d587815af31bda76cd91057053415b29
BLAKE2b-256 2389f40a57a195a8aa3083678c4b6ba455d8ca1601cb8dc2b4ed0f83315b6125

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 397.8 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 b3644e058d0229aece00ca84ba7c59a7992a3828bd64abfd2ae90e1b1387dba5
MD5 b468394dd188fc29eb5054f4fea57164
BLAKE2b-256 ca2be50d6e151e38fd18d04c10aa4953d53ffe6009581dc0118eb02c085f756a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f6bff1ecf1b0129d77ebcf76cebe4d16ae56a85f1ff1ef7040989b3500576805
MD5 a0a5b05ae78ffd3e78ffdb57a09daf21
BLAKE2b-256 279210ee74edb0a39f4a7af1cf271b3ac725c54f5c243c26fa5059cd794d15d7

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 475.2 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 982f01dc8438e75df7ee5e1b2f9dd430b8eefc6c14fd3344d95351d0cf32e99c
MD5 543080496cbda7317d1ca6ced332e158
BLAKE2b-256 4dcbc5cf794190a609cc43725d13fe036e6a7ea055347b679875454df5ea7cf7

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 397.0 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 8d37a0b83db0c0e00874cf78f927ddfadef12d54e7f42877d33c92431812f850
MD5 4d64842be28289050ddcae6e0ef04a89
BLAKE2b-256 9c689ac757c2a02da5602a73a037b7647cdd5c42204430d6cfe0a3aefe056497

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 10caace4785a7917ef9eb5d4721b9e8a060e208973225b6c379f9e757a981d16
MD5 1ccad35f3bcf84de2a69a234d90f792d
BLAKE2b-256 9f2d90bad728fab9d7c9475c05958ecbed504ecf0b9a474702c3d1936128ccda

See more details on using hashes here.

File details

Details for the file fastavro-1.4.2-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.2-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 472.4 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.2-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 6d9bde5d374da2f4f4ce67acc7d5a2e22a7d50f73a77ad60408819fa2b178462
MD5 363ba6a4b573e6277daf6221575627e1
BLAKE2b-256 6c8d6347e2a30069b9701627ae1515b1839d6e1204d056f76da66adaf2b9affc

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page