Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.7.tar.gz (728.2 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.7-cp310-cp310-win_amd64.whl (416.8 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.4.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.4.7-cp310-cp310-macosx_10_14_x86_64.whl (492.5 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.4.7-cp39-cp39-win_amd64.whl (415.8 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.7-cp39-cp39-macosx_10_14_x86_64.whl (492.3 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.7-cp38-cp38-win_amd64.whl (414.9 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.4.7-cp38-cp38-macosx_10_14_x86_64.whl (485.7 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.7-cp37-cp37m-win_amd64.whl (399.6 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.7-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.7-cp37-cp37m-macosx_10_14_x86_64.whl (477.4 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.4.7-cp36-cp36m-win_amd64.whl (398.8 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.4.7-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

fastavro-1.4.7-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

fastavro-1.4.7-cp36-cp36m-macosx_10_14_x86_64.whl (474.6 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.7.tar.gz.

File metadata

  • Download URL: fastavro-1.4.7.tar.gz
  • Upload date:
  • Size: 728.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7.tar.gz
Algorithm Hash digest
SHA256 a3835f2dba148f838a5412ad64cfcf47d3d8330cbd913a51f269c04984361f85
MD5 bcf9846c11351197f490d9d07bfe8946
BLAKE2b-256 28a0b8f0ba7a7f87326b26d4d3a8a7ca455bb243d309094855d8a242d6379c98

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 416.8 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 282b121831f50f343674cb4914875466cc5bbbb69b6d606422fbf7e3608693b9
MD5 3c11f71a0d7c9480a4f6877510728426
BLAKE2b-256 1e7c28605e17ebd6cdde38fb753bd6f36efa5dd889e6586e6f3b2076c5781717

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 59e8c9161f080602ac97c2964bde18a43435e3e4e6a84f83e0aac64b8e6ffa90
MD5 3c1f2ac3ec95d91ee5a0650763c127bc
BLAKE2b-256 4010741e8dfd3ec0cb36092cda9cad88bc0b16c15a17f8b230ed9d60c89fcb18

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 492.5 kB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 4af0fa6af388e984b0c70ebe57b486b7685ac0e9a0b9c58615fd3bf76c69e443
MD5 141291608022160f90a74fd7b820fdc3
BLAKE2b-256 29c658488f5b3202846e1705ee89a5e6a9d47a884f8c1036aaed0ad12405ddd9

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 415.8 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 fb2840e3974cec2b0f493ff7d85b88829e4d5dadf692b7a1c3f38ae7888ee0bb
MD5 dc1b98f4fb63749dd41529c5ff005093
BLAKE2b-256 513549bdfc68f58319d5ecd46cc0c7684bd20f9570e006b55dd268bcd9b80ae3

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a111a384a786b7f1fd6a8a8307da07ccf4d4c425084e2d61bae33ecfb60de405
MD5 9e5529a85b326c886ed67a1655aa2686
BLAKE2b-256 fd2fe50194eebb4e2fed2e88152fbae27263e52c5d4712131cd0caf8637b87e0

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1565df0863ee9ec8c67c2f3d91ded599d74f0ba7b29c727610bc13ca21da3021
MD5 fcd3050a0ee95f4965e529935cc0aa89
BLAKE2b-256 e8f191aea7f8c886cc94d06b53e777a12ae3198dbaa0abb843a1891bd50aae1b

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 492.3 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 7b3189eae69517d76d99e00c739617eb024fefb34dba6963dc8f2acbbce0bc1b
MD5 f1a0a3589937fe3c18f795f761a36f9b
BLAKE2b-256 6d4e84ef0d3d33c175f9d31f0f5283a62894add183adba49eee344a39919b2c8

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 414.9 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 dc40f13ddda42c087d2486807d27a9942d82c1830dd0ea1e2123c8354e4de74f
MD5 d9a6c9c45a97a7baa46aa7c38e6480d3
BLAKE2b-256 bf33bd51c3fac29a23c3008f87c3d187dd73ae1c98d1544a1e80c6cd07210171

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 764c2ba8f30c64e22646b747f6715ca2849ac9eef7af93b9c0561beed69d9dff
MD5 bf2475dd87026085f80cebb6ef364ccf
BLAKE2b-256 c073e147c53eb1ffda234b2ad8c42eed5558a4bb8d97c3859e8b53b235876f1f

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 96e7a253e73417e68e20106a36de28b3ceb8c22a069aa6d1f6323efdf1726816
MD5 70fff46dd645ffacf7b92740d92c975f
BLAKE2b-256 2998d02f3b5e99f4f673dfaa2decf2411d85cca455f1a29e3cca36b41e0c15ee

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 485.7 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0b9d6a4246a79ed215dadaeb17419275624a0b06323dca6b7e6e6d6b3fe2ed55
MD5 a62d89b54d4617bc3a364f151ecf92a3
BLAKE2b-256 38e082ffd90e09f60a168ea4ca105bf86a2224f5fc13221d4c9b5e4c2d13ce86

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 399.6 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 943457d32ba7d5e4fb9d3b965305159d50287c722bac754afbf0a789da7e7570
MD5 fa2ab942cdbb404a70146b58a6a8efe4
BLAKE2b-256 f4acdbedaedd9a2ef9519851008a9152e2cadfcdb50965044ecce922a8bf7915

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9f85e19477a25369e49a6b7c0d8f2c7a5b968acb6d278890e199c95221a1c770
MD5 64e7ef4ce1a2245f55b5b148c5ae3480
BLAKE2b-256 af2c41abe997e861f511f4c8c9e015430ee0263fad85612c3122774a9a8acaf0

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 39ba24fbec95d2b09154d598d57ca2f929e5c08b856024b886db830c867af299
MD5 5089309a15b3f1f334967a04226f6f37
BLAKE2b-256 6275614f4457884bd620d4378639e730e1f327dcc946366d42ac533597d507c4

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 477.4 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 d29a0736b5cc013c5b5c9f5cf4fca27f5fc4649a52ac5e2eca2589b27f4b7a64
MD5 e496207967bd0307df091346372840ca
BLAKE2b-256 b6071deaf1985a4531c443ea135a960ea9b8bcc45dc589fc7e4f0fa3f17471b3

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 398.8 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 5a41d1340bb45cf70b1bab9f4d5f99e6f3c022fa29f7a02764c0f13c9620dc5b
MD5 b6d6b116e2b0989ef9dabef59ba165d0
BLAKE2b-256 472deebb8162e22a8b4dfeaf251eeb158a1b6e78d03a5c6a184189668e1b1c9e

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7448b385d13ee7b90eff4bc86e8470e7837cebd0b3a1f8d530915427972f7197
MD5 f7a9d52746381693d0ae39de6efae996
BLAKE2b-256 e09d830b27112b16b4a513e2a32f959674a834797a76fc81386b4648d5d1b961

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.7-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 24cbb94230e6c855428ad182ef3a3a93f0019bce1f93b4a867f9783086ce0876
MD5 cfaeef91c20590d9df145a457e7de08a
BLAKE2b-256 bffe5606d94c4f887e546e712c9980146c8c1c9677546b6a009cd6cb7e3cd479

See more details on using hashes here.

File details

Details for the file fastavro-1.4.7-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.7-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 474.6 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.7-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 b713d680161cade96304eae68361e8235ad92c4405cd3fc45f16ebe07755667f
MD5 a2fca450e51bab603f1b6f8561e7b14c
BLAKE2b-256 eebf7c0a79f41bf0aee7dd302a660ec8f037eace4b62577f9285c0456c89e932

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page