Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

This version

1.4.9

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.9.tar.gz (732.4 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.9-cp310-cp310-win_amd64.whl (416.4 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.4.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.4.9-cp310-cp310-macosx_10_14_x86_64.whl (493.1 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.4.9-cp39-cp39-win_amd64.whl (415.4 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.9-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.9-cp39-cp39-macosx_10_14_x86_64.whl (492.6 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.9-cp38-cp38-win_amd64.whl (414.6 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.9-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.4.9-cp38-cp38-macosx_10_14_x86_64.whl (486.4 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.9-cp37-cp37m-win_amd64.whl (399.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.9-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.9-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.9-cp37-cp37m-macosx_10_14_x86_64.whl (477.7 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.9.tar.gz.

File metadata

  • Download URL: fastavro-1.4.9.tar.gz
  • Upload date:
  • Size: 732.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9.tar.gz
Algorithm Hash digest
SHA256 be3fec387eb2cdc9627060b5ae0690542c687dddc951b63fa11203553769ae5e
MD5 0db2c1518c8f0d94f004e497b0337043
BLAKE2b-256 9fd40a04211257324a27ef39e0309989f10d05227be63b601c7789a156b23623

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 416.4 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 927fd6148a8dd9646c129c0a0e8571aea829abc3cba04a3d5a4010a866934f4c
MD5 7724b86d1f8e38c2c6921da9bdb354b3
BLAKE2b-256 1f8b9ec309856dde0b40c1cdd479a53462330772bbbfbdf0a125238d4e1ef197

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1fc9c95b7c1d59c5a2d29be21075870a122152cf927d84587dafc96da6b2ac3d
MD5 9106e36299c350230e5795fbde82ff66
BLAKE2b-256 cbda4bd763ad01969400acc2dbb85bf2054689f53c66c492d64b28ec8ccf864e

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 493.1 kB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 2e64a77c529b638e89a879ff0211debfab5b2d114c26a2af29c81f6b013f395a
MD5 7c1e07aacd97c9669320640e00045e54
BLAKE2b-256 b4986f3da1e282612cc6b8f508269ef734d99c0faf21bd48a9bd67662ac32605

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 415.4 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 f9b04acaf06b16218b47985e92d8daa98c1116d58f3cff81a5b3cf39cef9afc0
MD5 3ab6b2a35ae5fdf7c8a84926cb23f37e
BLAKE2b-256 b5d1cf940cad8aaea9f0e10d1f4f4ae6a691040955224178adaeef27eaa93be3

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 32b804aa6920d80c0e94e1180d480f28f56c4b901849bd80ed180851752b5ce6
MD5 9eca205144f527d68df530c1df39b546
BLAKE2b-256 59f618ac1e81685ac63e557623459ef0b2475501fbab6cc027693c4e6778e964

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.9-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b98ef2bdb123b95945aa6d69d6a7d79f211df3274b2dd7786da7852ddec964d0
MD5 57f269468eb8ea3c1e2912e1f82fa723
BLAKE2b-256 cff79e86623a8706bd832f9d06794209dc8405ee0ae1de2ccef076a3ab80905a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 492.6 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 3759bdc77009ee1e2e76fc9f58b951c05c00a8600ef9ddbff59fee3cb0c9e235
MD5 df1602713bf51dc516f5fad319e9b50f
BLAKE2b-256 9cc274e57265a13d23b1197afe243bdacb1b21435b5bcf459e0889f8ba2cca62

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 414.6 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 fa9d8b47e0533c84152332ad491bb63bbae76a8a7a0df1caa821e0cbebf0fb70
MD5 f119fa6216fb7b7054a770e3cd65c12f
BLAKE2b-256 d48432c94456db041546579a504c20b1a5e165c2bd4607dc8902f48d2e21aec8

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6a9cd6d8471beb4020b4126fd04150ed7295f74ae7234d0dc9205b55c193851e
MD5 4e9b490eb0971085aab9d4b4bf2dae97
BLAKE2b-256 c04cdb5c5f639ea2cd826cc296fb32db29df5b73ac42243ded49155f8ca147da

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.9-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c7537e4df7782b03b9761e9338cef9fc7bfcc41100ab93c36c5c60fa568e724a
MD5 edfe1c881633d3e3c7714cf7514e9cb8
BLAKE2b-256 f7c0454de3a3e163c6419730fd658339bc0f4aef727b02888912149a9a18f032

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 486.4 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 9a6ada2d6e133a2319438248c2e023b6735747b249c5a79d5f08f9d431e5d226
MD5 b29f1dc590974db251f264b0f2e76f37
BLAKE2b-256 036a189deed9066012e8286050ac83a6fc863f60601b168619d5d213c0cf3d07

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 399.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 d6ccb77604903a0308316e696bb65a8943361af5f757d10985689656c9bce6ed
MD5 2b54ac18b5ef640a2cb8f99c5922434b
BLAKE2b-256 e8ac962dacc2f2ccc6a57173fc724d7ad4e61d4ef157b601c8c6f3c235e60a1c

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.9-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ce2c7747ce958115388872db0756d3eeb0d796084eea9b46dc3758ef32c4d952
MD5 d95cdf936fff30840da2fbce66582d2d
BLAKE2b-256 5498bca18a1b3d37a7174d5813ab7cdd1e41cb83a54fcb886b5a522a3d7029d3

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.9-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 37a77a1b5347a06416e236c77027c750aaeda29ef8189aa456eb2a2571274b43
MD5 37e6586c80f575070dc8ad536b307894
BLAKE2b-256 f4739ee21485f76f889ae061e794dd5d08c8c9f4a550fed98d33f2aeb99d7b6d

See more details on using hashes here.

File details

Details for the file fastavro-1.4.9-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.9-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 477.7 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.9-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 000b70c5109a61bdbfddeb2821a506de8f5333f243c608cbced61d44657d6c2f
MD5 5b529ad0e15c5f86173acf365dd01b01
BLAKE2b-256 11fdf07af5be6771fcc6f266cda9ddf18c42c5f2ac2ac3cf6e2d234257684a7b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page