Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

This version

1.8.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.8.3.tar.gz (799.3 kB view details)

Uploaded Source

Built Distributions

fastavro-1.8.3-cp311-cp311-win_amd64.whl (413.1 kB view details)

Uploaded CPython 3.11 Windows x86-64

fastavro-1.8.3-cp311-cp311-musllinux_1_1_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

fastavro-1.8.3-cp311-cp311-musllinux_1_1_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.1+ ARM64

fastavro-1.8.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

fastavro-1.8.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.9 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

fastavro-1.8.3-cp311-cp311-macosx_10_9_universal2.whl (931.0 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.8.3-cp310-cp310-win_amd64.whl (415.6 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.8.3-cp310-cp310-musllinux_1_1_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

fastavro-1.8.3-cp310-cp310-musllinux_1_1_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.1+ ARM64

fastavro-1.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.8.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.8.3-cp310-cp310-macosx_11_0_x86_64.whl (526.7 kB view details)

Uploaded CPython 3.10 macOS 11.0+ x86-64

fastavro-1.8.3-cp39-cp39-win_amd64.whl (463.8 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.8.3-cp39-cp39-musllinux_1_1_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

fastavro-1.8.3-cp39-cp39-musllinux_1_1_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 musllinux: musl 1.1+ ARM64

fastavro-1.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.8.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.8.3-cp39-cp39-macosx_11_0_x86_64.whl (539.0 kB view details)

Uploaded CPython 3.9 macOS 11.0+ x86-64

fastavro-1.8.3-cp38-cp38-win_amd64.whl (463.9 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.8.3-cp38-cp38-musllinux_1_1_x86_64.whl (3.0 MB view details)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

fastavro-1.8.3-cp38-cp38-musllinux_1_1_aarch64.whl (2.9 MB view details)

Uploaded CPython 3.8 musllinux: musl 1.1+ ARM64

fastavro-1.8.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.8.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.8.3-cp38-cp38-macosx_11_0_x86_64.whl (533.4 kB view details)

Uploaded CPython 3.8 macOS 11.0+ x86-64

File details

Details for the file fastavro-1.8.3.tar.gz.

File metadata

  • Download URL: fastavro-1.8.3.tar.gz
  • Upload date:
  • Size: 799.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.3.tar.gz
Algorithm Hash digest
SHA256 a6c2ec69516e908fce64d93a13e6e83afb880f2edb5ad3adaa1eb04c918de6d8
MD5 146270e2d155cd5b6611cfece1cc1e1e
BLAKE2b-256 6e52107038ee38bbd5cb2700b23dc72595c7a7d94677e2edf382f2b41e1c5aad

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.3-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 413.1 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.3-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 65e59420ce7a8cbb256363b9bc2b98fcd0c220723ec50541aa0aaf137dfa21fb
MD5 519622c5d5751959c8cf28f9c9d3873f
BLAKE2b-256 136017b670e82c6b57798c125a095a9eed660e0c9d20dcc73f8e922054d94f07

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp311-cp311-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 fa282c78485be06df1e25f0c9b6837de520a22838e7c9af95b58fc68c6c9ce34
MD5 cb35d9f36f4cef1fcb722f370896f18a
BLAKE2b-256 f4c635bcf205f7daec196bdd718c55f3b0cfa673591e584f2dbcc5ad2054bc54

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp311-cp311-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp311-cp311-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 dfc73765b294ef56f71c1bb064ee81efa1da13bb0b1134dd53674bbb89477c78
MD5 facdb94b0d4564c52b41af0b9342226f
BLAKE2b-256 48f0ec5b4b22b99174fb7d6ccef2992063745891396c8eeab2d18ec4a1f2f3d5

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 eb2702976cf9bf4e1c66bae3534f498a93272eaa4cf2ba24fe18aa29c5fab647
MD5 473df1b3cbca2963bb89d4b48bf3490a
BLAKE2b-256 7c515e3b6042b5db576bdad310d6b0ae6ff581fc60f9df6108e22155fe352a80

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2bf6f1debda2bf84f57bdeee289e38e1ea8b23722792b7bdec8be6b3bf4dac67
MD5 65827e94dfd9d1d6ea4e86ba48126381
BLAKE2b-256 b6d279473a6c8579acfd56ba0438961f274446849c663b43fcdabab713c957cf

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 4e5d8ad6dcf7cc4e15fc5f30344e4fcb1bac5c0c1b48ae88a46ceef470c04b0c
MD5 18be08a1318b279c2a40178727d5275e
BLAKE2b-256 2bd64f63ee4d5fefd87b18ac2a308bcbfa39496001e938dad4782bdeea8f0036

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.3-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 415.6 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.3-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 c38ba23be298b1df63eaadf0663e8c1dc3fe355608ba3ce769554f61cc20f2d8
MD5 9440a1d7eac040cdc667f446a654c317
BLAKE2b-256 e7d1c335c417e59ed544a56d25a173b912da78ed7022a38497a09ab0945f0fb3

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp310-cp310-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 b30bbe06289310ff60c32c0ab01c394354f5bcae421842f06915ee7e401232ee
MD5 ebb6097a3c9bdfa8efc17c40b3e1a251
BLAKE2b-256 f76653013c051efdfc28e8e5e4c47f98161c5d04ecc53d153ea74335b1a1aa58

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp310-cp310-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp310-cp310-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 4bf02deeea731910d55e24f3b44a848007b600ddd0b8861dab9075aa116b0da1
MD5 00412b13dd3edca5fd2bcd1a2c8d4970
BLAKE2b-256 a250a50de6c6734a0b1d771b7a309408f2d841f99acdf62ff7fc0376c5d9825c

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d107cde443d92be8b65007ef304a602702853925c4b9ce63b66b8cdf04938af0
MD5 e32c24fcc746caf7edfb736720331596
BLAKE2b-256 8dcd57c187a1a4104447bb3f3a23c42e6794db5c9f00ed5148032601ee27a00a

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 60bfc8b6d3a3e27ae68ce952f9c7a63001dd82f96a519ff25d105a2b61b4bae9
MD5 5af6100f8c6c17682da3d7101c8f1a3c
BLAKE2b-256 46ca0dd537cc7c260b655511cd9728614fb2d7cf77323d71181a85156e11dd30

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp310-cp310-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 0e7d98e27cfff61befa23b11422e72a9516fe571d87fd41a656074a958d1f5df
MD5 ec95df910547c00323b009ed0a528f8a
BLAKE2b-256 ffdb12aecbb11d7cd000e4d54ca8ab3f60457027c86c24273e59422c8d488e67

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.3-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 463.8 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.3-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 861efe9ad25fc26c3d360761d48930e6aad0cabe5ae888f92a721699bfe612ed
MD5 a97104ea5cb4a2b63a4cb0991d3bef39
BLAKE2b-256 ce7821d99797ea4b330c6511caf3e8d7ed67b4fdc08886aab52ec57e653a1e0e

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp39-cp39-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 7d2c093be35bcf77547d5cdeeefae2c18b88ae529fa3866da81f5c7c342fceb3
MD5 62a19fe51d2a2900b0b41f6c6a865313
BLAKE2b-256 e26d1fade505a3ba315b8942a28252da719c45188aeaaf1769e6e50ed114872f

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp39-cp39-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp39-cp39-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 c0c194340ad5c6a5a5347ba0170d1413c149cd87faddcc519d9fcdedadaa1619
MD5 352ae410950f9b2118d517012a6b3a44
BLAKE2b-256 faad34ddca0ac6050bcff08c8f600de36aa888b19bf955dd7bb10267875a99bb

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6035368cbcbbb1063c2d1ce763ed5a602f4b6af13b325e77a6b61e45f8172067
MD5 7897f3f6d6d8dbcc7d644956d842f811
BLAKE2b-256 663ee9e43ac3f18d986012e30ce571f7f4bf7effb4af9557de511609180efec2

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 0116bf82a10eb3553d61d6d884f18c8b21049463fdefaaea9275d8bad64a0f5b
MD5 38749572818f45cd8efd88e39c519de8
BLAKE2b-256 8866b5398d6f1e4804fb7de1c9239988e46f54af9057e174751f83ca8a039681

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp39-cp39-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 821b1de9785e266142a8658e25df52bceaa40936c087925688a4fad4dee0beb0
MD5 436e5d088507e86cac1d0c05d492cb9f
BLAKE2b-256 e0e78a2489a2eb97e2a51c579ca8d3d3458e5f23a30e7df1c729316acdf1d1b4

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.8.3-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 463.9 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.0

File hashes

Hashes for fastavro-1.8.3-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 7ca4e19db2ded435dd393f58f65297102e7329ca8ba31d03be9c480b34be9123
MD5 d35447caa45aea7907bbeeeffb1e03c3
BLAKE2b-256 e8a90fdd464d77072eafc9de34793082db0ea4a2a2aa334a04925781c92048be

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp38-cp38-musllinux_1_1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm Hash digest
SHA256 b9164d7cb1541d15587c3a446f17719bc1f20008a1df1583e55d8b5a323266b5
MD5 61ec2ce433067e4876fe4c8aad13103f
BLAKE2b-256 eee66186a5e5522efefd53d1b59605ff6e8d19b0a15e55e43898f656089454fa

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp38-cp38-musllinux_1_1_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp38-cp38-musllinux_1_1_aarch64.whl
Algorithm Hash digest
SHA256 b74fbcf860084576bd773169a18ddd140d06f9e85bb622756f557023947f179f
MD5 571a0cbb1e758a2ccbde1fbd1cfa7f50
BLAKE2b-256 dc6eae8dcba039bec5ffed95e8455c94dcfa3959b4fd950d54385a0bb72d1ba2

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3ce57e57ec56a235ab012fde3ce7eaa3846980a9026448fcb32cb065f2460514
MD5 2449574bac8712652fd1bf5e9d266d3a
BLAKE2b-256 a92aaf06dcb43345f87b7265211ab95f694a37015c4ae13464778dca5fd017af

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 d3f00ead48738e882832cc7ad87060365eb3eeace196ff9a5905a4caf0bab351
MD5 c88c5f9c740f4a11d98600713b941388
BLAKE2b-256 0997234d17c5547e218dc92eb09ee1a01bbdfb63ea36e9f8da503c9cc7b2b7f0

See more details on using hashes here.

File details

Details for the file fastavro-1.8.3-cp38-cp38-macosx_11_0_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.8.3-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm Hash digest
SHA256 4416fdf69c82364d737e77c2a6ab06eeb20375d84813c061789e20bc047132a5
MD5 1a1310712469c3a46223135e886d3c48
BLAKE2b-256 65b4e1fac2a9671c9529397668829f69ac025ce515e5adc9d5f5d86faecd4b71

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page