Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

This version

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.10.tar.gz (733.8 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.10-cp310-cp310-win_amd64.whl (417.9 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.4.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.4.10-cp310-cp310-macosx_10_14_x86_64.whl (494.6 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.4.10-cp39-cp39-win_amd64.whl (416.8 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.10-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.10-cp39-cp39-macosx_10_15_x86_64.whl (494.4 kB view details)

Uploaded CPython 3.9 macOS 10.15+ x86-64

fastavro-1.4.10-cp38-cp38-win_amd64.whl (416.8 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.10-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.4.10-cp38-cp38-macosx_10_14_x86_64.whl (488.0 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.10-cp37-cp37m-win_amd64.whl (403.3 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.10-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.10-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.10-cp37-cp37m-macosx_10_14_x86_64.whl (479.3 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.10.tar.gz.

File metadata

  • Download URL: fastavro-1.4.10.tar.gz
  • Upload date:
  • Size: 733.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10.tar.gz
Algorithm Hash digest
SHA256 a24f9dd803c44bfb599476b000f9bd0088f7ac2401e6c20818f38d8af12785a0
MD5 5725d59327372377bfed479979e4f5a7
BLAKE2b-256 328492ec704366b5d5b30ba4ac0f04efd5417fb2c60309f6062d85f725f29577

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 417.9 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 995525bdfbdfef205ea148b5bc6a9fe5ccf921931123c39d9aad75a2b661681e
MD5 c5995321357a9da7e46eebd97bd5034a
BLAKE2b-256 430a8450e76ace4d8eca236e04175ddabccaf4cfe58d02136679f6dbfd1c6847

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b7669302c9124b7cd6c1bdff80c77116b2290c984814077fb7d01d7389586054
MD5 37a76a068875d966111a029ad8ccab21
BLAKE2b-256 173fa107848ee51485618f6f9a99281998f59b34d81659500b9b8610e6c4d662

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 494.6 kB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 f225c81e869b3cefef6f2b478cd452693181ed7e645be3cea4d82024354ecaa0
MD5 d3e334eaa851cc274d8886557759d2de
BLAKE2b-256 ba8f5acc108407e67464d1fef119d52b27541b919c9403e0f487d5ff4e73d2c5

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 416.8 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 64cbd386e408b3bcb2de53b1f847163073eb0d0d0338db65f76051c6ba9a9bc1
MD5 69b9a76a0efdbd4c7b97448f28527f4c
BLAKE2b-256 8737200b64980ca2f4dbd2fb6ec9113d78317a01df3af301a3e3065867b7c6b0

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9660878ca01e3dbbee12385c5902a2b6b12ecbb5af4733d1026175a14b1ef67f
MD5 c9df7db531d643ce857fa2eb790ba62e
BLAKE2b-256 f5be653ef2e13c841931b76854a90d971fa29e5c1b3341c8968619541968d486

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.10-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 35f960dbba04647d8d6d5616f879622d2a1e8a84eb2d2e02a883a22e0803463a
MD5 492b3455f71a9b83ad4a2674abe985b4
BLAKE2b-256 e8184bc8dfb8b17e09e111e9a392f924c926b3271cd47caeb7999573d2dbb609

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp39-cp39-macosx_10_15_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp39-cp39-macosx_10_15_x86_64.whl
  • Upload date:
  • Size: 494.4 kB
  • Tags: CPython 3.9, macOS 10.15+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 0c6695753fa3035bbd0fa5cb21bf1b5dad39483c669b32ca0bb55fb07c1ccc87
MD5 6194daf2f465a80850454ae66f48a034
BLAKE2b-256 0727e1bd4d2d7bf70b47b7b011d4d8cb0e87a528afcf42322a2bdbcbd7f4b67a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 416.8 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 09f1dfdd8192ae09e0f477d1f024d8054fccdb099ad495d2a796bcee3cadebd1
MD5 0e36d355a84b63440e039376d51ad257
BLAKE2b-256 3a490157594f580974eeeedb3bc8bfa5f0fefb28ec4fdb58435985efa62fc68b

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a23763d73412c077aee401a0368c64cbc23859e26711dbae78a3cf0227f65165
MD5 863acccfd4046b83cf96aedacda70428
BLAKE2b-256 f5b6661873f94172661320b65e91c5e44f51211967ecae2d2331bf0e4b475bdf

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.10-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 0934490b0c3bcfd6bcbacbcb5144c6b5e4298cda209fbb17c856adf5405127dd
MD5 c217db88a9317d94d9de28ef7f23e65a
BLAKE2b-256 dd59eb8272a4e34578eb03282e45b075be8a46def47ccce4cb0494d93a3e337e

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 488.0 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 24c4a1a8cc92e135ecfcd9cbd1f6cfa088cbc74d78c18e02a609cb11fa33778d
MD5 d5991a32f61d4dbd3735a86306973b64
BLAKE2b-256 fad58e04a5e10c7338b985bb145322b24c429235dd66871ab62e4ef1491819e6

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 403.3 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 d5719adf6045fc743de5fa738d561a81e58dc782c94f1b16cb21b5dd6253e7fd
MD5 b5f0372e8dd5d4ead830924b9baac7f2
BLAKE2b-256 989ddb6b30f8c0dde8cd4773445eb1e120bec390b7f5d55d25dce4b0f43e36cc

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.10-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 10eb25378f37dc00c10e83c4c0442c1a6d1b871f74a6dfdfc12b6447962bbdd0
MD5 6f09f7210f6ab8de05f13d80f00957a7
BLAKE2b-256 1ca599c2959493ec7638b446ba22e5deeecad310d1093543dfc10f30dc33efa4

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.10-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c60965da6dc7a91e00ccd84d84797fad746555f44e8a816c4cc460fb231c44fe
MD5 5c36f9df50cf373806f8c8fa316ab1ad
BLAKE2b-256 52fb37ac3b5326dfc7dfbd80e87fb24673577f07dc1aade066e32d923b01ce9c

See more details on using hashes here.

File details

Details for the file fastavro-1.4.10-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.10-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 479.3 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.1 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.10-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 64244c53f1e4853184c2f7383d0332e1dcb34c38c05e6613530ade0378e8acfc
MD5 ff799a852b3cf40c5bf0e8f406d154d1
BLAKE2b-256 0392746407e4c6e88cc32696f410f598221d6a8fb855332f18808677882bc739

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page