Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

This version

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.3.tar.gz (725.3 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.3-cp39-cp39-win_amd64.whl (414.1 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.8 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.3-cp39-cp39-macosx_10_14_x86_64.whl (489.6 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.3-cp38-cp38-win_amd64.whl (413.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.9 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.4.3-cp38-cp38-macosx_10_14_x86_64.whl (483.5 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.3-cp37-cp37m-win_amd64.whl (398.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.3-cp37-cp37m-macosx_10_14_x86_64.whl (475.5 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.4.3-cp36-cp36m-win_amd64.whl (397.2 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.4.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

fastavro-1.4.3-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

fastavro-1.4.3-cp36-cp36m-macosx_10_14_x86_64.whl (472.7 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.3.tar.gz.

File metadata

  • Download URL: fastavro-1.4.3.tar.gz
  • Upload date:
  • Size: 725.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3.tar.gz
Algorithm Hash digest
SHA256 e9125c5f08e20094b296b8984ac1be9bc6668917152ac2fe044376dacb03012f
MD5 1845c817bdd3121ecbe0d3da7c3d8af8
BLAKE2b-256 9c2443ca45398f33ac536ae4ff4c8d345f2f7ff5ff6b9d0704f0e168055561dd

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 414.1 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 b41c5644eefd3616dccee1eafba50e548f759664014d5547617e35b6c013508b
MD5 e02dc141722996ee779be0cf3cde61ab
BLAKE2b-256 20d7aa5f6054525e910dadb404473049bf9b51d47196b1e3964b1ea9fa74597d

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3dc6ae436557c59df0a5f9b15c5c5dae09ce900223a2d7697403b6e54a9a84d4
MD5 b2d69f660df2bf2862580b629638e1cc
BLAKE2b-256 a27d3a5a1cba9addf0c4b6717b6d73ae8194b81f9e8e9e7abe6d20b65e312497

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a1be75642747e2dd707a0f581d3512d8046d825f4390cec57db803c3943ceba9
MD5 225882a2d0c1354f5d417698617b5d2a
BLAKE2b-256 f1759cc6e56cc63ba1604cd4b4213a30caa2e97e34184e66dd7d194ca6dce3d6

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 489.6 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 2997cb6089890b21e6ae46b3ef66aa0a4bae007ac1de4f4c2b13394800f4907e
MD5 2ef88290cea36bcd9c283641d57cd1b4
BLAKE2b-256 578b758b8d5f648779f138443195411e729507b96400c5ee62b1a3eac0b909a4

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 413.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 43a749cc1102d60aed36a1dab561321741929b5e045af6d8a55fe3bcc07e0547
MD5 699280d0b11ba44ae84c08a9a14d8bae
BLAKE2b-256 3a9f3efc5543777e2e343a6f08897a417b709568f708b5df1f799945f8b5e302

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 87042d0db94c3ed08d4a8a5d05589e717bb6284fb74da4a06e9e58c9e92647e0
MD5 09a4ce1bf062872adfcd08972834be6f
BLAKE2b-256 c2337b7f948e26e35e05043fe9c06466ea650fa3979c360fd457cb9851146a94

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f8587c520223f396680da89a0bae0d746f1e28672ae5b0caec309ee095b46b47
MD5 bbfdd4222a87d58b03c5435f3a9d2b04
BLAKE2b-256 d82e9cfbe1cf55424654aaaeca48d62a02c0525af9b50da7a3ea8ce52f94547e

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 483.5 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 e5dcf73833db68b94ca8902d1acdc21fde4c19e77a84748b1921170da080a356
MD5 c3996992f465a20c4d14cf8208632dc7
BLAKE2b-256 4338eed984e72756138dea1647fdcc262e1d055188579a91333cd8ed7c5368b8

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 398.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 18645ec353986f6e81834c0f7868af4b031dd59fb0a7eb6bd206cab23383c6cf
MD5 9575e7bb9a2f6d2f7c58cf3ba00eb11e
BLAKE2b-256 98fbce5e0ffa00aba60e3c03a0069fd8792cdac664e9ef538a2369836b59ee4f

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a0c52a1ca807b18eed32511d1fc78ce6a0191743a4b3c941150abc621a5ace7b
MD5 08a3f6d82304c5dd70d792d4ef3af601
BLAKE2b-256 a24b4ce366214bfdb740ff1fe82564b703558092e2f1d8d2d0f01e0242202f2d

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 50f5e039e0bd0601ee791d8c5fd3ba7f5cf62e1969d7f9503eb30000a56baece
MD5 8559ec37c4191222bbee8148d47a3872
BLAKE2b-256 c92b2418639d4e3bfcd554dd64f00718b07396c214d079e4b27650166502fc03

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 475.5 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 9915de901f4c90985363968780fa7fa6c153878e06decca3fed0ed16d50625bc
MD5 2d440167e96b7a0162225ba08a9a6c91
BLAKE2b-256 66421fea9328ca88d76303f03aedd484c754867db99c2b7841526837d4fd6be3

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 397.2 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 069414a730091fe71d5ee02d5a23f17aa453057c585e6da11b6ab6a5e3333f48
MD5 053a189f9a814e5cf1e13703234a84db
BLAKE2b-256 977200adfa16561f53a1e7fb2113c55c3f67d0c8e2aaee22db0232fd83a1a859

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 62724c6a3117c30afee5c2155167228b32cbc82a09f923f87cf913e20bcb782f
MD5 13d88f81ca3a90224b574495d0442413
BLAKE2b-256 1fb76b2f1a8d3854f6f9a0be133c6856499bb3f7bc21f0bf52ae067098020737

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.3-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 853c468c090168614993c45b1dc799fd5874581dcbb7826af784e7db7bc0109c
MD5 9b717464e990b998ff8fab9a3f6bebd8
BLAKE2b-256 f45d3b414aff5de88a40a4a3eb066fcedc02b19c15ecfa040c99740b5b0f8bae

See more details on using hashes here.

File details

Details for the file fastavro-1.4.3-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.3-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 472.7 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.3-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 b14dc6ad73e052558312fbd4837ed7ccf8ae24c959242200498ce781af01e053
MD5 b15afa6a3bb1b329e7760b310b8b316a
BLAKE2b-256 26b5718dc2b30912b07e8a1297613959e8925b42dcb479f4d2371fd2b01fae0f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page