Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

This version

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.4.tar.gz (726.0 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.4-cp39-cp39-win_amd64.whl (414.4 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.8 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.4-cp39-cp39-macosx_10_14_x86_64.whl (489.8 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.4-cp38-cp38-win_amd64.whl (413.4 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.9 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.4.4-cp38-cp38-macosx_10_14_x86_64.whl (483.7 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.4-cp37-cp37m-win_amd64.whl (398.4 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.4-cp37-cp37m-macosx_10_14_x86_64.whl (475.7 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.4.4-cp36-cp36m-win_amd64.whl (397.4 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.4.4-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

fastavro-1.4.4-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

fastavro-1.4.4-cp36-cp36m-macosx_10_14_x86_64.whl (472.9 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.4.tar.gz.

File metadata

  • Download URL: fastavro-1.4.4.tar.gz
  • Upload date:
  • Size: 726.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4.tar.gz
Algorithm Hash digest
SHA256 16fcc82844913804b05f28481972b850580ff3103b48f36c021d3b99019f9e3f
MD5 52f126c76e57120d89b8d81a4eb42f97
BLAKE2b-256 6cce4856f2e70cef8c9d0293bf087dd5fc1453fcb3a07040444e7e05e34c18ea

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 414.4 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 c74f8b48d4e4b36a9013ddb2cbaac68504cfdc48cdfe4753edfd017b5156e18a
MD5 5e3edd32d3e76db45f29d5e1f74d1cee
BLAKE2b-256 54973871e9d1806ce5a4c724d97b9e4d10d32e048815b81e69996272bb1c32d1

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 52b9fd197319a1b2eaeb9bfeb7313fa45ef92b49c3884c138b3ab90740232a66
MD5 820339ab9185fcb275a31b03d861f7b1
BLAKE2b-256 25496bb61be278a2d8542663fd6b651af14e8e81ceef2c026c10216698d0c346

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 30acd5c296155959f371bc4ccf7e76461fef5923c738fa34211b5a65c855aef2
MD5 8daae4d88e7e10bd6a9e799f59fe4311
BLAKE2b-256 454250e6c2699b798b828570321519b3db546295848fa862959d8b45c25a37f5

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 489.8 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 bb49ece8865c8515f34ff8e307799b8cdb32b6db31acf4174c09b5c8e7fc5280
MD5 32320ade7b8c5f9eb8196778f369cf28
BLAKE2b-256 8362bc166501dcc2e8ab4e3a152bad000b5d8cf7dbcfec7ed7e4d2c573bf6824

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 413.4 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 bfcff703061a9ecce06fc69ac8f994fd1ddb2d55a6ff585fe25a7b45bbda4011
MD5 ec20c539de03b86e76ba757bf219bd93
BLAKE2b-256 24b6e544371bd50ed50415b84ca95036a9141c96140056f3dcaecb2880d56021

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7844fc4f55da7b1948234a7e5a50230fa0f27ff74cdb5efbb107c5cd5db0f49a
MD5 1b48153dbe977b0fd6f1cb2fa82188b8
BLAKE2b-256 6a42dc873dcfc9efd35bbdc502e178e7eaa834767c12d56252ed07cb4948ec4b

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 24802411fa8557a6703ad1ccdea892f6b72730fc68ecb80214d7208f49a37ebc
MD5 63f90bf478bc8c7989622e90f93ffd32
BLAKE2b-256 8b492f91acb68e4abef6c06c1ec57133e7ec63343c359e1286f52b217e67c71a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 483.7 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 76aa6545443e1f1d6406c52876f3f7157076b91441943db300bc1c0f0daad6aa
MD5 0565014e22b681fc47dce889e194f3a1
BLAKE2b-256 9a1212fb8e281d26e9f9049c6f0217ae6e19864116d6d8ccf5e46116556677f4

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 398.4 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 6446204cf55299243c9d43b1c5e355c74e6980e61c9bb58138ea9f3a6c99d826
MD5 b1012bbdc7c3b7157b066e39044232eb
BLAKE2b-256 23a1896a52c2c679af1891ab73fcce83bd87fd34f5b34f16ae50cd1dbf75d919

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b53816eec734ca34518c7e748d6d538aca0d3c0111ec726ad9808a098bfdd209
MD5 59df0e4345b470279f6c97520e825281
BLAKE2b-256 09db8c8eda2864d1c1330a60ac362728d4e02c4ef60c7ad9fd87de24fc069848

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 836399e5ac80ef8a19e3a0484585c58e52268c87d77b3681615761a9439032ed
MD5 cd47c75bc5aa620c7e24eb27e11832da
BLAKE2b-256 8f25cd0b807743e5b1485c10337658a314e49236fc352e8a8ed42cb85e0814cf

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 475.7 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 b85f2569102f3b54a3973dd12f5bf5049c125488e70959360dd5cbb8c71d47c7
MD5 f1a87939a834d8c52a07d943a71b9af0
BLAKE2b-256 846b6a2367caf7ad72a8a8af40341983cac749dc1e62940aa0a02ee4139f0cb2

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 397.4 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 e03b80a9fb52b753d948788b0048b2a3f7551ba7f8584e60e90a1b3b5071fdd0
MD5 1ce4f75b4d8104fa8569869333127c74
BLAKE2b-256 013787056f1633f67bb6bc3e4725de6877dcb9a088f632b6ebc8501e00492396

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4f1e0093a5ea7f5b72997c91abfa7838cb536c9a1dca9df4c8b7f233e853e0ac
MD5 4d726a2417dddca350830dd2f883a198
BLAKE2b-256 89b331fbb0dcfa0d6a890558b4a904c0dcbc100735a0b8ff7b384d863bbe0f44

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.4-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a8391357b6857f80b850bb4708c4b8db0f3d1c1e83ab2ea888958d89f46efd26
MD5 f5a311dd07a2313b040f6981d0678712
BLAKE2b-256 b7cdfcc7d5f20c7a77703cfc2cf65feeb05f4dd8e69b196029490d55a5bf3194

See more details on using hashes here.

File details

Details for the file fastavro-1.4.4-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.4-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 472.9 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.4-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0a3e3fa7288db81c1ad521321ced7a895c6a7928511b27393c3f418d6e5f6946
MD5 384d8f02cd2be636da801ee031ab14d5
BLAKE2b-256 1b0027f847ab57d5a930838eb52ba15e15e1ac4d3d0f28b0e3c98003f1f95f8a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page