Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

This version

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.3.4.tar.gz (713.1 kB view details)

Uploaded Source

Built Distributions

fastavro-1.3.4-cp39-cp39-win_amd64.whl (404.9 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.3.4-cp39-cp39-manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9

fastavro-1.3.4-cp39-cp39-macosx_10_14_x86_64.whl (475.5 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.3.4-cp38-cp38-win_amd64.whl (404.3 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.3.4-cp38-cp38-manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8

fastavro-1.3.4-cp38-cp38-macosx_10_14_x86_64.whl (469.5 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.3.4-cp37-cp37m-win_amd64.whl (389.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.3.4-cp37-cp37m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m

fastavro-1.3.4-cp37-cp37m-macosx_10_14_x86_64.whl (462.3 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.3.4-cp36-cp36m-win_amd64.whl (388.3 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.3.4-cp36-cp36m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.4-cp36-cp36m-manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.4-cp36-cp36m-macosx_10_14_x86_64.whl (460.1 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.3.4.tar.gz.

File metadata

  • Download URL: fastavro-1.3.4.tar.gz
  • Upload date:
  • Size: 713.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4.tar.gz
Algorithm Hash digest
SHA256 4c0c2ed0c218d3e8737bd04d6cac2a3c1afd11090d3ac3a45fbaafd73cfe2cd7
MD5 aa3f88f42985cbbfaccdd332e7345ebb
BLAKE2b-256 47602dc9ce1af308e9b542f5fb96f47497b4bcb2b802e324fc4985d8e6dacce7

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 404.9 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 4fe9583e772cef1656bab4d7ee27a306a317f3f4fdf4cc45f2add0d919d20f8b
MD5 9ce113d149b7537df302d7dded79c0eb
BLAKE2b-256 9c757b19572c90cf1856c70a58ca99cf9043dfcfb8affc09f728824d845374ec

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.5 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e22f9a41949d5d330db4c959fb3bc0bbe226eefc3ed05d08d52b133b735ca5d0
MD5 5e2cb228ca8cdd5c24b3e3ce43b2522e
BLAKE2b-256 878ced9f68db926412708e920672f3dbd88cae6a0c2e9aa9b36a6449a019a1f8

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 475.5 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 86ed5ed392bfefa822608c1c08fc17eea702e0287c268e922b45f850df6d7435
MD5 cd5f1e42f60ebe86d2628b964d079770
BLAKE2b-256 0e674ee4909a13209ef0f7037ab3c34e12c3f06a6c4c074866db4f57fc828cd5

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 404.3 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 bd53092758f162080462d1dc170a25974204c0b31aba200798ed95f1429ca8aa
MD5 9427b95a4980d4fc96b19812eac20f4a
BLAKE2b-256 ab568d69ddfcf2c37e059134f088474585f5b3015a16e7acc839c59f797be160

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 43408a5b7db951fd4dd293d7253c308e8cec78f5f126dfe756b20e4fe5a5a47a
MD5 8100dfc43defde7f1e09875fd35025b2
BLAKE2b-256 64ed3feb00123910cba92ef58a8ca31fbadc55757f9662c9efbcc765639ba312

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 469.5 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 11ca52083cc79b1a3e7e9fbb2b84b810fe63bfe65c783ccb261081c2fcc28e34
MD5 87ac6a9cd662fd0a4e011196e795a1fa
BLAKE2b-256 9c2ae57569374ce0ce29460f77df415f6bcc801dfbf97ec01fbb74965e863918

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 389.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 48704347fb1b81fe2234f89583d3611500eb0b336712dd3ba053389c0a8d0584
MD5 f3d2b9acc07d86e52eba33e71be85ce6
BLAKE2b-256 37a320f2ed3d71e8b21246a470e0916024a271c9714f5344600051ae7c57fc1d

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 02406252b9c18b0a152d1536b4cbed8766192427da6e5e4eb9eb3d5e1fc96725
MD5 03ac47af170fa546cffd4caf9e3d81b7
BLAKE2b-256 e3ff8f147013c646128680a1b5c047f1db7ebbe171bb8ff3396937e640c0825d

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 462.3 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 fedcc1f1449825365d327714a65501171af6e40bd371356049218ca47deb7464
MD5 1c4398c6ffab40ba48e3a03fb4199d2c
BLAKE2b-256 edaeb84070125eaa4024be3c7ecbd6b367273f03354bb2d0d4da0512863f064f

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 388.3 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 e9d8277ee9972aeabaa72acfae262b1b310cee5a4d29a3347177661592733b9d
MD5 9c993b430afd37693a8820e616b05438
BLAKE2b-256 e690b0d48c7348326bfa0bea967d1dfe6ed8f3b02fa990fcaac404e2f81d3b8a

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 92ea0aa736d7a3c4ace86ca0971ceaecdac771fb3fc9b311cd24a1a20a8ab7a3
MD5 8eaebfe0590720dbe7b2e1026cc81c58
BLAKE2b-256 22a2bbae9555752c8f0992d8b7649ce43850cf0cb1ca024b47f48ec7630230e7

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 065d83f35e179654f1dd5a804d457753745b865cd2698a56bbc7fa33750ce908
MD5 9ec855df670d7f66566795bfddec23da
BLAKE2b-256 0a7aaa6a2c5c99c52ed2ee7aa0b7531f152d7efc209655d0ec84e183c27bae8b

See more details on using hashes here.

File details

Details for the file fastavro-1.3.4-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.4-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 460.1 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.4-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 3d9529bdfabfa38673768df1dd3146050a4c4e51924908f8a434ab61edea4067
MD5 e54d38348845f14e3bec4ba72f6c9cef
BLAKE2b-256 a84ebe8bc33c99b2e6814d629630dd89bc946dce7e62bb221a672b900e8dc2b8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page