Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

This version

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.8.tar.gz (728.2 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.8-cp310-cp310-win_amd64.whl (416.3 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.4.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.4.8-cp310-cp310-macosx_10_14_x86_64.whl (492.9 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.4.8-cp39-cp39-win_amd64.whl (415.3 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.8-cp39-cp39-macosx_10_14_x86_64.whl (492.4 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.8-cp38-cp38-win_amd64.whl (414.4 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.8-cp38-cp38-macosx_10_14_x86_64.whl (486.1 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.8-cp37-cp37m-win_amd64.whl (399.0 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.8-cp37-cp37m-macosx_10_14_x86_64.whl (477.5 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.8.tar.gz.

File metadata

  • Download URL: fastavro-1.4.8.tar.gz
  • Upload date:
  • Size: 728.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8.tar.gz
Algorithm Hash digest
SHA256 7ec4913adeed150e0981d598dd5aa49daa76abda7a1c3a1295aae6d7b0f42729
MD5 97383bebe0b029a0a0d528c4e639f977
BLAKE2b-256 5e2cfe8efdc33f4dab473ca031a288990a558d65efa9ea58b99036e81b889c58

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 416.3 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 98dbfabf42fd11d0bdf6f7af231043447c5e5f6d68427a8cbef31ef33739c0a4
MD5 1811249aa646809549dc5e69559e541c
BLAKE2b-256 9139d7eb8e373329a1779c99a429be23e553b6fcac10f2e4779e25953621e07d

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 dc95fcdbaa72c28c9444244d08fb7f02ff5495a8814244173549e90f7bd9733c
MD5 10269591258fd1ce1229887fb23b56a7
BLAKE2b-256 6264d462590a86584238879bc7605a977cddb66637833481a37c21e4412f76d2

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp310-cp310-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 492.9 kB
  • Tags: CPython 3.10, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 d4ede7a358e880fd89b4fbdd5ea9d22a7935b3b56198d5402705ffee45d17f26
MD5 1dab5fca8b88321bdc0f4b2edd864150
BLAKE2b-256 563a9ef137d3add2c052a6f2d67995a5f7d9a4b6c6fc8b0c2a1cfb84f73b8931

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 415.3 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 fda0dbbb3f4f104c16aa1a7ddd278bbd1f215ca300e63cee2cfd28f614675686
MD5 acc80c249010e2dfca0c3d239ec03019
BLAKE2b-256 da97cc1d1a488f5326cca0d466820867b68abbc13325d93ca2e3b32cf6bac455

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 09799e1ec32495c716abc6eddbd637e5b0cc5f5e1351fc61ffff0bd834b65f8a
MD5 b451a7dda4a4c5f9309d80782c2e5c22
BLAKE2b-256 069e115076ff6811fd493ce2f840fbf018cfda2d5b01597c638bd336c275baca

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.8-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 d704eece6437bf98a5bcd5d79f87f6f76a0df07bb050380d9aae528e8da2a469
MD5 ef2ce58f83a71117f59a6e8b76ca3124
BLAKE2b-256 5d19fa4aa0e5dd76486d3a289b4b7731516733fa00be095db27bb7d1c2d17f77

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 492.4 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0852fc0a2db7b08087c3d24962c1c3b73b02d27833ab779fe3737b3cc5d61967
MD5 8a31dbc547bceee912692e6746594806
BLAKE2b-256 8708200efe8dcd520b5addc10080de291616e9cefc3da08c9a1caf1eff90977a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 414.4 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 1ad54968eb42d5970247a99561e4876bb51e10717de89d0a2586dec107f85ef1
MD5 1d5d553ddb113a98ebfa23ea4477f4fe
BLAKE2b-256 baa24f7efed0e2b1cc898918246d81f576a061174bf45cffe525fcd21bc6c959

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9f6f63594a5d4e7aa3d123715aa5430bcb6cf36e9b1c079efdfa5f687385c0f2
MD5 73dcd1a1b169e7bad79537892d01e946
BLAKE2b-256 412f0b68bb116d4995280b3eced0abc36b7fda26c1aae0306af87e7f7e7ad121

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 486.1 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 eea2974a75f5c15691d37a0ccd505cb61e38580b38b0bdd17cc051b0414902fe
MD5 2c62cd14863d3addc0ecd5edd537dd8b
BLAKE2b-256 da8f47e77c758e1cce2e713e104dafbbb509deae1668d2df54c71e6cd85c18fb

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 399.0 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 8abce3e99ddbfd9613c41e97b292d43745a2fc311343f7ef58c6517aab4b71e5
MD5 3b8f1c5a1d30e4c7339a58d47b7d3d53
BLAKE2b-256 208774f88a7a37368ab0f80d805fb0c66a51b6f32371f7c23b6ab6f5751f8ed2

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d0126e8fff7294854157d3f2d0b9c056fbe24a941b01bef48badcf608feb241e
MD5 1b2fdbdf77d744cf55e3714aef0fca47
BLAKE2b-256 022b7ddfb8df6a79ef08e7574d12a7cbb0ec115051ef92780f4d4ca5302b673a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.8-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 13f1b7a92e014a4b43e6b2e08eec043b0e3fb1e713769d6585b658ad9fbe7593
MD5 34a04529d5ef501ee50b432e91926da3
BLAKE2b-256 fad1eed6b1c1ee5f261ebfe4a281a6389a2f7792a5b0980fb9473f0c3b3d0599

See more details on using hashes here.

File details

Details for the file fastavro-1.4.8-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.8-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 477.5 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.0

File hashes

Hashes for fastavro-1.4.8-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 355764158d31c7f4e0fba7be42ed9cfd228db5043efd1e2aeeb987c961a09d3a
MD5 ff08203d21aad972bd9d0aa73fb4a75f
BLAKE2b-256 1439afd651d762f2af8bcdebebe7daa6678f5395808af23b60021c529838fc90

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page