Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

This version

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.5.3.tar.gz (767.5 kB view details)

Uploaded Source

Built Distributions

fastavro-1.5.3-cp310-cp310-win_amd64.whl (427.5 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.5.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.5.3-cp310-cp310-macosx_10_14_x86_64.whl (497.2 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.5.3-cp39-cp39-win_amd64.whl (437.9 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.5.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.5.3-cp39-cp39-macosx_10_15_x86_64.whl (510.6 kB view details)

Uploaded CPython 3.9 macOS 10.15+ x86-64

fastavro-1.5.3-cp38-cp38-win_amd64.whl (437.7 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.5.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.5.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.5.3-cp38-cp38-macosx_10_15_x86_64.whl (503.8 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.5.3-cp37-cp37m-win_amd64.whl (423.9 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.5.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.5.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.5.3-cp37-cp37m-macosx_10_15_x86_64.whl (494.1 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.5.3.tar.gz.

File metadata

  • Download URL: fastavro-1.5.3.tar.gz
  • Upload date:
  • Size: 767.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.3.tar.gz
Algorithm Hash digest
SHA256 262a9f629448a0d4941a4871fd958b70a01f48a0106644b40e4fbaf984f5e89e
MD5 dc920a31d1b1ccfa180017003aa8d82d
BLAKE2b-256 910a91c3f999b2e72a3ea4360eb77c61a6b2f0e55c631ec65ee923806681182e

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.3-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 427.5 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.3-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 6ad6af14d5e927e0c5fbc5dcb7c7030ef8759c8011620344a57252c263d6dd5a
MD5 46961c19062466c2e46a62acb431fbf2
BLAKE2b-256 d3118ab431c6656a9297b5f27d4f3e469117237c011c3fc52421575cebfccc45

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2c14c72f79bcb689edea06ec276bfd5c4d4f17b8e32f545e32a6ee108c5f09de
MD5 1e99859408e8966beabf466d7afa7dc8
BLAKE2b-256 189a747bf59afc6890ffae6e6954e200cc6c74d5b14890d7733a07751d191110

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8ce66549bdbc9d43f40e1858796c1db59fa9f63f2368217071929c39f7ee7c1e
MD5 a5ecdd316f408267eb3617e86fc525a2
BLAKE2b-256 45629b10c313d3ef591a0e2e6b54f70418dfcff0ac6851aee41f9c2b8b20acca

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 bcaf1c9ead0aa1c7306cff229f2c207d714f7f765e0bb010e64c74d379008555
MD5 4387a3c99ae78b64c3c4f18b6c85d54b
BLAKE2b-256 8f54232b7be26776e912bcc7825a7e43b1ccbc164063432e498bb9004eae6598

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.3-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 437.9 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.3-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 14c23dc870785d860f1523e7be7ce4079dc36070f1bad18139304a5f4d654180
MD5 acfb97d23ca191b8e46e958a70b1806a
BLAKE2b-256 2b7b3804357781b1e1a611d69957c633ef37ea49d2cf030bcc249eae87fd5044

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 298e98c55e98f579df66e8258a942494a64b5340212968ed1b6bd9a68c52db5f
MD5 56ec76b6a6df668392919bb7940a68e6
BLAKE2b-256 e7b557645273a2dd6912ae0a5235bda398b4059bcce30e6f8b07fab02829aa6a

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9b399317d4f559d8f3ef695ef8db8ea230e4b489437d2a1e50307120e181e831
MD5 87aa4d7e9f22ce061d092ac71468a647
BLAKE2b-256 7c46be2b4acc09b47c3df49161cb27d0e829574f5c698f5644922342f99913d6

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp39-cp39-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 063f17e000b7a4174ad362ffc91b75de85970cc67fccfed2bd12e21908de6028
MD5 5249cc0a5d77ffde617afb9a137cee14
BLAKE2b-256 ef1fe58a93f994876419af8db9d9fa4257eda19d910e33ac56a6f7a5e4998d11

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.3-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 437.7 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.3-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 b0408d6b02abcbd4e8866b6b027d7cdd20d570e78177fb7c363fb3cc8ecb02a3
MD5 568cb8162469e670d65328a684225440
BLAKE2b-256 004f6c6da56df88fac7f4b9f55d703057120cd630c4f6ca42c0f45cf1b947346

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 259ade020a70870858b656219d25f566eb620e52967070f9e18b040f1c0cd482
MD5 3ee2699711433f435f94bc4656b30a6c
BLAKE2b-256 b2ff2af8a5256e79d1f678d0daadff75d64d3a6ce18e3bd462e2ec87443949c3

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 badcbd1c34f446f4b64ade84f9517ef7d52197e0ea9dd6b3775274f2271dba9a
MD5 f098f30dd70b5453abafa042512eac7b
BLAKE2b-256 a8eb1d59d894e409494885c4c9f1f2b5d777159c569a020eb6bc4e83021bc4bd

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 c5043cfbdfed27f1331860f1d215db838b92d575e8a78915c52c3f6c64d42176
MD5 c3d2533229123c41b17059b34029ff60
BLAKE2b-256 916c20f3a37f8dae4a91c703aa007977fec4c2b645f052cb30716795bb2964e1

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.3-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 423.9 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.3-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 86a33b2f819cad67f002250e9c6a62909fc31e048679da5f2ac02f4bbf0a5998
MD5 494d6066dcbcd6f3e03ca30f1006f483
BLAKE2b-256 a3ef09757ce48974672dd519e6922be201c467eda731023eeda65148a5c9f264

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 78ffb34790791a82edef9cce7e49e44faf1b6b0b1da12e9c7d45f31fd6655a7b
MD5 f805cdac0c9ee7d0f43ac1c0a54538a6
BLAKE2b-256 2d55bdcf1b5e513f501904fce7c246fdbed7674aa645a0e5e0f2eeef0186c2f0

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f8980f2bc1230aac509273938d23c544283a476bf84ac79b80540d503f941fe5
MD5 9ddfed8b7831e4994dce07ba7520af4e
BLAKE2b-256 b89578287a036ac854f1692fbde95867673971a2bbefb70e078a468baf1090ed

See more details on using hashes here.

File details

Details for the file fastavro-1.5.3-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.3-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 3a4fa9f72c9b134fcdb0ee30d4a838f1d30656ece72a95f18cc74db10148ee55
MD5 631dcd5d11861883f215e505a2806477
BLAKE2b-256 00bfe82c01eda466dd2f7f4c8b9cd2fea69c05ea6bad4a7cb32e9194c05214cf

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page