Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

This version

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.6.1.tar.gz (778.3 kB view details)

Uploaded Source

Built Distributions

fastavro-1.6.1-cp310-cp310-win_amd64.whl (435.7 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.6.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.6.1-cp310-cp310-macosx_10_14_x86_64.whl (506.7 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.6.1-cp39-cp39-win_amd64.whl (446.3 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.6.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.6.1-cp39-cp39-macosx_10_15_x86_64.whl (519.7 kB view details)

Uploaded CPython 3.9 macOS 10.15+ x86-64

fastavro-1.6.1-cp38-cp38-win_amd64.whl (446.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.6.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.6.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.6.1-cp38-cp38-macosx_10_15_x86_64.whl (513.6 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.6.1-cp37-cp37m-win_amd64.whl (432.5 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.6.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.6.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.6.1-cp37-cp37m-macosx_10_15_x86_64.whl (503.9 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.6.1.tar.gz.

File metadata

  • Download URL: fastavro-1.6.1.tar.gz
  • Upload date:
  • Size: 778.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.1.tar.gz
Algorithm Hash digest
SHA256 bc37a6edbde7a04a9df28ab838b94df7527e5eea648d61633233abba59205146
MD5 4bb0d2da1c5234976bbb31158ee20a75
BLAKE2b-256 ec93d07c9767bff6c0ab90a88be66abfc126071ed5f9437796ba6ecee6677e14

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.1-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 435.7 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 fd60999204f60d5bcc4e9a4461ce867efb23d271e955684b36c04970750e2ccb
MD5 f6966f8a29742b8b40a5091fc182b633
BLAKE2b-256 88fc51ed2c161580c5464923bde3de9fd41c142f69cea67ab60971373c5a78c3

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 98df91eff0c061ffeb722732ba61d4ea9c7afd0f5403534991a984e75e3b2272
MD5 7fd6c82b76e753973bd7ca589618a01b
BLAKE2b-256 34ca3141f7b6b4afcb51b801b79c1283b44bbeca66ae89e4e0d32c4fd02cfa6d

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1ed84fb27672e54b0bafecae1f128f6e2950828f451e59f93bb74eeccb31b496
MD5 666d523f35e6928db6b440d5b69537e8
BLAKE2b-256 3c7379b4da5d9d7f96012eefc58a670d926a23f5b51e5e2402a7cb96ba3459ff

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 de244146a3d7bc4e60c7be11f317054ee0e57ca807d85923bd8407981488ff18
MD5 136d3b03e2f41c7b823566e353ed6024
BLAKE2b-256 77ab5db043b2ef33ee9f002bf72a9cffbb9d2df4430ecceab3ff4adafda4290d

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 446.3 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 18f0ddf3cc4f09839c08eec67ed3be3d391894874cb87f856a0362540d18df17
MD5 08b4ca597e8e8231783f104b55baf53f
BLAKE2b-256 b52743efb4efcc98a6498909bb072a2748bb0633e2582afe0cc6cda38783ce6b

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1b34c4d33e22ada2fa19566a43976b3f5ca937ab6e054e65e1467bf67353f352
MD5 0103760b45d24ea588284231c4727f55
BLAKE2b-256 e0f9414269ac66429dc489909df0d4c05a9c5be8a2ba8bb904b64af0f57f0cd5

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f2ec2fa5f100e0f1010f550de6de8b1019a683c5034613772b4444d7c85bdb57
MD5 b86b5177b99926e7a171c789e430fa58
BLAKE2b-256 6ad39d002e98e3c045a8be0d97b2f2bc6fa0348dcda834462a07b13f5b9797d7

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp39-cp39-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 3e479b9e4b5e6184b57daab57f1992e649590e902d26b7e2ef50a605626e9053
MD5 295bba0d5333a72ddec4867e2edf1079
BLAKE2b-256 7bb6e0ad34326dd6abf8fea28f7c75c19dd641093b41b51887f6433000610895

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 446.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 ac31f490e3030a6c608fffa6e337bd3f1b00f227bcb098df88a6e3e38982dfa5
MD5 86689049133fb085b511fe7c17aba367
BLAKE2b-256 0ea1ce7c69503c72e4e8dd9dd9980130680494065967a39b3b5603381336b8e3

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b8b77e8f33e55d8f63d65f58a2b2c99248093e2da2e8a86f58320f5715fc3f54
MD5 3997932bba1c1c1f7569e776d314ea7a
BLAKE2b-256 f6894e2a25397e7a4f1453d547ed665d1a302553118a4ef5c45febb59b6893a2

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3dd8c98d410fbb46a475242cea1b86061d2568db89919eb9079a9a183f04d444
MD5 93b38ff6270f42a05b9ec6600af6c620
BLAKE2b-256 d715869316bdaba988e12f5641698c93b4efab7908de83296b1e96183cab1a0c

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 462c3177aa231f8b023d4af0a73f41517df8160399564fc750a711faa0889b59
MD5 41e6fb06f835005d58a3ca2368dd2c53
BLAKE2b-256 1cc0c491163d8dc5d0f5fce55f4a3951cc2f148cf39f4608bc2ac61716d51e51

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 432.5 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 3ef81602dc6ecac3d646d64fe13aeb11252f75b76141b5b3d755bbf548313718
MD5 9afc249a48ff6bb87ae164febc4dad7e
BLAKE2b-256 6b352304506b28981d26e41e00e6ab861f8fb87b7f46a400e2c821750a965a19

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 315084c45339845273193545a50f3d616787eb3231b0b1f3df229196b22f192e
MD5 194dab29d486a51c95bf6a80e80ac33a
BLAKE2b-256 74c94a29ccc9e41d7f6c9053bf3d0a6a53d7e16c8827064070a0f68d29dc7192

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 945d5f6e7a3076758b52fe1cebecfd86b66e6925a3a38732e93f3e1eab955256
MD5 587b5c9442322953fe7506a2d2290c6b
BLAKE2b-256 3a4046faa30739de91e13488264c39337921310e41ad8c7b1ad360d2187feb7c

See more details on using hashes here.

File details

Details for the file fastavro-1.6.1-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.1-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 19d4e83bf952fc2893d11d4c82cf04c3938c921a7c5bf277256cda1005e59f16
MD5 1d1c856e25e89a2e23bc80250310773d
BLAKE2b-256 b3161ca313938213627b3618b2cb94b57e323d98c0d658799fa958a159969745

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page