Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

This version

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.6.0.tar.gz (776.6 kB view details)

Uploaded Source

Built Distributions

fastavro-1.6.0-cp310-cp310-win_amd64.whl (435.5 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.6.0-cp310-cp310-macosx_10_14_x86_64.whl (506.3 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.6.0-cp39-cp39-win_amd64.whl (446.1 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.6.0-cp39-cp39-macosx_10_15_x86_64.whl (519.6 kB view details)

Uploaded CPython 3.9 macOS 10.15+ x86-64

fastavro-1.6.0-cp38-cp38-win_amd64.whl (446.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.6.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.6.0-cp38-cp38-macosx_10_15_x86_64.whl (513.4 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.6.0-cp37-cp37m-win_amd64.whl (432.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.6.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.6.0-cp37-cp37m-macosx_10_15_x86_64.whl (503.3 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.6.0.tar.gz.

File metadata

  • Download URL: fastavro-1.6.0.tar.gz
  • Upload date:
  • Size: 776.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.0.tar.gz
Algorithm Hash digest
SHA256 64759b161bbacb5fdcbd54bd5cee05d8510836e09290e958937334dbf1bb587f
MD5 6b0b20ccd8d05641c492211033938f8b
BLAKE2b-256 f928353bbd9a0a66cf2b212b08a0a302f57b448a74811882c5c47dc3af0fda71

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.0-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 435.5 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 a80eec11dfc879299f4e20e6c5b19307dc50ec20a6f431fcfe2f29c5ae63c575
MD5 e311280348d5cfef866d3c0bb9ccca0b
BLAKE2b-256 0ed101ff376efc764be20d89d0d842e6bbbbb00560f8612e17a76a98068a163e

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d1271f11304951d4b61415480ec53c2d44c297eb00fd2d95098171805a6350d7
MD5 e15e7279033ac6db513dc8242c166942
BLAKE2b-256 5559bc92c552d30f49e9d820d8e1d4faf48ae77b16b29d72285eb91cd1ea73e5

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8d18d3540aab5f34bb88ad58d2f45b2c64f10bd04ad126bb79024a1e07d6db2b
MD5 baa95859012d52ce639a150d95692908
BLAKE2b-256 648ff0c83c3fef20964289737a566ab030d4ce3e3c0c2471367c440a66366511

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 cbaa7e5db05694c70985c7ddc8afd9f7c9aef3e66e78cb18d05c70db221eaa1a
MD5 370e131008b47476449fa240e753c3a3
BLAKE2b-256 da85882df6470f2f8aba73211a1a1b0865b907622411c534c90e695a508b107c

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 446.1 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 4eee1bfa7a5d35d252a58c96b727c4645a0e0db3f4c44b94de444d3b086bbc48
MD5 660373cff709a8d8d5aa699c21526a53
BLAKE2b-256 fbbf28d439d921f1e092249124862f61dc54560361c26c52a4ccadc917c4b34e

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a69d35e66812b3aed7204d796c2a0f132d2d5a115d94cc1fb3f5da3577a74d97
MD5 09a40e02693edd794f7d56b6a79a6d97
BLAKE2b-256 4d43ce140e421302a5327c5896771b33c0ace45a9da017a85e326afb11d400b7

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8b788093cf45763e69cd8ffa2464a518edd13340a48e13c4211f85f25f91c972
MD5 7beeb2c29289604c89d498599dab3e8a
BLAKE2b-256 c17271b730fbf9587e9c36226be4d1d78c783f9cc2878d5cd8cf6e2fe5e35cc7

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp39-cp39-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 878b06ac7f6e411c849ceafd2cc7dd9e6eb3f7d4a5268aa6f02eb8460a069082
MD5 999d5a732061ac380347bc2a030b4283
BLAKE2b-256 8d8bdfe7c972497797e49d9145e9a74f59873ef4af1f339d0ae73d3bd395f5e5

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 446.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 ca4a3b4c39d2b280523111fb4ea09dd114ab78e7ee8adb166804624ce4b7ad98
MD5 26fde5a9d652985659715f3fbd97577f
BLAKE2b-256 6e7b9cb32c15c4b62604350950a1e254d562c6824b8453f246532c3bb69b1fca

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 970599975812c8df4aba06ac4e4e558211fc645f3ca44a8fab12b0f2152fd959
MD5 d62c64e9f08737932efb4d77da3a6bbc
BLAKE2b-256 95c56c7d6e6a51952339800d45392b4fde77c60e50ea6cceed32b0dc323768b7

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c039f259c61e75d920b2ce236d676a7dbf122ee136df92133542a99174491392
MD5 cb69eefe0cc5283cc7da659916a8a1a8
BLAKE2b-256 be7337023e9be8dd33d7f6f2f246685ed1c7c78ce8f9f77310a4a8fbe08864aa

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 7511fbf7e1251ae34965894cf59ad982612268a5043fbfeaf309db8f75c25489
MD5 bfb4fb7d4226689cdbaba5056ab81c98
BLAKE2b-256 dea2b5e45200c75425ef0f2e1415ca84b3b387dadf545335aa8a61831237811a

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.6.0-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 432.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.6.0-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 6ac6db0c8229e5c3f633208e9d7e9e016ee725cbc53a9e5993b4c070c189f506
MD5 50d273525f0a6de33c10e63eaaf42760
BLAKE2b-256 f2048f1c102d8e52826cfc353f3fc37b26544134dbd68e24ce0abe4b85ce4de1

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b78ce36f45cd582c8d8b5c8ade764b378d0f3581c1e62fba895945984dec107a
MD5 09c7c0872e861ac4929e3c19dbb891ec
BLAKE2b-256 f9737f97b20e38afee3487ff1f1e303b2832fe952eebbcab871c0f1b27450ead

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 209afeb9dcdf624b10f223c17d3469fefd9669bdaceaf5a3fb47cb74a107e443
MD5 5a451b5e14849b9493a73ccbb64726dc
BLAKE2b-256 4225b6b866fb9204110ba7c2f8b9288d549704cf252c186bc7cafef8d22c8b25

See more details on using hashes here.

File details

Details for the file fastavro-1.6.0-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.6.0-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 77354c17dc36cd21a3127004fd83854e799899b26f42fbe4a8bce3424c493551
MD5 286091c8a510216c7f4479b4bb4f8e7d
BLAKE2b-256 079e3814cf8c786a2034511c09116cdff32c7a5281a61be5e1445ab5700fab0d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page