Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

This version

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.5.2.tar.gz (761.6 kB view details)

Uploaded Source

Built Distributions

fastavro-1.5.2-cp310-cp310-win_amd64.whl (425.2 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.5.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.5.2-cp310-cp310-macosx_10_14_x86_64.whl (493.8 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.5.2-cp39-cp39-win_amd64.whl (435.4 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.5.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.5.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.5.2-cp39-cp39-macosx_10_15_x86_64.whl (507.0 kB view details)

Uploaded CPython 3.9 macOS 10.15+ x86-64

fastavro-1.5.2-cp38-cp38-win_amd64.whl (435.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.5.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.5.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.5.2-cp38-cp38-macosx_10_15_x86_64.whl (500.2 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.5.2-cp37-cp37m-win_amd64.whl (421.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.5.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.5.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.5.2-cp37-cp37m-macosx_10_15_x86_64.whl (490.4 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.5.2.tar.gz.

File metadata

  • Download URL: fastavro-1.5.2.tar.gz
  • Upload date:
  • Size: 761.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.2.tar.gz
Algorithm Hash digest
SHA256 ee7ffc6882834499fd53877a08cdadbb107821a559bb78278dbbb3685ae92788
MD5 fbbda3193996a3803ce858bbee240d6d
BLAKE2b-256 024845f18b3182cb94c91891f639901c7a200c97dad9de2b77f7f1e2a0529438

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.2-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 425.2 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.2-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 cc162420a6619debb8aabffc961e8f9aa442dfb0246115c421a0fa108c5e0796
MD5 63a2c20046b0f3e1fe4584bead486dd6
BLAKE2b-256 769436062f35e663f0e0e74f83c49e0596709305c14ddef934745f6fdfcf1b0d

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7e4ccdf1142a01b5a10b9451cc7f2e82b0bf4d2593921b9aa8d66a4758f42ee7
MD5 6cc34b40ed3fa85f08d5c2040b34d433
BLAKE2b-256 3ef2edee76c3431f7ea4c66ef4d3d8ba2311874564fcad69d73f95b076bef708

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 5de3473c759c884f88c0e4e078538b9199416183bd4f49a6baf39468d7c0900c
MD5 9b5517738503d95c43e52dc23e93005f
BLAKE2b-256 38280b9e1509b43911baa099fc8c84b0c62e3aae12f54b6777d09d1a666fd661

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 4c447c81bf9002aa0258bb8e208698744b4e22362548d5423f718c92c22ce572
MD5 2866357fc6b5dad22d455d6612ce79b0
BLAKE2b-256 c056412c0e0374a9666187f3c107ff93b8dc081b0c90393c20f8002ae0a2608b

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.2-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 435.4 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.2-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 db4f1a65dfa6d26b4be74d9aacd13467a17928ada2c15096b401288671689e44
MD5 689caaa1410480c7b377ed7ceef8f13a
BLAKE2b-256 aa7afa01fa7698b028a856ca553ef1368e7a152d87f7819e92e399d8d3ae3534

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 778d134d144d025f72ed98f60b5a63d363f473c5b19a32f4173e2e11e7f30be9
MD5 45530db72ccd506ede10ba2520d62ce0
BLAKE2b-256 b952e148700748ad4dc25c424c3105090b2351961759d1a6d52386f1b0d278c7

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7699d1283062f11a3bbfb705f1f2d3fae0f7a1238ae3ca6234c7c9208b0a11a6
MD5 f8e1a0e874a9fbac8b77336c96a324b5
BLAKE2b-256 20bde2d159b946415bd7a2074c56f40fff5519d86f21e2214e0a507bb453d534

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp39-cp39-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 e6f23f1f4e2726408a35d2f2b279c29da41a989351a0f8c28ec62348f01be977
MD5 c240db8cee50731eaefafb946cc664d1
BLAKE2b-256 a1cf534c94b13e5140badd67dc17144c9f0c03a2ac2499c0e68f58f6f82e5433

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 435.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 45b7bfabf42bb3882071599eef49228039febbb1b8801b3a65bf32d717eea075
MD5 c7c1c628d37d4dc99c1afa0e7533173d
BLAKE2b-256 3ea78b822c991e4ff10dad9a93a40cce1b6d24b5d19c003abe4a0ae2987b9435

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 49398a773fab076725abf8926525092c455f2cb2ae58127d69fcc132d970fd6e
MD5 929c72f4fe2bc09685ba448e7b5079ad
BLAKE2b-256 600cfca173d2316e249753aa5f0e76c2eb42464754e2726fa2646e85f21e7021

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 746d97e41d24c838c70a44cb4e6a7e4109f20923fda5a6de18482e31db06ccdd
MD5 4f02cc3e17bea5406f9bea6ab6fc2d44
BLAKE2b-256 3d98dc43cbb095cf1a023a9970fdbe4abab45ed71833dbd31f718976043c7f8f

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 90517b77f9d8362efe8deaa3da06673f0ef961e973bd9daa49aba998d8ca849e
MD5 36a8f620cd5e582797c68b19c5e3e3e0
BLAKE2b-256 72396b419f817878e075e95062e8054b2c2396e7781719c4b80160d53c11a32c

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 421.2 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 faa8c9cd4920c2e4e4131dae2e5fb050cc64533c22384d5dbba55c6a6463771e
MD5 0a8b27644e1b1575df2ff43891599b02
BLAKE2b-256 2de2ed3f260b31c52c3e13f87a1d2dc0aee56ab1a85379ad4059988c99c0b911

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2c4777ebd022aecb05829b52359351746fb8ac1f3958385e524addde22d212c4
MD5 64358c15b52be1f0e02ded5e06c0c9c8
BLAKE2b-256 087d8dd8cfa17b0f950e34b86f5bd5984ae4c6fc9c8df648f51e4ff0215edfcd

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3600c5b8ccc0c5dfab5927486d71ba79ac155b0eeecce517d86e191a5186110c
MD5 22a9cb681e75b8d8c8ca2e02857b7504
BLAKE2b-256 91ab4de599a923f4b1801575b6838d079c94dfee21be740386eb79e0f2a7fd49

See more details on using hashes here.

File details

Details for the file fastavro-1.5.2-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.2-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 8bfd03691fb57771b0f4d4a241bd5d5f2e622cbd55e88ab3718a58b7e8e00bdb
MD5 e49f802db7f1034df11d0d6941c99251
BLAKE2b-256 33ac004d5ce705c9db5b545af047e461744d0ae6a3c19b4063a64510eeae4381

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page