Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.7
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

This version

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.5.4.tar.gz (772.9 kB view details)

Uploaded Source

Built Distributions

fastavro-1.5.4-cp310-cp310-win_amd64.whl (432.4 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.5.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.5.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.5 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.5.4-cp310-cp310-macosx_10_14_x86_64.whl (502.9 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.5.4-cp39-cp39-win_amd64.whl (442.3 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.5.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.5.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.5.4-cp39-cp39-macosx_10_15_x86_64.whl (515.4 kB view details)

Uploaded CPython 3.9 macOS 10.15+ x86-64

fastavro-1.5.4-cp38-cp38-win_amd64.whl (442.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.5.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.5.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.5.4-cp38-cp38-macosx_10_15_x86_64.whl (509.5 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.5.4-cp37-cp37m-win_amd64.whl (428.6 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.5.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.5.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.5.4-cp37-cp37m-macosx_10_15_x86_64.whl (499.5 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.5.4.tar.gz.

File metadata

  • Download URL: fastavro-1.5.4.tar.gz
  • Upload date:
  • Size: 772.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.4.tar.gz
Algorithm Hash digest
SHA256 d86f72c966713fb699570a18f7960cf4110b069c70681d7538be8d671c9db7c8
MD5 1b60f312c94fb41800d8150be525936f
BLAKE2b-256 5161ef23d955780d54f0124926e8ea66f3f2476d61664805ec8d9d4b1a4906f1

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.4-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 432.4 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.4-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 2402428b26d3c08a58acfa723833e19fb75077872bcb2475a4c81195cdae6a5d
MD5 6e2a7242fda54b1b06633f4d7af142a5
BLAKE2b-256 01105e04b18a66526755dfb4db1a0f0d1083368a16f65876d4c1d9a347966da6

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 cd44636d7ff8365a57b88707b747371fffb676c8c1f68c0d423ec36623888668
MD5 f8f2c353ccf1d7c835608f359321f38a
BLAKE2b-256 a85cdfe2f5e12cb26ba0599b601a2fd6761bc7babdca2d746e31e844c9a7ff6c

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8459faec46e34f2dfeb9b70ee8c36e935e626cff8608d675724718987a5f9ce5
MD5 647633a07e86d55af13d89638fc4f513
BLAKE2b-256 b45cd766d3c747793c21987a89f3779cc3cd144db74c44607316bb3f8c4bbd6d

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 d316cc476b2b24ef06402b8bfa047f8f72a9d6df2de777bb30d9ededda7e3a02
MD5 230a1230f932e64d6de8720dec3dd922
BLAKE2b-256 2df74c2ac6e760e4a2cccf176280aff26b81aa468c91a6ff17fe20589cdd74b3

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.4-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 442.3 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.4-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 18250aa2ab0f7a095b1865565cf9976ea4605c201129636e6defe24ec3ef112c
MD5 bc48598de79a364bafd96161059d8958
BLAKE2b-256 3a82ef2447f33d0a1daf578f9b1ff588fd2a0cf9c7ac68dc7990ed35a3cb03fc

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 06a7b5602dfa032c92f20ca90b8bde88251573773e501bedf5e8b76b9feb14a3
MD5 1e74b6276a04106e905df42c9c8fb5a2
BLAKE2b-256 113db4e9b9dbadf249e534e034c73b64912673200d62064190d545e09791cd78

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 bd021ec850fd30020b7c4fa868466fb7f95450f1f06eac92bd2204cbd8e45fb8
MD5 7a6ba87f6c6f87ff8c1b3c50fe8b8486
BLAKE2b-256 c22e563a51391adf46c2f3378db668b8b9df38c0a22140d91aee3d187728b7d4

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp39-cp39-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 1f7685f3a3c38352abab432bad2f9f2229a0e5f5f8548831e887c30f8396f2e9
MD5 bc256fb60af699fa53448bb82f886448
BLAKE2b-256 0626a2de00dd7b9e01ccb48f3fa461a281e87cba872c8ea85517a7d7f6b27a03

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.4-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 442.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.4-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 b6c30299a49b11f42251cb81c8e15db67750642eac7ba5c194a5ee95c83ebb11
MD5 d32ea1b40c8bd511c45b4ae2fe4d49fb
BLAKE2b-256 0ba158922a466befad70553478c10617f8f5e790181a326fc55c29a81d8d9da2

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a3d190aee86ab73caa1aa550eba850be2ca5dd29d814b38720f4e300184e01d5
MD5 3cf94f9b0c7de000ac494bbe0b7927a0
BLAKE2b-256 459925b80026784ba744dff37b21b1cdbfa3087448d40523e2d04bb8babda393

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 80fe920229ab1f40eccb1b4918481cdd8a20e5e7dce19308ab38b23732da8a70
MD5 308858c3dd0dae054ad69f83b42e9671
BLAKE2b-256 aeaa7932b0e4ecedf54c3e2fc8d84c2559fb02809fc9e4bba38fe42ca26e0526

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 f7d5bc76c03c692d9acea0e5d5baceec19e1c059b26cb8ae9f4481e842be95a5
MD5 712697076b68eb6770f8dff693886c9b
BLAKE2b-256 9ec46c2b95837c1ab865a1b21910929fa98f753cd25c00f5cd7b089462069c41

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.5.4-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 428.6 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.0

File hashes

Hashes for fastavro-1.5.4-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 1a2f2465efd0e7de557c4034e8d4d88a132750cfa51e1582362a1b3a1a9fa911
MD5 a62087c37f8d27217d3d686da09e3abd
BLAKE2b-256 bc75c59476db261c4200e0f09533e409a3568eeb54d5704a3afdbbb295fbc7bd

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2e93a5eecb28cc35d670c9c4df70223fa9bcd6d9ca21b38b1b7ae13ece60c7fb
MD5 799bf149ce80748c15c72e9d901d9246
BLAKE2b-256 aa3bc27dcf523d6e67ee790b241c09666109daa65a40f39e588eb9f87baaccf9

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 5217e9713a3ea03205532394fba4d743749155b04b10b12a12fc26d225b89792
MD5 0413cd15a7b1f0ef42f4555598986aaa
BLAKE2b-256 077bec425ba0a404e6ccda7284bf973d52bdf1149b81f8d546af09f8a970ac6f

See more details on using hashes here.

File details

Details for the file fastavro-1.5.4-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.5.4-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 5afc14398f4191d1a807aa59d2fba5ed869b31343679ec43dbc289db0a8e35c5
MD5 e0c3a9e2d7f8875cb3a9531802e6b99b
BLAKE2b-256 210284213cf612455e41b21d78b2b37c4ce464beb98e085766fda19c7ed52d19

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page