Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

This version

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.1.tar.gz (721.4 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.1-cp39-cp39-win_amd64.whl (411.0 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.1-cp39-cp39-macosx_10_14_x86_64.whl (485.6 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.1-cp38-cp38-win_amd64.whl (410.0 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.1-cp38-cp38-macosx_10_14_x86_64.whl (479.6 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.1-cp37-cp37m-win_amd64.whl (395.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.1-cp37-cp37m-macosx_10_14_x86_64.whl (471.7 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.4.1-cp36-cp36m-win_amd64.whl (394.3 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.4.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

fastavro-1.4.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

fastavro-1.4.1-cp36-cp36m-macosx_10_14_x86_64.whl (468.9 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.1.tar.gz.

File metadata

  • Download URL: fastavro-1.4.1.tar.gz
  • Upload date:
  • Size: 721.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1.tar.gz
Algorithm Hash digest
SHA256 237a741668316a2aadb14ba0532666a305dd14b4043aace89bcb0c6419c08162
MD5 e2274f748e2b92ee95e8bbcb3bf446a7
BLAKE2b-256 6b74ea2e40fb661dcfcca3ab7744f8719869954514c69c1be69b409393860668

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 411.0 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 3e804c4fc9875314aa41901055941b199f87aeb1c880cc6fe3ad258fa08b24c6
MD5 9c005db708c3e83f14fca49c12903d53
BLAKE2b-256 20f957990543c9a976882aea77bf9dab7515ba21fb8a794662172876d0240e4a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 cad3ecbac1fe1d319c617ff01104639795020cb72faf7e30ebe802c1d60ec915
MD5 c7081832e6ca165a97907b8e5b729d1c
BLAKE2b-256 6ab4a81ae7a4202801edc315882317567b1dc2ff9c2539652e593fb4c441a51d

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 485.6 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 72c81690cef6ed9c87a146eaf9f608150bf78fd537a7e796780381b4a7baabb8
MD5 4d20aae2dcc76ee0dfa326f7a959d2c7
BLAKE2b-256 5f0f1f7153022531157f1661fc12c1257c83eaf54c6cc32122673c5c7388ef55

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 410.0 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 6d7d4032ecb28bef3dd41e8c91c986df351e3323f526c901d1cbb53425617756
MD5 69b69f4ae821e21d137082b7e214c7f8
BLAKE2b-256 01c54bd3a633e42161b6546d7e3848a38f96b546091c740bec9bc3555c789c83

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 741cd757b7789e6ab821a1de02c1e18dfada417e1cfccae3f20dc2aadd6654fc
MD5 9595675992a78d42f890a0dbebdd8303
BLAKE2b-256 2ad87d0688ca0066be9a5c5ab8c1ccd1ba10ced11a08e9fe87a46feea3020c55

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 479.6 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 a8c5df1aa6c29409bbfe571504e87bf23553c88083a83febb42a74ec0cabe2ba
MD5 cb91a763e985c5f6da0609f73fd349ca
BLAKE2b-256 d87547d829b27cf1e97bfacfa6b5eca91cdb63858313200558baa13da488cb6b

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 395.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 5bd8a134daff2ea5ef0d72a528ca42dfe6c01deac7103dedf77d1a55936a981e
MD5 16da056790dcd73d6d94c6195cf35a91
BLAKE2b-256 50af7e16ca8f17c49db5abf3279365250e78ce75ba762ab4ec5f5dff5023aee7

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 453676a26e99f2f3af7f57c4236ceeee4e453ab7b9bd5f09e9c89bad5e572c78
MD5 2247c2a81b26f69f89b1c041ceb9c663
BLAKE2b-256 52d18f5c8611026f0ddcd86a8e2f965998e0c159af980c31efba72342c69f3e4

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 471.7 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 a4b6c2b2d126bf6246bead79cba43e17f07a1b99a25ccceb315c6e75ab7c9d39
MD5 5a9ba1e9203d40b59dc2328a6f9730a0
BLAKE2b-256 feb1a82c965ee6173bdd41e74f5afcf448523ce7b3cc25ecb1fc153f06ad4600

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 394.3 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 948e69da16f4bf20bea65805ea210d793eae55f5f24f8be2c4d18c2a773aaf7b
MD5 71839b0ae498ccb3f777997b9f051918
BLAKE2b-256 9843bef38859869a3384d977ea32fb2e13e1cc2717284be4e978c7b02f8c5294

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c298d2b2389049dfc6a06326a0a2f9809c7045e8cadb82b9e7e948fd32270547
MD5 cee8cc7c900ab6b574262d5512059d45
BLAKE2b-256 5f3292ca3cf96a562a7fdcac4c5f5634f9ce783eaef3326f548025435207083c

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 18ed93c24ba96ed0e6cacf50170805188cc53a5e50c40fc94316387a98aa497b
MD5 cc79214fb24b9b850dc2555dc8e6417a
BLAKE2b-256 409a674db31a2f58168514c544fae4ad2b490058ea0fafbddaace0d8c4468506

See more details on using hashes here.

File details

Details for the file fastavro-1.4.1-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.1-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 468.9 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.1-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 3f99237de0f853f083a0f9929f54155b408a5c4b04fcfa8c59e589aa853f2111
MD5 6660300f9e939c73ffa0ae06356b9492
BLAKE2b-256 4c2ca09ae67b6e5bd01d199c4b03c78f8f872a8f02b941ae98abea67070eadae

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page