Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

This version

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.3.1.tar.gz (700.8 kB view details)

Uploaded Source

Built Distributions

fastavro-1.3.1-cp39-cp39-win_amd64.whl (398.3 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.3.1-cp39-cp39-manylinux2014_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.9

fastavro-1.3.1-cp39-cp39-macosx_10_14_x86_64.whl (467.2 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.3.1-cp38-cp38-win_amd64.whl (397.8 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.3.1-cp38-cp38-manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8

fastavro-1.3.1-cp38-cp38-manylinux2014_aarch64.whl (1.9 MB view details)

Uploaded CPython 3.8

fastavro-1.3.1-cp38-cp38-macosx_10_14_x86_64.whl (462.1 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.3.1-cp37-cp37m-win_amd64.whl (382.9 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.3.1-cp37-cp37m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m

fastavro-1.3.1-cp37-cp37m-macosx_10_14_x86_64.whl (454.6 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.3.1-cp36-cp36m-win_amd64.whl (382.2 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.3.1-cp36-cp36m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.1-cp36-cp36m-manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.1-cp36-cp36m-macosx_10_14_x86_64.whl (472.3 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.3.1.tar.gz.

File metadata

  • Download URL: fastavro-1.3.1.tar.gz
  • Upload date:
  • Size: 700.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1.tar.gz
Algorithm Hash digest
SHA256 13da1a914d0914aeadeb72f9b032fd0dc1e68a513319e7537fb785726dd5adc2
MD5 adbd03ca8a12101b13156469bc8e1105
BLAKE2b-256 81759133ad9dda8d93184d73101b244e253e78f50e7b153b4dd32d6f8468af4e

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 398.3 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 9ce9363be4e8ed0d646495ede99c69e8b299a6d99fbd9406e08da1961cdcd7bd
MD5 9eff8f38f5a6970bd6a1e25237e470ee
BLAKE2b-256 6e68967d33953f9a07fa2298e484e22b4f52c30192f0f52e35f8805f509b696a

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 36e20cce2dd47eac5f28b237794a2606d3a06c43a0297866c740d273046416a6
MD5 ff36983ef6319f4e9cf1479152aeaa28
BLAKE2b-256 966ff7842aff22927dd638506555d722ec5394dbc487451c293a1d0481cc9227

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 467.2 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 1c2d3c6897dde7202ae4a2c8b510c3970d082fc6282a6d53a95ae21d2d0e4c6b
MD5 c00dcacf8cd96f6a701da9ce6397fbed
BLAKE2b-256 e63aae24deea6628210946498c72a51b3cc1de5fa96390f1d90b258fff39002c

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 397.8 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 1fc3d5625b6075c878f6d9d7b52c026669daefd2255ef2aae56cd287cc32d1b1
MD5 450f3644b6f9ad0218e33fce0e623aaa
BLAKE2b-256 486378b28972cfc4421c8f36afbe7b3a14a867d275ca7e262161f72a93e6a490

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c921b1b0b6694887b37184a9bb56bb332f53b768dcc2de2760955d9213095e95
MD5 0f7f021562393b0a17988759331f98ef
BLAKE2b-256 313f7fe01bf5af3236de3d8a553379aa52e534cc98b45aaf5e84dcc03ab52fad

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 45dcfe5d9429ac1f527b56361972aff09d983aeaced0e5416598573e7c109646
MD5 a6fbdd3c85eaac6a78590c8552396518
BLAKE2b-256 89a43f5ed648f02c9be8ff187cee309e07345ec37e36d817635211a5761432a8

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 462.1 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 46a13791cedba330732a70648be1481bcd0a64a8f2e35944391466f24a66102f
MD5 330a40f2f66be1edc1d5782a5524d199
BLAKE2b-256 8e8b391ec76346e57a29c80d35a1564b30e8ec35e5aa420484869a2cc6e42cf0

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 382.9 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 7fa80c06cbf6de780dc4b5180d73393f33882e98aa047acb5262e775e2f5be0e
MD5 c2d7f3443b558dc6b731bc8fd873a05d
BLAKE2b-256 55511976567ffc62db1c8258a0ab9a531a5ffc6e068893836e52cd9da08d66d6

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0ab2e54cf55d5793d6b255f990027e4ec3a83d3b6c1e61b68974eb05d16aebe6
MD5 e72c41bb190110cee88c37cb8a7172e3
BLAKE2b-256 04857f70850d0893de47cad0252bcd1647b3e57f93d2997e36f5743c38617db2

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 454.6 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 dba861535a4f15d95ef2b97978408a49ffd3ccafc0ec7b44470a510568d4a31b
MD5 bbc52ada31539de38563afe9934268ec
BLAKE2b-256 28f74fcce0ba0ae573396f93d0da17981e03db5c77f14274fb7f23c669d4fb95

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 382.2 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 15b4435bd1b22a25996b5c2f748bb77b1ce394a0d121ef67a94848cf03760f38
MD5 647090662ddab6959d886687cb461734
BLAKE2b-256 0e8d183fbdc681b8b14faada01372bca673c167807db1f7ef3d2b1ee0b95c8c8

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d5a3cfcf376bbc635a904a776451ee63545823975712c2a3399cd41e94f0f3f7
MD5 e941c45ed2d064e42fec4954725e9019
BLAKE2b-256 6e2040f9b8df7691ccd5bb452e789507708804df55b7fd4041936c7c84d2ebea

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b139f12a381dcfeb1e4891705210daad498739243665795649ba6eaf019989f9
MD5 fd38ff9cb65189f45e188c6c720d7423
BLAKE2b-256 9a7623c4a961ebc186a46f1992c295c9b641c0892a3aa959c780dc47426f35a8

See more details on using hashes here.

File details

Details for the file fastavro-1.3.1-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.1-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 472.3 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.1-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 97b1aa31ec0bb4e834ca8f16a2262db029b443bb59015f87e2e533e5c88f06e0
MD5 4ce4f92273cfa9cdac80dc6a82d69299
BLAKE2b-256 0e47b59504324c4c72106e0f2944a5e16dce634de120aa09b34e65becf4bd396

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page