Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

This version

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.3.0.tar.gz (670.9 kB view details)

Uploaded Source

Built Distributions

fastavro-1.3.0-cp39-cp39-win_amd64.whl (377.5 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.3.0-cp39-cp39-manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.9

fastavro-1.3.0-cp39-cp39-macosx_10_14_x86_64.whl (436.7 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.3.0-cp38-cp38-win_amd64.whl (377.3 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.3.0-cp38-cp38-manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.8

fastavro-1.3.0-cp38-cp38-manylinux2014_aarch64.whl (1.7 MB view details)

Uploaded CPython 3.8

fastavro-1.3.0-cp38-cp38-macosx_10_14_x86_64.whl (431.6 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.3.0-cp37-cp37m-win_amd64.whl (363.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.3.0-cp37-cp37m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.7m

fastavro-1.3.0-cp37-cp37m-macosx_10_14_x86_64.whl (424.4 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.3.0-cp36-cp36m-win_amd64.whl (362.3 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.3.0-cp36-cp36m-manylinux2014_x86_64.whl (2.0 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.0-cp36-cp36m-manylinux2014_aarch64.whl (1.4 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.0-cp36-cp36m-macosx_10_14_x86_64.whl (439.6 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.3.0.tar.gz.

File metadata

  • Download URL: fastavro-1.3.0.tar.gz
  • Upload date:
  • Size: 670.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0.tar.gz
Algorithm Hash digest
SHA256 f401be0ca4b1b17bda3ed69faaf967829ec8586cb6da6482b1f3bc9246a6fea6
MD5 eae0ebb2568f045739a714f39f9f8857
BLAKE2b-256 66adeaf93f82ccba90eee2325513ca54f5ab9700a5642c32a6bf0a50afdde243

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 377.5 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 93305112901f990fcd21597af521f9499ccd2990f2b67f195d8e215a658717df
MD5 5c9ee6fda9a3383ef71890c934708a05
BLAKE2b-256 cf5e00244e61388a7e9044027725f617aea6785f569784d6cba1ba051e30b20e

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4ea3399090de1d2e1212393d38be8c8c9da3725ed81dd6a2c0c1002d2b3eccca
MD5 6385c542e9d6bdac92bba1099bc427fa
BLAKE2b-256 025c6c55a24ddaca3043bac62b2a3ad381ddf697af5cec433c577774f3be3949

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 436.7 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 f95b98dcc393aa76c353f2297545fd38e2c5d8361a58c553e272a1aea787e2c1
MD5 9f6dfc1c129432f3bcd0cb7a4c1fc1d9
BLAKE2b-256 d31df3217085e0ec67a262b8e6f4e8cf05844c7f920fcc11f0bc79f97bd8b323

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 377.3 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 a84f39b45c767c444c35e4f40d811001f1fedd1418b92b0bf727486827476483
MD5 ca1cbe1392939126f3211fc3ad771a03
BLAKE2b-256 778add8bec44f661422de2d924d5fc41692f032d98a4e2a87ecbc43af78a3495

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 70a67fc8ba6e07c807d21c284a90962edf7e30ef02973961e7f49da11610a22d
MD5 c4875aca08744696e8cfae2071351ae5
BLAKE2b-256 c3f4e5bb0fe5ce233c6c1dc179b1c2911c493bca974797bca4ca92bf93e56fd1

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2a4c2439182ac2f0d080e758dfe56ccfbb9af48da76fbf1ccb7a12deb6913a7b
MD5 56a112d049d5fed1fa211feaaaa56a8a
BLAKE2b-256 9e524126c1af26840ee2ba03715e692a1088697daef3ea06d6d192367f4a99ad

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 431.6 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 ba0987332ac3dd928586e1afc5b5c82808a68e8036a51dbc4307edee50f992a8
MD5 c36e603ee9005afb466bb256da8d01da
BLAKE2b-256 e912fcbb56ba82bf8be1a9b6ab16052c267f6123305eb3514ad6f77fbc11a478

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 363.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 e71c099e9f8b57cb4f5e6d51c2f463f181f39a14a4a1024d99e373ee403bac8d
MD5 7eda289bd45f0529765ee6674ac12aee
BLAKE2b-256 4edd97a3aef8b62ff3a79afc5760f3463960d4e646186965f8cf6a117d560c08

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f4e2739f5277a14bd1efdd323d12e126d5fbb7f70e46c8f626bfbe79324b78e3
MD5 e356e6c4772e5190c0ec25e1aa0ce53e
BLAKE2b-256 40a9c44dce0ba7cb6e59b4172e180d02dc10f3f91d1fcfe8548af7f7aabdc319

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 424.4 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 0ba301ee8098cd6657eae67837d21bc47739e291dfc973e396a00190991d069b
MD5 424a876280ab1371f1827ae30dfc0bbd
BLAKE2b-256 2e50951c867fb9a26673e5fd41fa9c44d670b51bfe5441fe3ed08caf92f41a61

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 362.3 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 f2c600140e25ee36c1fd58d49601ca577c19b4f8b056f87b1d5ba19b346e2804
MD5 de506701648f83d519e77f237b12446b
BLAKE2b-256 91dcb623cec3c380799c96c5570863666a83e019a7385e91cf9d584c1dfaff2a

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1c85781543032c17b9c28a1b87835900b36e21b3693509edcf9ca806ab575bfe
MD5 2bfdc926569d41588aa743189dee720f
BLAKE2b-256 eab86aaa3d132a7c9a0055746933f11b9212e7ef836739d9134be793938f05e4

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e3d98c9f0a5f2217f9571712e855d42dd0de6dea4fed1e489f43098e4f84d163
MD5 f1343904a0a1d5c84c731e1800c1a012
BLAKE2b-256 b14817102a9753b00e141b17cb4e4177e25f025278212887b27ecb9b48620785

See more details on using hashes here.

File details

Details for the file fastavro-1.3.0-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.0-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 439.6 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 62ab5bc64dd12e7792ab9585429f3c3a34a5c7ca63846d79de7afe565dcc2c84
MD5 68f2790f3213fb6c76bef4a2ff523ef8
BLAKE2b-256 b8f0f949476a676b928ab89f385590f0e4f6b59234e2bef614eb8d98f81797ca

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page