Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

This version

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.4.5.tar.gz (727.6 kB view details)

Uploaded Source

Built Distributions

fastavro-1.4.5-cp39-cp39-win_amd64.whl (415.1 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.4.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.4.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.8 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.4.5-cp39-cp39-macosx_10_14_x86_64.whl (490.7 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.4.5-cp38-cp38-win_amd64.whl (414.1 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.4.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.4.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.9 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.4.5-cp38-cp38-macosx_10_14_x86_64.whl (484.7 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.4.5-cp37-cp37m-win_amd64.whl (398.9 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.4.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.4.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.4.5-cp37-cp37m-macosx_10_14_x86_64.whl (476.3 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.4.5-cp36-cp36m-win_amd64.whl (398.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.4.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

fastavro-1.4.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

fastavro-1.4.5-cp36-cp36m-macosx_10_14_x86_64.whl (473.6 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.4.5.tar.gz.

File metadata

  • Download URL: fastavro-1.4.5.tar.gz
  • Upload date:
  • Size: 727.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5.tar.gz
Algorithm Hash digest
SHA256 fb5b4b46bd9f89be07921752bac952daa872a79f94fadf01f3679da906824424
MD5 2879187d191f6a9b9952e4295892693c
BLAKE2b-256 37db1cfae958348f817953660bec6c31b5ccb4dd6128c5c503a45011fc0917a7

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 415.1 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 4043ca61362601dbc9b0b51e852207130677842e93d95b5a5327c3ad478f14e1
MD5 50a937d944e9a3d42ccb20c0bb84a661
BLAKE2b-256 60c1c17f3a90bcb5574735cade294c29b409d01377ad8b0f975ba44ac00f828a

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 fafe37983605ed74a5ca8063951f6d5984ad871e0ff895f14afa81a6d88c316e
MD5 2f00c19c46becb240504eabaa2e9923d
BLAKE2b-256 0238a6f70656f1366989a7a3afe63b7567f2ca63a90f2f255311050da6e2d59c

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 61603a2b46cd6ca06a250fec3011c40400683c9b57d934f3e4d5cc3fe0f3d254
MD5 d8525ad6341c67af5a90015b97199f00
BLAKE2b-256 79b48443a0394152d28035b3c1818afdd83f594a3314669c1ecf3ff39f7a4176

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 490.7 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 246cc9043b31a004e13204fef72917bdb4886b752defe08eb32da3d87123802f
MD5 52c91ac6adf3e380acbc734a05067e18
BLAKE2b-256 ef14604c39c8360684b7b118b3da6657cb7fcdaf5ca7c82e29636567cf7f4c14

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 414.1 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 4dfcd2401759a80da6ceb4e46af660626d77c235b903db09c74d031f8e02a219
MD5 dc44c3a8e11cdd378c4b25fd98b82e18
BLAKE2b-256 65b4b8afce2ed8bf8c8fdf26a381df8c91fcd91caf141f36423730200d0c899b

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a60af4595b68838ce736aec0e9221d3e50dd29f26c6409f63f41256bfb6ef5c8
MD5 d4bb4bf37534e3790481f1fc3ffedf41
BLAKE2b-256 6016ec2d3c292536cad49d8c19d97916d8e9303e30967e96f8e8b89ecfb73772

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9cad773eccad51271faacafb7a4bb8c387e456eb419090cb529c116a4d918a89
MD5 34d515f67fb6a83b9b7380156a04d69b
BLAKE2b-256 9de2cb0d61cad32874378699f1a527c6d15f1cb37c54b438ccfafdb1584c160e

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 484.7 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 fe37f2e2e8e1170abe50dd113a730ffcb64d75b5a68997da2b22bb8545439642
MD5 6208f25aa9a58d284eddc12c90841339
BLAKE2b-256 40d82c6ebb6586f9b1531f5ecd0a1e92e792a09e0a2fa94ba50c8e6c1bb58ed8

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 398.9 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 5d8163fe404c1c48ce973772b5a00d1f696ae04c5e3ae4aa63c89276e617af63
MD5 1028e703adc28d9054540dab33aff4ff
BLAKE2b-256 7dd54069f798baae10879d56b26134d8eea80affb59a127b40a1c14464a69b75

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 06b531f1be2878c3eef7efdc68d637a3bfe2b6b074edde30ab6321bbb5ac1961
MD5 6f244f5e5b55b68e129df3d8ff11d4a3
BLAKE2b-256 36ef49449d135b68041bd5361d096c980221ef7306bbc02fd600c0a533d1a97d

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 431d382657812d9b24f2d16663841721187e7e89b137b8b34fdeee7809d66e88
MD5 ec97f0d9d3d0358d3f810116b678334a
BLAKE2b-256 fbb019d54cafbecf27f2130a62cf646ca414f2c15440ea26a14221644690e40f

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 476.3 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 e7ea611a956c508bd97d2a2bc17c3fcdec48bd1151ad05b0e251bf2e50994665
MD5 fb6726dc4842c28bc35638febec6d181
BLAKE2b-256 a57121ed822a3a572963dfbcac6b3e560cb9a9276c260718e0c9cd1a8e2d3876

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 398.0 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 6187bf2a37c2ca33a78bc424e874869b35593fcd059988487faa02f6b7b57f73
MD5 3c9b22ffc0b51ee01a001a45b94ca3c5
BLAKE2b-256 e89325f7fe80ca6a88fe521cdb053ef908b390acae93ff5570277afd33d5c29b

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a6b95e003a4c6668c102bd39cdeda7bdf38c16db46926a0b9f92800836e437f7
MD5 a3ae3b7be6748bc42c9cf870e2896af2
BLAKE2b-256 717124fa3dbedc7cd79c402558d70f8a96de1e2d1bf390abe48b3ecc38996b41

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.4.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7ee068012da1d262f2ae60798ad142cb7dd91eac21d04f379553924b91a46fff
MD5 7c78c3e84f343436db6b36bc8263b098
BLAKE2b-256 ef67c2b18e71b24253aa0502aea75e6d0d344c0de4284f3729ebfa7231630f28

See more details on using hashes here.

File details

Details for the file fastavro-1.4.5-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.4.5-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 473.6 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.4.5-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 48e6907088c058342635b577427c8d78ab18c8d20ca58a0c5365e5eb3157c6e2
MD5 e1b6fa7386e7fda959657cce1dcd5d48
BLAKE2b-256 520a318157d9b0b27c3b1623d2fd94a8e3a729b3d983aca951e292d70b542d5c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page