Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml

    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

This version

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.4.tar.gz (451.4 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.4-cp37-cp37m-win_amd64.whl (282.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.4-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.4-cp37-cp37m-macosx_10_12_x86_64.whl (325.9 kB view details)

Uploaded CPython 3.7m macOS 10.12+ x86-64

fastavro-0.21.4-cp36-cp36m-win_amd64.whl (282.1 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.4-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.4-cp36-cp36m-macosx_10_12_x86_64.whl (324.5 kB view details)

Uploaded CPython 3.6m macOS 10.12+ x86-64

fastavro-0.21.4-cp35-cp35m-win_amd64.whl (274.9 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.4-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.4-cp35-cp35m-macosx_10_12_x86_64.whl (323.1 kB view details)

Uploaded CPython 3.5m macOS 10.12+ x86-64

fastavro-0.21.4-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.4-cp34-cp34m-macosx_10_12_x86_64.whl (332.5 kB view details)

Uploaded CPython 3.4m macOS 10.12+ x86-64

fastavro-0.21.4-cp27-cp27m-win_amd64.whl (277.6 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.4-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.4-cp27-cp27m-macosx_10_12_x86_64.whl (327.8 kB view details)

Uploaded CPython 2.7m macOS 10.12+ x86-64

File details

Details for the file fastavro-0.21.4.tar.gz.

File metadata

  • Download URL: fastavro-0.21.4.tar.gz
  • Upload date:
  • Size: 451.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.21.4.tar.gz
Algorithm Hash digest
SHA256 02aa0af491f726fc59bfed417020226fd00d8d766762e25ef08ad935a9b98a7d
MD5 fea3137d66a598fdb4de6c14e2943880
BLAKE2b-256 378d58f3188f5c445fab00c0e4a1f35a9bafa6f4f42dfe9081b29e7ce9a4b51f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 443b8f926927011058d420adad0a8823915beb7bb31364641f59df7ac7d0afc9
MD5 298e94701f005c6655d5bb93c84e3679
BLAKE2b-256 911ca66bded90a24e3ecb381a7624c0f8cdb3fb575a2a4b6d80752ee731424cb

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 cd9f60490fb8b198662d9a1f2271b94d88274bf9fd86515087b89c9cc11f0892
MD5 2cd322ee4f24c5b6fe3bb4e459571799
BLAKE2b-256 fec884e47948ab82ef1993414d6350cc442c5a13396bf4c571939e40b8fbccb9

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp37-cp37m-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp37-cp37m-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 3a33902ad81ed848d83e9549a1871175763ed8be24976ab4420c6bb11d988aff
MD5 3757757d282ccf1826133d9cc012d9eb
BLAKE2b-256 a1e59a2cbb0a2b11927733b4d1f35ef2bdd7f54e54a5711c8fbc7b6472e0c8c7

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp36-cp36m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 8f132954e943b2fc78907c4fd1d7d7bc080245c0a0ce9a465592e50d6a0b73f0
MD5 37b0e1ad201d24c13c8e14a4e43e821a
BLAKE2b-256 3aa5b357909eb300ae3a8499f1718b3887b379e743553bcc2dc2ed325902072b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 f1340475cb4c47ef6f61532f49d76c5846ce139d6704aa5ed30ca5f100a7e8b3
MD5 fea52b26fd79e3ebb4f684547b655ef9
BLAKE2b-256 497f78c7e263021259454d1c92951c6e9769a809a572b554ef7409a1cd027283

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp36-cp36m-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp36-cp36m-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 d155fece5cf37fb8127d5344859e0ce32b82c4f6332ec70c717c6dee12bca563
MD5 935a4a44a5d79fb96f4ef28846fdfe29
BLAKE2b-256 4d3a61691905316026f41a1fd8e9e3638e0d34b07f618a7a11026f2915aea494

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp35-cp35m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 369ae473de1fee3dca2fab25cad3fea128e40d5b35f9e0fd500cd8a7f96a53e4
MD5 294758b98c3c4263954f366125cce971
BLAKE2b-256 b2dc982f002a670f9ee5dcbc7bedcb6d374adc802db4a9bfb3e3e80d43d19c01

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 a0375604f298989304c46041c0eab4740f9de9d2e3d520374b7a609cfa328579
MD5 f8df22e4bda89ddcfeda71b85230d242
BLAKE2b-256 ba7508e2b98424ce1bcf34d77dd51cf0890b91daf88fc616c92dcd004fd6e532

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp35-cp35m-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp35-cp35m-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 2c201d45149bfd28fd6aec309aade2503bba23b0e2224ab473235965b02b31ec
MD5 cde4f42c5eea3d8a692dd0a8143a0626
BLAKE2b-256 a8eaa993b7818d0b8b4301e495db0a81bf79613ad490e96c25ee3cbb1fc06104

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 887d5480df4a6c835b4e690fa3b531c2d1f018a6a54c5a8cfab022fcee0be735
MD5 1a3c1922b14c2e94373ee33fde1aadaa
BLAKE2b-256 860ebf7e6af8dcdd1028f8e1c269a2c8340bb4264462e05d6d14a594812a59bf

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp34-cp34m-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp34-cp34m-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 8802d5bff22aee1d1330bcff669bd9bae24432dbbbe06be99a31f5d39dcd2b06
MD5 d8d08d92f138dbc90d6e412429d61bc0
BLAKE2b-256 0d153ff1765adf5a96606d2f79d4b6fe9117ae90b204ab8f9954d4a148ba0ab3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp27-cp27m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 fc8e311341b6b267221ee199e217d0e3347bea41d54ced0775d0bc1a364e12f6
MD5 183e6d3f9420e11ed3ffb491e2d720f2
BLAKE2b-256 d3547014774956d691e12bc6fd8af27aa83fb87affbce68210ffda2868d26ba3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 17b325ba3e0d61427506892b28602c029be48365363d4a6d4a338e594d3d928d
MD5 93c7b3ac91aed63a20bedbf78eaff1b9
BLAKE2b-256 c729fcb47a815d61ecb677ab0dc902f8e35fc92059b6118b2599c1a639d97fd0

See more details on using hashes here.

File details

Details for the file fastavro-0.21.4-cp27-cp27m-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.4-cp27-cp27m-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 04bf2ff06eb0fee2d15f7c0bc20b96a1843e39cb6c3afe979b09f718ed8a8b67
MD5 8d5ff8fb7c81e9ad962d70167e01b280
BLAKE2b-256 5d8f8ae4d4710f5e9c3fc47e0bb8620e14c90b3666346c71f311c421e1965d6e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page