Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml

    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

This version

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.2.tar.gz (450.9 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.2-cp37-cp37m-win_amd64.whl (282.0 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.2-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.2-cp36-cp36m-win_amd64.whl (281.8 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.2-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.2-cp35-cp35m-win_amd64.whl (274.8 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.2-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.2-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.2-cp27-cp27m-win_amd64.whl (276.7 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.2-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

File details

Details for the file fastavro-0.21.2.tar.gz.

File metadata

  • Download URL: fastavro-0.21.2.tar.gz
  • Upload date:
  • Size: 450.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.21.2.tar.gz
Algorithm Hash digest
SHA256 8cb16df617a3702c1be556eb3d2fdfa1654360f06b80696477715d95d7417da8
MD5 9089ecc5e603898a41a11eb86c433ca4
BLAKE2b-256 fd5b53363fd4f3a6bf72952532152f834928bde8e747e70cfd5ba03453b7e1e0

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 e01da6535eeb55c210f041c5983edb85a6cfbb619d4f22b1f38eb97fa03f0e2c
MD5 524ef5cc1947e1a32766626deb74555a
BLAKE2b-256 51cb9889935620b4bbd4d0ebd4a777d0a2d7b3c453c2f3f1cb3c1f44ab95f600

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 cda7626d63e28390f9ab4bcb742e754406de5c606e03240681e0ea6585daaa56
MD5 26d75045bd652a5d43957630e6d2e86f
BLAKE2b-256 a530c2498a009cf999a23fe47aec15546ed09e84fc33d1aa5f16fd12f60b72a9

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp36-cp36m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 f332a5b0135d073afda33aa6cab18eb1e4de8d64a1031da5b449c150fae632b3
MD5 2cedf7e32cdc91f45b4a0bafec5dbe33
BLAKE2b-256 cb3d931640d09f1be28352a1b41bf6215f1b134b2cee43b8ca182761ec2e91ee

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 f30522dc5f52d28403838c6d9931232fa938ad592ba27a0d1b7496fc40fa4658
MD5 7afc3065f50f52c2c26e79a60709baf7
BLAKE2b-256 93b72aa46b27ac1ec1c57e1e3dc227e13150fd7ae2e44f4c3e951dacc9ebf95c

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp35-cp35m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 591d21343e982672687a77fb384c6df130e54506a8560eed1b9fc390e2493f1b
MD5 878d9cf02462e62af91308438088b5b6
BLAKE2b-256 5da1aa7ba1dbfdb6e3b977ea2ac2e4f5511490821b9e82ac8fa5aba9484dc2d1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 531ada8064014cdcd4dc98356982436edf370e3458b42016ead5d7793349035f
MD5 aa4c3e0d2bc3477f3f243744b77f399d
BLAKE2b-256 472adfa824e37c969dffb50236cb224ee50c715883ba4a6b81664ba37b161131

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8286742ca1da2c202b9b10a5f8c2402cfdf38fbf9214e8d6c355d01c85d2e347
MD5 d1fd1bdd0227275986a35ad41f6a51c3
BLAKE2b-256 a9083e3477a7117f6c5b2f48c1a97ee7b88a0afdb90707d23e1fc4bb6434e138

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp27-cp27m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 9fe4068f60b6461865245d4ca3d126704fb35fb0461e321a0f04d8ff09c61694
MD5 ead32a33c87807e08eaf4bf5c50ec3f7
BLAKE2b-256 dd8c29ef5b89775294f68f7c2db1fe3935f93c65231f81e0664f627c0d2a17f6

See more details on using hashes here.

File details

Details for the file fastavro-0.21.2-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.2-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 51280b525ea8e483939dc62693371bf3915616e427fe15d792220c2073c0ba0e
MD5 5ebd6585ef95276c4548888cbed0f5f8
BLAKE2b-256 f68a749f48a0d2afd02dc0a7cc22cadb7e896f70b48fcef57f722938a521a3f3

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page