Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml

    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

This version

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.3.tar.gz (451.3 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.3-cp37-cp37m-win_amd64.whl (282.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.3-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.3-cp36-cp36m-win_amd64.whl (282.1 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.3-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.3-cp35-cp35m-win_amd64.whl (274.9 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.3-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.3-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.3-cp27-cp27m-win_amd64.whl (277.6 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.3-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

File details

Details for the file fastavro-0.21.3.tar.gz.

File metadata

  • Download URL: fastavro-0.21.3.tar.gz
  • Upload date:
  • Size: 451.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.21.3.tar.gz
Algorithm Hash digest
SHA256 910cbd2b001d928b5d5ceb690ab44cb77445488a1c9c2050bf5acb90571d99bc
MD5 4bad586a59802b954b4db1566112662e
BLAKE2b-256 eafe156463ec870b6f393832f59617c85d3e70af7c3699590ed9c435eecd898e

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 bbb57f696a6f1c507fefa34cdc6073025a9ababf9c1fa054b689bbf292281627
MD5 bcb685161e1e540ce732cc61e63f0a4c
BLAKE2b-256 acf7711b6163c81204ad0733b3b149f1b6d0763d62bb31706243831a65034cb3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 7f34ac676307af1cc5532d7deae96f27ddff9658ebc0d73f9634c3eb135d35a6
MD5 5f0566792d7eca38007f9f00cd68940a
BLAKE2b-256 e52c0a1c5d7b0d53c45936c91e7853ae7422240174f710de41d23d8559e2e2ae

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp36-cp36m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 33fa1737f772d4b5778f72b19cd0b96652fca25bc15cf190601493e48cdd6ce8
MD5 d28dfdf96a943776973661df40838e67
BLAKE2b-256 6034c2100acabf17ad0246b6af7493bc5812d83924f54a6bc433e53ea2677ac1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 6d62a391259bbcf98947dab8bbacc209adafcff29f1a70bb3090951546e9047f
MD5 76abd28e6dff97e9ac92ed301b2bccb4
BLAKE2b-256 3ce2dbe75d256fb5a9c7bf7aba713cc8d98ffe97a9f9c4cf5cbe2862889ab25a

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp35-cp35m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 e125d9c1fd5cec73ba880f29afc5bc3e29d5a70435cb2f143d0e827acf59f815
MD5 bfdf97bc115f04e391d2363c27132c36
BLAKE2b-256 160e41c454f62f2aa0046e2e692adc8e5dd24f94e903d1871eefa85db03db144

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 ed7559541f8d6da2d9caa8150fbd432a3929fd204bec122bf323959482235afe
MD5 c8c71dd62efb4e23e82a0103a2ecdadf
BLAKE2b-256 711e9e52e1d2dcf565eb8626a0b5b0600ece5ce29544604a0a73366a68bd76cb

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 4a3e756a4d69deabfe9e4947f998d2ef9939d5b0cc3d81f4f93658d27ce89b6b
MD5 02f7b57db0de943bfac4be1afb7427cb
BLAKE2b-256 21848d9a3b3e5f608b5f18d77373313b07dcd5f6d4fb22d074c8ec838c2d3451

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp27-cp27m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 f52c9e6e471df28ec6adebb8a11fae4e7a2e8bd199a87ffc583fb0801f2beb3c
MD5 7bcf9180076fdb9caded4bec23b57add
BLAKE2b-256 5f346751f5edce782478ac4a88cf2808f148fe169a86dbf5427f412f417880f1

See more details on using hashes here.

File details

Details for the file fastavro-0.21.3-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.3-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 fb1bb4da38da6923624dbea403fba85f30eeadeb3a58abc7641afd305630c06a
MD5 bc483c73aa312c34aae29a5f4da41a41
BLAKE2b-256 9b898eebd1bca67e1d5d7d044b7486b61f3f3cec4c76514fad4043148b8f1d5d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page