Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml

    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

This version

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.1.tar.gz (450.8 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.1-cp37-cp37m-win_amd64.whl (282.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.1-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.1-cp36-cp36m-win_amd64.whl (281.9 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.1-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.1-cp35-cp35m-win_amd64.whl (274.9 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.1-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.1-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.21.1-cp27-cp27m-win_amd64.whl (276.9 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.1-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

File details

Details for the file fastavro-0.21.1.tar.gz.

File metadata

  • Download URL: fastavro-0.21.1.tar.gz
  • Upload date:
  • Size: 450.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.21.1.tar.gz
Algorithm Hash digest
SHA256 d9377c054b2e8831696504063dea1cfe8c7e318b93334a528dddb76e2e3e4068
MD5 01458004992164fd1ea6e1cb50b21e3b
BLAKE2b-256 fd529b6b163f53171ff8fcc4b42f7f4ba33b8bbd252d0af614cc9d422bef7060

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 3ea7315c3dc4f0a1f49cd64c4a6521e419f3305804e622dc480ebe0f60dd3280
MD5 fa3d4187666289bed4b9d25c166d82f1
BLAKE2b-256 e49a1020e3d2068aeaa8bb2b422350f2910a63415490b2db73f7d3b671f4a41c

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 b42a6980f67c4df5da9e7513df6b667c730c5cade0704b06f8a7879a48adde23
MD5 6339dda0f96651f5f1a08c9a7eee95fd
BLAKE2b-256 5d1ef6d90b47bd3a80b3474051963ba1b6edcf18c16a3cdcb770b73d824b17e3

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp36-cp36m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 7d3f584b96e4ef78dbc7d2eafb943b980d1e4ce0200124b5f29972d3e31e63d2
MD5 7e993ba17aab62cbd67eec5eb1402bfc
BLAKE2b-256 343f36da9cb38e748e4301a54c2a4dc13b2dafdd3ad91acf030f2c71e26fc2cd

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8a6981610d28833e2a970213c105380db4be71cde839e70343460948632bb826
MD5 fe77183fb45b6d30dfc88ef1db1cd353
BLAKE2b-256 cde3bd9e2abc31fa6bf2eb8ea98e0a9a80e89316194da01987e4a6996d1677fb

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp35-cp35m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 de071af0cee71aadafa07b4c2af4808b506e4d8409b427e171fca837d62b5b99
MD5 108b37af0afd9fa0e5c01df18de49084
BLAKE2b-256 0b0a830e7c3ebd70c92bf1f9d165d5029bd376816ec461d358fb3d1b2b7b20c8

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 65e7a8b36e5e697d59ba870fe4460db54fd2e7055d129efd5e21cdd95d91bbb7
MD5 19f1ec5ee47cc510b5bb4516b5c2781f
BLAKE2b-256 d0cc7c86dfe746322ecea82ff4c06a33a4d05946a3835fc976a0015e33478d92

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2fbdd21259d3b73dea4663beb36f240075ec471c17ee174b2525864a5cc5ff4b
MD5 9e71bf0215e7ca1262f8e27e155d2563
BLAKE2b-256 ffd01ef6d26b9f9e8517e1d0b4b09a6720ed4ebcfab0a7f9c65001a717e58923

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp27-cp27m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 56a3fa01e388f286a68d69f4052fd34b5eb5d7b4e7f40f2797cc18b6cdb188a1
MD5 11b89da067a3eaad3b7b162f429d5164
BLAKE2b-256 707d73ba6cbbb5eab79a8b0a72d4855995306facddc89973450f4a2a82604716

See more details on using hashes here.

File details

Details for the file fastavro-0.21.1-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.21.1-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 1505e69769adf3d7610200a86c9b12d70cdaf404207c6fe6d151e5c364c12da4
MD5 db1f1cc1d4eb23092e797e4e9c0c7b18
BLAKE2b-256 0c7a9a05bf6b9695d41c5c3d0c4455f52bad96666cddc2005c6b8fff0653195b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page