Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

# fastavro [![Build Status](https://travis-ci.org/fastavro/fastavro.svg?branch=master)](https://travis-ci.org/fastavro/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/fastavro/fastavro/branch/master/graph/badge.svg)](https://codecov.io/gh/fastavro/fastavro)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.5

  • Python 3.6

  • Python 3.7

  • PyPy

  • PyPy3

## Supported Features

  • File Writer

  • File Reader (iterating via records or blocks)

  • Schemaless Writer

  • Schemaless Reader

  • Snappy and Deflate codecs

  • Schema resolution

  • Aliases

  • Logical Types

## Missing Features

  • Anything involving Avro’s RPC features

  • Parsing schemas into the canonical form

  • Schema fingerprinting

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.

  • Get approval from discussing on the github issue before opening the pull request

  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won’t test on all versions of python or on pypy, so it’s possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/fastavro/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml

    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/fastavro/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/fastavro/fastavro)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

This version

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.21.20.tar.gz (478.2 kB view details)

Uploaded Source

Built Distributions

fastavro-0.21.20-cp37-cp37m-win_amd64.whl (299.7 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-0.21.20-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

fastavro-0.21.20-cp37-cp37m-macosx_10_13_x86_64.whl (359.7 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

fastavro-0.21.20-cp36-cp36m-win_amd64.whl (298.8 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.21.20-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.21.20-cp36-cp36m-macosx_10_13_x86_64.whl (375.3 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

fastavro-0.21.20-cp35-cp35m-win_amd64.whl (281.9 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.21.20-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.21.20-cp35-cp35m-macosx_10_13_x86_64.whl (340.0 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

fastavro-0.21.20-cp27-cp27mu-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7mu

fastavro-0.21.20-cp27-cp27m-win_amd64.whl (286.1 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.21.20-cp27-cp27m-manylinux1_x86_64.whl (1.0 MB view details)

Uploaded CPython 2.7m

fastavro-0.21.20-cp27-cp27m-macosx_10_13_x86_64.whl (342.5 kB view details)

Uploaded CPython 2.7m macOS 10.13+ x86-64

File details

Details for the file fastavro-0.21.20.tar.gz.

File metadata

  • Download URL: fastavro-0.21.20.tar.gz
  • Upload date:
  • Size: 478.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20.tar.gz
Algorithm Hash digest
SHA256 a3452d8a8b0029a3ba5c95090ac56d7546988e023542c27b37f29c2719af221f
MD5 104d0c85ac05735029498248088b8b3b
BLAKE2b-256 07e5578bf2243d4fd36bd7f3f0551830fca57dec0048f65d90fe79c52aaf6b06

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 299.7 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 6d6939618e7b7d57a86f892286429631e3c5a637ee1c9693d29264799f64a2fb
MD5 5ace47b9db7d854c089e4df52e45741b
BLAKE2b-256 a352ae10b7310dfc4de26c78402511715385b9336dc643d55340980da86b6b02

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 305e475851070536c717adb9581d84372ae576c2b253653011b6594f7668cbd5
MD5 bceb22263e532288e2c500f93114db0f
BLAKE2b-256 6960f475973a193aeb2cccda1857f4ae88405fdf933929840105be7d154bf139

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 359.7 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 1657aa2c117f18c377968be94949ac6df47f353c7f7a2f28b125b26b4400186b
MD5 bd9df7eed241657c54a5c7f8f961205b
BLAKE2b-256 c6e024f256757944dc6d0a5d657c5b44a7c56e3e1a0221f45d0a701d5ab52812

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 298.8 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 d617e42e8a86de23abbd4cd7637638a47f66515f8049797ff27f6817b8bf38ec
MD5 1126d3cd41ae61e3a86d539521c5e86c
BLAKE2b-256 d0be2e02a742f5cbeb30e1e2c75ece3662456cf214d51a63fd27d79802e86e2f

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 475614a33fc50cd2fd9e8712e5cb3e50fa47146a07d6d6d7a2fb7f3bb90b6366
MD5 3cc8b56b08817953c7a70464f98c7420
BLAKE2b-256 466d01ccc3f3cdace803ed0616359d071d1c499ac827cda3f816adee94c2bb6b

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 375.3 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 a1f8f5c6836b43557cefa1b3a7dddc81f61c67f4a7806815b5a9936c98e795f4
MD5 6436747be604f8006a9dbdae838b0f15
BLAKE2b-256 bf606c0a581b5a84ae8c8d1991ae5ee9ec1f6ee8dad833de306d85504a2df749

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 281.9 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 1fe4eede0caee564f6922ae7e26b6836a28630b294ec023ca9dd3991af6be4eb
MD5 22e63940ff205dce36e1600deb0181b1
BLAKE2b-256 93e506a975191f46dc4b0e04618fc1fbf0c889f5d4ce9f859ddf66e078b16467

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 071859b06ad7e4e4c4e62709e987fb1319c9d148e884f5fce27e3a73ed9ea584
MD5 87c3aa8ab05d60941a4a92badb8ff037
BLAKE2b-256 3b6e34f65ae1376ea15a16c8ec3818b299a83993d5359a140ba2c4eac2c20797

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 340.0 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 81016e87d975e146ff2ddb0ffad729685ef8d7fc9d962fb9a87ceb4961d113b0
MD5 79c01ac2eecc3660118ca2ac635f3239
BLAKE2b-256 7de010a66d516fc228746537653dcf6b65509ee7da8d46b5284624ff1d2d15f5

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp27-cp27mu-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp27-cp27mu-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7mu
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 710b49acb293058fafb3e4a954e75120eb8f6b8906ac7a537369ee5542ce50aa
MD5 ddbfdf545c9f1a97243ba4876e06fd8d
BLAKE2b-256 c931fda0abfd038684f8fd9bd42c7fb73a37f3eccabc6edf5bbfd746553e0362

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp27-cp27m-win_amd64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp27-cp27m-win_amd64.whl
  • Upload date:
  • Size: 286.1 kB
  • Tags: CPython 2.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 5b991c5084a85e82673ed4727e33a0a014c47b20522d2ead0a3c65760dfa0e18
MD5 211ec91b690a6caad5eb4481d6224363
BLAKE2b-256 53060571f56acd61236ec3d44e4922068c047e17263b006efb037a1883872485

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp27-cp27m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 2.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 26312419af3754d8a11384feaea3f2765d0ceb8ed360d467b6a25db267b82b39
MD5 17ee4dfde24443023d2ff1d281993d2d
BLAKE2b-256 ec7d550a5bc093e42a86c311ab97d0e5a5591e49f21e795b9643a029c3965284

See more details on using hashes here.

File details

Details for the file fastavro-0.21.20-cp27-cp27m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: fastavro-0.21.20-cp27-cp27m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 342.5 kB
  • Tags: CPython 2.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0

File hashes

Hashes for fastavro-0.21.20-cp27-cp27m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 460b013b00e15d398c73a1362950f484955529047aad90fb04cacc7c3c62e702
MD5 c10583fbeebd97437e8478d1563b063d
BLAKE2b-256 8b4f8e1fbee34b4c6804fb49a72f494d2c01d04319a2a159e7fc25537a62c1fc

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page