Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

# fastavro [![Build Status](https://travis-ci.org/tebeka/fastavro.svg?branch=master)](https://travis-ci.org/tebeka/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • PyPy

  • PyPy3

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Hacking

As recommended by Cython, the C files output is distributed. This has the advantage that the end user does not need to have Cython installed. However it means that every time you change fastavro you need to run make.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml

    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/tebeka/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/tebeka/fastavro)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

This version

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.17.10.tar.gz (443.0 kB view details)

Uploaded Source

Built Distributions

fastavro-0.17.10-cp36-cp36m-win_amd64.whl (278.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.17.10-cp36-cp36m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m

fastavro-0.17.10-cp35-cp35m-win_amd64.whl (270.9 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.17.10-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.17.10-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.17.10-cp27-cp27m-win_amd64.whl (276.1 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.17.10-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

File details

Details for the file fastavro-0.17.10.tar.gz.

File metadata

  • Download URL: fastavro-0.17.10.tar.gz
  • Upload date:
  • Size: 443.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.17.10.tar.gz
Algorithm Hash digest
SHA256 5f9b0145a00cd5b737fdc173708431c202a0f6c9e6385f43809b124fbfcffb83
MD5 dcfe4860053d80fe7def3e7eff16d529
BLAKE2b-256 9d372a3cef693885236264e3b1c1339ff2afb63853a11dac2d433892ff93c8e4

See more details on using hashes here.

File details

Details for the file fastavro-0.17.10-cp36-cp36m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.17.10-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 1310a71975dc00f15345dfe7cdcef2792e0362359ae94c3acc2b63a723ec8ec8
MD5 7be4d1664e698dc76495e45c5526863f
BLAKE2b-256 939caf4d3114bdc26b4cb26f7f43f5859d253c0593111e1bbcd44323a1ba936a

See more details on using hashes here.

File details

Details for the file fastavro-0.17.10-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.17.10-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 5e96785bcea54814c1ffb4efa13e03456a55fffb87b9ccac883523c055a6cdc1
MD5 1c22a6cb70a0fc11e90a3c06d9079ea7
BLAKE2b-256 63ff7a58ef4a1884f125ac30beced41f1497c367d715844913f2e17f8fcef16c

See more details on using hashes here.

File details

Details for the file fastavro-0.17.10-cp35-cp35m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.17.10-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 927c91bf22eae2129dd857b47d1cdc019ded9134dca1d036385f499c027161e3
MD5 4806ca95a65954d2bbecb5706e5790a5
BLAKE2b-256 f11db9beb8299c8a9c45303c2833a69f965cd0421fc0f27cb2ec76fea9a8810c

See more details on using hashes here.

File details

Details for the file fastavro-0.17.10-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.17.10-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 4aec221a3de46f67d4139c97f95041f6ecdd31e80a15ec09f5d44cde703142e8
MD5 cab03c27fc024ff35c945bdbb4ed5be2
BLAKE2b-256 7ab04c25dc4f5ebf9c49fd3969ce011e7e1a1de8fe46b1e332620dcf5bb3f88b

See more details on using hashes here.

File details

Details for the file fastavro-0.17.10-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.17.10-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 3bd1af436f4c868fad06956f9ab2bb2b778d2e9b93a3deb8758f54abd7aa79a3
MD5 92c332afbf90f0c2021afcbd2ec98c09
BLAKE2b-256 e961c62ac711d599619ef1d3d1a7e1cf4e52fc1fd924f8309a7f791e24f6848c

See more details on using hashes here.

File details

Details for the file fastavro-0.17.10-cp27-cp27m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.17.10-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 eb11d8a96ff3f3f6a141f1d77e847bc06626f8e3c260ca10c35b4032cbb56bf8
MD5 06ced49d6e2dabae2e9b7c8b2067f05f
BLAKE2b-256 ef388f0d829393264f628773ee32607ad2549121c115de544526f9b2c653dce2

See more details on using hashes here.

File details

Details for the file fastavro-0.17.10-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.17.10-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 4acbfeee3733e4ca87c8a523bf80363713c24f93581ea9dc2d9108effdabc369
MD5 c5fa070636fb9f3f632ca0c47215a584
BLAKE2b-256 62e8e398a6d810b8801c0409f93ae8da96687351d4bc5f4b06e553371c669354

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page