Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

# fastavro [![Build Status](https://travis-ci.org/tebeka/fastavro.svg?branch=master)](https://travis-ci.org/tebeka/fastavro) [![Documentation Status](https://readthedocs.org/projects/fastavro/badge/?version=latest)](http://fastavro.readthedocs.io/en/latest/?badge=latest)

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 2.7

  • Python 3.4

  • Python 3.5

  • Python 3.6

  • PyPy

  • PyPy3

[Cython]: http://cython.org/

# Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

# Installing fastavro is available both on [PyPi](http://pypi.python.org/pypi)

pip install fastavro

and on [conda-forge](https://conda-forge.github.io) conda channel.

conda install -c conda-forge fastavro

# Hacking

As recommended by Cython, the C files output is distributed. This has the advantage that the end user does not need to have Cython installed. However it means that every time you change fastavro you need to run make.

### Releasing

We release both to [pypi][pypi] and to [conda-forge][conda-forge].

We assume you have [twine][twine] installed and that you’ve created your own fork of [fastavro-feedstock][feedstock].

  • Make sure the tests pass

  • Run make tag

  • Copy the windows build artifacts for the new version from https://ci.appveyor.com/project/scottbelden/fastavro to the dist folder

  • Copy the linux build artifacts for the new version from https://github.com/tebeka/fastavro/releases/tag/ to the dist folder

  • Run make publish

  • Note the sha signature emitted at the above

  • Switch to feedstock directory and edit recipe/meta.yaml

    • Update version and sha256 variables at the top of the file

    • Run python recipe/test_recipe.py

    • Submit a [PR][pr]

[conda-forge]: https://conda-forge.org/ [feedstock]: https://github.com/conda-forge/fastavro-feedstock [pr]: https://conda-forge.org/#update_recipe [pypi]: https://pypi-hypernode.com/pypi [twine]: https://pypi-hypernode.com/pypi/twine

# Changes

See the [ChangeLog]

[ChangeLog]: https://github.com/tebeka/fastavro/blob/master/ChangeLog

# Contact

[Project Home](https://github.com/tebeka/fastavro)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

This version

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.18.0.tar.gz (445.6 kB view details)

Uploaded Source

Built Distributions

fastavro-0.18.0-cp36-cp36m-win_amd64.whl (277.8 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-0.18.0-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.6m

fastavro-0.18.0-cp35-cp35m-win_amd64.whl (270.4 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-0.18.0-cp35-cp35m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m

fastavro-0.18.0-cp34-cp34m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.4m

fastavro-0.18.0-cp27-cp27m-win_amd64.whl (275.8 kB view details)

Uploaded CPython 2.7m Windows x86-64

fastavro-0.18.0-cp27-cp27m-manylinux1_x86_64.whl (1.1 MB view details)

Uploaded CPython 2.7m

File details

Details for the file fastavro-0.18.0.tar.gz.

File metadata

  • Download URL: fastavro-0.18.0.tar.gz
  • Upload date:
  • Size: 445.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.18.0.tar.gz
Algorithm Hash digest
SHA256 5fc261ef01534a2a310953c63e4629a6cb3aec8538cc744a3cfa5a54107db18f
MD5 e91097957eda924da5d6124b09c44f94
BLAKE2b-256 951bfbae76a43afc7eec823a2346cb2b5ae18c2c872a10a44b63ab0d6e862dee

See more details on using hashes here.

File details

Details for the file fastavro-0.18.0-cp36-cp36m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.18.0-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 7eff8cc864c11265a97764c4f1b3dd691a8f4a971390492b989cc2df8a092e3e
MD5 39e34062ba76d4c42596565aa348eca6
BLAKE2b-256 54ecca96247c6d4efcd3158ea04635a4dd33621e39dbe460364ae69d2534847c

See more details on using hashes here.

File details

Details for the file fastavro-0.18.0-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.18.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 1f2e255c36938a3e2f1d664882014977d7aae4efcca1a2c7a86215fddc04d9cf
MD5 8db764d4347267d23e1a9961d94c5c26
BLAKE2b-256 6fcf157eb10f8324ceee6351e3802e8b58fd3e28165703d138995f0989ae1220

See more details on using hashes here.

File details

Details for the file fastavro-0.18.0-cp35-cp35m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.18.0-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 15db5c608caf2cb0fb21f45fb5c453bb67a85dc41c86b18eb8c6ab698f502102
MD5 57805152fcde550d44606a99bf391d70
BLAKE2b-256 af90352ec255bf90c3eb6bf4381da480e50d89068c200bca7bee236d23503e00

See more details on using hashes here.

File details

Details for the file fastavro-0.18.0-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.18.0-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 20bde7c8ad6997979c1cb2bdee3203f1a69075ce841a5807892be83be67bc66b
MD5 c21f7112c3cd1042fb54f53f7c438a5b
BLAKE2b-256 3aa979685b3518671e6618a9923a72a3c09c4a52e2875b9544d43846a5ce94f4

See more details on using hashes here.

File details

Details for the file fastavro-0.18.0-cp34-cp34m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.18.0-cp34-cp34m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 a28b11dd7fe04af31048914883797e2bba7f5916d753b95a41e84e87f10e2431
MD5 c7494542dceeb4717ab714ac709ae969
BLAKE2b-256 7e75dffcd85894206e37290fb0524a3ad743193dc8a193fbe9abf3b498c56464

See more details on using hashes here.

File details

Details for the file fastavro-0.18.0-cp27-cp27m-win_amd64.whl.

File metadata

File hashes

Hashes for fastavro-0.18.0-cp27-cp27m-win_amd64.whl
Algorithm Hash digest
SHA256 ccb2a3c4018edc39a122544b07e8d78dd022108f465341b1419810c52090b74a
MD5 d299dcd1760c065d7bbfca2aa7560413
BLAKE2b-256 2816880cf16dafcafa61cc6324364304f6e3c0ca656e80d62ce9d9faac16d75c

See more details on using hashes here.

File details

Details for the file fastavro-0.18.0-cp27-cp27m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-0.18.0-cp27-cp27m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 98d99c090ae3f19aaac0b870e5c5e2631b4bc58e798b369ef236bfabd2658060
MD5 ebfa93f51aab41bacded7850513f634f
BLAKE2b-256 8973ba96b3a9bdfc2085a9579ad718f5a9e1ccf7ae0c1dcba2844fe6f6f90980

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page