Skip to main content

Fast read/write of AVRO files

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.5
  • Python 3.6
  • Python 3.7
  • Python 3.8
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types

Missing Features

  • Anything involving Avro's RPC features
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.0.0.tar.gz (658.4 kB view details)

Uploaded Source

Built Distributions

fastavro-1.0.0-cp38-cp38-win_amd64.whl (353.2 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.0.0-cp38-cp38-win32.whl (310.3 kB view details)

Uploaded CPython 3.8 Windows x86

fastavro-1.0.0-cp38-cp38-manylinux2014_x86_64.whl (1.7 MB view details)

Uploaded CPython 3.8

fastavro-1.0.0-cp38-cp38-macosx_10_14_x86_64.whl (435.2 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.0.0-cp37-cp37m-win_amd64.whl (341.8 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.0.0-cp37-cp37m-win32.whl (301.0 kB view details)

Uploaded CPython 3.7m Windows x86

fastavro-1.0.0-cp37-cp37m-manylinux2014_x86_64.whl (1.4 MB view details)

Uploaded CPython 3.7m

fastavro-1.0.0-cp37-cp37m-macosx_10_14_x86_64.whl (426.4 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.0.0-cp36-cp36m-win_amd64.whl (340.5 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.0.0-cp36-cp36m-win32.whl (300.5 kB view details)

Uploaded CPython 3.6m Windows x86

fastavro-1.0.0-cp36-cp36m-manylinux2014_x86_64.whl (1.4 MB view details)

Uploaded CPython 3.6m

fastavro-1.0.0-cp36-cp36m-macosx_10_14_x86_64.whl (445.9 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

fastavro-1.0.0-cp35-cp35m-win_amd64.whl (323.0 kB view details)

Uploaded CPython 3.5m Windows x86-64

fastavro-1.0.0-cp35-cp35m-win32.whl (284.1 kB view details)

Uploaded CPython 3.5m Windows x86

fastavro-1.0.0-cp35-cp35m-manylinux2014_x86_64.whl (1.4 MB view details)

Uploaded CPython 3.5m

fastavro-1.0.0-cp35-cp35m-macosx_10_14_x86_64.whl (405.7 kB view details)

Uploaded CPython 3.5m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.0.0.tar.gz.

File metadata

  • Download URL: fastavro-1.0.0.tar.gz
  • Upload date:
  • Size: 658.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0.tar.gz
Algorithm Hash digest
SHA256 c37c640f0e843880c6db5f314dd284d4c8ce79d40b9cd00194042244aa8bd626
MD5 aae0e7664ccc0899c187490d7d15b9bf
BLAKE2b-256 19b13fc23eb5c5f32ec69dc072bc383c79ed013eb1fb178cae4e0b9576527ea5

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 353.2 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 644bbb7162f21ccba5beb420ef2c1ad9580dc60c1347f9aa523c1427b6842cc3
MD5 81077684924818ece71f2aa2b86a7de8
BLAKE2b-256 6aa4e895d783a189538a06e2009541015b49b93d32b38c69c55379b3bc4d6f1b

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp38-cp38-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp38-cp38-win32.whl
  • Upload date:
  • Size: 310.3 kB
  • Tags: CPython 3.8, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp38-cp38-win32.whl
Algorithm Hash digest
SHA256 fc26506e34a630fbb344c82ca63c38a66cbf0c43b919a3f94f80cb561c7c2649
MD5 d9172324cdfaf1c96b388779ee02ec90
BLAKE2b-256 da5dae34119750839543908b062629d9046450b0dd7564075f36ac45c97b53cc

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2bad4a2e861e62ba4a802f479c1ad2985b9f9be05ffbd582c4ba05f91c086161
MD5 aaa9614f5e0eea5cdcb1a252ada110c3
BLAKE2b-256 1e4f8d113ce4e51f8488f036244d8b03b239b96d7c89bc4a23ebcc61ad1f7d31

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 435.2 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 69e1ba0cfe6d5377785bf2d7aa9313e953713ceefed68160d5016d96b5e79b09
MD5 0fa1b751838f3e548bed11cde69c5690
BLAKE2b-256 f3085f8db6664b8042ec645291eca2b9ed353c3def69b79db5e57bb68166ee89

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 341.8 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 9509e54b5ae30473e891ae6fd9a1a419d7069a89700da526d22f6629dbe09641
MD5 d8de2f4e99b28ebbf138e41cfa2cb47c
BLAKE2b-256 cd4e43be438796b8f76c4b8b1046dffd291a889c36ca59aec1127153ce202cd1

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp37-cp37m-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 301.0 kB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 20bacd72d6ae789626ccc48bad5a577dd67805e9f831aa44c2a481541d2c3ac2
MD5 fdb5148d4e198c2218fc685f0fcc7fe8
BLAKE2b-256 24e545fefc9ae155e4e2b7e8809acbdcf8b524ca94602352b562b96f8c2d143d

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ceb63afa7e8185692b13259b9c2368c11bda3b2dad989b13dd9119d417aba132
MD5 8cd19d6df1c645ec0e308ab002c63994
BLAKE2b-256 166fca36eba7cdaf611505b302af19f75e4f93508fa225c32d641b257d2c05d4

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 426.4 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 46da3161cacc69c5163c8a2c06f469d8a2adaa753cdbd2e6524e6c191bdc1f2b
MD5 baed25c9ecd3198824eea13a6d6d3375
BLAKE2b-256 402c7fd3e140078c0a9b6fb53f8c58db110639ae490ae7bc901dcb209386cc40

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 340.5 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 c3f922c9b65b25fb11ab8741c120c9545cfeff88b65aad5810bd08ebc5562274
MD5 815505a7bb890784ccdec7e71983ee5e
BLAKE2b-256 5a961653909b8d4a836897ce76e825784d6e2de4449426bc0dc721d28c9edce0

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp36-cp36m-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp36-cp36m-win32.whl
  • Upload date:
  • Size: 300.5 kB
  • Tags: CPython 3.6m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp36-cp36m-win32.whl
Algorithm Hash digest
SHA256 5c93781a6b11c46352242b7a0129f91d32c0e73274153954bec0c0c66921af8b
MD5 5d48cd762dd11ce39077ee638565a8d1
BLAKE2b-256 b59958821782dfbefc45cd1721f67ddf231675e115482d54a899980ff77d2c45

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 aef80e52d41c7ad47519316ea864a3a06ac02d2d1e9d00242aaef455ece615a6
MD5 f55aed2a326d922dbe34974b675aab7e
BLAKE2b-256 1e0d70f138f0f8b565099d85c40ed975c1e66e9fd585fda32d5d07dbda5a5eba

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 445.9 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 b39e26759bc9c06c7b501432dd626252e049dfce13d8739e550aae767c5cf816
MD5 9e1666f779ffcd7c22f4db7fb98b82e7
BLAKE2b-256 56d8a099e4b0af4e635beae0162179c2f91c40595492f76e1db2d6962b83031c

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 323.0 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 873804317274824dcb78e40a7a04aaa62de121a412b7abcc4d8939b7e7d11582
MD5 c49cf2d92552bd4b582496ea38d87fb8
BLAKE2b-256 e69b5e0ee24acc1712a0f47a37761b47eede861b8ddffe764e1e113bc5148228

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp35-cp35m-win32.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp35-cp35m-win32.whl
  • Upload date:
  • Size: 284.1 kB
  • Tags: CPython 3.5m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp35-cp35m-win32.whl
Algorithm Hash digest
SHA256 0369acf12ee20accead91f39c0368d4352c5b4acf1333c5da9d8ec854bec62d2
MD5 f9014d4d5830e628a320b340583a646f
BLAKE2b-256 169e8b4179cf3763c9feb3a647fffb3f8ba6e30988f8a94089cb1bc184fa0557

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp35-cp35m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp35-cp35m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp35-cp35m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 af26feaff7800a0e23150702a1cb6732c2e8cb5f0e071e036734b44f2f8df253
MD5 1421151646559a281530771e1bbca372
BLAKE2b-256 0c19f14d9d7a6a6c80e3e5ac52bc49aa3afd26e4f0493ee47a5376f986b1fe2e

See more details on using hashes here.

File details

Details for the file fastavro-1.0.0-cp35-cp35m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.0.0-cp35-cp35m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 405.7 kB
  • Tags: CPython 3.5m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0

File hashes

Hashes for fastavro-1.0.0-cp35-cp35m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 06f3c029b34f56a30c6aef3a49df39426b548e8522cac456d617704c77a2eaf1
MD5 ba879dcd40a60d2f816c408aa7d22557
BLAKE2b-256 fd4472bd0d0458609735607cd68521d97d54d47f9720c4ea04d0aa4ac9e2dace

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page