Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

This version

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.3.3.tar.gz (706.6 kB view details)

Uploaded Source

Built Distributions

fastavro-1.3.3-cp39-cp39-win_amd64.whl (401.7 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.3.3-cp39-cp39-manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.9

fastavro-1.3.3-cp39-cp39-macosx_10_14_x86_64.whl (471.5 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.3.3-cp38-cp38-win_amd64.whl (401.1 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.3.3-cp38-cp38-manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.8

fastavro-1.3.3-cp38-cp38-macosx_10_14_x86_64.whl (466.1 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.3.3-cp37-cp37m-win_amd64.whl (386.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.3.3-cp37-cp37m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m

fastavro-1.3.3-cp37-cp37m-macosx_10_14_x86_64.whl (458.8 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.3.3-cp36-cp36m-win_amd64.whl (385.6 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.3.3-cp36-cp36m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.3-cp36-cp36m-manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.3-cp36-cp36m-macosx_10_14_x86_64.whl (456.4 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.3.3.tar.gz.

File metadata

  • Download URL: fastavro-1.3.3.tar.gz
  • Upload date:
  • Size: 706.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3.tar.gz
Algorithm Hash digest
SHA256 1d9b18a86ecc3837a919c3190f2f538b7bcf179e0ad2524d7626699773bf2945
MD5 345bfc8fbd3ae12c835a78fb22f076fd
BLAKE2b-256 5ec031ace5e3c7c7ff9f2879c7eed21cbd558bf35ee0fd3e590537e6517c7968

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 401.7 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 03fd673f0407bb2e49b2008dc28cadde7291fd4b9ae2a0711a353dcb0c5c2488
MD5 3dfdbbeeaca8a86343c0f7660f26c537
BLAKE2b-256 7a3c9ef4d950592f8d6fad1df4e3f5c524ddc2b005dd0ac61e95ab9a884e5b97

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.4 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2c53b804e58ffc7446437bd5c6fb36a031830381fc33d939dc5e1214b4265fa8
MD5 b1779a970ac04bb4a2144f2a737a2a46
BLAKE2b-256 e5164eb40908d69ba15132aef3260cf757daa303d68d9a754e00224caa4ff521

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 471.5 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 9e4aff00a5afcaa78228909709dce3a42dc58cfea37919349c68c1272a8e2f29
MD5 6fb6ecfd0dfba74277a4be7a591151f5
BLAKE2b-256 bd1e3747c858960a5811e52a9f18c74ab17b04ef348e806ef0282f2891962404

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 401.1 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 f266c0e76d26dd90412fb7eb7b1a6417157c0faf48daa0d8a945ac24da5a407c
MD5 c59e7fa9b821e9cfc95fc2dce8a028f4
BLAKE2b-256 5d8834ed3bf929a2c352c5cacc4bc3b65032385956c97456cdd0a7380ad0849c

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 328d2cfdda8535a133ebba6689db8e30a146a0873c446151a0bd16de547bdcc3
MD5 4b4dd0d61c01d2dd7b4dd2819822bd60
BLAKE2b-256 4862ba035067b8e74df7c14e2bcca4e26559174838fd78728e6814a63bface42

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 466.1 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 c532503cc4abd7bf41227a68bd9cf162eabe36d749f2ccff5d26efcdc2233900
MD5 a1359651c1951c3a60d68fe74eb932df
BLAKE2b-256 8a22014e1047745fcf98f5d8e6f33cf268ce2fb00b4fdad05901d28bf582b8a5

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 386.2 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 63aaae1d599c084debc7db628a8b911bc78a9848a86c3eef1be02b6c32ead1ec
MD5 420ce3579fe294d84b31b25c1978ff64
BLAKE2b-256 da11243e6368164992ed41a52bbcec1a0d4df7a3b473d1811919e4de5e3aede3

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a61b478d1ec6788c8fc4ec4e17b8735c9fa59ed2d6780a3a31c94678beb14dbe
MD5 492a03690bf42b464ba6327ca5eccc25
BLAKE2b-256 a87ab52db23bf1e278fce361e9b32d7599e465f349b63106b9708fcaf0f7fa50

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 458.8 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 c80340db54e866b13626f0282b0af9189220978d3b4e384f499ebd8118bfb943
MD5 bf1bc0fe87a9d616c5ddbc46bf59b003
BLAKE2b-256 7fe799bd281eeddd0becc7cedf37f00b8a8ff535aa3d3567776343b114577cb1

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 385.6 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 d91f3394ab636b860e447af2102f60a0acbec4f8beb21b9fd8292e972295ef1f
MD5 3470e2d462ebbc48807a3e8904e73fc0
BLAKE2b-256 ede562035f7631707c755391d2895193b81dcdc2376d69674be74c4614a4c603

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 02cd188c97cdc72459a06370f4a3fa0b10626c9fcea077f250f1eac9e8d7e519
MD5 92a371148c58cd7e99698c31836997d1
BLAKE2b-256 65753e3721d9e4cbbf47bd4e460a2537fde7c9a03f2c5c4f907714ae6947f312

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 996f2675eb1469535b5f5749c6a045b895bafa7938ff21996afa0f6dcc476fe5
MD5 25ca8ef5aebda03b0b740f6e866317d2
BLAKE2b-256 d51e97edd337a8f082176ec2ecc7d18b4f0e3f6f3ccf4c98ca02430961c07060

See more details on using hashes here.

File details

Details for the file fastavro-1.3.3-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.3-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 456.4 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.3-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 aff437d0a2cab602ecffffa85da9289ddfd05b0a4fe77a139133a355cb603cb7
MD5 68bbf1d3269cb78ac2c87246ac336c6a
BLAKE2b-256 814a394becb8d0761a3a777fc69806bd9ca2c5949d304d072cadc3779f740917

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page