Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.6
  • Python 3.7
  • Python 3.8
  • Python 3.9
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPi

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as github issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the github issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a docker container. This won't test on all versions of python or on pypy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out FASTAVRO_USE_CYTHON=1 python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not availabe, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependecies set up.

Releasing

We release both to pypi and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.6
Classifiers

Release history Release notifications | RSS feed

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

This version

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.3.5.tar.gz (717.3 kB view details)

Uploaded Source

Built Distributions

fastavro-1.3.5-cp39-cp39-win_amd64.whl (408.6 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.3.5-cp39-cp39-manylinux2014_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9

fastavro-1.3.5-cp39-cp39-macosx_10_14_x86_64.whl (481.6 kB view details)

Uploaded CPython 3.9 macOS 10.14+ x86-64

fastavro-1.3.5-cp38-cp38-win_amd64.whl (408.0 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.3.5-cp38-cp38-manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8

fastavro-1.3.5-cp38-cp38-macosx_10_14_x86_64.whl (475.4 kB view details)

Uploaded CPython 3.8 macOS 10.14+ x86-64

fastavro-1.3.5-cp37-cp37m-win_amd64.whl (392.6 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.3.5-cp37-cp37m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m

fastavro-1.3.5-cp37-cp37m-macosx_10_14_x86_64.whl (468.2 kB view details)

Uploaded CPython 3.7m macOS 10.14+ x86-64

fastavro-1.3.5-cp36-cp36m-win_amd64.whl (391.9 kB view details)

Uploaded CPython 3.6m Windows x86-64

fastavro-1.3.5-cp36-cp36m-manylinux2014_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.5-cp36-cp36m-manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded CPython 3.6m

fastavro-1.3.5-cp36-cp36m-macosx_10_14_x86_64.whl (466.0 kB view details)

Uploaded CPython 3.6m macOS 10.14+ x86-64

File details

Details for the file fastavro-1.3.5.tar.gz.

File metadata

  • Download URL: fastavro-1.3.5.tar.gz
  • Upload date:
  • Size: 717.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5.tar.gz
Algorithm Hash digest
SHA256 02e3377dfb6f0ae152209cc2f92ca20fbdd75764a9ffef3662bf4682e9fe559e
MD5 75225b2d94fa8cad9139db890e3e078c
BLAKE2b-256 4b94bdd76c942a755f8af1afbe77e2f59eb5aa7a4752b09efd980fb4f83fe2fa

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 408.6 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 ffee71f4f82b53c620dbfb39f234c4ba5b3f6144f80b236e797c1b561898f94a
MD5 d02732c60b6c2d557530a3903f183849
BLAKE2b-256 c323692e3af7aeac1a510b2dce160de4933ac0a9281cfd90f8507f581b18b64d

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp39-cp39-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.5 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 1812e0f61b3a18ccd9cf6370c2e691b39a3cadbdc45d98560828206848b2b0f1
MD5 b9ee265cf33c16a111d1fc2167bb4aa8
BLAKE2b-256 7a08d21dfceb6694ff138fcf4c87279a30d4abb00bdcbaebed07b72ad3fe2000

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp39-cp39-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp39-cp39-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 481.6 kB
  • Tags: CPython 3.9, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp39-cp39-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 c59ecef619d7739c6c86a0b7fbdc3bf274b3529c3ae2b4497e9496b51a32090f
MD5 818e73d6fa1d36b8445be785ee5af291
BLAKE2b-256 46c456bcf6a5bd56520d35e938a6ccb77be9da3d538831772062ae0ba472ffaf

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 408.0 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 1187b1f353679e5740351c4a050f41191cdad1ff9277faf6e83d9de2dd20a8df
MD5 68ce5357ad1ea42f2fdc375f4da278fe
BLAKE2b-256 2a58c102043c88d9e06994f6bd2b75530d42ca1ca4e6f90df0da9c740e65718b

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8bb762d702d96abfc630860a671371a7aa94b09ffd3c155d1b2ce016119d062e
MD5 6566c1bf283e8a44cf92e32e2a7e36df
BLAKE2b-256 b41653c5069875977a23099bdbb05a647e75a3d662fd0d7e24c2d485e3b6deec

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp38-cp38-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp38-cp38-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 475.4 kB
  • Tags: CPython 3.8, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 43c268586699f37ed282b8cb1dcfd7f0cf0f2d19cfe8b507de4fa93683cc0530
MD5 bc9144952f0dcee8e4139053c1a558e0
BLAKE2b-256 e3248e1c4d4c582d230919429a50e144ebaca114db260efe1fbe40c9f2e9d538

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 392.6 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 fb3a5aa3e2666700c268be2da67b14d6ca7019eeffc3fa25134414a19733dd6a
MD5 5deedd9c7ac0c4b5c48b8c374684eb77
BLAKE2b-256 513c1306298050dd02ad29e429f67d96b55fe33ac5f47ccb28511758ff022bc2

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 80343cbf1c90d93a078265cc3ec6333914d4ef1a088714e2f3ab71c2771407b1
MD5 aed17e1427285e51d46683265f99b2d7
BLAKE2b-256 c5c79ef9fda5e178aa5f5cda00c0d7505749506c540f9caf876d23c6cf915bf9

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp37-cp37m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp37-cp37m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 468.2 kB
  • Tags: CPython 3.7m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 722d514ddbfa926ce2cceddb3b0c4bab87167f2d45ae254334c12f6a1634d1a0
MD5 02bd09fd934c78b79ceb2125f204b7e9
BLAKE2b-256 ebbf82d4323972288f5b903d64ec1bf9d6472116f02f3f94ce82d5ae435d056c

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 391.9 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 fbd85b63f88ac7023fcb1828250fdc9395fddec8d226dd527ad29a8512787785
MD5 c0b1bca4806a0e25dce813853a1be7ea
BLAKE2b-256 a00a1c00db576871f753bfcad2c34310b2e2e799612fa38f73147530ad3c7f38

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 68fc157d1624ed5f09a2b8a41b19f7f7a8ddb6716e26b9af66b4d9613f1d462f
MD5 922f6e7d7f52c95dce89ccdc9f3f0dae
BLAKE2b-256 5845fb4d873076846f194a84a620ee2a05b406ee33a1886ea3a08a2107fafee0

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e39d9d41a441597f795c3ecd5404c4b4a57f83c44040bc3b58d6c268faaa3546
MD5 8cf0ee4134da246cc5b19c7ac53bcccc
BLAKE2b-256 7d1e09c5680d9bbe617ddfd74e638d4686457fafc3c66a1d01c79b667a3a851a

See more details on using hashes here.

File details

Details for the file fastavro-1.3.5-cp36-cp36m-macosx_10_14_x86_64.whl.

File metadata

  • Download URL: fastavro-1.3.5-cp36-cp36m-macosx_10_14_x86_64.whl
  • Upload date:
  • Size: 466.0 kB
  • Tags: CPython 3.6m, macOS 10.14+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.0

File hashes

Hashes for fastavro-1.3.5-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 a3d5d8ff3ea47d7ab39b253736c5546417ad4765cf336f63a533363720268c4e
MD5 d219e0073408fcbdaccee3c9540b3b8c
BLAKE2b-256 604e0259923be4dba78d2bbda9d6947f3c1b41f5c80ff5f157a02261bfd62465

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page