Skip to main content

Fast read/write of AVRO files

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.7
  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.7.0.tar.gz (787.7 kB view details)

Uploaded Source

Built Distributions

fastavro-1.7.0-cp311-cp311-win_amd64.whl (400.6 kB view details)

Uploaded CPython 3.11 Windows x86-64

fastavro-1.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

fastavro-1.7.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

fastavro-1.7.0-cp311-cp311-macosx_10_9_universal2.whl (915.1 kB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.7.0-cp310-cp310-win_amd64.whl (440.8 kB view details)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.7.0-cp310-cp310-macosx_10_14_x86_64.whl (516.8 kB view details)

Uploaded CPython 3.10 macOS 10.14+ x86-64

fastavro-1.7.0-cp39-cp39-win_amd64.whl (451.5 kB view details)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.7.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.7.0-cp39-cp39-macosx_10_15_x86_64.whl (529.4 kB view details)

Uploaded CPython 3.9 macOS 10.15+ x86-64

fastavro-1.7.0-cp38-cp38-win_amd64.whl (451.9 kB view details)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.7.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.7.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.7.0-cp38-cp38-macosx_10_15_x86_64.whl (525.0 kB view details)

Uploaded CPython 3.8 macOS 10.15+ x86-64

fastavro-1.7.0-cp37-cp37m-win_amd64.whl (437.2 kB view details)

Uploaded CPython 3.7m Windows x86-64

fastavro-1.7.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

fastavro-1.7.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.4 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

fastavro-1.7.0-cp37-cp37m-macosx_10_15_x86_64.whl (514.0 kB view details)

Uploaded CPython 3.7m macOS 10.15+ x86-64

File details

Details for the file fastavro-1.7.0.tar.gz.

File metadata

  • Download URL: fastavro-1.7.0.tar.gz
  • Upload date:
  • Size: 787.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.0.tar.gz
Algorithm Hash digest
SHA256 4b1205f46489b4032d3155c1ab44d9824be0c7454df98d3a5bd22b78b98f23c8
MD5 aa8bfdbd973cd1417efcbfa51cf1aced
BLAKE2b-256 f534c33c65671c97fb792242ac6114fe451eea24ed2b6450321206fb8a05e1ec

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.0-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 400.6 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.0-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 63d0d2a2bb3e85d006c834633be51b105a50b0dc7cc8423b06f30601b532adf4
MD5 1433f36047534ecac77620c9963e7de6
BLAKE2b-256 5bbae2a17dea09fef7b26d8541fa162179f99a180f201c6fc3f957120e88f5a6

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7aab162f02e64bf82d0282430a3c6ec7a36982b1c5d479e7dcc278e6d62a84b8
MD5 35cabb44fbe488ec05b22279c0e50a9e
BLAKE2b-256 2377437bd494784d949697d202407d2a86800349e68dff26e1a3f329b9733d61

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7c664302f94064adeb93403c61c74e07b9710526403eba3b59169f99bb99c55c
MD5 92007eb2e7cbebcf78bf6697bdab1fed
BLAKE2b-256 4c0e4c501bd23fbe287723e35e831d314a249f04b560513c9301c2fa94bf85f5

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp311-cp311-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp311-cp311-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 4da6d83abd04a804310667f8d1fc23d44e9ed91ed9a9bc9c1fcd906e0c403b12
MD5 18569494ccb52c806050d5260bcecf38
BLAKE2b-256 e9495b04610c010bc015dfa394d7072b8717691d0031fb8f5fcc0be533f387b8

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.0-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 440.8 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 54b60e79506a456bcfc940560fa2c73a7a8e3ddc58a1ae4d94fdd99f6b02aef0
MD5 2d591f9749af75d00a1a2307b788955e
BLAKE2b-256 4f0fc352ce8760045f51527d200712ce55203a57d442a239436e92e2103b535d

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c20cf6cd8098bb93c2cffd53d03ccea1dcf9ec594a5c83963acf29a2882f8693
MD5 a5cf7c02a7b2dadff962b4ae8b1b7f3e
BLAKE2b-256 5da4b94f6ae1fdb2bdb5470d0f7a562c3e35cbed4753a2847e63563f03ae9b89

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 216132bc54da19e97e1531dd69c86282408d4c797749d83b01b3a00862a180de
MD5 6a4c9e260c8ba13fda26d76e8d7b1113
BLAKE2b-256 3aeb63fa25d0474e9d126ecb38e04c53d01079ed47455550a14427290cbcfdf8

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp310-cp310-macosx_10_14_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp310-cp310-macosx_10_14_x86_64.whl
Algorithm Hash digest
SHA256 ab3387a06e272980fa034f5c62f7063977b77df6416d3d30a4d3b49cc8827566
MD5 ae68a7fce9c791aa1d5ed4da76741a34
BLAKE2b-256 28fd2ff4026464e0a231a126836e7affcbf0eefab6a304ba6f73f2d7df4b89d8

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 451.5 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 492e8902063aa1c73170e11c18495fcaa86b71eae3513ef83ba438ca02b16b34
MD5 456a7058978611622bc79c5560ab5a0d
BLAKE2b-256 a184f0c3aa921fe1d22d30d42ec42a7b11630f909a72801472da1f8a9e7a6544

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f7348d318858203bd108e6bcde177d8a6f0590b52bc624d815f26fb6c37029bb
MD5 a6818c6ed2776cf34a90e3f47ef9c23e
BLAKE2b-256 26cf5d439794e02c243b7ebe8c21124cb3e94aa324051ba71f7c49cedd942e76

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9e4463b00242e4baf52d499aeefab46a26d9dd18d808d4219cd4d21089da540e
MD5 a543418f4cb537d16fc7c4d97e1cf5b4
BLAKE2b-256 f1992a5484fa65c6411d32be7d283b0b069e4c3d39ec995766941a4601dd126e

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp39-cp39-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp39-cp39-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 91cc9be740abca894082af4fe3ab9db057d4e5fa783cfa9a94c02ec041bf4346
MD5 93e23e575fdf56f98564fb7963049b53
BLAKE2b-256 5bbcd614a31abde6bed4559ad373b58e2029f88e6219823429377d59f2b22c2a

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 451.9 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 601f8c2ec166bd721f4b12eafe195dd1373d3f8dce4fe2425abd2df3e3968ac7
MD5 3e5634f73e5b77446f78a399d4161095
BLAKE2b-256 f0c66d58c3c7a0be5d1826ed0ce9c870afcdfdf1f39a5f5e6b2ad397502dfe3d

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 202424a5a7831b773f0f2cc2f82e89ed1726051fd5994f13dc678514144e10d4
MD5 68b7a4bd3e74545764e1bcb2e2aa6bec
BLAKE2b-256 e220463775c7b84e5dc275f236b77167e0c3f04358a5599c3b28458fe7147b5d

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 a7e35b94b692f8cca0096c89abf1937efed66252dea0b3b3165babfb3c289fb7
MD5 464ada10a3008aa069b28cb7766cf69e
BLAKE2b-256 2ecc2aeebd088903960ffe490772d11946a2e49b3fc6c745a4b1a97ee22c5185

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp38-cp38-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp38-cp38-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 a0ebd5c1269085179de4b3f072de274fb66a471ecbc5245bd8684e6f94943c2f
MD5 b11c5d255e578547fc8659057cf932fb
BLAKE2b-256 b45b5a28f635f44bd76c772c2f342deeee7a4de1eb595a460c3f5806bd00beed

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: fastavro-1.7.0-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 437.2 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.0

File hashes

Hashes for fastavro-1.7.0-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 749206f1cec3d7429546e49db5708f4571497d35181b6b334c4844133f230515
MD5 77f96433191bb45008a7b2bba832a91c
BLAKE2b-256 3bbfc6d785458cb587d90c6f85ed726d8b59c077eea1fb9426a5b43000a1416f

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b905634b5763ca08c9f7b51486e2c3ae7907f5d9bc48208c69b16ccbe8455e90
MD5 24a21338a47cd109c01687417275f66a
BLAKE2b-256 fd66d88d20d7b1c6d277d323b4243fb15f65c48c61c1d2254cff274e145d283f

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7df57d31cb58770a9066790250e9f4ec91242c69e1dc62ea732a6fb2406a8f96
MD5 c192113235f3b8f17c4c652a1b757002
BLAKE2b-256 6391e54fee1025283064a5a011693beffbfbc6300245f7d69c609eecd99e5686

See more details on using hashes here.

File details

Details for the file fastavro-1.7.0-cp37-cp37m-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for fastavro-1.7.0-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 a07a3245049529d6d9028d664361cc4990e74d035d2303875295e2f7b97eba09
MD5 e0ad80780b5f20fafeeb2d4eb733d09c
BLAKE2b-256 dccdc06dcb6bde9a33a954b51d554d99bbae3c86df752cdecddb6cf07a857860

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page