fastavro

Fast iteration of AVRO files

Project description

fastavro

The current Python avro package is packed with features but dog slow.

On a test case of about 10K records, it takes about 14sec to iterate over all of them. In comparison the JAVA avro SDK does it in about 1.9sec.

fastavro is less feature complete than avro, however it’s much faster. It iterates over the same 10K records in 2.9sec, and if you use it with PyPy it’ll do it in 1.5sec (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

If you have Cython installed, then fastavro will be even faster. For the same 10K records it’ll run in about 1.7sec.

Usage

from fastavro import iter_avro

with open('some-file.avro', 'rb') as fo:
    avro = iter_avro(fo)
    schema = avro.schema

    for record in avro:
        process_record(record)

You can also use the fastavro module from the command line to dump avro files. Each record will be dumped to standard output in one line of JSON.

python -m fastavro weather.avro

Limitations

Support only iteration
- No writing for you!
Supports only null and deflate codecs
- avro also supports snappy
No reader schema

Contact

Miki Tebeka <miki.tebeka@gmail.com> https://bitbucket.org/tebeka/fastavro

Project details

Release history Release notifications | RSS feed

1.9.7

Sep 6, 2024

1.9.6

Sep 5, 2024

1.9.5

Jul 8, 2024

1.9.4

Feb 13, 2024

1.9.3

Jan 9, 2024

1.9.2

Dec 21, 2023

1.9.1

Dec 6, 2023

1.9.0

Oct 28, 2023

1.8.4

Oct 3, 2023

1.8.3

Sep 7, 2023

1.8.2

Jul 19, 2023

1.8.1

Jul 17, 2023

1.8.0

Jul 6, 2023

1.7.4

May 4, 2023

1.7.3

Mar 8, 2023

1.7.2

Feb 22, 2023

1.7.1

Jan 27, 2023

1.7.0

Oct 26, 2022

1.6.1

Sep 9, 2022

1.6.0

Aug 15, 2022

1.5.4

Jul 29, 2022

1.5.3

Jul 19, 2022

1.5.2

Jun 27, 2022

1.5.1

Jun 8, 2022

1.5.0

Jun 7, 2022

1.4.12

May 18, 2022

1.4.11

Apr 27, 2022

1.4.10

Mar 4, 2022

1.4.9

Jan 8, 2022

1.4.8

Dec 27, 2021

1.4.7

Oct 29, 2021

1.4.6

Oct 23, 2021

1.4.5

Sep 23, 2021

1.4.4

Jul 22, 2021

1.4.3

Jul 16, 2021

1.4.2

Jun 28, 2021

1.4.1

May 18, 2021

1.4.0

Apr 16, 2021

1.3.5

Mar 31, 2021

1.3.4

Mar 20, 2021

1.3.3

Mar 13, 2021

1.3.2

Feb 14, 2021

1.3.1

Feb 6, 2021

1.3.0

Jan 21, 2021

1.2.4

Jan 17, 2021

1.2.3

Dec 24, 2020

1.2.2

Dec 23, 2020

1.2.1

Dec 2, 2020

1.2.0

Nov 19, 2020

1.1.1

Nov 17, 2020

1.1.0

Oct 30, 2020

1.0.0.post1

Aug 26, 2020

1.0.0 yanked

Aug 23, 2020

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

Aug 18, 2020

0.24.1

Aug 16, 2020

0.24.0

Jul 30, 2020

0.23.6

Jul 12, 2020

0.23.5

Jun 22, 2020

0.23.4

May 15, 2020

0.23.3

Apr 29, 2020

0.23.2

Apr 18, 2020

0.23.1

Apr 3, 2020

0.23.0

Mar 23, 2020

0.22.13

Mar 3, 2020

0.22.12

Feb 27, 2020

0.22.11

Feb 26, 2020

0.22.10

Feb 24, 2020

0.22.9

Dec 20, 2019

0.22.8

Dec 16, 2019

0.22.7

Nov 6, 2019

0.22.6

Nov 3, 2019

0.22.5

Sep 19, 2019

0.22.4

Aug 26, 2019

0.22.3

Jul 12, 2019

0.22.2

Jun 28, 2019

0.22.1

Jun 14, 2019

0.22.0

Jun 13, 2019

0.21.24

May 29, 2019

0.21.23

May 7, 2019

0.21.22

Apr 27, 2019

0.21.21

Apr 19, 2019

0.21.20

Apr 3, 2019

0.21.19

Mar 3, 2019

0.21.18

Feb 14, 2019

0.21.17

Jan 22, 2019

0.21.16

Dec 22, 2018

0.21.15

Dec 10, 2018

0.21.14

Nov 17, 2018

0.21.13

Nov 12, 2018

0.21.12

Nov 1, 2018

0.21.11

Oct 30, 2018

0.21.10

Oct 25, 2018

0.21.9

Oct 9, 2018

0.21.8

Sep 25, 2018

0.21.7

Sep 17, 2018

0.21.6

Sep 16, 2018

0.21.5

Sep 4, 2018

0.21.4

Jul 25, 2018

0.21.3

Jul 12, 2018

0.21.2

Jul 12, 2018

0.21.1

Jul 10, 2018

0.21.0

Jul 9, 2018

0.21.0rc1 pre-release

Jul 6, 2018

0.20.0

Jul 3, 2018

0.19.9

Jun 29, 2018

0.19.8

Jun 27, 2018

0.19.7

Jun 13, 2018

0.19.6

May 31, 2018

0.19.5

May 29, 2018

0.19.4

May 22, 2018

0.19.3

May 20, 2018

0.19.2

May 18, 2018

0.19.1

May 16, 2018

0.19.0

May 14, 2018

0.18.2

May 8, 2018

0.18.1

May 2, 2018

0.18.0

Apr 27, 2018

0.17.10

Mar 30, 2018

0.17.9

Mar 1, 2018

0.17.8

Feb 12, 2018

0.17.7

Feb 1, 2018

0.17.6

Feb 1, 2018

0.17.5

Jan 24, 2018

0.17.4

Jan 22, 2018

0.17.3

Jan 19, 2018

0.17.2

Jan 18, 2018

0.17.1

Dec 27, 2017

0.17.0

Dec 26, 2017

0.16.7

Dec 24, 2017

0.16.6

Dec 13, 2017

0.16.5

Dec 12, 2017

0.16.4

Dec 5, 2017

0.16.3

Nov 29, 2017

0.16.1

Nov 27, 2017

0.14.11

Nov 7, 2017

0.14.10

Sep 18, 2017

0.14.9

Sep 10, 2017

0.14.8

Sep 2, 2017

0.14.7

Aug 8, 2017

0.14.6

Aug 2, 2017

0.14.5

Jul 15, 2017

0.14.3

Jun 24, 2017

0.14.2

Jun 8, 2017

0.14.1

Jun 7, 2017

0.14.0

Jun 3, 2017

0.13.0

May 2, 2017

0.12.2

Apr 19, 2017

0.12.1

Dec 9, 2016

0.12.0

Dec 8, 2016

0.11.1

Nov 25, 2016

0.11.0

Oct 20, 2016

0.10.2

Aug 1, 2016

0.10.1

Jul 3, 2016

0.9.11

Jun 12, 2016

0.9.10

Jun 7, 2016

0.9.9

Feb 13, 2016

0.9.8

Jan 15, 2016

0.9.7

Dec 27, 2015

0.9.6

Oct 14, 2015

0.9.5

Oct 4, 2015

0.9.4

Oct 1, 2015

0.9.3

Sep 1, 2015

0.9.2

Aug 25, 2015

0.9.1

Aug 21, 2015

0.9.0

Aug 20, 2015

0.8.8

Aug 18, 2015

0.8.7

Aug 15, 2015

0.8.6

Aug 13, 2015

0.8.5

Aug 5, 2015

0.8.4

Aug 3, 2015

0.8.3

Jul 16, 2015

0.8.2

Jul 14, 2015

0.8.1

Jun 2, 2015

0.8.0

May 4, 2015

0.7.10

Apr 28, 2015

0.7.9

Aug 28, 2014

0.7.8

May 20, 2014

0.7.7

Mar 27, 2013

0.7.6

Mar 27, 2013

0.7.5

Mar 23, 2013

0.7.4

Mar 2, 2013

0.7.3

Feb 19, 2013

0.7.2

Jan 8, 2013

0.7.1

Dec 11, 2012

0.7.0

Dec 11, 2012

0.6.10

Oct 6, 2012

0.6.9

Oct 6, 2012

0.6.8

Jul 14, 2012

0.6.7

Apr 29, 2012

0.6.6

Apr 29, 2012

0.6.5

Apr 25, 2012

0.6.4

Apr 14, 2012

0.6.3

Mar 16, 2012

0.6.2

Mar 13, 2012

0.6.1

Mar 12, 2012

0.6.0

Mar 10, 2012

0.5.0

Feb 23, 2012

0.4.2

Jan 30, 2012

0.4.1

Jan 25, 2012

0.4.0

Jan 25, 2012

0.3.2

Jan 24, 2012

0.3.1

Jan 24, 2012

0.3.0

Jan 23, 2012

This version

0.2.2

Jan 11, 2012

0.2.1

Jan 11, 2012

0.2.0

Jan 11, 2012

0.1.0

Jan 5, 2012

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-0.2.2.tar.gz (42.3 kB view details)

Uploaded Jan 11, 2012 Source

Built Distribution

fastavro-0.2.2-py2.7-linux-x86_64.egg (100.0 kB view details)

Uploaded Jan 11, 2012 Source

File details

Details for the file fastavro-0.2.2.tar.gz.

File metadata

Download URL: fastavro-0.2.2.tar.gz
Upload date: Jan 11, 2012
Size: 42.3 kB
Tags: Source
Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.2.2.tar.gz
Algorithm	Hash digest
SHA256	`40e4eff91b74bae582160531c7dc8afb67b7c4713087cfe282e9d82bd43bcc9e`
MD5	`1f511cf7b28e6559514efbb7c3be6e5e`
BLAKE2b-256	`b438b6ca250674e5e60e8abb628c87e34c4d40c2cb581b3169740961cb60739a`

See more details on using hashes here.

File details

Details for the file fastavro-0.2.2-py2.7-linux-x86_64.egg.

File metadata

Download URL: fastavro-0.2.2-py2.7-linux-x86_64.egg
Upload date: Jan 11, 2012
Size: 100.0 kB
Tags: Source
Uploaded using Trusted Publishing? No

File hashes

Hashes for fastavro-0.2.2-py2.7-linux-x86_64.egg
Algorithm	Hash digest
SHA256	`0ca302c396e58d3a02b11d3ded39ca0c2ca9e3ad28701e7462c8eba7737a8ac9`
MD5	`c3a4b759a6ccc14dee442690a48643d6`
BLAKE2b-256	`4722a836e27e65231b799e945a05e2786df57e289b44a464f06399cce725f671`