fastparquet

Python support for Parquet file format

These details have not been verified by PyPI

Project links

Homepage

Project description

https://github.com/dask/fastparquet/actions/workflows/main.yaml/badge.svg

https://readthedocs.org/projects/fastparquet/badge/?version=latest

fastparquet is a python implementation of the parquet format, aiming integrate into python-based big data work-flows. It is used implicitly by the projects Dask, Pandas and intake-parquet.

We offer a high degree of support for the features of the parquet format, and very competitive performance, in a small install size and codebase.

Details of this project, how to use it and comparisons to other work can be found in the documentation.

Requirements

(all development is against recent versions in the default anaconda channels and/or conda-forge)

Required:

numpy
pandas
cython >= 0.29.23 (if building from pyx files)
cramjam
fsspec

Supported compression algorithms:

Available by default:
- gzip
- snappy
- brotli
- lz4
- zstandard
Optionally supported
- lzo

Installation

Install using conda, to get the latest compiled version:

conda install -c conda-forge fastparquet

or install from PyPI:

pip install fastparquet

You may wish to install numpy first, to help pip’s resolver. This may install an appropriate wheel, or compile from source. For the latter, you will need a suitable C compiler toolchain on your system.

You can also install latest version from github:

pip install git+https://github.com/dask/fastparquet

in which case you should also have cython to be able to rebuild the C files.

Usage

Please refer to the documentation.

Reading

from fastparquet import ParquetFile
pf = ParquetFile('myfile.parq')
df = pf.to_pandas()
df2 = pf.to_pandas(['col1', 'col2'], categories=['col1'])

You may specify which columns to load, which of those to keep as categoricals (if the data uses dictionary encoding). The file-path can be a single file, a metadata file pointing to other data files, or a directory (tree) containing data files. The latter is what is typically output by hive/spark.

Writing

from fastparquet import write
write('outfile.parq', df)
write('outfile2.parq', df, row_group_offsets=[0, 10000, 20000],
      compression='GZIP', file_scheme='hive')

The default is to produce a single output file with a single row-group (i.e., logical segment) and no compression. At the moment, only simple data-types and plain encoding are supported, so expect performance to be similar to numpy.savez.

History

This project forked in October 2016 from parquet-python, which was not designed for vectorised loading of big data or parallel access.

Project details

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

2024.11.0

Nov 12, 2024

2024.5.0

May 21, 2024

2024.2.0

Feb 8, 2024

2023.10.1

Oct 26, 2023

2023.10.0

Oct 25, 2023

2023.8.0

Aug 30, 2023

2023.7.0

Jul 1, 2023

2023.4.0

Apr 27, 2023

2023.2.0

Feb 8, 2023

2023.1.0

Jan 19, 2023

This version

2022.12.0

Dec 5, 2022

2022.11.0

Nov 17, 2022

0.8.3

Aug 28, 2022

0.8.2

Aug 19, 2022

0.8.1

Apr 1, 2022

0.8.0

Jan 26, 2022

0.7.2

Nov 22, 2021

0.7.1

Aug 3, 2021

0.7.0

Jul 16, 2021

0.6.3

May 13, 2021

0.6.2

May 12, 2021

0.6.1

May 11, 2021

0.6.0.post1

May 6, 2021

0.6.0

May 6, 2021

0.5.0

Dec 29, 2020

0.4.2 yanked

Dec 14, 2020

Reason this release was yanked:

deps not updated

0.4.1

Jul 16, 2020

0.4.0

May 12, 2020

0.3.3

Feb 5, 2020

0.3.2

Aug 1, 2019

0.3.1

Apr 25, 2019

0.3.0

Mar 30, 2019

0.2.1

Dec 18, 2018

0.2.0

Nov 22, 2018

0.1.6

Aug 19, 2018

0.1.5

Apr 1, 2018

0.1.4

Jan 27, 2018

0.1.3

Oct 8, 2017

0.1.2

Aug 28, 2017

0.1.1

Jul 21, 2017

0.1.0

Jun 13, 2017

0.0.6

May 4, 2017

0.0.5

Feb 16, 2017

0.0.4.post1

Dec 27, 2016

0.0.4

Dec 27, 2016

0.0.3

Dec 1, 2016

0.0.2

Nov 15, 2016

0.0.1.post2

Nov 1, 2016

0.0.1.post1

Nov 1, 2016

0.0.1

Nov 1, 2016

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastparquet-2022.12.0.tar.gz (391.8 kB view details)

Uploaded Dec 5, 2022 Source

Built Distributions

fastparquet-2022.12.0-cp311-cp311-win_amd64.whl (614.0 kB view details)

Uploaded Dec 5, 2022 CPython 3.11 Windows x86-64

fastparquet-2022.12.0-cp311-cp311-musllinux_1_1_x86_64.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.11 musllinux: musl 1.1+ x86-64

fastparquet-2022.12.0-cp311-cp311-musllinux_1_1_i686.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.11 musllinux: musl 1.1+ i686

fastparquet-2022.12.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.11 manylinux: glibc 2.17+ x86-64

fastparquet-2022.12.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.11 manylinux: glibc 2.17+ ARM64

fastparquet-2022.12.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.11 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

fastparquet-2022.12.0-cp311-cp311-macosx_11_0_arm64.whl (576.5 kB view details)

Uploaded Dec 5, 2022 CPython 3.11 macOS 11.0+ ARM64

fastparquet-2022.12.0-cp311-cp311-macosx_10_9_universal2.whl (780.9 kB view details)

Uploaded Dec 5, 2022 CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastparquet-2022.12.0-cp310-cp310-win_amd64.whl (616.1 kB view details)

Uploaded Dec 5, 2022 CPython 3.10 Windows x86-64

fastparquet-2022.12.0-cp310-cp310-musllinux_1_1_x86_64.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.10 musllinux: musl 1.1+ x86-64

fastparquet-2022.12.0-cp310-cp310-musllinux_1_1_i686.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.10 musllinux: musl 1.1+ i686

fastparquet-2022.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.10 manylinux: glibc 2.17+ x86-64

fastparquet-2022.12.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.10 manylinux: glibc 2.17+ ARM64

fastparquet-2022.12.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.10 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

fastparquet-2022.12.0-cp310-cp310-macosx_11_0_arm64.whl (580.6 kB view details)

Uploaded Dec 5, 2022 CPython 3.10 macOS 11.0+ ARM64

fastparquet-2022.12.0-cp310-cp310-macosx_10_9_universal2.whl (788.1 kB view details)

Uploaded Dec 5, 2022 CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64)

fastparquet-2022.12.0-cp39-cp39-win_amd64.whl (618.3 kB view details)

Uploaded Dec 5, 2022 CPython 3.9 Windows x86-64

fastparquet-2022.12.0-cp39-cp39-musllinux_1_1_x86_64.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.9 musllinux: musl 1.1+ x86-64

fastparquet-2022.12.0-cp39-cp39-musllinux_1_1_i686.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.9 musllinux: musl 1.1+ i686

fastparquet-2022.12.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.9 manylinux: glibc 2.17+ x86-64

fastparquet-2022.12.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.9 manylinux: glibc 2.17+ ARM64

fastparquet-2022.12.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.9 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

fastparquet-2022.12.0-cp39-cp39-macosx_11_0_arm64.whl (581.0 kB view details)

Uploaded Dec 5, 2022 CPython 3.9 macOS 11.0+ ARM64

fastparquet-2022.12.0-cp39-cp39-macosx_10_9_universal2.whl (790.3 kB view details)

Uploaded Dec 5, 2022 CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64)

fastparquet-2022.12.0-cp38-cp38-win_amd64.whl (628.1 kB view details)

Uploaded Dec 5, 2022 CPython 3.8 Windows x86-64

fastparquet-2022.12.0-cp38-cp38-musllinux_1_1_x86_64.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.8 musllinux: musl 1.1+ x86-64

fastparquet-2022.12.0-cp38-cp38-musllinux_1_1_i686.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.8 musllinux: musl 1.1+ i686

fastparquet-2022.12.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Dec 5, 2022 CPython 3.8 manylinux: glibc 2.17+ x86-64

fastparquet-2022.12.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.8 manylinux: glibc 2.17+ ARM64

fastparquet-2022.12.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (1.5 MB view details)

Uploaded Dec 5, 2022 CPython 3.8 manylinux: glibc 2.17+ i686 manylinux: glibc 2.5+ i686

fastparquet-2022.12.0-cp38-cp38-macosx_11_0_arm64.whl (586.3 kB view details)

Uploaded Dec 5, 2022 CPython 3.8 macOS 11.0+ ARM64

fastparquet-2022.12.0-cp38-cp38-macosx_10_9_universal2.whl (791.4 kB view details)

Uploaded Dec 5, 2022 CPython 3.8 macOS 10.9+ universal2 (ARM64, x86-64)

File details

Details for the file fastparquet-2022.12.0.tar.gz.

File metadata

Download URL: fastparquet-2022.12.0.tar.gz
Upload date: Dec 5, 2022
Size: 391.8 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for fastparquet-2022.12.0.tar.gz
Algorithm	Hash digest
SHA256	`6c32349355652ab35a082d79e76e809ec0a9b68546d321a0003dda762fb40a18`
MD5	`f4505d3367e68313272a165435bf74e6`
BLAKE2b-256	`b600961bc35ab1e9a49a7e805d201493a7bad279e9a29e51a526acb91f08648d`

Algorithm	Hash digest
SHA256	`e6f0b4c04db5d465f9e47f29ee40c213cd41c29ed290807de6953963d855205d`
MD5	`5ca4df56479b5abb1b70762edebb54ca`
BLAKE2b-256	`eaaf0daa375e4a933d7b77d49bd4b53bc9582cf95233e2a377674b1103127edc`

Algorithm	Hash digest
SHA256	`46c3f482106742623eadcfbcd18597758e7210a1d811aa1606c555e8c3aec2b6`
MD5	`8cfe4cb148104196744b50b6e2b64f57`
BLAKE2b-256	`cb6ba60226b986d4cfb24ecf920dde357f1e6d5b6ef6461bfbc94836527056dd`

Algorithm	Hash digest
SHA256	`eb7e2b452e36fbf9b8ed71be7e5200c7e5e4f9f8bd50c697f26638d95c18413c`
MD5	`47338578e4f65b3d347badee5bbbd31c`
BLAKE2b-256	`d2640ca3c84b9e281f5aa700bb8476eb905e3dcf63de52d6bee578438fd27e27`

Algorithm	Hash digest
SHA256	`eea0f24d786ece9cae64a2ddcf06b504436229af08f04c0ec76210cd428eb79c`
MD5	`1efad638616da6a5aa3398ac171eb8ec`
BLAKE2b-256	`eb31b1bfd20ff9474a61f0a25acde3dc1f80fea5afa8560aed9279d8c8a4cf27`

Algorithm	Hash digest
SHA256	`cf2d772db1bc2dfad641eb7b75adc1e09fe31a59536dbef619f7bdb8f3f0f1d8`
MD5	`0087405885c377075fdb475684ab370b`
BLAKE2b-256	`f156c99e6cb17b6539a2906660070ce7cc513a1989a339e7a718917b84065a47`

Algorithm	Hash digest
SHA256	`45371a5393e27c8823d05c5f857d1354c947c5fb2fc9f314006e50bba3d1d506`
MD5	`f1774122a2dec9a98089734bde45a408`
BLAKE2b-256	`3910e63756dc3a5aa556b914b0b05b8820e0eed4406a42b3b81d3955d9f1d983`

Algorithm	Hash digest
SHA256	`f47d1652044966abdfa6b235a8bd586461ece984575fae1b49a609128c795d42`
MD5	`2b370e69b953e602f38062bcf7d8da47`
BLAKE2b-256	`b4480118a8e0dd2e5ee31335b73755a215eb842f5fc3b08528a6ad9cfa3d87da`

Algorithm	Hash digest
SHA256	`3e2927999ef7f2a94b09a00c2f61e1bb86d24ca3d2a259bba449eef938b049f8`
MD5	`b292f2e20e91a067549cc7c0c75a8070`
BLAKE2b-256	`75b5369a69abcedad84362bd40056b0d372a6d53d8e3c01d563d1e4eca45046b`

Algorithm	Hash digest
SHA256	`ac746c782fbb8a8ec6155da8467e02a65ba781d652a5347c986b24466c7d17c6`
MD5	`91c166dec8f4ac08fcb90ee54fe88977`
BLAKE2b-256	`1b0a1b0d5035561c207cfd8a3fa2c1df60b117282fbdf8e3db258f783824b8ac`

Algorithm	Hash digest
SHA256	`7257c7dc1f39bd27fc5f964d1fd017dc378e972dc35278aafe1468fd2e364a97`
MD5	`370eeb3e5bd1fb7eec2af89acdc0df1d`
BLAKE2b-256	`d6240c826ce4b21d25c7b2bc27da2e5b9be0d095a24bd235f1e6518584b8ba20`

Algorithm	Hash digest
SHA256	`5f4ec6e91600626ad4a60476bbdda154807458ef54141d64ad5e2a7b436a5d0f`
MD5	`0d0e43c15b50335ae83f4318939a2e92`
BLAKE2b-256	`d0d13ffab48bef5f6f15940627c08a376a7dbcba03a230b8c7033ab6efaaf482`

Algorithm	Hash digest
SHA256	`8cb46f8282f5a23684b1c31380c6cec6987233dbaa55b81cb75c1f4cc82a0111`
MD5	`e20fe77cf8b97fec6969763980508260`
BLAKE2b-256	`4abe911d01624a63a8f9527f34d9b38d7eb293209f8d011d06d792c7fa3a19f4`

Algorithm	Hash digest
SHA256	`dcde27d4461e094036e01c21df833b2274afd025311b631ca635fa11dc06e83e`
MD5	`ae8b1fe70a095f2cf2976173a9518ba4`
BLAKE2b-256	`325b22afb87f7a089d69d6882586fecaa3a4df9dc9c7e26bb41a03e91cf52244`

Algorithm	Hash digest
SHA256	`2ab2ae2e6dd22e23a31586daced4fe2128ddea8e4abb25c685fce34cc36a537f`
MD5	`f9fdd59b2bf91d24ffca209123ce7b99`
BLAKE2b-256	`63f38ee8ccba81c6b40de85d666434d8e3b39c3cceaa1b1e8df1b49db00a3d24`

Algorithm	Hash digest
SHA256	`a23be3c71c2e21cbb87cccafc08458815d5d61921e3a05659469e8f071fe0cc2`
MD5	`b7c4580db93dbf7285aaa8ebbf54935a`
BLAKE2b-256	`4e5fe1387d79c4721abe19214ddd4bde8e9b5d4b9182d901ea595005003a849f`

Algorithm	Hash digest
SHA256	`b7c95a7e1bc2bc91b72704a2869f99f10334b5c40a365bce1b61f46ceffc7e3c`
MD5	`9ef16a188394943629459927f24b62ba`
BLAKE2b-256	`9649296318fb86bf5a9c35f1a969be898481032e9d7c06516e9de7b263383048`

Algorithm	Hash digest
SHA256	`b7be4935c57ce4db9ec5cf7703a15312a3b722179f1645c3976b36660a25c11c`
MD5	`170fb5bf0137850b9dbdf8e3c6379084`
BLAKE2b-256	`4d6489a50e7eff54cfbd5b746693ecbf506daf5ff9cd89b6deb902771ef67975`

Algorithm	Hash digest
SHA256	`a9b0e6227d676a5394a8a3f46c0105db31e3013d6c64fb9901f36294bf5d5eb3`
MD5	`6204ea9d466d4df2da097aaeae1937c0`
BLAKE2b-256	`8f50ed78f0abb1b7f97c67f30da26ca5015e85104891106d7cd55ba369cabb51`

Algorithm	Hash digest
SHA256	`27f511c62e47ab9a3fe49ed9f426f886f388d4aa4df3eb7e747353f18c78cac2`
MD5	`58430949aa18c3f741a3a4493182ef14`
BLAKE2b-256	`007031d4534c53b425370ae31a831cc252fdd345b1ce13c27322d7f9021960d3`

Algorithm	Hash digest
SHA256	`82458131bb7619b23ab670d0fa3c8379663ce59042a4281820ca5c8bb3f72374`
MD5	`2338f2d07be83ae78d77a91fca4eb53c`
BLAKE2b-256	`c33f7636ec0fc17925955a20833c2c872b50d339afc5b97798d2ecb62a4ef3cf`

fastparquet 2022.12.0

Navigation

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Project description

Requirements

Installation

Usage

History

Project details

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Algorithm	Hash digest
SHA256	`14cc3c50fca76df6a7aa58f69376e77f547865bd946a5405b70b32e8a3126ec7`
MD5	`07590da801e5c69daeaa6405db98db24`
BLAKE2b-256	`881dc1fe3d109c4496d36bf0920da0282cb992fe6fee59936e167b7857ab23cc`

Algorithm	Hash digest
SHA256	`aa69af2509d0a5a822abf26fc53247940b2c2f4cb614c6eff7ce8118dd61d665`
MD5	`b704edd257ccaed3bc3943d3313885c2`
BLAKE2b-256	`f925e9d8a313afb853b28a8b1381f97137f3f85c8fdbc0b81ff834dcb2894009`

Algorithm	Hash digest
SHA256	`01508f83a3952de94b927fdeabf908d9d39608e67270dcd1c9a3f5d56b0baa03`
MD5	`47ce81856a830acb9746b2b4b24f3062`
BLAKE2b-256	`e853237887ea900de9f70e1b512831947828b16f2bf77f4a5f7cf53966f54c40`

Algorithm	Hash digest
SHA256	`a0d252e9342458d6c2574103ac0224b3a9bf44da6ebb5e08509da84d25cbffa3`
MD5	`b464e2ebe19abd8c4148fcc6e75d89a0`
BLAKE2b-256	`5c1880e3c8f6dd97248c3020fc5c1b5fb8f39fecfaeec67b3ea232be361d018f`

Algorithm	Hash digest
SHA256	`0eb862ab28e3ac9479e06faa5153357e60ca54869126519fbbda557b725b0a0c`
MD5	`7ec02dbf7abeed224bdcad55987e8951`
BLAKE2b-256	`87136cbfc1726fe37b4c590390fd33fac0a1f1c1a57e5aeccda7310fc8716c40`

Algorithm	Hash digest
SHA256	`540eafe9a811736f09dc77712dfd1adde197020e521159ae95eed65315afc03d`
MD5	`1e5799b92cb0319424e6d0427f4e9670`
BLAKE2b-256	`94736513a22b99c58a4b819752bf461aef53c58ebf1e12e988183f1fbddb955c`

Algorithm	Hash digest
SHA256	`2f0c0af41ed731976b90bacfd763f1b2d90d7653a1bab9fa8c6261f8380a6744`
MD5	`fe8c3a78836d1993933ecb4f5ed66ead`
BLAKE2b-256	`fa1b12969b8dd50114278a3dc94aca9541c8ee4bc33618eda0fee36365b657e3`

Algorithm	Hash digest
SHA256	`8f6a42c67398c4b9d502d5f8e39d569d0cca30fa21bbc230f436915be3ed086d`
MD5	`eecca8f728f43f224efe96bf56207aa7`
BLAKE2b-256	`5e5cadd0f42af077da6ec6fa4b853971c8c857ad2199a3b9a6cda7ea49f82914`

Algorithm	Hash digest
SHA256	`35ad000a4e969b87931f52d37cd5a9241f0daa3d9fb4be2f52f95e592c5ba9cc`
MD5	`36f530c2876b05f887e385d478502221`
BLAKE2b-256	`eaf186a8fd2e1bff8f7cf99e5dfddeef81ea90e174391591b78f0e176f834404`

Algorithm	Hash digest
SHA256	`76cf692c2390cef70cccd7122f4aca0a2dcd9a53b0b5622e9f4b9b0e6f9da63c`
MD5	`41d0c83e064203be97ce211251d0724e`
BLAKE2b-256	`9e64539cc45e669cd43b05beafb4f6442120f9dbb4d35872d260c468c778773d`