Python support for Parquet file format
Project description
fastparquet is a python implementation of the parquet format, aiming integrate into python-based big data work-flows.
Not all parts of the parquet-format have been implemented yet or tested e.g. see the Todos linked below. With that said, fastparquet is capable of reading all the data files from the parquet-compatability project.
Introduction
This software is alpha, expect frequent API changes and breakages.
A list of expected features and their status in this branch can be found in this issue. Please feel free to comment on that list as to missing items and priorities.
In the meantime, the more eyes on this code, the more example files and the more use cases the better.
Requirements
(all development is against recent versions in the default anaconda channels)
Required:
numba
numpy
pandas
Optional (compression algorithms; gzip is always available):
snappy
lzo
brotli
Installation
Install from github:
> pip install git+https://github.com/martindurant/fastparquet
Assuming the requirements have been met. Numba should be installed using conda, and a conda package of this package will be forthcoming.
Usage
Reading
from fastparquet import ParquetFile
pf = ParquetFile('myfile.parq')
df = pf.to_pandas()
df2 = pf.to_pandas(['col1', 'col2'], categories=['col1'])You may specify which columns to load, which of those to keep as categoricals (if the data uses dictionary encoding). The file-path can be a single file, a metadata file pointing to other data files, or a directory (tree) containing data files. The latter is what is typically output by hive/spark.
Writing
from fastparquet import write
write('outfile.parq', df)
write('outfile2.parq', df, partitions=[0, 10000, 20000],
compression='GZIP', file_scheme='hive')The default is to produce a single output file with a single row-group (i.e., logical segment) and no compression. At the moment, only simple data-types and plain encoding are supported, so expect performance to be similar to numpy.savez.
History
Since the second week of October, this fork of parquet-python has been undergoing considerable redevelopment. The aim is to have a small and simple and performant library for reading and writing the parquet format from python.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for fastparquet-0.0.1.post2-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3c43149f04c27a108e94d76d395cd7eda3adec5415f7bac80a5b2b7ae1a96043 |
|
| MD5 | ffddfbdb4dbf35a0cf7ce3eeab323c2c |
|
| BLAKE2b-256 | d6be0c86872702f01747eeeb72f832cd1669163576833cbc014e80e9f371793a |
