Skip to main content

Python support for Parquet file format

Project description

https://travis-ci.org/jcrobak/parquet-python.svg?branch=master

fastparquet is a python implementation of the parquet format, aiming integrate into python-based big data work-flows.

Not all parts of the parquet-format have been implemented yet or tested e.g. see the Todos linked below. With that said, fastparquet is capable of reading all the data files from the parquet-compatability project.

Introduction

Details of this project can be found in the documentation.

The original plan listing expected features can be found in this issue. Please feel free to comment on that list as to missing items and priorities, or raise new issues with bugs or requests.

Requirements

(all development is against recent versions in the default anaconda channels)

Required:

  • numba (requires LLVM 4.0.x)

  • numpy

  • pandas

  • cython

  • six

Optional (compression algorithms; gzip is always available):

  • snappy (aka python-snappy)

  • lzo

  • brotli

  • lz4

  • zstandard

Installation

Install using conda:

conda install -c conda-forge fastparquet

install from pypi:

pip install fastparquet

or install latest version from github:

pip install git+https://github.com/dask/fastparquet

For the pip methods, numba must have been previously installed (using conda).

Usage

Reading

from fastparquet import ParquetFile
pf = ParquetFile('myfile.parq')
df = pf.to_pandas()
df2 = pf.to_pandas(['col1', 'col2'], categories=['col1'])

You may specify which columns to load, which of those to keep as categoricals (if the data uses dictionary encoding). The file-path can be a single file, a metadata file pointing to other data files, or a directory (tree) containing data files. The latter is what is typically output by hive/spark.

Writing

from fastparquet import write
write('outfile.parq', df)
write('outfile2.parq', df, row_group_offsets=[0, 10000, 20000],
      compression='GZIP', file_scheme='hive')

The default is to produce a single output file with a single row-group (i.e., logical segment) and no compression. At the moment, only simple data-types and plain encoding are supported, so expect performance to be similar to numpy.savez.

History

Since early October 2016, this fork of parquet-python has been undergoing considerable redevelopment. The aim is to have a small and simple and performant library for reading and writing the parquet format from python.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastparquet-0.3.0.tar.gz (148.3 kB view details)

Uploaded Source

Built Distribution

fastparquet-0.3.0-cp36-cp36m-macosx_10_7_x86_64.whl (178.8 kB view details)

Uploaded CPython 3.6m macOS 10.7+ x86-64

File details

Details for the file fastparquet-0.3.0.tar.gz.

File metadata

  • Download URL: fastparquet-0.3.0.tar.gz
  • Upload date:
  • Size: 148.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/3.6.8

File hashes

Hashes for fastparquet-0.3.0.tar.gz
Algorithm Hash digest
SHA256 cab1de69c2f941c1e56b70933ed981d3577e176b3fbcd699b3b2d0b09363885c
MD5 3c902aa3dfc4439451996ff144cc7b8e
BLAKE2b-256 669b2488c574e12a2e77ab624dcf7bc6b7c42159a9428919b4dd6d5a78a84855

See more details on using hashes here.

Provenance

File details

Details for the file fastparquet-0.3.0-cp36-cp36m-macosx_10_7_x86_64.whl.

File metadata

  • Download URL: fastparquet-0.3.0-cp36-cp36m-macosx_10_7_x86_64.whl
  • Upload date:
  • Size: 178.8 kB
  • Tags: CPython 3.6m, macOS 10.7+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/3.6.8

File hashes

Hashes for fastparquet-0.3.0-cp36-cp36m-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 8333cd705f05edc62fdc1859d160e24b24a6f22b04af6cb7e6546482216ff395
MD5 b43c1913eaa9e74599d66a25149f3606
BLAKE2b-256 b2f4f7a34885c02e1f1cd9ae3be3277fe5aefb2d0e184cdb2df95a464dae0fea

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page