Skip to main content

Functions to make reference descriptions for ReferenceFileSystem

Project description

kerchunk

Cloud-friendly access to archival data

Docs Tests Pypi Conda-forge

Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly in-situ access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.

Why Kerchunk:

We provide the following things:

  • completely serverless architecture
  • metadata consolidation, so you can understand a many-file dataset (metadata plus physical storage) in a single read
  • read from all of the storage backends supported by fsspec, including object storage (s3, gcs, abfs, alibaba), http, cloud user storage (dropbox, gdrive) and network protocols (ftp, ssh, hdfs, smb...)
  • loading of various file types (currently netcdf4/HDF, grib2, tiff, fits, zarr), potentially heterogeneous within a single dataset, without a need to go via the specific driver (e.g., no need for h5py)
  • asynchronous concurrent fetch of many data chunks in one go, amortizing the cost of latency
  • parallel access with a library like zarr without any locks
  • logical datasets viewing many (>~millions) data files, and direct access/subselection to them via coordinate indexing across an arbitrary number of dimensions
logo

For further information, please see the documentation.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kerchunk-0.2.7.tar.gz (709.1 kB view details)

Uploaded Source

Built Distribution

kerchunk-0.2.7-py3-none-any.whl (62.9 kB view details)

Uploaded Python 3

File details

Details for the file kerchunk-0.2.7.tar.gz.

File metadata

  • Download URL: kerchunk-0.2.7.tar.gz
  • Upload date:
  • Size: 709.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.9

File hashes

Hashes for kerchunk-0.2.7.tar.gz
Algorithm Hash digest
SHA256 0425aa0fbf56f898053ee4c4dd40b35cea12d2fc986e036086e99a4ad16bd4e6
MD5 d2aea7a6c226b04dcf59e5bd8f692a54
BLAKE2b-256 af8aabf6a85db39c046db06081d34aa128980785b70c8b4b6618e44ebfefee11

See more details on using hashes here.

File details

Details for the file kerchunk-0.2.7-py3-none-any.whl.

File metadata

  • Download URL: kerchunk-0.2.7-py3-none-any.whl
  • Upload date:
  • Size: 62.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.10.9

File hashes

Hashes for kerchunk-0.2.7-py3-none-any.whl
Algorithm Hash digest
SHA256 9c0b4f721d0d6fef93fb5ffd3e0906d7a776bb19fb8347c02449899972c9b48c
MD5 a23e8d82b923dded4405e4acf10025fc
BLAKE2b-256 e1c91280fa083aee51224327a44ffcd6037b1c6ed914159e46757be631b3f776

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page