Functions to make reference descriptions for ReferenceFileSystem

Project description

kerchunk

Cloud-friendly access to archival data

Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage.
It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly in-situ access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.

(formerly known as fsspec-reference-maker)

For further information, please see the documentation pages.

Project details

Release history Release notifications | RSS feed

0.2.6

Jul 8, 2024

0.2.5

May 10, 2024

0.2.4

Mar 5, 2024

0.2.3

Feb 5, 2024

0.2.2

Oct 25, 2023

0.2.1

Oct 21, 2023

0.2.0

Aug 30, 2023

0.1.2

May 12, 2023

0.1.1

Apr 17, 2023

0.1.0

Jan 19, 2023

0.0.9

Oct 24, 2022

0.0.8

Sep 9, 2022

0.0.7

Jul 27, 2022

0.0.6

Mar 22, 2022

This version

0.0.5

Oct 20, 2021

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kerchunk-0.0.5.tar.gz (37.9 kB view details)

Uploaded Oct 20, 2021 Source

Built Distribution

kerchunk-0.0.5-py3-none-any.whl (24.2 kB view details)

Uploaded Oct 20, 2021 Python 3

File details

Details for the file kerchunk-0.0.5.tar.gz.

File metadata

Download URL: kerchunk-0.0.5.tar.gz
Upload date: Oct 20, 2021
Size: 37.9 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.24.0 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8

File hashes

Hashes for kerchunk-0.0.5.tar.gz
Algorithm	Hash digest
SHA256	`eb5de054215d13e733a0c71c91ae03d5d54124d446365319cfe1aa8e116ecb7e`
MD5	`3a9d916812aad603c42af69db9845d76`
BLAKE2b-256	`10232e3dc294776e2e03cb1d5380ab7d1f35ea5f674d3c2b8f9f4d249e44ab61`

See more details on using hashes here.

File details

Details for the file kerchunk-0.0.5-py3-none-any.whl.

File metadata

Download URL: kerchunk-0.0.5-py3-none-any.whl
Upload date: Oct 20, 2021
Size: 24.2 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.24.0 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8

File hashes

Hashes for kerchunk-0.0.5-py3-none-any.whl
Algorithm	Hash digest
SHA256	`757ddc6e5c32106ac17021ae0349ecad6c93dcd832abbceb0eafc2793028b474`
MD5	`f1e86fc3fb63a26be2604678c5fcc599`
BLAKE2b-256	`60a630e0577702fa66ca828970da8ee57ed68cdbc68983ee1a7f3eedc0ef5f7c`