An intake plugin for parsing an ESM (Earth System Model) Collection/catalog and loading assets (netCDF files and/or Zarr stores) into xarray datasets.
Project description
Intake-esm
Motivation
Project efforts such as the Coupled Model Intercomparison Project (CMIP) and the Community Earth System Model (CESM) Large Ensemble Project produce a huge of amount climate data persisted on tape, disk storage, object storage components across multiple (in the order of ~ 300,000) data assets. These data assets are stored in netCDF and more recently Zarr formats. Finding, investigating, loading these assets into data array containers such as xarray can be a daunting task due to the large number of files a user may be interested in. Intake-esm aims to address these issues by providing necessary functionality for searching, discovering, data access/loading.
Overview
intake-esm is a data cataloging utility built on top of intake, pandas, and xarray, and it’s pretty awesome!
Opening an ESM collection definition file: An ESM (Earth System Model) collection file is a JSON file that conforms to the ESM Collection Specification. When provided a link/path to an esm collection file, intake-esm establishes a link to a database (CSV file) that contains data assets locations and associated metadata (i.e., which experiement, model, the come from). The collection JSON file can be stored on a local filesystem or can be hosted on a remote server.
>>> import intake >>> col_url = "https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json" >>> col = intake.open_esm_datastore(col_url)Search and Discovery: intake-esm provides functionality to execute queries against the database:
>>> cat = col.search(experiment_id=['historical', 'ssp585'], table_id='Oyr', ... variable_id='o2', grid_label='gn')Access: when the user is satisfied with the results of their query, they can ask intake-esm to load data assets (netCDF/HDF files and/or Zarr stores) into xarray datasets:
>>> dset_dict = cat.to_dataset_dict(zarr_kwargs={'consolidated': True, 'decode_times': False}, ... cdf_kwargs={'chunks': {}, 'decode_times': False})
See documentation for more information.
Installation
Intake-esm can be installed from PyPI with pip:
pip install intake-esmIt is also available from conda-forge for conda installations:
conda install -c conda-forge intake-esmRelease history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file intake-esm-2019.12.13.tar.gz.
File metadata
- Download URL: intake-esm-2019.12.13.tar.gz
- Upload date:
- Size: 229.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8b4684e2ffcf8705e4cf2305dfe9453a19006838120583be97cb57824149544e |
|
| MD5 | 0b89c80e3c60c46a489673f07fe9d842 |
|
| BLAKE2b-256 | ee88b0c1edb998ac204d86cd64cdd6182de9a69b48d79b9e4485f65421d7fe48 |
File details
Details for the file intake_esm-2019.12.13-py3-none-any.whl.
File metadata
- Download URL: intake_esm-2019.12.13-py3-none-any.whl
- Upload date:
- Size: 17.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1ae423da8ae7466904f992df8ae6a7b11610dd8a46d0b50ca8f34868814bf08a |
|
| MD5 | 8d3f9113d70236798e305a50a1b2278b |
|
| BLAKE2b-256 | 078476cffc7f36433b68f9ee9311df7746d98aa124a29659155256e9a626e18f |
