Bookshelf

bookshelf is how Climate Resource reuses datasets across projects

The bookshelf represents a shared collection of curated datasets or Books. Each Book is a preprocessed, versioned dataset including the notebooks used to produce it. As the underlying datasets or processing are updated, new Books can be created (with an updated version). A single dataset may produce multiple Resources if different representations are useful. These Books can be deployed to a shared Bookshelf so that they are accessible by other users.

Users are able to use specific Books within other projects. The dataset and associated metadata is fetched and cached locally. Specific versions of Books can also be pinned for reproducibility purposes.

This repository contains the notebooks that are used to generate the Books as well as a CLI tool for managing these datasets.

This is a prototype and will likely change in future. Other potential ideas:

• Deployed data are made available via api.climateresource.com.au so that they can be consumed queried smartly
• Simple web page to allow querying the data

Each Book consists of a datapackage description of the metadata. This datapackage contains the associated Resources and their hashes. Each Resource is fetched when it is first used and then cached for later use

Full documentation can be found at: bookshelf.readthedocs.io. We recommend reading the docs there because the internal documentation links don't render correctly on GitLab's viewer.

Installation

bookshelf can be installed via pip:

pip install bookshelf

For developers

For development, we rely on poetry for all our dependency management. To get started, you will need to make sure that poetry is installed (instructions here, we found that pipx and pip worked better to install on a Mac).

For all of work, we use our Makefile. You can read the instructions out and run the commands by hand if you wish, but we generally discourage this because it can be error prone. In order to create your environment, run make virtual-environment.

If there are any issues, the messages from the Makefile should guide you through. If not, please raise an issue in the issue tracker.

For the rest of our developer docs, please see .