roocs-utils 0.4.2

roocs-utils

A package containing common components for the roocs project

• Free software: BSD - see LICENSE file in top-level package directory

• Documentation: https://roocs-utils.readthedocs.io.

$ export ABCUNIT_DB_SETTINGS="dbname=<name> user=<user> host=<host> password=<pwd>"

$ pip install abcunit-backend

Creating batches

Once the list of datasets is collated a number of batches must be created:

$ python roocs_utils/catalog_maker/cli.py create-batches -p c3s-cmip6

The option -p is required to specify the project.

Creating catalog entries

Once the batches are created, the catalog maker can be run - either locally or on lotus. The settings for how many datasets to be included in a batch and the maximum duration of each job on lotus can also be changed in roocs_utils/etc/roocs.ini.

Each batch can be run idependently, e.g. running batch 1 locally:

$ python roocs_utils/catalog_maker/cli.py run -p c3s-cmip6 -b 1 -r local

or running all batches on lotus:

$ python roocs_utils/catalog_maker/cli.py run -p c3s-cmip6 -r lotus

This creates a table in the database containing an ordered dictionary of the entry for each file in each dataset if successful, or the error traceback if there is an Exception raised.

Viewing entries and errors

To view the records:

$ python roocs_utils/catalog_maker/cli.py list -p c3s-cmip6

With many entries, this may take a while.

To just get a count of how many files have been scanned:

$ python roocs_utils/catalog_maker/cli.py list -p c3s-cmip6 -c

To see any errors:

$ python roocs_utils/catalog_maker/cli.py show-errors -p c3s-cmip6

To see just a count of errors:

$ python roocs_utils/catalog_maker/cli.py show-errors -p c3s-cmip6 -c

Each count will show how many files and how many datasets have been successful/failed.

The list count will also show the total numbers of datasets/files in the database - including errors. The error count will show whether there are any datasets that have files which have succeeded and failed i.e. that are partially scanned.

Writing to CSV

The final command is to write the entries to a csv file.

$ python roocs_utils/catalog_maker/cli.py write -p c3s-cmip6

The csv file will be generated in the csv_dir specified in roocs_utils/etc/roocs.ini and will have the name “{project}_{version_stamp}.csv”. e.g. c3s-cmip6_v20210414.csv

A yaml file will be created the catalog_dir specified in roocs_utils/etc/roocs.ini. It will have the name c3s.yml and will contain the below for each project scanned and which is using the same catalog_dir:

sources:
  c3s-cmip6:
    args:
      urlpath:
    cache:
    - argkey: urlpath
      type: file
    description: c3s-cmip6 datasets
    driver: intake.source.csv.CSVSource
    metadata:
      last_updated:

urlpath and last_updated for a project will be updated very time the csv file is written for the project.

Deleting the table of results

In order to delete all entries in the table of results

$ python roocs_utils/catalog_maker/cli.py clean -p c3s-cmip6

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

• Cookiecutter: https://github.com/audreyr/cookiecutter

• cookiecutter-pypackage: https://github.com/audreyr/cookiecutter-pypackage