sruns-monitor 1.2.0

Use case

You have one or more Illumina sequencers that are writing to a mounted filesystem such as NFS. You need a way to detect when there is a new, completed sequencing run directory and then relocate it to redundant storage. Downstream tools need to be able to know when a tarred run directory is available to post-processing (i.e. demultiplexing, QC, read alignment, etc.).

How it works

Sequencing Runs Monitor solves the aforementioned challenges through the use of Google Cloud Platform services and by tracking workflow state. Sequencing runs are tarred with gzip compression and then uploaded to Google Cloud Storage. Workflow state is tracked locally via SQLite and optionally in the NoSQL database Google Firestore for redundancy and to allow downstream clients to query sequencing run records.

Note: while you don’t need to use a Google compute instance to run the monitor script, the documentation here assumes that you are since it is the recommended way. That’s due to the fact that the monitor must interact with certain GCP services, and hence must be running with proper Google credentials (i.e. a service account).

The monitor script is named srun-mon (a shim that calls the main function in launch_monitor.py). When running this, you must provide a path to a JSON configuration file, described in detail further below. You should set up your compute instance to run this script as a daemon service.

The workflow is fitted into two tasks: the tar task and the upload task. When the monitor detects a new sequencing run, it executes the workflow in a child process. The workflow is smart enough to detect which task to begin with, thanks to the local SQLite database. This database has a record for each sequencing run and tracks which workflow tasks have been completed, and whether the workflow is running.

The workflow tasks

The workflow is split up into two tasks, each of which can be run on its own (i.e. if the workflow needs to be restarted from where it left off).

The configuration file

This is a small JSON file that lets the monitor know things such as which GCP bucket and Firestore collection to use, for example. The possible keys are:

• name: The name of the monitor. The name will appear in the subject line if email notification is configured, as well as in other places, i.e. log messages.

• completed_runs_dir: The directory to move a run directory to after it has completed the workflow. This directory will be created if it doesn’t yet exist. Defaults to a folder by the name ‘SRM_COMPLETED` that resides within the same directory as the one being watched. Note that at present, there isn’t a means to clean out the completed runs directory, but that will come in a future release.

• cycle_pause_sec: The number of seconds to wait in-between scans of watchdir. Defaults to 60.

• firestore_collection: The name of the Google Firestore collection to use for persistent workflow state that downstream tools can query. If it doesn’t exist yet, it will be created. If this parameter is not provided, support for Firestore is turned off.

• gcp_bucket_basedir: The directory in gcp_bucket_name in which to store all uploaded files. Defaults to the root directory.

• gcp_bucket_name: (Required) The name of the Google Cloud Storage bucket to which tarred run directories will be uploaded.

• sentinal_file_age_minutes: How old in minutes the sentinal file, i.e. CopyComplete.txt, should be before initiating any tasks, such as tarring the run directory. Illumina Support recommends 15 minutes, which is thus the default. This helps to ensure that the Illumina Universal Copy Services (UCS) running on the sequencer has had ample time to complete once the sentinal file appears.

• sqlite_db: The name of the local SQLite database to use for tracking workflow state. Defaults to sruns.db if not specified.

• sweep_age_sec: When a run in the completed runs directory is older than this many seconds, remove it. Defaults to 604800 (1 week).

• task_runtime_limit_sec: The number of seconds a child process is allowed to run before being killed. This is meant to serve as a safety mechanism to prevent errant child processes from consuming resources in the event that this does happen due to unforeseen circumstances. An email notification will be sent out in this case to alert about the errant process and the sequencing run it was associated with. The number of seconds you set for this depends on several factors, such as run size and network speed. It is suggested to use two days (172800 seconds) at least to be conservative.

• watchdir: (Required) The directory to monitor for new sequencing runs.

The user-supplied configuration file is validated in the Monitor against a built-in schema.

Tracking workflow state

The state of the workflow for a given run directory is tracked both locally in a SQLite database as well as Google Firestore - a NoSQL database. Local state is tracked for the purpose of being able to restart workflows if a child process ever crashes, or if the node goes down. Firestore is used to enable downstream applications to query the collection (whose name is specified in your configuration file) to do their own post-processing as desired. For example, an external tool could query the collection and ask if a given run is completed yet. Completed in this sense means that the run was tarred and uploded to a Google bucket. Then, the tool could tell where the tarfile blob is located.

Installation and setup

This works in later versions of Python 3 only:

pip3 install sruns-monitor

It is recommended to start your compute instance (that will run the monitor) using a service account with the following roles:

• roles/storage.objectAdmin

• roles/datastore.owner

Running Test Cases

Each module has associated test cases. There are both unit tests and functional tests.

python3 -m unittest

monitor_integration_tests.py