ssub 0.2.0

Use case

smon has done its job to persistantly store the raw sequencing runs in a Google Storage bucket. Now another tool is necessary that can automatially start demultiplexing. However, demultiplexing generally mustn’t begin until it has a SampleSheet. But once a SampleSheet is readily available, demultiplexing needs to start and the results need to be uploaded to Google Storage.

How it works

SampleSheets Subscriber (ssub) solves the aforementioned challenges by utilizing the power of GCP events and triggers. At a high level, it works as follows:

1. User/application uploads a samplesheet to a dedicated bucket. The sample sheet naming convention is ${RUN_NAME }.csv.

2. Google Storage immediately fires off an event to a Pub/Sub topic (whenever there is a new SampleSheet or when an existing one is overwritten).

3. Meanwhile, ssub is running on a compute instance as a daemon process. It is subscribed to that same Pub/Sub topic. ssub polls the topic for new messages regularly, i.e. once a minute.

4. When ssub receives a new message, the script parses information about the event.

5. ssub will the query the Firestore collection - the same one used by smon - for a document whose name is equal to the samplesheet name (minus the .csv part). ssub will then download both the samplesheet and the run tarball. The samplesheet location is provided in the Pub/Sub message; the raw run tarball location is provided within the Firestore document.

6. ssub will then kick off bcl2fastq.

7. Demultiplexing results are output in a folder name ‘demux’ within the local run directory.

8. ssub will upload the demux folder to the same Google Storage folder that contains the raw sequencing run.

9. ssub will update the relevant Firestore document to add the location to the demux folder in Google Storage.

All processing happens within a sub-directory of the calling directory that is named ssub_runs.

Reanalysis

Reruns of the dmeultiplexing pipeline may be necessary for various reasons, i.e. the demultiplexing results are no longer available and need to be regenerated, or there was a missing barcode in the SampleSheet, or there was an incorrect barcode in the SampleSheet …

Reanalysis is easily accomplished simply by re-uploading the SampleSheet, overwriting the previous one, to Google Storage. Google Storage will assign a generation number to the SampleSheet. Think of the generation number as a type of versioning number that Google Storage assigns to each object each time that the object changes. Even re-uploading the same exact same file again produces a new generation number.

Internally, ssub does all of it’s processing (file downloads, analysis) within a local directory path named after the run and the generation number of the SampleSheet. Thus, it’s perfectly fine for a user to upload an incorrect SampleSheet, and then to immediately afterwards upload the correct one. In such a scenario, there will be two runs of the pipeline, and they won’t interfere with each other. You will notice, however, that there will be two sets of demultplexing results uploaded to Google Storage, each of which exist within a folder named after the original generation number.

The configuration file

This is a small JSON file that lets the monitor know things such as which GCP bucket and Firestore collection to use, for example. The possible keys are:

• name: The client name of the subscriber. The name will appear in the subject line if email notification is configured, as well as in other places, i.e. log messages.

• cycle_pause_sec: The number of seconds to wait in-between polls of the Pub/Sub topic. Defaults to 60.

• firestore_collection: The name of the Google Firestore collection to use for persistent workflow state that downstream tools can query. If it doesn’t exist yet, it will be created. If this parameter is not provided, support for Firestore is turned off.

• sweep_age_sec: When an analysis directory (within the ssub_runs directory)

  is older than this many seconds, remove it. Defaults to 604800 (1 week).

The user-supplied configuration file is validated against a built-in schema.

Installation

In a later version of Python3, run:

pip3 install ssub

It is recommended to start your compute instance (that will run the monitor) using a service account with the following roles:

• roles/storage.objectAdmin

• roles/datastore.owner

Alternatively, give your compute instance the cloud-platform scope.

Deployment:

It is suggested to use the Dockerfile that comes in the respository.