SWIRL: Scientific Workflow Intermediate Representation Language

SWIRL is a low-level intermediate representation language for distributed scientific workflows. Unlike other product-agnostic workflow languages, SWIRL is not intended for human interaction but serves as a low-level compilation target for distributed workflow execution plans. It models the execution plan of a location-aware workflow graph as a distributed system with send/receive communication primitives. The optimised SWIRL representation can then be compiled into one or more self-contained executable bundles, making it adaptable to specific execution environments and embracing heterogeneity. This repository contains the reference SWIRL Compiler toolchain, called swirlc, written in Python 3 and relying on the ANTLR parser generator.

Install

PyPI

The swirlc package is available on PyPi, so you can install it using pip.

pip install swirlc

Please note that the SWIRL compiler requires python >= 3.8. Then you can use it through the swirlc CLI.

Docker

The SWIRL Docker image is available on Docker Hub. The script below gives an example of swirlc CLI invocation in a Docker container

mkdir -p SWIRL
docker run                          \
        --user $(id -u):$(id -g)    \
        --volume $(pwd):/data       \
        --workdir /data             \
        alphaunito/swirlc:latest    \
    swirlc                          \
        COMMAND                     \
        [OPTION]                     

Use SWIRL

The swirlc CLI offers two primary functionalities: translate and compile. The former translates a distributed workflow written in one of the supported high-level languages into a low-level SWIRL intermediate representation. The latter compiles a SWIRL representation into a target executable bundle.

Translate

The swirlc translate command can be used to translate an existing, high-level workflow representation into a SWIRL low-level intermediate representation, usually stored in a *.swirl file, and a *.yml file containing a set of metadata. As an example, the following command can be used to generate a SWIRL representation from a DAX workflow generated by the Pegasus WMS:

swirlc translate --language dax [DAX_DIRECTORY]

Note that the DAX_DIRECTORY must contain four files: replica.yml, sites.yml, transformations.yml and workflow.yml.

Compile

The swirlc compile command can be used to create executable traces in different programming languages from a SWIRL representation. The result of the command is an executable bundle ready to run on highly-distributed execution environments (e.g., HPC, Cloud, or Edge). As an example, the following command can be used to generate a Python-based executable bundle from a SWIRL_FILE and a METADATA_FILE:

swirlc compile [SWIRL_FILE] [METADATA_FILE]

Note that all the target locations need to have the Python interpreter installed.

SWIRL Team

Iacopo Colonnelli iacopo.colonnelli@unito.it (Designer and maintainer)
Doriana Medić doriana.medic@unito.it (Designer and maintainer)
Alberto Mulone alberto.mulone@unito.it (Maintainer)