Hera is a Python framework for constructing and submitting Argo Workflows. The main goal of Hera is to make Argo Workflows more accessible by abstracting away some setup that is typically necessary for constructing Argo workflows.
Project description
Hera (hera-workflows)
The Argo was constructed by the shipwright Argus,
and its crew were specially protected by the goddess Hera.
(https://en.wikipedia.org/wiki/Argo)
Hera is a Python framework for constructing and submitting Argo Workflows. The main goal of Hera is to make the Argo ecosystem accessible by simplifying workflow construction and submission.
You can watch the introductory Hera presentation at the "Argo Workflows and Events Community Meeting 20 Oct 2021" here!
Table of content
Requirements
Hera requires an Argo server to be deployed to a Kubernetes cluster. Currently, Hera assumes that the Argo server sits behind an authentication layer that can authenticate workflow submission requests by using the Bearer token on the request. To learn how to deploy Argo to your own Kubernetes cluster you can follow the Argo Workflows guide!
Another option for workflow submission without the authentication layer is using port forwarding to your Argo server
deployment and submitting workflows to localhost:2746 (2746 is the default, but you are free to use yours). Please
refer to the documentation of Argo Workflows to see the
command for port forward!
Note Since the deprecation of tokens being automatically created for ServiceAccounts and Argo using Bearer tokens in place, it is necessary to use
--auth=serverand/or--auth=clientwhen setting up Argo Workflows on Kubernetes v1.24+ in order for hera-workflows to communicate to the Argo Server.
Installation
| Source | Command |
|---|---|
| PyPi | pip install hera-workflows |
| Conda | conda install -c conda-forge hera-workflows |
| GitHub repo | python -m pip install git+https://github.com/argoproj-labs/hera-workflows --ignore-installed/pip install . |
Examples
from hera import Task, Workflow
def say(message: str):
print(message)
with Workflow("diamond") as w:
a = Task('a', say, ['This is task A!'])
b = Task('b', say, ['This is task B!'])
c = Task('c', say, ['This is task C!'])
d = Task('d', say, ['This is task D!'])
a >> [b, c] >> d
w.create()
See the examples directory for a collection of Argo workflow construction and submission via Hera!
Contributing
If you plan to submit contributions to Hera you can install Hera in a virtual environment managed by poetry:
poetry install
Once the dependencies are installed, you can use the various make targets to replicate the CI jobs.
make help
check-codegen Check if the code is up to date
ci Run all the CI checks
codegen Generate all the code
events-models Generate the Events models portion of Argo Workflows
events-service Generate the events service option of Hera
examples Generate all the examples
format Format and sort imports for source, tests, examples, etc.
help Showcase the help instructions for all the available `make` commands
lint Run a `lint` process on Hera and report problems
models Generate all the Argo Workflows models
services Generate the services of Hera
test Run tests for Hera
workflows-models Generate the Workflows models portion of Argo Workflows
workflows-service Generate the Workflows service option of Hera
Also, see the contributing guide!
Comparison
There are other libraries currently available for structuring and submitting Argo Workflows:
- Couler, which aims to provide a unified interface for constructing and managing workflows on different workflow engines;
- Argo Python DSL, which allows you to programmaticaly define Argo worfklows using Python.
While the aforementioned libraries provide amazing functionality for Argo workflow construction and submission, they require an advanced understanding of Argo concepts. When Dyno Therapeutics started using Argo Workflows, it was challenging to construct and submit experimental machine learning workflows. Scientists and engineers at Dyno Therapeutics used a lot of time for workflow definition rather than the implementation of the atomic unit of execution - the Python function - that performed, for instance, model training.
Hera presents a much simpler interface for task and workflow construction, empowering users to focus on their own executable payloads rather than workflow setup. Here's a side by side comparison of Hera, Argo Python DSL, and Couler:
| Hera | Couler | Argo Python DSL |
|---|---|---|
from hera import Task, Workflow
def say(message: str):
print(message)
with Workflow("diamond") as w:
a = Task('a', say, ['This is task A!'])
b = Task('b', say, ['This is task B!'])
c = Task('c', say, ['This is task C!'])
d = Task('d', say, ['This is task D!'])
a >> [b, c] >> d
w.create()
|
import couler.argo as couler
from couler.argo_submitter import ArgoSubmitter
def job(name):
couler.run_container(
image="docker/whalesay:latest",
command=["cowsay"],
args=[name],
step_name=name,
)
def diamond():
couler.dag(
[
[lambda: job(name="A")],
[lambda: job(name="A"), lambda: job(name="B")], # A -> B
[lambda: job(name="A"), lambda: job(name="C")], # A -> C
[lambda: job(name="B"), lambda: job(name="D")], # B -> D
[lambda: job(name="C"), lambda: job(name="D")], # C -> D
]
)
diamond()
submitter = ArgoSubmitter()
couler.run(submitter=submitter)
|
from argo.workflows.dsl import Workflow
from argo.workflows.dsl.tasks import *
from argo.workflows.dsl.templates import *
class DagDiamond(Workflow):
@task
@parameter(name="message", value="A")
def A(self, message: V1alpha1Parameter) -> V1alpha1Template:
return self.echo(message=message)
@task
@parameter(name="message", value="B")
@dependencies(["A"])
def B(self, message: V1alpha1Parameter) -> V1alpha1Template:
return self.echo(message=message)
@task
@parameter(name="message", value="C")
@dependencies(["A"])
def C(self, message: V1alpha1Parameter) -> V1alpha1Template:
return self.echo(message=message)
@task
@parameter(name="message", value="D")
@dependencies(["B", "C"])
def D(self, message: V1alpha1Parameter) -> V1alpha1Template:
return self.echo(message=message)
@template
@inputs.parameter(name="message")
def echo(self, message: V1alpha1Parameter) -> V1Container:
container = V1Container(
image="alpine:3.7",
name="echo",
command=["echo", "{{inputs.parameters.message}}"],
)
return container
|
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file hera_workflows-5.0.0rc1.tar.gz.
File metadata
- Download URL: hera_workflows-5.0.0rc1.tar.gz
- Upload date:
- Size: 198.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.16
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a5ecefc8036deea250dfa5775a98175f27b6815d7bc7005a108538361caf74ba |
|
| MD5 | bc12d9eb24d332b52f7b0f9cf187f077 |
|
| BLAKE2b-256 | 91d527d68a9b4685bcee6119ecb91cc67f461e353fa26cc4e59fa04e32cf1825 |
File details
Details for the file hera_workflows-5.0.0rc1-py3-none-any.whl.
File metadata
- Download URL: hera_workflows-5.0.0rc1-py3-none-any.whl
- Upload date:
- Size: 226.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.16
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a64d4e16e65d82b6da04e887f9c6255da5aeee2d9b4661858f0d8157aa494e7c |
|
| MD5 | 291aab1e445e6bbfec34c3c783ec7e5a |
|
| BLAKE2b-256 | 9508bc44958b6c1ede0b54a971bbe8ff1a8aba3e12a10a4df221cadcb7fdfc45 |
