REANA-Commons.
Project description
REANA-Commons
About
REANA-Commons is a component of the REANA reusable and reproducible research data analysis platform. It provides common utilities and schemas shared by the REANA cluster components.
Features
- common API clients for internal communication
- centralised OpenAPI specifications for REANA components
- AMQP connection management and communication
- utility functions for cluster components
Usage
The detailed information on how to install and use REANA can be found in docs.reana.io.
Useful links
- REANA project home page
- REANA user documentation
- REANA user support forum
- REANA-Commons releases
- REANA-Commons developer documentation
- REANA-Commons known issues
- REANA-Commons source code
Changelog
0.9.8 (2024-03-01)
Build
- python: change extra names to comply with PEP 685 (#446) (9dad6da)
- python: require smart-open<7 for Python 3.6 (#446) (17fd581)
- python: restore snakemake
reportsextra (#446) (904178f)
Continuous integration
0.9.7 (2024-02-20)
Build
Documentation
0.9.6 (2024-02-13)
Features
- config: allow customisation of runtime group name (#440) (5cec305)
- snakemake: upgrade to Snakemake 7.32.4 (#435) (20ae9ce)
Bug fixes
Code refactoring
Continuous integration
- commitlint: addition of commit message linter (#432) (a67906f)
- commitlint: check for the presence of concrete PR number (#438) (d3035dc)
- release-please: initial configuration (#432) (687f2f4)
- shellcheck: check all shell scripts recursively (#436) (709a685)
- shellcheck: fix exit code propagation (#438) (85d9a2a)
0.9.5 (2023-12-15)
- Fixes installation by pinning
bravado-coreto versions lower than 6.1.1.
0.9.4 (2023-11-30)
- Changes the REANA specification schema to use the
draft-07version of the JSON Schema specification. - Changes validation of REANA specification to expose functions for loading workflow input parameters and workflow specifications.
- Changes validation of REANA specification to make the
environmentproperty mandatory for the steps of serial workflows. - Changes validation of REANA specification to raise a warning for unexpected properties for the steps of serial workflows.
- Changes CVMFS support to allow users to automatically mount any available repository.
- Fixes the mounting of CVMFS volumes for the REANA deployments that use non-default Kubernetes namespace.
0.9.3 (2023-09-26)
- Adds support for Python 3.12.
- Adds the OpenAPI specification support for
prune_workspaceendpoint that allows to delete files that are neither inputs nor outputs from the workspace. - Adds support for
tests.filesinreana.yamlallowing to specify Gherkin feature files for testing runnable examples. - Changes the OpenAPI specification to include the
run_stopped_atproperty in the workflow progress information returned by the workflow list and workflow status endpoints. - Changes the OpenAPI specification to include the
maximum_interactive_session_inactivity_periodvalue to theinfoendpoint. - Changes the email sending utility to allow configuring authentication and encryption options.
- Changes validation of REANA specification to emit warnings about unknown properties.
- Fixes the verbs used to describe changes to the status of a workflow in order to avoid incorrect grammatical phrases such as
workflow has been failed. - Fixes the loading of Snakemake and CWL workflow specifications when no parameters are specified.
- Fixes the OpenAPI specification of GitLab OAuth endpoint return statuses.
- Fixes container image names to be Podman-compatible.
- Fixes the email sending utility to not send emails when notifications are disabled globally.
0.9.2.1 (2023-07-19)
- Changes
PyYAMLdependency version bounds in order to fix installation on Python 3.10+.
0.9.2 (2023-02-10)
- Fixes
wcmatchdependency version specification.
0.9.1 (2023-01-18)
- Changes Kerberos renew container's configuration to log each ticket renewal.
0.9.0 (2022-12-13)
- Adds support for Python 3.11.
- Adds support for Rucio.
- Adds REANA specification validation and loading logic from
reana-client. - Adds common utility functions for managing workspace files.
- Adds OpenAPI specification support for
launchendpoint that allows running workflows from remote sources. - Adds OpenAPI specification support for
get_workflow_retention_rulesendpoint that allows to retrieve the workspace file retention rules of a workflow. - Adds generation of Kerberos init and renew container's configuration.
- Adds support for Unicode characters inside email body.
- Changes OpenAPI specification to include missing response schema elements and some other small enhancements.
- Changes the Kubernetes Python client to use the
networking/v1API. - Changes REANA specification loading functionality to allow specifying different working directories.
- Changes REANA specification to allow enabling Kerberos for the whole workflow.
- Changes REANA specification to allow specifying
retention_daysfor the workflow. - Changes REANA specification to allow specifying
slurm_partitionandslurm_timefor Slurm compute backend jobs. - Changes the loading of Snakemake specifications to preserve the current working directory.
- Fixes the submission of jobs by stripping potential leading and trailing whitespaces in Docker image names.
0.8.5 (2022-02-23)
- Adds
retry_countparameter to WorkflowSubmissionPublisher.
0.8.4 (2022-02-08)
- Adds new configuration variable to toggle Kubernetes security context. (
K8S_USE_SECURITY_CONTEXT) - Changes installation to revert
Yadagedependency versions.
0.8.3 (2022-02-04)
- Changes installation to remove upper version pin on
kombu.
0.8.2 (2022-02-01)
- Adds support for Python 3.10.
- Adds workflow name validation utility.
- Changes
Snakemakeloaded specification to include compute backends. - Changes OpenAPI specification with respect to return supported compute backends in
infoendpoint. - Fixes file system usage calculation on CephFS shares in
get_disk_usageutility function.
0.8.1 (2021-12-21)
- Adds OpenAPI specification support for
kubernetes_job_timeouthandling. - Changes OpenAPI specification for cluster health status endpoint.
- Changes Yadage dependencies to allow 0.21.x patchlevel-version updates.
- Changes installation to require Python-3.6 or higher versions.
0.8.0 (2021-11-22)
- Adds
get_disk_usageutility function to calculate disk usage for a directory. - Adds
Yadageworkflow specification loading utilities. - Adds workspace validation utilities.
- Adds
Snakemakeworkflow engine integration. - Adds custom objects API instance to k8s client.
- Adds available worklow engines configuration.
- Adds environment variable to define time between job controller connection checks.
- Adds cluster health status endpoint.
- Adds OpenAPI specifications with respect to user quotas.
- Changes
workflow-submissionqueue as a priority queue and allows to set the priority number on workflow submission. - Changes OpenAPI specifications with respect to turning
workspacesendpoint intoinfo. - Changes publisher logging level on error callback.
- Removes support for Python 2.
0.7.5 (2021-07-02)
- Adds support for glob patterns when listing workflow files.
- Adds support for specifying
kubernetes_memory_limitfor Kubernetes compute backend jobs.
0.7.4 (2021-03-17)
- Adds new functions to serialise/deserialise job commands between REANA components.
- Changes
reana_readyfunction location to REANA-Server.
0.7.3 (2021-02-22)
- Adds new configuration variable to toggle runtime user jobs clean up depending on their statuses. (
REANA_RUNTIME_KUBERNETES_KEEP_ALIVE_JOBS_WITH_STATUSES) - Adds central class to instantiate workflow engines with more resilience. (
workflow_engine.create_workflow_engine_command)
0.7.2 (2021-02-02)
- Adds support for Python 3.9.
- Fixes minor code warnings.
- Fixes a helper function that calculates directory hashes.
- Changes OpenAPI specifications with respect to sign-up form.
- Changes OpenAPI specifications with respect to email confirmation.
- Changes CI system to include Python flake8 checker.
0.7.1 (2020-11-09)
- Adds support for restarting yadage workflows (through
accept_metadiroperational option). - Allows
htcondor_max_runtimeandhtcondor_accounting_groupto be specified for HTC jobs. - Adds new field in REANA-Server OpenAPI spec to return server version.
- Changes CI system from Travis to GitHub Actions.
0.7.0 (2020-10-20)
- Adds new utility to send emails.
- Adds centralised validation utility for workflow operational options.
- Adds new configuration variable to set the maximum number of running workflows. (
REANA_MAX_CONCURRENT_BATCH_WORKFLOWS) - Adds new configuration variable to set prefix of REANA cluster component names. (
REANA_COMPONENT_PREFIX) - Adds new configuration variable for the runtime pod node selector label. (
REANA_RUNTIME_KUBERNETES_NODE_LABEL) - Adds new configuration variable to define the Kubernetes namespace in which REANA infrastructure components run. (
REANA_INFRASTRUCTURE_KUBERNETES_NAMESPACE) - Adds new configuration variable to define the Kubernetes namespace in which REANA runtime components components run. (
REANA_RUNTIME_KUBERNETES_NAMESPACE) - Adds possibility to specify unpacked container images for running jobs.
- Adds support for
initfilesoperational option for the Yadage workflow engine. - Fixes memory leak in Bravado client instantiation.
- Changes CephFS Persistent Volume Claim name. (
REANA_SHARED_PVC_NAME) - Changes default logging level to
INFO. - Changes default CVMFS volume list to include LHCb Gaudi related workflows.
- Changes code formatting to respect
blackcoding style. - Changes underlying requirements to use Kubernetes Python library version 11.
- Changes underlying requirements to use latest CVMFS CSI driver version.
- Changes documentation to single-page layout.
0.6.1 (2020-05-25)
- Upgrades Kubernetes Python client.
0.6.0 (2019-12-19)
- Adds new API for Gitlab integration.
- Adds new Kubernetes client API for ingresses.
- Adds new APIs for management of user secrets.
- Adds EOS storage Kubernetes configuration.
- Adds HTCondor and Slurm compute backends.
- Adds support for streaming file uploads.
- Allows unpacked CVMFS and CMS open data volumes.
- Adds Serial workflow step name and compute backend.
- Adds support for Python 3.8.
0.5.0 (2019-04-16)
- Centralises log level and log format configuration.
- Adds new utility to inspect the disk usage on a given workspace.
(
get_workspace_disk_usage) - Introduces the module to share Celery tasks accross REANA
components. (
tasks.py) - Introduces common Celery task to determine whether REANA can
execute new workflows depending on a set of conditions
such as running job count. (
reana_ready,check_predefined_conditions,check_running_job_count) - Allows the AMQP consumer to be configurable with multiple queues.
- Introduces new queue for workflow submission. (
workflow-submission) - Introduces new publisher for workflow submissions.
(
WorkflowSubmissionPublisher) - Centralises Kubernetes API client configuration and initialisation.
- Adds Kubernetes specific configuration for CVMFS volumes as utils.
- Introduces a new method,
copy_openapi_specs, to automatically move validated OpenAPI specifications from components to REANA Commonsopenapi_specificationsdirectory. - Centralises interactive session types.
- Introduces central REANA errors through the
errors.pymodule. - Skips SSL verification for all HTTPS requests performed with the
BaseAPIClient.
0.4.0 (2018-11-06)
- Aggregates OpenAPI specifications of REANA components.
- Improves AMQP re-connection handling. Switches from
pikatokombu. - Enhances test suite and increases code coverage.
- Changes license to MIT.
0.3.1 (2018-09-04)
- Adds parameter expansion and validation utilities for parametrised Serial workflows.
0.3.0 (2018-08-10)
- Initial public release.
- Provides basic AMQP pub/sub methods for REANA components.
- Utilities for caching used in different REANA components.
- Click formatting helpers.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
reana-commons-0.9.8.tar.gz
(83.6 kB
view hashes)
