This package provides a stable interface for interactions between Snakemake and its executor plugins.
Project description
Stable interfaces and functionality for Snakemake executor plugins
This package provides a stable interface for interactions between Snakemake and its executor plugins (WIP).
Plugins should implement the following skeleton to comply with this interface:
from dataclasses import dataclass, field
from typing import List, Generator, Optional
from snakemake_interface_executor_plugins.executors.base import SubmittedJobInfo
from snakemake_interface_executor_plugins.executors.remote import RemoteExecutor
from snakemake_interface_executor_plugins import ExecutorSettingsBase, CommonSettings
from snakemake_interface_executor_plugins.workflow import WorkflowExecutorInterface
from snakemake_interface_executor_plugins.logging import LoggerExecutorInterface
from snakemake_interface_executor_plugins.jobs import (
JobExecutorInterface,
)
# Optional:
# Define additional settings for your executor.
# They will occur in the Snakemake CLI as --<executor-name>-<param-name>
# Omit this class if you don't need any.
# Make sure that all defined fields are Optional and specify a default value
# of None or anything else that makes sense in your case.
@dataclass
class ExecutorSettings(ExecutorSettingsBase):
myparam: Optional[int]=field(
default=None,
metadata={
"help": "Some help text",
# Optionally request that setting is also available for specification
# via an environment variable. The variable will be named automatically as
# SNAKEMAKE_<executor-name>_<param-name>, all upper case.
# This mechanism should only be used for passwords and usernames.
# For other items, we rather recommend to let people use a profile
# for setting defaults
# (https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles).
"env_var": False,
# Optionally specify that setting is required when the executor is in use.
"required": True
}
)
# Required:
# Specify common settings shared by various executors.
common_settings = CommonSettings(
# define whether your executor plugin executes locally
# or remotely. In virtually all cases, it will be remote execution
# (cluster, cloud, etc.). Only Snakemake's standard execution
# plugins (snakemake-executor-plugin-dryrun, snakemake-executor-plugin-local)
# are expected to specify False here.
non_local_exec=True
)
# Required:
# Implementation of your executor
class Executor(RemoteExecutor):
def __init__(
self,
workflow: WorkflowExecutorInterface,
logger: LoggerExecutorInterface,
):
super().__init__(
workflow,
logger,
# configure behavior of RemoteExecutor below
# whether arguments for setting the remote provider shall be passed to jobs
pass_default_remote_provider_args=True,
# whether arguments for setting default resources shall be passed to jobs
pass_default_resources_args=True,
# whether environment variables shall be passed to jobs
pass_envvar_declarations_to_cmd=True,
# specify initial amount of seconds to sleep before checking for job status
init_sleep_seconds=0,
)
# access workflow
self.workflow
# access executor specific settings
self.workflow.executor_settings
# IMPORTANT: in your plugin, only access methods and properties of
# Snakemake objects (like Workflow, Persistence, etc.) that are
# defined in the interfaces found in the
# snakemake-interface-executor-plugins and the
# snakemake-interface-common package.
# Other parts of those objects are NOT guaranteed to remain
# stable across new releases.
# To ensure that the used interfaces are not changing, you should
# depend on these packages as >=a.b.c,<d with d=a+1 (i.e. pin the
# dependency on this package to be at least the version at time
# of development and less than the next major version which would
# introduce breaking changes).
# In case of errors outside of jobs, please raise a WorkflowError
def run_job(self, job: JobExecutorInterface):
# Implement here how to run a job.
# You can access the job's resources, etc.
# via the job object.
# After submitting the job, you have to call
# self.report_job_submission(job_info).
# with job_info being of type
# snakemake_interface_executor_plugins.executors.base.SubmittedJobInfo.
# If required, make sure to pass the job's id to the job_info object, as keyword
# argument 'external_job_id'.
...
async def check_active_jobs(
self, active_jobs: List[SubmittedJobInfo]
) -> Generator[SubmittedJobInfo, None, None]:
# Check the status of active jobs.
# You have to iterate over the given list active_jobs.
# If you provided it above, each will have its external_jobid set according
# to the information you provided at submission time.
# For jobs that have finished successfully, you have to call
# self.report_job_success(active_job).
# For jobs that have errored, you have to call
# self.report_job_error(active_job).
# This will also take care of providing a proper error message.
# Usually there is no need to perform additional logging here.
# Jobs that are still running have to be yielded.
#
# For queries to the remote middleware, please use
# self.status_rate_limiter like this:
#
# async with self.status_rate_limiter:
# # query remote middleware here
#
# To modify the time until the next call of this method,
# you can set self.next_sleep_seconds here.
...
def cancel_jobs(self, active_jobs: List[SubmittedJobInfo]):
# Cancel all active jobs.
# This method is called when Snakemake is interrupted.
...
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for snakemake_interface_executor_plugins-4.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 243ad5efe6815c622a015d574af3eadbb7f3db0ecfff41c79a2dcea7036949aa |
|
MD5 | 11aad754cd18d016790f0276a39d4792 |
|
BLAKE2b-256 | 27c7725047ba2aac548f9be2c2f338c97c5e0466a22744c29773434fe85064ec |
Close
Hashes for snakemake_interface_executor_plugins-4.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f4a3a74851f45d213057eb4471cf989c7494958736b7266ce5b0dcf5866ca70 |
|
MD5 | 8f93166a3e07db3a7f0a1d50909d2698 |
|
BLAKE2b-256 | f7f429f8cbd6b1aaf3563b1268f859019cf274fcedbcd1822fbf36899a933e86 |