This package provides a stable interface for interactions between Snakemake and its executor plugins.
Project description
Stable interfaces and functionality for Snakemake executor plugins
This package provides a stable interface for interactions between Snakemake and its executor plugins (WIP).
Plugins should implement the following skeleton to comply with this interface:
from dataclasses import dataclass, field
from typing import List, Generator, Optional
from snakemake_interface_executor_plugins.executors.base import SubmittedJobInfo
from snakemake_interface_executor_plugins.executors.remote import RemoteExecutor
from snakemake_interface_executor_plugins import ExecutorSettingsBase, CommonSettings
from snakemake_interface_executor_plugins.workflow import WorkflowExecutorInterface
from snakemake_interface_executor_plugins.logging import LoggerExecutorInterface
from snakemake_interface_executor_plugins.jobs import (
ExecutorJobInterface,
)
# Optional:
# Define additional settings for your executor.
# They will occur in the Snakemake CLI as --<executor-name>-<param-name>
# Omit this class if you don't need any.
# Make sure that all defined fields are Optional and specify a default value
# of None or anything else that makes sense in your case.
@dataclass
class ExecutorSettings(ExecutorSettingsBase):
myparam: Optional[int]=field(
default=None,
metadata={
"help": "Some help text",
# Optionally request that setting is also available for specification
# via an environment variable. The variable will be named automatically as
# SNAKEMAKE_<executor-name>_<param-name>, all upper case.
# This mechanism should only be used for passwords and usernames.
# For other items, we rather recommend to let people use a profile
# for setting defaults
# (https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles).
"env_var": False,
# Optionally specify that setting is required when the executor is in use.
"required": True
}
)
# Required:
# Specify common settings shared by various executors.
common_settings = CommonSettings(
# define whether your executor plugin executes locally
# or remotely. In virtually all cases, it will be remote execution
# (cluster, cloud, etc.). Only Snakemake's standard execution
# plugins (snakemake-executor-plugin-dryrun, snakemake-executor-plugin-local)
# are expected to specify False here.
non_local_exec=True
)
# Required:
# Implementation of your executor
class Executor(RemoteExecutor):
def __init__(
self,
workflow: WorkflowExecutorInterface,
logger: LoggerExecutorInterface,
):
super().__init__(
workflow,
logger,
# configure behavior of RemoteExecutor below
# whether arguments for setting the remote provider shall be passed to jobs
pass_default_remote_provider_args=True,
# whether arguments for setting default resources shall be passed to jobs
pass_default_resources_args=True,
# whether environment variables shall be passed to jobs
pass_envvar_declarations_to_cmd=True,
# specify initial amount of seconds to sleep before checking for job status
init_sleep_seconds=0,
)
# access workflow
self.workflow
# access executor specific settings
self.workflow.executor_settings
# IMPORTANT: in your plugin, only access methods and properties of
# Snakemake objects (like Workflow, Persistence, etc.) that are
# defined in the interfaces found in the
# snakemake-interface-executor-plugins and the
# snakemake-interface-common package.
# Other parts of those objects are NOT guaranteed to remain
# stable across new releases.
# To ensure that the used interfaces are not changing, you should
# depend on these packages as >=a.b.c,<d with d=a+1 (i.e. pin the
# dependency on this package to be at least the version at time
# of development and less than the next major version which would
# introduce breaking changes).
# In case of errors outside of jobs, please raise a WorkflowError
def run_job(self, job: ExecutorJobInterface):
# Implement here how to run a job.
# You can access the job's resources, etc.
# via the job object.
# After submitting the job, you have to call
# self.report_job_submission(job_info).
# with job_info being of type
# snakemake_interface_executor_plugins.executors.base.SubmittedJobInfo.
# If required, make sure to pass the job's id to the job_info object, as keyword
# argument 'external_job_id'.
...
async def check_active_jobs(
self, active_jobs: List[SubmittedJobInfo]
) -> Generator[SubmittedJobInfo, None, None]:
# Check the status of active jobs.
# You have to iterate over the given list active_jobs.
# If you provided it above, each will have its external_jobid set according
# to the information you provided at submission time.
# For jobs that have finished successfully, you have to call
# self.report_job_success(active_job).
# For jobs that have errored, you have to call
# self.report_job_error(active_job).
# This will also take care of providing a proper error message.
# Usually there is no need to perform additional logging here.
# Jobs that are still running have to be yielded.
#
# For queries to the remote middleware, please use
# self.status_rate_limiter like this:
#
# async with self.status_rate_limiter:
# # query remote middleware here
#
# To modify the time until the next call of this method,
# you can set self.next_sleep_seconds here.
...
def cancel_jobs(self, active_jobs: List[SubmittedJobInfo]):
# Cancel all active jobs.
# This method is called when Snakemake is interrupted.
...
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for snakemake_interface_executor_plugins-3.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4cd8691e9d9f82c5c56d2f07851d5ad2e8f7a7c89c31181f4ecd0d1e49e0005 |
|
MD5 | 0b8f247bd5b5d6b70d7dc7b8542b6322 |
|
BLAKE2b-256 | 189528b075e9f15d947ebdbeef7d2a95d087dc1657ad92a42b63337ea7625a48 |
Close
Hashes for snakemake_interface_executor_plugins-3.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d12c2f9006a7bdea9c4594289f370232af65bab6e7d8cdae12e04c280e50a1f |
|
MD5 | 9d4a41664aeb5aa9de24d0dd29eb4583 |
|
BLAKE2b-256 | 49ee8d9e9399d1364dde6a3da721e2e33ec9b121be2e5944a23fbf9cb7c536a1 |