Framework for task processing executors and configuration
Project description
Task Processing
Interfaces and shared infrastructure for generic task processing (also known as taskproc) at Yelp.
Developer Setup
Pre-requisites
Running examples
hello-world.py is a very simple annotated example that launches a task to echo hello world. From the root of the repository, run:
docker-compose -f examples/cluster/docker-compose.yaml \
run playground examples/hello-world.py
This will bring up a single master, single agent Mesos cluster using Docker Compose and launch a single task which will print "hello world" to the sandbox's stdout before terminating.
Other examples available include:
-
async.py Example of the async task runner.
-
dynamo_persistence.py Example that shows how task events may be persisted to DynamoDB using the
statefulplugin. -
file_persistence.py Example that shows how task events may be persisted to disk using the
statefulplugin. -
promise.py Example that shows how the promise/future task runner (not yet implemented) may be used.
-
subscription.py Example of the subscription task runner.
-
sync.py Brief example using the sync task runner.
-
timeout.py Example that shows how to timeout a task execution using the
timeoutplugin. -
retry.py Example that shows how to retry a task on failure using the
retryplugin. -
task_logging.py Example that shows how to fetch task logs from Mesos agents using the
loggingplugin.
Running tests
From the root of the repository, run:
make
Repository Structure
/interfaces
Event
Runner
TaskExecutor
/plugins
Plugins can be chained to create a task execution pipeline with more than one property. Please refer to persistence/retry/timeout examples.
mesos
Implements all required interfaces to talk to Mesos deployment. This plugin uses PyMesos to communicate with Mesos.
timeout
Implements an executor to timeout task execution.
retrying
Implements an executor to retry task execution upon failure.
logging
Implements an executor to retrieve task logs from Mesos agents. Note that it has to be the immediate upstream executor of the mesos executor.
Configuration options
- authentication_principal Mesos principal
- credential_secret_file path to file containing Mesos secret
- mesos_address host:port to connect to Mesos cluster
- event_translator a fucntion that maps Mesos-specific events to
Eventobjects
stateful
TODO: documentation
/runners
Runners provide specific concurrency semantics and are supposed to be platform independent.
Sync
Running a task is a blocking operation. sync runners block until the running task has completed or a stop event is received.
Async
Provide callbacks for different events in tasks' lifecycle. async runners allow tasks to specify one or more EventHandlers which consist of predicates and callbacks. Predicates are evaluated when an update is received from the task (e.g. that it has terminated and whether or not it has succeded) and if the predicate passes, the callback is called.
Promise/Future
Running a task returns future object.
Subscription
Provide a queue object and receive all events in there.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for task_processing-1.3.0-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4acd1a3b93e93136ea52ba18a03b2e4b65fd41e37937d5f4761e91e269b1c613 |
|
| MD5 | 9cbab7159122524a1b8d22886653fa56 |
|
| BLAKE2b-256 | 3aa22ddbc64f19b68e18c238b48f6d00529a22793713de2d21e61ac495887b29 |
