Framework for task processing executors and configuration

Project description

Task Processing

Interfaces and shared infrastructure for generic task processing (also known as taskproc) at Yelp.

Developer Setup

Pre-requisites

Running tests

From the root of the repository, run:

make

Repository Structure

/interfaces

Event

Runner

TaskExecutor

/plugins

Plugins can be chained to create a task execution pipeline with more than one property.

Kubernetes

Implements all required interfaces to talk to Kubernetes. This plugin uses kubernetes-client to communicate with Kubernetes.

/runners

Runners provide specific concurrency semantics and are supposed to be platform independent.

Sync

Running a task is a blocking operation. sync runners block until the running task has completed or a stop event is received.

Async

Provide callbacks for different events in tasks' lifecycle. async runners allow tasks to specify one or more EventHandlers which consist of predicates and callbacks. Predicates are evaluated when an update is received from the task (e.g. that it has terminated and whether or not it has succeded) and if the predicate passes, the callback is called.

Promise/Future

Running a task returns future object.

Subscription

Provide a queue object and receive all events in there.

Project details

Release history Release notifications | RSS feed

1.3.0

Oct 28, 2024

1.2.0

Jul 11, 2024

1.1.0

Jun 6, 2024

This version

1.0.0

May 13, 2024

0.14.0

May 9, 2024

0.13.0

Jan 25, 2024

0.12.2

Jul 7, 2023

0.12.1

Jun 12, 2023

0.12.0

Jun 9, 2023

0.11.0

Apr 24, 2023

0.10.1

Mar 15, 2023

0.9.1

Aug 16, 2022

0.9.0

Aug 15, 2022

0.8.0

Aug 2, 2022

0.7.0

Jul 25, 2022

0.6.0

Jul 22, 2022

0.5.0

Jul 22, 2022

0.4.0

Apr 14, 2022

0.3.0

Dec 3, 2021

0.2.0

Oct 26, 2021

0.1.26

Sep 10, 2021

0.1.25

Sep 7, 2021

0.1.24

Aug 19, 2021

0.1.23

Aug 18, 2021

0.1.22

Aug 18, 2021

0.1.21

Aug 11, 2021

0.1.19

Aug 2, 2021

0.1.18

Aug 2, 2021

0.1.17

Jul 26, 2021

0.1.16

Jul 23, 2021

0.1.15

Jul 23, 2021

0.1.14

Jul 21, 2021

0.1.13

Jul 16, 2021

0.1.12

Jun 7, 2021

0.1.9

May 6, 2020

0.1.8

Nov 1, 2019

0.1.7

Aug 6, 2019

0.1.6

Jul 31, 2019

0.1.5

Mar 28, 2019

0.1.4

Dec 5, 2018

0.1.3

Sep 28, 2018

0.1.2

Sep 10, 2018

0.1.1

Sep 4, 2018

0.1.0

Aug 29, 2018

0.0.17

Aug 13, 2018

0.0.16

Aug 9, 2018

0.0.15

Aug 6, 2018

0.0.14

Aug 1, 2018

0.0.12

Jul 18, 2018

0.0.11

Jun 20, 2018

0.0.10

Jun 11, 2018

0.0.9

Jun 1, 2018

0.0.8

May 8, 2018

0.0.7

Apr 10, 2018

0.0.6

Jan 29, 2018

0.0.5

Nov 10, 2017

0.0.4

Nov 10, 2017

0.0.3

Oct 18, 2017

0.0.2

Oct 9, 2017

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

task_processing-1.0.0.tar.gz (28.2 kB view details)

Uploaded May 13, 2024 Source

Built Distribution

task_processing-1.0.0-py2.py3-none-any.whl (34.5 kB view details)

Uploaded May 13, 2024 Python 2 Python 3

File details

Details for the file task_processing-1.0.0.tar.gz.

File metadata

Download URL: task_processing-1.0.0.tar.gz
Upload date: May 13, 2024
Size: 28.2 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.0.0 CPython/3.8.19

File hashes

Hashes for task_processing-1.0.0.tar.gz
Algorithm	Hash digest
SHA256	`25e34cc3778034451504ff32f2a2f339505175cb3236460e9c70ef223f218456`
MD5	`cd3e88ee9bc1a2eb164ff77a652682ec`
BLAKE2b-256	`313f2009e1e0e335ee905ea0cd66dfa944ba88cf03d9e60f7dc8ff1f4ad6cee2`

See more details on using hashes here.

Provenance

File details

Details for the file task_processing-1.0.0-py2.py3-none-any.whl.

File metadata

Download URL: task_processing-1.0.0-py2.py3-none-any.whl
Upload date: May 13, 2024
Size: 34.5 kB
Tags: Python 2, Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.0.0 CPython/3.8.19

File hashes

Hashes for task_processing-1.0.0-py2.py3-none-any.whl
Algorithm	Hash digest
SHA256	`8a2f3a7f0d1bc1918f4c047c1ccc1d9c9957364abeb3668bc727147f31a115b3`
MD5	`8f012b539d07b2d25293d40b528bd386`
BLAKE2b-256	`ccb50c495a7adeaab9f69c9b38a4b7ace99a7ee34de5c9176f0f215d25db9be0`