A library of composable Python executors

These details have not been verified by PyPI

Project links

Homepage

Project description

more-executors

This library is intended for use with the concurrent.futures module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

Futures with implicit retry
Futures with implicit cancel on executor shutdown
Futures with implicit cancel after timeout
Futures with transformed output values (sync & async)
Futures resolved by a caller-provided polling function
Throttle the number of futures running at once
Synchronous executor
Bridge concurrent.futures with asyncio
Convenience API for creating executors

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

Changelog

v1.18.0

Reduced log verbosity (#115)
Fixed deadlock when awaiting a future whose executor was garbage collected (#114)

v1.17.0

Exception tracebacks are now propagated correctly on python2 via exception_info

v1.16.0

API break: removed new_default methods in retry module
Minor usability improvements to retry API
Introduced flat_bind (#97)

v1.15.0

Fixed possible deadlock in CancelOnShutdownExecutor (#98)
Fixed Executors.bind with functools.partial (#96)
Fixed ThrottleExecutor thread leak when shutdown() is never called (#93)

v1.14.0

API break: removed Executors.wrap class method
Fixed thread leaks when shutdown() is never called (#87)
Refactors to avoid pylint errors from client code (#86)

v1.13.0

Introduced Executors.bind

v1.12.0

Introduced FlatMapExecutor

v1.11.0

Fixed hangs on executor shutdown

v1.10.0

Improved RetryPolicy API
Fixed a race condition leading to RetryExecutor hangs
Added logger argument to each executor

v1.9.0

Introduced ThrottleExecutor

v1.8.0

Fixed missing long_description in package

v1.7.0

Revised TimeoutExecutor concept to "cancel after timeout"
Introduced AsyncioExecutor

v1.6.0

Introduce TimeoutExecutor
Use monotonic clock in RetryExecutor
Avoid some uninterruptible sleeps on Python 2.x
Minor improvements to logging

License

GPLv3

Project details

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

2.11.4

Feb 1, 2023

2.11.3

Dec 13, 2022

2.11.2

Dec 4, 2022

2.11.1

Jun 5, 2022

2.11.0

Apr 27, 2022

2.10.1

Mar 23, 2022

2.10.0

Feb 27, 2022

2.9.0

Feb 21, 2022

2.8.2

Feb 16, 2022

2.8.1

Jul 21, 2021

2.8.0

Jul 14, 2021

2.7.0

Jul 11, 2021

2.6.0

Jun 19, 2021

2.5.1

Jun 1, 2020

2.5.0

Sep 23, 2019

2.4.0

Sep 15, 2019

2.3.1

Sep 11, 2019

2.3.0

Sep 7, 2019

2.2.0

Aug 31, 2019

2.1.2

Jun 26, 2019

2.1.1

Jun 16, 2019

2.1.0

Jun 5, 2019

2.0.2

May 3, 2019

2.0.1

Apr 6, 2019

2.0.0

Mar 17, 2019

1.20.2

Feb 23, 2019

1.20.1

Feb 15, 2019

1.20.0

Jan 18, 2019

1.19.1

Jan 16, 2019

1.19.0

Jan 16, 2019

This version

1.18.0

Jan 15, 2019

1.17.2

Jan 9, 2019

1.17.1

Jan 9, 2019

1.16.0

Jan 6, 2019

1.15.0

Jan 3, 2019

1.14.0

Dec 26, 2018

1.13.0

Dec 15, 2018

1.12.1

Dec 9, 2018

1.12.0

Dec 9, 2018

1.11.0

Apr 28, 2018

1.10.0

Apr 28, 2018

1.9.0

Apr 15, 2018

1.8.0

Apr 7, 2018

1.7.0

Apr 7, 2018

1.6.0

Mar 22, 2018

1.5.0

Mar 18, 2018

1.4.0

Mar 18, 2018

1.3.0

Mar 17, 2018

1.2.0

Mar 14, 2018

1.1.0

Mar 12, 2018

1.0.0

Mar 11, 2018

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

more-executors-1.18.0.tar.gz (15.5 kB view details)

Uploaded Jan 15, 2019 Source

Built Distribution

more_executors-1.18.0-py2.py3-none-any.whl (35.0 kB view details)

Uploaded Jan 15, 2019 Python 2 Python 3

File details

Details for the file more-executors-1.18.0.tar.gz.

File metadata

Download URL: more-executors-1.18.0.tar.gz
Upload date: Jan 15, 2019
Size: 15.5 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.29.1 CPython/3.6.3

File hashes

Hashes for more-executors-1.18.0.tar.gz
Algorithm	Hash digest
SHA256	`f495b4c388f701e15da99985db11e5bd56062eb863d1bc90a6c67f8cd88f1a2f`
MD5	`efd4010dad42e9d846c5115013643008`
BLAKE2b-256	`46ae774b267b41de7f46cfb1175e8d46374cb8978336eca66b9647573f22923e`

See more details on using hashes here.

Provenance

File details

Details for the file more_executors-1.18.0-py2.py3-none-any.whl.

File metadata

Download URL: more_executors-1.18.0-py2.py3-none-any.whl
Upload date: Jan 15, 2019
Size: 35.0 kB
Tags: Python 2, Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.29.1 CPython/3.6.3

File hashes

Hashes for more_executors-1.18.0-py2.py3-none-any.whl
Algorithm	Hash digest
SHA256	`6a8be5458125a2b2ec342e0b6712ef5cfbfdc0f12551f56ccc0dd89be0a29d09`
MD5	`18c4acf37550ade7318f15268176a226`
BLAKE2b-256	`45f85c8a3489a56678147d6000d79123914fc5928089fec4391f78da5d778f5c`