Skip to main content

A library of composable Python executors and futures

Project description

more-executors

Build Status Maintainability Test Coverage

This library is intended for use with the concurrent.futures module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

  • Futures with implicit retry
  • Futures with implicit cancel on executor shutdown
  • Futures with implicit cancel after timeout
  • Futures with transformed output values (sync & async)
  • Futures resolved by a caller-provided polling function
  • Throttle the number of futures running at once
  • Synchronous executor
  • Bridge concurrent.futures with asyncio
  • Convenience API for creating executors
  • Instrumented with Prometheus

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

Development

virtualenv and pip may be used to locally install this project from source:

virtualenv ~/dev/python
. ~/dev/python/bin/activate

git clone https://github.com/rohanpm/more-executors
cd more-executors

pip install --editable .

Autotests may be run with pytest:

pip install -r test-requirements.txt
py.test

Submit pull requests against https://github.com/rohanpm/more-executors.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

more-executors-2.10.0.tar.gz (51.0 kB view details)

Uploaded Source

Built Distribution

more_executors-2.10.0-py2.py3-none-any.whl (70.4 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file more-executors-2.10.0.tar.gz.

File metadata

  • Download URL: more-executors-2.10.0.tar.gz
  • Upload date:
  • Size: 51.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.24.0 requests-toolbelt/0.8.0 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/20.0.1 rfc3986/2.0.0 colorama/0.4.4 CPython/3.8.5

File hashes

Hashes for more-executors-2.10.0.tar.gz
Algorithm Hash digest
SHA256 b65e77c07d4eec046f6872e314670fc8192106ce1f47c9bd23570af609c29de4
MD5 63f547d282beca9db4b8ea58ab62f0a7
BLAKE2b-256 056b6fe4a3db9e7ed323e51bf8203a88dd1bc05adab502e5c64351be8c0924c1

See more details on using hashes here.

Provenance

File details

Details for the file more_executors-2.10.0-py2.py3-none-any.whl.

File metadata

  • Download URL: more_executors-2.10.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 70.4 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.24.0 requests-toolbelt/0.8.0 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/20.0.1 rfc3986/2.0.0 colorama/0.4.4 CPython/3.8.5

File hashes

Hashes for more_executors-2.10.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 170cdf50289f33e75379e79b1b68a0121be860a79f272d8d372c9cedeae1eb15
MD5 fa8dd3c2b470ef2467f9e31397212f97
BLAKE2b-256 0f3a61036243a924e0abac60b99eef58410965cb24958993fc73ed93f4694173

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page