Skip to main content

A library of composable Python executors

Project description

Build Status Coverage Status

This library is intended for use with the `concurrent.futures <https://docs.python.org/3/library/concurrent.futures.html>`__ module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

  • Futures with implicit retry

  • Futures with transformed output values

  • Futures resolved by a caller-provided polling function

  • Convenience API for creating executors

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

License

GPLv3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

more-executors-1.3.0.tar.gz (11.8 kB view details)

Uploaded Source

Built Distribution

more_executors-1.3.0-py2.py3-none-any.whl (27.4 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file more-executors-1.3.0.tar.gz.

File metadata

File hashes

Hashes for more-executors-1.3.0.tar.gz
Algorithm Hash digest
SHA256 5e75d582913581c24b9c60c038d346ddf0878944f5d845c59746878e976129b4
MD5 87a8336fddc292d2e4ca1701ac9e06e0
BLAKE2b-256 610fb7cc6662bed599a63b21256ec59ea09ff3ad5d207feb5be4cd684a245e9f

See more details on using hashes here.

Provenance

File details

Details for the file more_executors-1.3.0-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for more_executors-1.3.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 9f13fced651daa3c974d804fea5be182bcd3ad01c8f15e37752d76cd4e25effe
MD5 2ee9d187e10a18267c73c594c4b15531
BLAKE2b-256 2f9a7c47bcca3b4caff4e773f92404283b87c511c77f3522ac7c3cf872ff90e0

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page