Skip to main content

A library of composable Python executors

Project description

Build Status Coverage Status

This library is intended for use with the `concurrent.futures <https://docs.python.org/3/library/concurrent.futures.html>`__ module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

  • Futures with implicit retry

  • Futures with transformed output values

  • Futures resolved by a caller-provided polling function

  • Convenience API for creating executors

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

License

GPLv3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

more-executors-1.2.0.tar.gz (11.2 kB view details)

Uploaded Source

Built Distribution

more_executors-1.2.0-py2.py3-none-any.whl (26.7 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file more-executors-1.2.0.tar.gz.

File metadata

File hashes

Hashes for more-executors-1.2.0.tar.gz
Algorithm Hash digest
SHA256 677eaeedbba21d08ecbc68b93d3e0d5dbaf42ec662ccca84e7698a2e00e642a7
MD5 9e9bd943f57f3f829c08a9f1f988b1b1
BLAKE2b-256 8e25609476a7d49ceaa2a78203615c17d81f6fe4b8cfd8b5fca6d415736930d0

See more details on using hashes here.

Provenance

File details

Details for the file more_executors-1.2.0-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for more_executors-1.2.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 102d23ca622bbf4b308191e8a4a30f2239adb5814a0caefaf59754021431a868
MD5 67a388110d75a231d2fce0491b50a830
BLAKE2b-256 26b7bfe337822c608df9b162e8d8e73c98f33d15afe00360254ed787289d4570

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page