Skip to main content

A library of composable Python executors

Project description

Build Status Coverage Status

This library is intended for use with the `concurrent.futures <https://docs.python.org/3/library/concurrent.futures.html>`__ module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

  • Futures with implicit retry

  • Futures with implicit cancel on executor shutdown

  • Futures with implicit cancel after timeout

  • Futures with transformed output values

  • Futures resolved by a caller-provided polling function

  • Throttle the number of futures running at once

  • Synchronous executor

  • Bridge concurrent.futures with asyncio

  • Convenience API for creating executors

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

Changelog

v1.9.0

  • Introduced ThrottleExecutor

v1.8.0

  • Fixed missing long_description in package

v1.7.0

  • Revised TimeoutExecutor concept to “cancel after timeout”

  • Introduced AsyncioExecutor

v1.6.0

  • Introduce TimeoutExecutor

  • Use monotonic clock in RetryExecutor

  • Avoid some uninterruptible sleeps on Python 2.x

  • Minor improvements to logging

License

GPLv3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

more-executors-1.9.0.tar.gz (15.4 kB view details)

Uploaded Source

Built Distribution

more_executors-1.9.0-py2.py3-none-any.whl (33.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file more-executors-1.9.0.tar.gz.

File metadata

File hashes

Hashes for more-executors-1.9.0.tar.gz
Algorithm Hash digest
SHA256 d9f97f3dc16bcba1f24f56933da0c3a9f83c383bc5903cb25ed735205d7321ed
MD5 c3ad63930330b2d37ff7fde75e7ae7b6
BLAKE2b-256 772f9a54f425f2a2566cedd1ac15107a125a2f5c97959c086f639226cc433fca

See more details on using hashes here.

Provenance

File details

Details for the file more_executors-1.9.0-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for more_executors-1.9.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 40a4753218f1423aa9ffb23f3631de06ad259c0a483f066830960eaa817a3fdb
MD5 64a02045fe2aeb40594b7fc81e5545fc
BLAKE2b-256 932014c6e87b3df701a3d1fc5f31e68369c835c11c8f5a7c363974a6848e4e77

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page