Skip to main content

A library of composable Python executors

Project description

more-executors

Build Status Coverage Status

This library is intended for use with the concurrent.futures module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.

Features

  • Futures with implicit retry
  • Futures with implicit cancel on executor shutdown
  • Futures with implicit cancel after timeout
  • Futures with transformed output values (sync & async)
  • Futures resolved by a caller-provided polling function
  • Throttle the number of futures running at once
  • Synchronous executor
  • Bridge concurrent.futures with asyncio
  • Convenience API for creating executors

See the API documentation for detailed information on usage.

Example

This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.

import requests
from concurrent.futures import as_completed
from more_executors import Executors


def get_json(response):
    response.raise_for_status()
    return (response.url, response.json())


def fetch_urls(urls):
    # Configure an executor:
    # - run up to 4 requests concurrently, in separate threads
    # - run get_json on each response
    # - retry up to several minutes on any errors
    executor = Executors.\
        thread_pool(max_workers=4).\
        with_map(get_json).\
        with_retry()

    # Submit requests for each given URL
    futures = [executor.submit(requests.get, url)
               for url in urls]

    # Futures API works as normal; we can block on the completed
    # futures and map/retry happens implicitly
    for future in as_completed(futures):
        (url, data) = future.result()
        do_something(url, data)

Changelog

v1.17.0

  • Exception tracebacks are now propagated correctly on python2 via exception_info

v1.16.0

  • API break: removed new_default methods in retry module
  • Minor usability improvements to retry API
  • Introduced flat_bind (#97)

v1.15.0

  • Fixed possible deadlock in CancelOnShutdownExecutor (#98)
  • Fixed Executors.bind with functools.partial (#96)
  • Fixed ThrottleExecutor thread leak when shutdown() is never called (#93)

v1.14.0

  • API break: removed Executors.wrap class method
  • Fixed thread leaks when shutdown() is never called (#87)
  • Refactors to avoid pylint errors from client code (#86)

v1.13.0

  • Introduced Executors.bind

v1.12.0

  • Introduced FlatMapExecutor

v1.11.0

  • Fixed hangs on executor shutdown

v1.10.0

  • Improved RetryPolicy API
  • Fixed a race condition leading to RetryExecutor hangs
  • Added logger argument to each executor

v1.9.0

  • Introduced ThrottleExecutor

v1.8.0

  • Fixed missing long_description in package

v1.7.0

  • Revised TimeoutExecutor concept to "cancel after timeout"
  • Introduced AsyncioExecutor

v1.6.0

  • Introduce TimeoutExecutor
  • Use monotonic clock in RetryExecutor
  • Avoid some uninterruptible sleeps on Python 2.x
  • Minor improvements to logging

License

GPLv3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

more-executors-1.17.1.tar.gz (15.4 kB view details)

Uploaded Source

Built Distribution

more_executors-1.17.1-py2.py3-none-any.whl (34.8 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file more-executors-1.17.1.tar.gz.

File metadata

  • Download URL: more-executors-1.17.1.tar.gz
  • Upload date:
  • Size: 15.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.29.0 CPython/3.6.3

File hashes

Hashes for more-executors-1.17.1.tar.gz
Algorithm Hash digest
SHA256 b9f1ff4feffabe00d310040e2ceeba15d854f8ae4e2f52923a336a7cc81fd095
MD5 68c8d40c929dc3071eb7e20e6c4d2e4f
BLAKE2b-256 b10726440e6ef6d2a8fb1dc56dccf74c9b3af6d9f6a7a5a9b5a7638590bda000

See more details on using hashes here.

Provenance

File details

Details for the file more_executors-1.17.1-py2.py3-none-any.whl.

File metadata

  • Download URL: more_executors-1.17.1-py2.py3-none-any.whl
  • Upload date:
  • Size: 34.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.29.0 CPython/3.6.3

File hashes

Hashes for more_executors-1.17.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 4c02e0fd6d690018c4860d34d9325a2cc4df3ce09868a08453ea554c25238adb
MD5 208dd30b83849f8c54880f082a0fc60d
BLAKE2b-256 635c1c72cca2b3ac738aa7e9af765a17d5dd7096f8cc1ffbd3337029ab418f28

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page