A library of composable Python executors and futures
Project description
more-executors
This library is intended for use with the
concurrent.futures
module. It includes a collection of Executor implementations in order to
extend the behavior of Future objects.
Features
- Futures with implicit retry
- Futures with implicit cancel on executor shutdown
- Futures with implicit cancel after timeout
- Futures with transformed output values (sync & async)
- Futures resolved by a caller-provided polling function
- Throttle the number of futures running at once
- Synchronous executor
- Bridge
concurrent.futureswithasyncio - Convenience API for creating executors
See the API documentation for detailed information on usage.
Example
This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.
import requests
from concurrent.futures import as_completed
from more_executors import Executors
def get_json(response):
response.raise_for_status()
return (response.url, response.json())
def fetch_urls(urls):
# Configure an executor:
# - run up to 4 requests concurrently, in separate threads
# - run get_json on each response
# - retry up to several minutes on any errors
executor = Executors.\
thread_pool(max_workers=4).\
with_map(get_json).\
with_retry()
# Submit requests for each given URL
futures = [executor.submit(requests.get, url)
for url in urls]
# Futures API works as normal; we can block on the completed
# futures and map/retry happens implicitly
for future in as_completed(futures):
(url, data) = future.result()
do_something(url, data)
Changelog
v1.19.0
- Fixed TimeoutExecutor thread leak when
shutdown()is never called - Introduced
more_executors.futuresmodule for composing futures
v1.18.0
- Reduced log verbosity (#115)
- Fixed deadlock when awaiting a future whose executor was garbage collected (#114)
v1.17.0
- Exception tracebacks are now propagated correctly on python2
via
exception_info
v1.16.0
- API break: removed
new_defaultmethods in retry module - Minor usability improvements to retry API
- Introduced flat_bind (#97)
v1.15.0
- Fixed possible deadlock in CancelOnShutdownExecutor (#98)
- Fixed
Executors.bindwithfunctools.partial(#96) - Fixed ThrottleExecutor thread leak when
shutdown()is never called (#93)
v1.14.0
- API break: removed
Executors.wrapclass method - Fixed thread leaks when
shutdown()is never called (#87) - Refactors to avoid pylint errors from client code (#86)
v1.13.0
- Introduced Executors.bind
v1.12.0
- Introduced FlatMapExecutor
v1.11.0
- Fixed hangs on executor shutdown
v1.10.0
- Improved RetryPolicy API
- Fixed a race condition leading to RetryExecutor hangs
- Added
loggerargument to each executor
v1.9.0
- Introduced ThrottleExecutor
v1.8.0
- Fixed missing long_description in package
v1.7.0
- Revised TimeoutExecutor concept to "cancel after timeout"
- Introduced AsyncioExecutor
v1.6.0
- Introduce TimeoutExecutor
- Use monotonic clock in RetryExecutor
- Avoid some uninterruptible sleeps on Python 2.x
- Minor improvements to logging
License
GPLv3
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
more-executors-1.19.0.tar.gz
(15.6 kB
view details)
Built Distribution
File details
Details for the file more-executors-1.19.0.tar.gz.
File metadata
- Download URL: more-executors-1.19.0.tar.gz
- Upload date:
- Size: 15.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.29.1 CPython/3.6.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0f7bc9a44166a3378722e8fa9eb4751a42cb42e408ef99b52a784803452267be |
|
| MD5 | d6a6f068563d7b3dc7fc0cd14c35be5c |
|
| BLAKE2b-256 | 13499adec286c4769455f71f6fc4703fbeb62bb84389c24d191423a426f968e4 |
Provenance
File details
Details for the file more_executors-1.19.0-py2.py3-none-any.whl.
File metadata
- Download URL: more_executors-1.19.0-py2.py3-none-any.whl
- Upload date:
- Size: 35.1 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.29.1 CPython/3.6.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b890ee2be9e2195764ae55865750097eec233e01d5c892c7b2360bce9aa6b8ff |
|
| MD5 | 39b068bd165a92857e375685c37b1d73 |
|
| BLAKE2b-256 | 7a421e2d64218f7be189e174c49ee3c6e74a64d0b5b511f4255befb9ae254e37 |
