A library of composable Python executors
Project description
This library is intended for use with the `concurrent.futures <https://docs.python.org/3/library/concurrent.futures.html>`__ module. It includes a collection of Executor implementations in order to extend the behavior of Future objects.
Features
Futures with implicit retry
Futures with transformed output values
Futures resolved by a caller-provided polling function
Convenience API for creating executors
See the API documentation for detailed information on usage.
Example
This example combines the map and retry executors to create futures for HTTP requests running concurrently, decoding JSON responses within the future and retrying on error.
import requests
from concurrent.futures import as_completed
from more_executors import Executors
def get_json(response):
response.raise_for_status()
return (response.url, response.json())
def fetch_urls(urls):
# Configure an executor:
# - run up to 4 requests concurrently, in separate threads
# - run get_json on each response
# - retry up to several minutes on any errors
executor = Executors.\
thread_pool(max_workers=4).\
with_map(get_json).\
with_retry()
# Submit requests for each given URL
futures = [executor.submit(requests.get, url)
for url in urls]
# Futures API works as normal; we can block on the completed
# futures and map/retry happens implicitly
for future in as_completed(futures):
(url, data) = future.result()
do_something(url, data)License
GPLv3
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for more_executors-1.2.0-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 102d23ca622bbf4b308191e8a4a30f2239adb5814a0caefaf59754021431a868 |
|
| MD5 | 67a388110d75a231d2fce0491b50a830 |
|
| BLAKE2b-256 | 26b7bfe337822c608df9b162e8d8e73c98f33d15afe00360254ed787289d4570 |
