A robust implementation of concurrent.futures.ProcessPoolExecutor
Project description
# Reusable Process Pool Executor [](https://travis-ci.org/tomMoral/loky) [](https://ci.appveyor.com/project/tomMoral/loky/branch/master) [](https://codecov.io/gh/tomMoral/loky)
### Goal
The aim of this project is to provide a robust, cross-platform and
cross-version implementation of the `ProcessPoolExecutor` class of
`concurrent.futures`. It notably features:
* __Deadlock free implementation__: one of the major concern in
standard `multiprocessing` and `concurrent.futures` libraries is the
ability of the `Pool/Executor` to handle crashes of worker
processes. This library intends to fix those possible deadlocks and
send back meaningful errors.
* __Consistent spawn behavior__: All processes are started using
fork/exec on POSIX systems. This ensures safer interactions with
third party libraries.
* __Reusable executor__: strategy to avoid respawning a complete
executor every time. A singleton executor instance can be reused (and
dynamically resized if necessary) across consecutive calls to limit
spawning and shutdown overhead. The worker processes can be shutdown
automatically after a configurable idling timeout to free system
resources.
* __Transparent cloudpickle integration__: to call interactively
defined functions and lambda expressions in parallel. It is also
possible to register a custom pickler implementation to handle
inter-process communications.
* __No need for if ``if __name__ == "__main__":`` in scripts__: thanks
to the use of ``cloudpickle`` to call functions defined in the
``__main__`` module, it is not required to protect the code calling
parallel functions under Windows.
### Usage
```python
import os
from time import sleep
from loky import get_reusable_executor
def say_hello(k):
pid = os.getpid()
print("Hello from {} with arg {}".format(pid, k))
sleep(.01)
return pid
# Create an executor with 4 worker processes, that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(say_hello, 1)
print("Got results:", res.result())
results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4
executor.shutdown(wait=True)
```
### Acknowledgement
This work is supported by the Center for Data Science, funded by the IDEX
Paris-Saclay, ANR-11-IDEX-0003-02
### Goal
The aim of this project is to provide a robust, cross-platform and
cross-version implementation of the `ProcessPoolExecutor` class of
`concurrent.futures`. It notably features:
* __Deadlock free implementation__: one of the major concern in
standard `multiprocessing` and `concurrent.futures` libraries is the
ability of the `Pool/Executor` to handle crashes of worker
processes. This library intends to fix those possible deadlocks and
send back meaningful errors.
* __Consistent spawn behavior__: All processes are started using
fork/exec on POSIX systems. This ensures safer interactions with
third party libraries.
* __Reusable executor__: strategy to avoid respawning a complete
executor every time. A singleton executor instance can be reused (and
dynamically resized if necessary) across consecutive calls to limit
spawning and shutdown overhead. The worker processes can be shutdown
automatically after a configurable idling timeout to free system
resources.
* __Transparent cloudpickle integration__: to call interactively
defined functions and lambda expressions in parallel. It is also
possible to register a custom pickler implementation to handle
inter-process communications.
* __No need for if ``if __name__ == "__main__":`` in scripts__: thanks
to the use of ``cloudpickle`` to call functions defined in the
``__main__`` module, it is not required to protect the code calling
parallel functions under Windows.
### Usage
```python
import os
from time import sleep
from loky import get_reusable_executor
def say_hello(k):
pid = os.getpid()
print("Hello from {} with arg {}".format(pid, k))
sleep(.01)
return pid
# Create an executor with 4 worker processes, that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(say_hello, 1)
print("Got results:", res.result())
results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4
executor.shutdown(wait=True)
```
### Acknowledgement
This work is supported by the Center for Data Science, funded by the IDEX
Paris-Saclay, ANR-11-IDEX-0003-02
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
loky-1.1.1.tar.gz
(69.3 kB
view details)
Built Distribution
loky-1.1.1-py2.py3-none-any.whl
(54.0 kB
view details)
File details
Details for the file loky-1.1.1.tar.gz.
File metadata
- Download URL: loky-1.1.1.tar.gz
- Upload date:
- Size: 69.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 179d094f6e60cec366982cd316a9f8ebe60993461a67661ed72602c2d61f6e01 |
|
| MD5 | 65cee62bb7f7d99592b0137727f6b2b1 |
|
| BLAKE2b-256 | 5f94c2e18ae8ac496a38ece756b3465ac11550b7996ef07e5c38f914aa2c131c |
Provenance
File details
Details for the file loky-1.1.1-py2.py3-none-any.whl.
File metadata
- Download URL: loky-1.1.1-py2.py3-none-any.whl
- Upload date:
- Size: 54.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1e19fa12a796bdd7f2ef560104f964f70a15e879bdd6fec9f2bd979cd41a5d7a |
|
| MD5 | 03c1b24e5eaa84aa4b5b04479734dfe2 |
|
| BLAKE2b-256 | ce359d762d9505fac201fbacde84450c8f39b5f71379d674e7f6dbfedfe13a0d |
