A robust implementation of concurrent.futures.ProcessPoolExecutor
Project description
Reusable Process Pool Executor 
Goal
The aim of this project is to provide a robust, cross-platform and
cross-version implementation of the ProcessPoolExecutor class of
concurrent.futures. It notably features:
-
Consistent and robust spawn behavior: All processes are started using fork + exec on POSIX systems. This ensures safer interactions with third party libraries. On the contrary,
multiprocessing.Pooluses fork without exec by default, causing third party runtimes to crash (e.g. OpenMP, macOS Accelerate...). -
Reusable executor: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.
-
Transparent cloudpickle integration: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.
-
No need for
if __name__ == "__main__":in scripts: thanks to the use ofcloudpickleto call functions defined in the__main__module, it is not required to protect the code calling parallel functions under Windows. -
Deadlock free implementation: one of the major concern in standard
multiprocessingandconcurrent.futuresmodules is the ability of thePool/Executorto handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors. Note that the implementation ofconcurrent.futures.ProcessPoolExecutorthat comes with Python 3.7+ is as robust as the executor from loky but the later also works for older versions of Python.
Installation
The recommended way to install loky is with pip,
pip install loky
loky can also be installed from sources using
python setup.py install
Usage
import os
from time import sleep
from loky import get_reusable_executor
def say_hello(k):
pid = os.getpid()
print("Hello from {} with arg {}".format(pid, k))
sleep(.01)
return pid
# Create an executor with 4 worker processes, that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(say_hello, 1)
print("Got results:", res.result())
results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4
For more advance usage, see our documentation
Workflow to contribute
To contribute to loky, first create an account on github. Once this is done, fork the loky repository to have your own repository, clone it using 'git clone' on the computers where you want to work. Make your changes in your clone, push them to your github account, test them on several computers, and when you are happy with them, send a pull request to the main repository.
Running the test suite
To run the test suite, you need the pytest (version >= 3) and psutil
modules. Run the test suite using:
pip install -e .
pytest .
from the root of the project.
Acknowledgement
This work is supported by the Center for Data Science, funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file loky-2.3.0.tar.gz.
File metadata
- Download URL: loky-2.3.0.tar.gz
- Upload date:
- Size: 85.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8e2549bb92b92036ffa22fe6286ec544ab95160dd17a4d1426c50ab648c558a1 |
|
| MD5 | 27df7c840324c9f4d7f97c8734d3a50f |
|
| BLAKE2b-256 | 1e8db182adb3432d8b9169a9d2bb5ee6bcc620f8538b0701a77fea9d74a30748 |
Provenance
File details
Details for the file loky-2.3.0-py2.py3-none-any.whl.
File metadata
- Download URL: loky-2.3.0-py2.py3-none-any.whl
- Upload date:
- Size: 59.8 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.7.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 300416cf5fc91ff67659a5acd8de26a334fc3670b3c7e44a3b0eba8437942be7 |
|
| MD5 | 9dcbb65dffd049846d79a71f639389ff |
|
| BLAKE2b-256 | 199cffe3c9be02dcc0937eb88956f5a0c06be74c67a896aa4fbeb42fde3142f2 |
