A robust implementation of concurrent.futures.ProcessPoolExecutor
Project description
Reusable Process Pool Executor
==============================
|Build Status| |Build status| |codecov|
Goal
~~~~
The aim of this project is to provide a robust, cross-platform and
cross-version implementation of the :class:`~concurrent.futures.ProcessPoolExecutor` class of
:mod:`concurrent.futures`. It notably features:
- **Deadlock free implementation**: one of the major concern in standard :class:`multiprocessing.pool.Pool` and in :class:`concurrent.futures.ProcessPoolExecutor` is their ability to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors.
- **Consistent spawn behavior**: All processes are started using fork/exec on POSIX systems. This ensures safer interactions with third party libraries.
- **Reusable executor**: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.
- **Transparent :mod:`cloudpickle` integration**: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.
- **No need for ``if __name__ == "__main__":`` in scripts**: thanks to the use of :mod:`cloudpickle <https://github.com/cloudpipe/cloudpickle>` to call functions defined in the :mod:`__main__` module, it is not required to protect the code calling parallel functions under Windows.
Usage
~~~~~
The basic usage of :mod:`loky` relies on the :func:`~loky.get_reusable_executor`, which internally manages a custom :class:`~concurrent.futures.ProcessPoolExecutor` object, which is reused or re-spawned depending on the context.
.. code:: python
import os
from time import sleep
from loky import get_reusable_executor
def say_hello(k):
pid = os.getpid()
print("Hello from {} with arg {}".format(pid, k))
sleep(.01)
return pid
# Create an executor with 4 worker processes, that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(say_hello, 1)
print("Got results:", res.result())
results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4
Acknowledgement
~~~~~~~~~~~~~~~
This work is supported by the Center for Data Science, funded by the
IDEX Paris-Saclay, ANR-11-IDEX-0003-02
.. |Build Status| image:: https://travis-ci.org/tomMoral/loky.svg?branch=master
:target: https://travis-ci.org/tomMoral/loky
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/oifqilb5sb0p7fdp/branch/master?svg=true
:target: https://ci.appveyor.com/project/tomMoral/loky/branch/master
.. |codecov| image:: https://codecov.io/gh/tomMoral/loky/branch/master/graph/badge.svg
:target: https://codecov.io/gh/tomMoral/loky
.. _cloudpickle: https://github.com/cloudpipe/cloudpickle
==============================
|Build Status| |Build status| |codecov|
Goal
~~~~
The aim of this project is to provide a robust, cross-platform and
cross-version implementation of the :class:`~concurrent.futures.ProcessPoolExecutor` class of
:mod:`concurrent.futures`. It notably features:
- **Deadlock free implementation**: one of the major concern in standard :class:`multiprocessing.pool.Pool` and in :class:`concurrent.futures.ProcessPoolExecutor` is their ability to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors.
- **Consistent spawn behavior**: All processes are started using fork/exec on POSIX systems. This ensures safer interactions with third party libraries.
- **Reusable executor**: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.
- **Transparent :mod:`cloudpickle` integration**: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.
- **No need for ``if __name__ == "__main__":`` in scripts**: thanks to the use of :mod:`cloudpickle <https://github.com/cloudpipe/cloudpickle>` to call functions defined in the :mod:`__main__` module, it is not required to protect the code calling parallel functions under Windows.
Usage
~~~~~
The basic usage of :mod:`loky` relies on the :func:`~loky.get_reusable_executor`, which internally manages a custom :class:`~concurrent.futures.ProcessPoolExecutor` object, which is reused or re-spawned depending on the context.
.. code:: python
import os
from time import sleep
from loky import get_reusable_executor
def say_hello(k):
pid = os.getpid()
print("Hello from {} with arg {}".format(pid, k))
sleep(.01)
return pid
# Create an executor with 4 worker processes, that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(say_hello, 1)
print("Got results:", res.result())
results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4
Acknowledgement
~~~~~~~~~~~~~~~
This work is supported by the Center for Data Science, funded by the
IDEX Paris-Saclay, ANR-11-IDEX-0003-02
.. |Build Status| image:: https://travis-ci.org/tomMoral/loky.svg?branch=master
:target: https://travis-ci.org/tomMoral/loky
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/oifqilb5sb0p7fdp/branch/master?svg=true
:target: https://ci.appveyor.com/project/tomMoral/loky/branch/master
.. |codecov| image:: https://codecov.io/gh/tomMoral/loky/branch/master/graph/badge.svg
:target: https://codecov.io/gh/tomMoral/loky
.. _cloudpickle: https://github.com/cloudpipe/cloudpickle
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
loky-2.1.1.tar.gz
(76.2 kB
view details)
Built Distribution
loky-2.1.1-py2.py3-none-any.whl
(56.6 kB
view details)
File details
Details for the file loky-2.1.1.tar.gz.
File metadata
- Download URL: loky-2.1.1.tar.gz
- Upload date:
- Size: 76.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 177411fe8d3f9f55b562c63dcfdabfd466477a4e46d0d84f90d85a7b29303782 |
|
| MD5 | ea7b3b02e9a11d90b0357be4570f3d18 |
|
| BLAKE2b-256 | 932fafcfa106d9cb15e2823de2ba8b82d3d8382393dd2020b5cd553ff9497466 |
Provenance
File details
Details for the file loky-2.1.1-py2.py3-none-any.whl.
File metadata
- Download URL: loky-2.1.1-py2.py3-none-any.whl
- Upload date:
- Size: 56.6 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 9f64e87921c24e45d875670d574d4d41c6dd595a6220f56f328cfc0c9896bd97 |
|
| MD5 | a50c9930cce2080ed6d502317ea8d1b3 |
|
| BLAKE2b-256 | 95aaf9b09ebb9f4ed9b9f2c774e32cda51c3fc4692a0b6e8c91577df7791164d |
