Skip to main content

A robust implementation of concurrent.futures.ProcessPoolExecutor

Project description

Reusable Process Pool Executor Build Status codecov

Goal

The aim of this project is to provide a robust, cross-platform and cross-version implementation of the ProcessPoolExecutor class of concurrent.futures. It notably features:

  • Consistent and robust spawn behavior: All processes are started using fork + exec on POSIX systems. This ensures safer interactions with third party libraries. On the contrary, multiprocessing.Pool uses fork without exec by default, causing third party runtimes to crash (e.g. OpenMP, macOS Accelerate...).

  • Reusable executor: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.

  • Transparent cloudpickle integration: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.

  • No need for if __name__ == "__main__": in scripts: thanks to the use of cloudpickle to call functions defined in the __main__ module, it is not required to protect the code calling parallel functions under Windows.

  • Deadlock free implementation: one of the major concern in standard multiprocessing and concurrent.futures modules is the ability of the Pool/Executor to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors. Note that the implementation of concurrent.futures.ProcessPoolExecutor that comes with Python 3.7+ is as robust as the executor from loky but the later also works for older versions of Python.

Installation

The recommended way to install loky is with pip,

pip install loky

loky can also be installed from sources using

python setup.py install

Note that loky has an optional dependency on psutil to allow early memory leak detections.

Usage

import os
from time import sleep
from loky import get_reusable_executor


def say_hello(k):
    pid = os.getpid()
    print("Hello from {} with arg {}".format(pid, k))
    sleep(.01)
    return pid


# Create an executor with 4 worker processes, that will
# automatically shutdown after idling for 2s
executor = get_reusable_executor(max_workers=4, timeout=2)

res = executor.submit(say_hello, 1)
print("Got results:", res.result())

results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("Number of used processes:", n_workers)
assert n_workers == 4

For more advance usage, see our documentation

Workflow to contribute

To contribute to loky, first create an account on github. Once this is done, fork the loky repository to have your own repository, clone it using 'git clone' on the computers where you want to work. Make your changes in your clone, push them to your github account, test them on several computers, and when you are happy with them, send a pull request to the main repository.

Running the test suite

To run the test suite, you need the pytest (version >= 3) and psutil modules. Run the test suite using:

    pip install -e .
    pytest .

from the root of the project.

Acknowledgement

This work is supported by the Center for Data Science, funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

loky-2.8.0.tar.gz (119.9 kB view details)

Uploaded Source

Built Distribution

loky-2.8.0-py2.py3-none-any.whl (65.8 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file loky-2.8.0.tar.gz.

File metadata

  • Download URL: loky-2.8.0.tar.gz
  • Upload date:
  • Size: 119.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/46.1.3.post20200330 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.8.2

File hashes

Hashes for loky-2.8.0.tar.gz
Algorithm Hash digest
SHA256 76e0431a8f124c267e6a95df66426265e5d35d75310638950871ba76dc12b545
MD5 1db75c07a289d5b39fe2fb909a0ab9a8
BLAKE2b-256 ac02815e43e69592a8a0eabc3d3d99f3e83eb6e401bc248046767f33417d9f08

See more details on using hashes here.

Provenance

File details

Details for the file loky-2.8.0-py2.py3-none-any.whl.

File metadata

  • Download URL: loky-2.8.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 65.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/46.1.3.post20200330 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.8.2

File hashes

Hashes for loky-2.8.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 7c4d012c277393e22135cc137231b632d185e4482a29ea60afaf34e3b687c4c1
MD5 2500eeeddc200a54443be5064cbc9c6f
BLAKE2b-256 c2615a6958c714772d0481556e49e42b197b4e4a01a3acad57f6334a5159fd74

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page