pympipool - scale functions over multiple compute nodes using mpi4py
Project description
pympipool
Scale functions over multiple compute nodes using mpi4py
Functionality
Write a python test file like pool.py:
import numpy as np
from pympipool import Pool
def calc(i):
return np.array(i ** 2)
with Pool(cores=2) as p:
print(p.map(function=calc, lst=[1, 2, 3, 4]))
You can execute the python file pool.py in a serial python process:
python pool.py
>>> [array(1), array(4), array(9), array(16)]
Internally pympipool uses mpi4py to distribute the four calculation to two processors cores=2.
Installation
As pympipool requires openmpi and mpi4py it is highly recommended to install it via conda:
conda install -c conda-forge pympipool
Alternatively, it is also possible to pympipool via pip:
pip install pympipool
Changelog
0.2.0
- Communicate via zmq rather than
stdinandstdout, this enables support formpichandopenmpi. - Add error handling to propagate the
Exception, when it is raised by mapping the function to the arguments.
0.1.0
- Major switch of the communication interface between the serial python process and the mpi parallel python process.
Previously, functions were converted to source code using
inspect.getsource()anddillwas used to convert the sourcecode to an binary blob which could then be transferred between the processes. In the new version, the function is directly pickled usingcloudpickleascloudpicklesupports both pickle by reference and pickle by value. Here the pickle by value functionality is used to pickle the functions which is be communicated. - The documentation is updated to reflect the changes in the updated version.
0.0.2
- output of the function which is mapped to the arguments is suppressed, as
stdoutinterferes with the communication ofpympipool. Consequently, the output ofprintstatements is no longer visible. - support for python 3.11 is added
mpi4pycompatibility is updated from3.1.3to3.1.4dillcompatibility is updated from0.3.5.1to0.3.6tqdmcompatibility is updated from4.64.0to4.64.1
0.0.1
- initial release
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pympipool-0.2.0.tar.gz
(21.8 kB
view details)
Built Distribution
pympipool-0.2.0-py3-none-any.whl
(11.5 kB
view details)
File details
Details for the file pympipool-0.2.0.tar.gz.
File metadata
- Download URL: pympipool-0.2.0.tar.gz
- Upload date:
- Size: 21.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.16
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3bf9bce5825eef51729b17c5e5d8a0048a0353197fa72cfc6a290ab6231ebeaf |
|
| MD5 | 7a6e62fe5b70a7ce704d2d349f6242d5 |
|
| BLAKE2b-256 | b0e6b7613087f6b336c82073c65c62e63f8b9969c5ddf48599051c0b344d0e79 |
File details
Details for the file pympipool-0.2.0-py3-none-any.whl.
File metadata
- Download URL: pympipool-0.2.0-py3-none-any.whl
- Upload date:
- Size: 11.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.16
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 272b2ab9c468499fdce50c9a46e4b50ef9b57985b461625e6a2f0a919ca0ec21 |
|
| MD5 | 607d7860436d321aef10cf7707e7bb07 |
|
| BLAKE2b-256 | b375f4ead57ef154989849359ae7076c0fd6ceeffbea0acf263c3ed627a96465 |
