ZProc - Process on steroids
Project description
ZProc - Multi Processing on steroids
P.S.A.: Not suitable for use in production. In beta stage right now. Thanks.
Behold, the power of ZProc:
import zproc
ctx = zproc.Context(wait=True) # wait for processes in this context
ctx.state["cookies"] = 0
@zproc.atomic
def eat_cookie(state):
"""Eat a cookie atomically."""
state["cookies"] -= 1
print("nom nom nom")
@zproc.atomic
def bake_cookie(state):
"""Bake a cookie atomically."""
state["cookies"] += 1
print("Here's a cookie!")
@ctx.call_when_change("cookies")
def cookie_eater(state):
"""Eat cookies as they're baked."""
eat_cookie(state)
@ctx.process
def cookie_baker(state):
"""Bake some cookies."""
for i in range(5):
bake_cookie(state)
output
Here's a cookie!
Here's a cookie!
nom nom nom
Here's a cookie!
nom nom nom
Here's a cookie!
nom nom nom
Here's a cookie!
nom nom nom
nom nom nom
(baker and eater run in different processes)
Install
pip install zproc
License: MIT License (MIT)
Requires: Python >=3.5
Documentation ( 
)
Read the docs
Examples
Why use ZProc?
At the surface, it's just a better API for Message passing parallelism (using ZMQ).
Message passing can be tedious because of all the manual wiring involved.
ZProc lets you do message passing parallelism with a more pythonic, safe, easy-to-use interface.
It does that by providing a global dict called state.
The state is not a shared object.
It works purely on message passing.
It also supports a fair bit of reactive programming, using state watchers.
Behind the covers, it uses the Actor Model.
It also borrows the autoretry feature of Celery, but unlike
Celery it doesn't need a broker.
Bottom line, you'll have a lot more fun doing parallel/concurrent programming using ZProc, than anything else.
Features
-
🌠 Asynchronous paradigms without
async def- Build a combination of synchronous and asynchronous systems, with ease.
- By watching for changes in state, without Busy Waiting.
- 🔖
-
🌠 Process management
- Process Factory
- Remembers to kill processes when exiting, for general peace. (even when they're nested)
- Keeps a record of processes created using ZProc.
- 🔖
-
🌠 Atomic Operations
- Perform an arbitrary number of operations on state as a single, atomic operation.
- 🔖
Caveats
-
The state only gets updated if you do it directly.
This means that if you mutate objects inside the state, they wont get reflected in the global state. -
The state should be pickle-able.
-
It runs an extra Process for managing the state.
Its fairly lightweight though, and shouldn't add too much weight to your application.
FAQ
-
Fast?
- Above all, ZProc is written for safety and the ease of use.
- However, since its written using ZMQ, it's plenty fast for most stuff.
- Run -> 🔖 for a taste.
-
Stable?
- Mostly. However, since it's still in the
0.x.xstage, you can expect some API changes.
- Mostly. However, since it's still in the
-
Production ready?
- Please don't use it in production right now.
-
Windows compatible?
- Probably?
Inner Workings
-
ZProc uses a Server, which is responsible for storing and communicating the state.
- This isolates our resource (state), and makes it safer to do atomic operations.
-
The process(s) communicate through ZMQ sockets, over
ipc://.- The clients (Proceses) use a
ZMQ_DEALERsocket. - The Server uses a
ZMQ_ROUTERsocket.
- The clients (Proceses) use a
-
If a Process wishes to watch the state, it subscribes to a global publish message.
- The zproc server publishes the state at every state update.
(using
ZMQ_PUBsocket) - A Process may subscribe to this message and
filter out the event it needs (using
ZMQ_SUBsocket). - zmq sockets block your application efficiently till an update occurs, eliminating the need for Busy Waiting.
- The zproc server publishes the state at every state update.
(using
(Busy waiting refers to the while True: <check condition> approach).
Local development
git clone https://github.com/pycampers/zproc.git
cd zproc
pipenv install
Build documentation
Assuming you have sphinx installed (Linux)
cd docs
pipenv run ./build.sh
ZProc in the wild
Thanks
- Thanks to pieter hintjens, for his work on the ZeroMQ library and for his amazing book.
- Thanks to tblib, ZProc can raise First-class Exceptions from the zproc server!
- Thanks to psutil, ZProc can handle nested procesess!
- Thanks to Kennith Rietz. His setup.py was used to host this project on pypi. Plus lot of documentation is blatantly copied from his documentation on requests
ZProc is short for Zero Process.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file zproc-0.9.1.tar.gz.
File metadata
- Download URL: zproc-0.9.1.tar.gz
- Upload date:
- Size: 19.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.5.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 5d95b404ba2299df513a23fdfb65107e7214656468da60eca252435c246b34ac |
|
| MD5 | 87fd29c2ecb3fae5fa7a04ab170e0d05 |
|
| BLAKE2b-256 | 9f16708f02d460016c7f7f89510d196286891f4858c26c7c4b8aff3c6441cdca |
File details
Details for the file zproc-0.9.1-py2.py3-none-any.whl.
File metadata
- Download URL: zproc-0.9.1-py2.py3-none-any.whl
- Upload date:
- Size: 32.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.5.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e7a6b25050b47bf764fc1ca9ec8e92523af04ac61162be75bad02df33ccc0612 |
|
| MD5 | 94bc8d9491742a0f904a5dd895276c74 |
|
| BLAKE2b-256 | f1d459110d6f12f538d4cfa43f5c98dafb5cb2e0631ea7acb262bbf445d6359d |
