ZProc - Process on steroids

Multi-Tasking how it should've been.

To make utterly perfect MT programs (and I mean that literally), we don't need mutexes, locks, or any other form of inter-thread communication except messages sent across ZeroMQ sockets.

P.S. ZProc is short for Zero Process

Behold, the power of ZProc:

import zproc

ctx = zproc.Context(background=True)
ctx.state["cookies"] = 0


@zproc.atomic
def eat_cookie(state):
    state["cookies"] -= 1
    print("nom nom nom")


@zproc.atomic
def bake_cookie(state):
    state["cookies"] += 1
    print("Here's a cookie!")


@ctx.call_when_change("cookies")
def cookie_eater(state):
    eat_cookie(state)


@ctx.processify()
def cookie_baker(state):
    for i in range(5):
        bake_cookie(state)

output

Here's a cookie!
Here's a cookie!
nom nom nom
Here's a cookie!
nom nom nom
Here's a cookie!
nom nom nom
Here's a cookie!
nom nom nom
nom nom nom

Notice how the outputs are asynchronous, because the baker and eater run in different processes.

print(cookie_eater)
print(cookie_baker)

output

<ZeroProcess pid: 2555 target: <function cookie_eater at 0x7f5b4542c9d8> uuid: e74521ae-76ca-11e8-bd1f-7c7a912e12b5>
<ZeroProcess pid: 2556 target: <function cookie_baker at 0x7f5b4542c950> uuid: e74521ae-76ca-11e8-bd1f-7c7a912e12b5>

If two ZeroProcess instances have the same uuid, that means they share the same state.

Install

pip install zproc

License: MIT License (MIT)
Requires: Python >=3.5

Documentation ( )

Read the docs

Examples

Backstory

Traditional Multi Processing involved shared memory (global variables).
However, it was proven that shared memory tends to violate the laws of Physics.

🔖 <- Joe Armstrong, the creator of Erlang.

The solution presented by Erlang, (and ZMQ) is the notion of message passing for achieving parallelism.
However, Message passing can be tedious, and often un-pythonic.

This is where ZProc comes in.

It provides you a middle ground between message passing and shared memory.

It does message passing, without you ever knowing that it's doing it.

It also borrows the concept of state and props from reactJS.

Hence, provides you a global dict called state.
The state is not a shared object.
It works purely on message passing.

Unlike reactJS, you are free to mutate the state, up-to a certain extent.

Behind the covers, it simulates the Actor Model.
ZProc doesn't blindly follow it, but you can think of it as such.

It also borrows the autoretry feature of Celery, but unlike Celery it doesn't need a broker.

The zen of zero

The Ø in ØMQ is all about trade-offs. On the one hand, this strange name lowers ØMQ’s visibility on Google and Twitter. On the other hand, it annoys the heck out of some Danish folk who write us things like “ØMG røtfl”, and “Ø is not a funny-looking zero!” and “Rødgrød med Fløde!” (which is apparently an insult that means “May your neighbours be the direct descendants of Grendel!”). Seems like a fair trade.

Originally, the zero in ØMQ was meant to signify “zero broker” and (as close to) “zero latency” (as possible). Since then, it has come to encompass different goals: zero administration, zero cost, zero waste. More generally, “zero” refers to the culture of minimalism that permeates the project. We add power by removing complexity rather than by exposing new functionality.

Features

• 🌠   Global State w/o shared memory

  • Globally synchronized state (dict), without using shared memory.
  • 🔖

• 🌠   Asynchronous paradigms without async def

  • Build any combination of synchronous and asynchronous systems.
  • watch for changes in state, without Busy Waiting.
  • 🔖

• 🌠   Process management

  • Process Factory
  • Remembers to kill processes when exiting, for general peace. (even when they're nested)
  • Keeps a record of processes created using ZProc.
  • 🔖

• 🌠   Atomic Operations

  • Perform an arbitrary number of operations on state as a single, atomic operation.
  • 🔖

Caveats

• The state only gets updated if you do it directly.
  This means that if you mutate objects inside the state, they wont get reflected in the global state.
• The state should be pickle-able
• It runs an extra Process for managing the state.
  Its fairly lightweight though, and shouldn't add too much weight to your application.

FAQ

• Fast?

  • Above all, ZProc is written for safety and ease of use.
  • However, since its written using ZMQ, it's plenty fast for most stuff.
  • Run -> 🔖 for a taste.

• Stable?

  • The code itself is stable, but the API is quite unstable.

• Production ready?

  • Please don't use it in production right now.

• Real?

  • YES. It works.
  • 🔖

• Windows compatible?

  • Probably?

Inner Workings

• Zproc uses a Server, which is responsible for storing and communicating the state.

  • This isolates our resource (state), eliminating the need for locks.

• The process(s) communicate through ZMQ sockets, over ipc://.

  • The clients (Proceses) use a ZMQ_DEALER socket.
  • The Server uses a ZMQ_ROUTER socket.

• If a Process wishes to watch the state, it subscribes to a global publish message.

  • The zproc server publishes the state at every state update. (using ZMQ_PUB socket)
  • A Process may subscribe to this message and filter out the event it needs (using ZMQ_SUB socket).
  • zmq sockets block your application efficiently till an update occurs, eliminating the need for Busy Waiting.

(Busy waiting refers to the while True: <check condition> approach).

Local development

git clone https://github.com/pycampers/zproc.git
cd zproc
pipenv install

Build documentation

Assuming you have sphinx installed (Linux)

cd docs
pipenv run ./build.sh

Thanks

• Thanks to pieter hintjens, for his work on the ZeroMQ library and for his amazing book.
• Thanks to tblib, ZProc can raise First-class Exceptions from the zproc server!
• Thanks to psutil, ZProc can handle nested procesess!
• Thanks to Kennith Rietz. His setup.py was used to host this project on pypi. Plus lot of documentation is blatantly copied from his documentation on requests

🐍🏕️