Skip to main content

ZProc - Process on steroids

Project description


# ZProc - Process on steroids
##### Multi-Processing how it should've been.

ZProc is short for [Zero](http://zguide.zeromq.org/page:all#The-Zen-of-Zero) - [Process](https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.Process)

>generally, "zero" refers to the culture of minimalism that permeates the project. We add power by removing complexity rather than by exposing new functionality.

ZProc aims to reduce the pain of multi-processing with the aid of following -

- ☄ Global State management
- Literally allows you to use a global, shared, mutable, state object (`dict`) without having to worry about the atomicity of your operations and race conditions.
- Made possible using the zproc server. Read [Inner Workings](https://github.com/pycampers/zproc#inner-workings) .

- ☄ State synchronization
- Sync-ing global state across all processes (without shared varialbes!)

- ☄ Async/Synchronous paradims without `async def`
- Give you the freedom to build any combination of synchronous and asynchronous systems.
- Read [here](http://zproc.readthedocs.io/en/latest/source/zproc.html#zproc.zproc.ZeroState) for more. Basically,
> Allows you to watch for changes in your state, without having to worry about irrelevant details

- ☄️ Process management
- [Process Factory](http://zproc.readthedocs.io/en/latest/source/zproc.html#zproc.zproc.Context.process_factory)
- Remembers to kill processes when exiting, for general peace.
- Keeps a record of processes created using ZProc. Read [here](http://zproc.readthedocs.io/en/latest/source/zproc.html#zproc.zproc.Context) for more

- ☄ Batch Operations
- Allows you to perform a bunch of operations on state as a singe, atomic operation.
- Read more here.


# Documentation

[Read the docs](http://zproc.readthedocs.io/en/latest/)


# Examples

The simplest way to see zproc in action is to skim through the examples:

They should be pretty self-explanatory. I am happy to make these examples better, since I use them as test-cases for testing ZProc.

These can be found under the `examples` directory.


# Is this
- production-ready?
- No. But it will soon be!
- I work on it regulary since its the backbone of a number of my own projects.
- fast?
- plenty, since its written with ZMQ.
- See `luck_test.py` example for a taste.
- NOT enough for very large applications, but enough for small to medium sized applications.
- stable?
- Mostly.
- I tend to make some fast changes that end up breaking my code.
- I am actively writing tests to help with this though.
- Real?
- YES. It works. See it [here](https://github.com/pycampers/muro) in action.
- Windows / Mac compatible?
- I honestly don't know. Please tell me.

# Inner Workings

- The process(s) communicate over zmq sockets, over `ipc://`.

- Zproc runs a zproc server, which is responsible for storing and managing the state.
- update the state whenever another process updates it.
- transmitt the state whenever a process needs to access it.

- Overall, the zproc server helps create an illusion of a global shared state when actually, it isn't shared at all!

- If a process wishes to synchronize at a certain condition, it can attach a handler to the zproc server.

- The zproc server will check the condition on all state-changes.

- If the condition is met, the zproc server shall open a tunnel to the application and send the state back.

- zmq sockets block your application until that tunnel is opened.

# Caveats

- The state only gets updated if you do it directly. This means that if you mutate objects inside the state, they wont get updated in global state.

- It runs an extra daemonic server for managing the state. Its fairly lightweight though, and shouldn't add too much weight to your application.

- The state should be pickle-able

# Known issues

- Processes inside processes are known to create wierd behavior like
- not being able to access state
- not shutting down properly on exit


# Install
`pip install zproc `

# Build documentation

assuming you have sphinx installed (Linux)
```
cd docs
./build.sh
```

# Thanks

- Thanks to [tblib](https://github.com/ionelmc/python-tblib) ZProc can raise First-class Exceptions from the zproc server!


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

zproc-0.2.9.tar.gz (11.0 kB view details)

Uploaded Source

Built Distribution

zproc-0.2.9-py2.py3-none-any.whl (20.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file zproc-0.2.9.tar.gz.

File metadata

  • Download URL: zproc-0.2.9.tar.gz
  • Upload date:
  • Size: 11.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for zproc-0.2.9.tar.gz
Algorithm Hash digest
SHA256 e383aba4cc8b2e85f718c69c94cc4c31a5db66ea24f00270d8b61fd8c83550cc
MD5 18a3b8077a4cd0fb0c85970817357853
BLAKE2b-256 19c73ce2adcd03800a9da6b9a9761eff352f3faca14f223944e838fe7d6fc1c1

See more details on using hashes here.

File details

Details for the file zproc-0.2.9-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for zproc-0.2.9-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 889fce0533d05bba025c3bf5446cf83c7e21c47ca0afbc0cacfc6f50580243d3
MD5 615bc5854f3847ccaf408fb0241bbf43
BLAKE2b-256 ed2a003859693a55f3d00e44d41f61dcf1073b2a868d17083a3a7d3e5bdb524a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page