Python multiprocessing fork
Project description
billiard
About
billiard is a fork of the Python 2.7 multiprocessing package. The multiprocessing package itself is a renamed and updated version of R Oudkerk’s pyprocessing package. This standalone variant is intended to be compatible with Python 2.4 and 2.5, and will draw it’s fixes/improvements from python-trunk.
This package would not be possible if not for the contributions of not only the current maintainers but all of the contributors to the original pyprocessing package listed here
Also it is a fork of the multiprocessin backport package by Christian Heims.
It includes the no-execv patch contributed by sbt.
And the Pool improvements previously located in Celery.
Bug reporting
Please report bugs related to multiprocessing at the Python bug tracker. Issues related to billiard should be reported at http://github.com/celery/billiard/issues.
Changes
2.7.3.9 - 2012-06-03
Environment variable MP_MAIN_FILE envvar is now set to the path of the __main__ module when execv is enabled.
Pool: Errors occuring in the TaskHandler are now reported.
2.7.3.8 - 2012-06-01
Can now be installed on Py 3.2
Issue #12091: simplify ApplyResult and MapResult with threading.Event
Patch by Charles-Francois Natali
Pool: Support running without TimeoutHandler thread.
The with_*_thread arguments has also been replaced with a single threads=True argument.
Two new pool callbacks:
on_timeout_set(job, soft, hard)
Applied when a task is executed with a timeout.
on_timeout_cancel(job)
Applied when a timeout is cancelled (the job completed)
2.7.3.7 - 2012-05-21
Fixes Python 2.5 support.
2.7.3.6 - 2012-05-21
Pool: Can now be used in an event loop, without starting the supporting threads (TimeoutHandler still not supported)
To facilitate this the pool has gained the following keyword arguments:
with_task_thread
with_result_thread
with_supervisor_thread
on_process_up
Callback called with Process instance as argument whenever a new worker process is added.
Used to add new process fds to the eventloop:
def on_process_up(proc): hub.add_reader(proc.sentinel, pool.maintain_pool)on_process_down
Callback called with Process instance as argument whenever a new worker process is found dead.
Used to remove process fds from the eventloop:
def on_process_down(proc): hub.remove(proc.sentinel)semaphore
Sets the semaphore used to protect from adding new items to the pool when no processes available. The default is a threaded one, so this can be used to change to an async semaphore.
And the following attributes:
- ``readers`` A map of ``fd`` -> ``callback``, to be registered in an eventloop. Currently this is only the result outqueue with a callback that processes all currently incoming results.And the following methods:
- ``did_start_ok`` To be called after starting the pool, and after setting up the eventloop with the pool fds, to ensure that the worker processes didn't immediately exit caused by an error (internal/memory). - ``maintain_pool`` Public version of ``_maintain_pool`` that handles max restarts.Pool: Process too frequent restart protection now only counts if the process had a non-successful exitcode.
This to take into account the maxtasksperchild option, and allowing processes to exit cleanly on their own.
Pool: New options max_restart + max_restart_freq
This means that the supervisor can’t restart processes faster than max_restart’ times per max_restart_freq seconds (like the Erlang supervisor maxR & maxT settings).
The pool is closed and joined if the max restart frequency is exceeded, where previously it would keep restarting at an unlimited rate, possibly crashing the system.
The current default value is to stop if it exceeds 100 * process_count restarts in 1 seconds. This may change later.
It will only count processes with an unsuccessful exit code, this is to take into account the maxtasksperchild setting and code that voluntarily exits.
Pool: The WorkerLostError message now includes the exitcode of the process that disappeared.
2.7.3.5 - 2012-05-09
Now always cleans up after sys.exc_info() to avoid cyclic references.
ExceptionInfo without arguments now defaults to sys.exc_info.
Forking can now be disabled using the MULTIPROCESSING_FORKING_DISABLE environment variable.
Also this envvar is set so that the behavior is inherited after execv.
The semaphore cleanup process started when execv is used now sets a useful process name if the setproctitle module is installed.
Sets the FORKED_BY_MULTIPROCESSING environment variable if forking is disabled.
2.7.3.4 - 2012-04-27
Added billiard.ensure_multiprocessing()
Raises NotImplementedError if the platform does not support multiprocessing (e.g. Jython).
2.7.3.3 - 2012-04-23
PyPy now falls back to using its internal _multiprocessing module, so everything works except for forking_enable(False) (which silently degrades).
Fixed Python 2.5 compat issues.
Uses more with statements
Merged some of the changes from the Python 3 branch.
2.7.3.2 - 2012-04-20
Now installs on PyPy/Jython (but does not work).
2.7.3.1 - 2012-04-20
Python 2.5 support added.
2.7.3.0 - 2012-04-20
Updated from Python 2.7.3
Python 2.4 support removed, now only supports 2.5, 2.6 and 2.7. (may consider py3k support at some point).
Pool improvments from Celery.
no-execv patch added (http://bugs.python.org/issue8713)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for billiard-2.7.3.9.win32-py2.7.exe
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 509d0d5b1470a9f2d2cb67e22726949cf31318e7667b7029f1f85142f4f90070 |
|
| MD5 | 572d6b3392423fc955ed5ccbc2ba17f9 |
|
| BLAKE2b-256 | fd3db0743ae82648cde85531274a658549156ff425a1e21fbc54dfe7736b28b2 |
Hashes for billiard-2.7.3.9-py2.7-win32.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d768ee7d92052efb4171277b11e0bb201be1c0aa5a5ece17a1c0a9e5c7286e31 |
|
| MD5 | c97b384ce4e21501caad6733abcddca2 |
|
| BLAKE2b-256 | 4afa5e30878d430400a5e47bcd1607ee5d85b11b99db490673579275cd45c2c5 |
Hashes for billiard-2.7.3.9-py2.7-macosx-10.7-universal.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a3d1b8620980e25ceeed6986389e32fb4dd6f80ae23998cddc757112711aae59 |
|
| MD5 | 6334e8ebc1e8894297499eb3aa9eaad0 |
|
| BLAKE2b-256 | aee197fdc2eeac3d28ae5cda3950275318f1e79d6940ee714a87144e9f233d73 |
Hashes for billiard-2.7.3.9-py2.7-macosx-10.6-universal.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | dd59c87ec92177b71e1d2856def313f3323963c302bfaf5f7e5980048ac5a58c |
|
| MD5 | 92b4a803c460654568f05a5e8a4caee5 |
|
| BLAKE2b-256 | 6c5bffb73a9b38c463568fdf8395a6e5f4e97e467dafee19889bae80446b7f3b |
Hashes for billiard-2.7.3.9-py2.7-linux-i686.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ad0818d5ffb746f593acc0a62a934cb3477dc479239d34f03596ff6961b204b8 |
|
| MD5 | edce5f214dfb54d792bc90e4bd32c294 |
|
| BLAKE2b-256 | f261c9db7fd9af5af4e3f2cc28d2d71989aed8f12d8587c1ebdf36f2993f6eed |
Hashes for billiard-2.7.3.9-py2.6-macosx-10.7-universal.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 9aa08f072f241d23fabf529ab9be30207a34ad011bdf5d96de78f41c591dae91 |
|
| MD5 | 05c12aece8604dc1cd35e75186c12e61 |
|
| BLAKE2b-256 | 6e70720983cd72d2943ddba6e4e9f20879dde90be24a240c3901710cdafec255 |
Hashes for billiard-2.7.3.9-py2.6-linux-i686.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cc6160aef869ad140a98b324691d7ea74fa72ee57a72ef8af5703403803e1c3d |
|
| MD5 | 70827112cfa4e3b378bcfb0326127857 |
|
| BLAKE2b-256 | 54542623c9e5ba39eb2d4f0871baaf2557b09de17e6ed8a61c2d26db079f88ae |
Hashes for billiard-2.7.3.9-py2.5-macosx-10.7-universal.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 23e0c6c7568118d15dfb8c617df02c619bed0725b6392388701f6f788d63e61f |
|
| MD5 | f9d7de743f786d8b1ae5182082fe1d26 |
|
| BLAKE2b-256 | 4e3364ceb4f6de9eda0d450f81a5bf74cfdfd7f389b55c30794018e6480a963b |
Hashes for billiard-2.7.3.9-py2.5-linux-i686.egg
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 88c7cafbd23c4a863979a09e8e43a650b1f5bd26c262b16ad3351c2a11217a0e |
|
| MD5 | 6de7b1918ee0c8d012b731601ca6dca4 |
|
| BLAKE2b-256 | 9c639e3475e95579c6caa5bae0f9c2028c16e3583512b01a9710c289613aa60c |
