Skip to main content

Python multiprocessing fork

Project description

billiard

About

billiard is a fork of the Python 2.7 multiprocessing package. The multiprocessing package itself is a renamed and updated version of R Oudkerk’s pyprocessing package. This standalone variant is intended to be compatible with Python 2.4 and 2.5, and will draw it’s fixes/improvements from python-trunk.

  • This package would not be possible if not for the contributions of not only the current maintainers but all of the contributors to the original pyprocessing package listed here

  • Also it is a fork of the multiprocessin backport package by Christian Heims.

  • It includes the no-execv patch contributed by sbt.

  • And the Pool improvements previously located in Celery.

Bug reporting

Please report bugs related to multiprocessing at the Python bug tracker. Issues related to billiard should be reported at http://github.com/celery/billiard/issues.

Changes

2.7.3.9 - 2012-06-03

  • Environment variable MP_MAIN_FILE envvar is now set to the path of the __main__ module when execv is enabled.

  • Pool: Errors occuring in the TaskHandler are now reported.

2.7.3.8 - 2012-06-01

  • Can now be installed on Py 3.2

  • Issue #12091: simplify ApplyResult and MapResult with threading.Event

    Patch by Charles-Francois Natali

  • Pool: Support running without TimeoutHandler thread.

    • The with_*_thread arguments has also been replaced with a single threads=True argument.

    • Two new pool callbacks:

      • on_timeout_set(job, soft, hard)

        Applied when a task is executed with a timeout.

      • on_timeout_cancel(job)

        Applied when a timeout is cancelled (the job completed)

2.7.3.7 - 2012-05-21

  • Fixes Python 2.5 support.

2.7.3.6 - 2012-05-21

  • Pool: Can now be used in an event loop, without starting the supporting threads (TimeoutHandler still not supported)

    To facilitate this the pool has gained the following keyword arguments:

    • with_task_thread

    • with_result_thread

    • with_supervisor_thread

    • on_process_up

      Callback called with Process instance as argument whenever a new worker process is added.

      Used to add new process fds to the eventloop:

      def on_process_up(proc):
          hub.add_reader(proc.sentinel, pool.maintain_pool)
    • on_process_down

      Callback called with Process instance as argument whenever a new worker process is found dead.

      Used to remove process fds from the eventloop:

      def on_process_down(proc):
          hub.remove(proc.sentinel)
    • semaphore

      Sets the semaphore used to protect from adding new items to the pool when no processes available. The default is a threaded one, so this can be used to change to an async semaphore.

    And the following attributes:

    - ``readers``
    
        A map of ``fd`` -> ``callback``, to be registered in an eventloop.
        Currently this is only the result outqueue with a callback
        that processes all currently incoming results.

    And the following methods:

    - ``did_start_ok``
    
        To be called after starting the pool, and after setting up the
        eventloop with the pool fds, to ensure that the worker processes
        didn't immediately exit caused by an error (internal/memory).
    
    - ``maintain_pool``
    
        Public version of ``_maintain_pool`` that handles max restarts.
  • Pool: Process too frequent restart protection now only counts if the process had a non-successful exitcode.

    This to take into account the maxtasksperchild option, and allowing processes to exit cleanly on their own.

  • Pool: New options max_restart + max_restart_freq

    This means that the supervisor can’t restart processes faster than max_restart’ times per max_restart_freq seconds (like the Erlang supervisor maxR & maxT settings).

    The pool is closed and joined if the max restart frequency is exceeded, where previously it would keep restarting at an unlimited rate, possibly crashing the system.

    The current default value is to stop if it exceeds 100 * process_count restarts in 1 seconds. This may change later.

    It will only count processes with an unsuccessful exit code, this is to take into account the maxtasksperchild setting and code that voluntarily exits.

  • Pool: The WorkerLostError message now includes the exitcode of the process that disappeared.

2.7.3.5 - 2012-05-09

  • Now always cleans up after sys.exc_info() to avoid cyclic references.

  • ExceptionInfo without arguments now defaults to sys.exc_info.

  • Forking can now be disabled using the MULTIPROCESSING_FORKING_DISABLE environment variable.

    Also this envvar is set so that the behavior is inherited after execv.

  • The semaphore cleanup process started when execv is used now sets a useful process name if the setproctitle module is installed.

  • Sets the FORKED_BY_MULTIPROCESSING environment variable if forking is disabled.

2.7.3.4 - 2012-04-27

  • Added billiard.ensure_multiprocessing()

    Raises NotImplementedError if the platform does not support multiprocessing (e.g. Jython).

2.7.3.3 - 2012-04-23

  • PyPy now falls back to using its internal _multiprocessing module, so everything works except for forking_enable(False) (which silently degrades).

  • Fixed Python 2.5 compat issues.

  • Uses more with statements

  • Merged some of the changes from the Python 3 branch.

2.7.3.2 - 2012-04-20

  • Now installs on PyPy/Jython (but does not work).

2.7.3.1 - 2012-04-20

  • Python 2.5 support added.

2.7.3.0 - 2012-04-20

  • Updated from Python 2.7.3

  • Python 2.4 support removed, now only supports 2.5, 2.6 and 2.7. (may consider py3k support at some point).

  • Pool improvments from Celery.

  • no-execv patch added (http://bugs.python.org/issue8713)

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

billiard-2.7.3.9.tar.gz (121.6 kB view details)

Uploaded Source

Built Distributions

billiard-2.7.3.9.win32-py2.7.exe (299.0 kB view details)

Uploaded Source

billiard-2.7.3.9-py2.7-win32.egg (199.6 kB view details)

Uploaded Source

billiard-2.7.3.9-py2.7-linux-i686.egg (218.1 kB view details)

Uploaded Source

billiard-2.7.3.9-py2.6-linux-i686.egg (218.7 kB view details)

Uploaded Source

billiard-2.7.3.9-py2.5-linux-i686.egg (218.3 kB view details)

Uploaded Source

File details

Details for the file billiard-2.7.3.9.tar.gz.

File metadata

  • Download URL: billiard-2.7.3.9.tar.gz
  • Upload date:
  • Size: 121.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for billiard-2.7.3.9.tar.gz
Algorithm Hash digest
SHA256 f15b1cbc4d216cae3b32bc534aabb421529a455a3921b155124c7ea61d8af213
MD5 beaab52135f9c3e06229bd873bab01db
BLAKE2b-256 8e0f1b8c98e1aaf491b7eb3961603db74bcc0fbe2e5b9c1ff90271458d83bb1f

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9.win32-py2.7.exe.

File metadata

File hashes

Hashes for billiard-2.7.3.9.win32-py2.7.exe
Algorithm Hash digest
SHA256 509d0d5b1470a9f2d2cb67e22726949cf31318e7667b7029f1f85142f4f90070
MD5 572d6b3392423fc955ed5ccbc2ba17f9
BLAKE2b-256 fd3db0743ae82648cde85531274a658549156ff425a1e21fbc54dfe7736b28b2

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.7-win32.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.7-win32.egg
Algorithm Hash digest
SHA256 d768ee7d92052efb4171277b11e0bb201be1c0aa5a5ece17a1c0a9e5c7286e31
MD5 c97b384ce4e21501caad6733abcddca2
BLAKE2b-256 4afa5e30878d430400a5e47bcd1607ee5d85b11b99db490673579275cd45c2c5

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.7-macosx-10.7-universal.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.7-macosx-10.7-universal.egg
Algorithm Hash digest
SHA256 a3d1b8620980e25ceeed6986389e32fb4dd6f80ae23998cddc757112711aae59
MD5 6334e8ebc1e8894297499eb3aa9eaad0
BLAKE2b-256 aee197fdc2eeac3d28ae5cda3950275318f1e79d6940ee714a87144e9f233d73

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.7-macosx-10.6-universal.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.7-macosx-10.6-universal.egg
Algorithm Hash digest
SHA256 dd59c87ec92177b71e1d2856def313f3323963c302bfaf5f7e5980048ac5a58c
MD5 92b4a803c460654568f05a5e8a4caee5
BLAKE2b-256 6c5bffb73a9b38c463568fdf8395a6e5f4e97e467dafee19889bae80446b7f3b

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.7-linux-i686.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.7-linux-i686.egg
Algorithm Hash digest
SHA256 ad0818d5ffb746f593acc0a62a934cb3477dc479239d34f03596ff6961b204b8
MD5 edce5f214dfb54d792bc90e4bd32c294
BLAKE2b-256 f261c9db7fd9af5af4e3f2cc28d2d71989aed8f12d8587c1ebdf36f2993f6eed

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.6-macosx-10.7-universal.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.6-macosx-10.7-universal.egg
Algorithm Hash digest
SHA256 9aa08f072f241d23fabf529ab9be30207a34ad011bdf5d96de78f41c591dae91
MD5 05c12aece8604dc1cd35e75186c12e61
BLAKE2b-256 6e70720983cd72d2943ddba6e4e9f20879dde90be24a240c3901710cdafec255

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.6-linux-i686.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.6-linux-i686.egg
Algorithm Hash digest
SHA256 cc6160aef869ad140a98b324691d7ea74fa72ee57a72ef8af5703403803e1c3d
MD5 70827112cfa4e3b378bcfb0326127857
BLAKE2b-256 54542623c9e5ba39eb2d4f0871baaf2557b09de17e6ed8a61c2d26db079f88ae

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.5-macosx-10.7-universal.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.5-macosx-10.7-universal.egg
Algorithm Hash digest
SHA256 23e0c6c7568118d15dfb8c617df02c619bed0725b6392388701f6f788d63e61f
MD5 f9d7de743f786d8b1ae5182082fe1d26
BLAKE2b-256 4e3364ceb4f6de9eda0d450f81a5bf74cfdfd7f389b55c30794018e6480a963b

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.9-py2.5-linux-i686.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.9-py2.5-linux-i686.egg
Algorithm Hash digest
SHA256 88c7cafbd23c4a863979a09e8e43a650b1f5bd26c262b16ad3351c2a11217a0e
MD5 6de7b1918ee0c8d012b731601ca6dca4
BLAKE2b-256 9c639e3475e95579c6caa5bae0f9c2028c16e3583512b01a9710c289613aa60c

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page