Skip to main content

Python multiprocessing fork with improvements and bugfixes

Project description

billiard

About

billiard is a fork of the Python 2.7 multiprocessing package. The multiprocessing package itself is a renamed and updated version of R Oudkerk’s pyprocessing package. This standalone variant is intended to be compatible with Python 2.4 and 2.5, and will draw it’s fixes/improvements from python-trunk.

  • This package would not be possible if not for the contributions of not only the current maintainers but all of the contributors to the original pyprocessing package listed here

  • Also it is a fork of the multiprocessin backport package by Christian Heims.

  • It includes the no-execv patch contributed by R. Oudkerk.

  • And the Pool improvements previously located in Celery.

Bug reporting

Please report bugs related to multiprocessing at the Python bug tracker. Issues related to billiard should be reported at http://github.com/celery/billiard/issues.

Changes

2.7.3.18 - 2012-11-05

  • [Pool] Fix for check_timeouts if not set.

    Fix contributed by Dmitry Sukhov

  • Fixed pickle problem with Traceback.

    Code.frame.__loader__ is now ignored as it may be set to an unpickleable object.

  • The Django old-layout warning was always showing.

2.7.3.17 - 2012-09-26

  • Fixes typo

2.7.3.16 - 2012-09-26

  • Windows: Fixes for SemLock._rebuild (Issue #24).

  • Pool: Job terminated with terminate_job now raises billiard.exceptions.Terminated.

2.7.3.15 - 2012-09-21

  • Windows: Fixes unpickling of SemLock when using fallback.

  • Windows: Fixes installation when no C compiler.

2.7.3.14 - 2012-09-20

  • Installation now works again for Python 3.

2.7.3.13 - 2012-09-14

  • Merged with Python trunk (many authors, many fixes: see Python changelog in trunk).

  • Using execv now also works with older Django projects using setup_environ (Issue #10).

  • Billiard now installs with a warning that the C extension could not be built if a compiler is not installed or the build fails in some other way.

    It really is recommended to have the C extension installed when running with force execv, but this change also makes it easier to install.

  • Pool: Hard timeouts now sends KILL shortly after TERM so that C extensions cannot block the signal.

    Python signal handlers are called in the interpreter, so they cannot be called while a C extension is blocking the interpreter from running.

  • Now uses a timeout value for Thread.join that doesn’t exceed the maximum on some platforms.

  • Fixed bug in the SemLock fallback used when C extensions not installed.

    Fix contributed by Mher Movsisyan.

  • Pool: Now sets a Process.index attribute for every process in the pool.

    This number will always be between 0 and concurrency-1, and can be used to e.g. create a logfile for each process in the pool without creating a new logfile whenever a process is replaced.

2.7.3.12 - 2012-08-05

  • Fixed Python 2.5 compatibility issue.

  • New Pool.terminate_job(pid) to terminate a job without raising WorkerLostError

2.7.3.11 - 2012-08-01

  • Adds support for FreeBSD 7+

    Fix contributed by koobs.

  • Pool: New argument allow_restart is now required to enable the pool process sentinel that is required to restart the pool.

    It’s disabled by default, which reduces the number of file descriptors/semaphores required to run the pool.

  • Pool: Now emits a warning if a worker process exited with error-code.

    But not if the error code is 155, which is now returned if the worker process was recycled (maxtasksperchild).

  • Python 3 compatibility fixes.

  • Python 2.5 compatibility fixes.

2.7.3.10 - 2012-06-26

  • The TimeLimitExceeded exception string representation only included the seconds as a number, it now gives a more human friendly description.

  • Fixed typo in LaxBoundedSemaphore.shrink.

  • Pool: ResultHandler.handle_event no longer requires any arguments.

  • setup.py bdist now works

2.7.3.9 - 2012-06-03

  • Environment variable MP_MAIN_FILE envvar is now set to the path of the __main__ module when execv is enabled.

  • Pool: Errors occurring in the TaskHandler are now reported.

2.7.3.8 - 2012-06-01

  • Can now be installed on Py 3.2

  • Issue #12091: simplify ApplyResult and MapResult with threading.Event

    Patch by Charles-Francois Natali

  • Pool: Support running without TimeoutHandler thread.

    • The with_*_thread arguments has also been replaced with a single threads=True argument.

    • Two new pool callbacks:

      • on_timeout_set(job, soft, hard)

        Applied when a task is executed with a timeout.

      • on_timeout_cancel(job)

        Applied when a timeout is cancelled (the job completed)

2.7.3.7 - 2012-05-21

  • Fixes Python 2.5 support.

2.7.3.6 - 2012-05-21

  • Pool: Can now be used in an event loop, without starting the supporting threads (TimeoutHandler still not supported)

    To facilitate this the pool has gained the following keyword arguments:

    • with_task_thread

    • with_result_thread

    • with_supervisor_thread

    • on_process_up

      Callback called with Process instance as argument whenever a new worker process is added.

      Used to add new process fds to the eventloop:

      def on_process_up(proc):
          hub.add_reader(proc.sentinel, pool.maintain_pool)
    • on_process_down

      Callback called with Process instance as argument whenever a new worker process is found dead.

      Used to remove process fds from the eventloop:

      def on_process_down(proc):
          hub.remove(proc.sentinel)
    • semaphore

      Sets the semaphore used to protect from adding new items to the pool when no processes available. The default is a threaded one, so this can be used to change to an async semaphore.

    And the following attributes:

    - ``readers``
    
        A map of ``fd`` -> ``callback``, to be registered in an eventloop.
        Currently this is only the result outqueue with a callback
        that processes all currently incoming results.

    And the following methods:

    - ``did_start_ok``
    
        To be called after starting the pool, and after setting up the
        eventloop with the pool fds, to ensure that the worker processes
        didn't immediately exit caused by an error (internal/memory).
    
    - ``maintain_pool``
    
        Public version of ``_maintain_pool`` that handles max restarts.
  • Pool: Process too frequent restart protection now only counts if the process had a non-successful exit-code.

    This to take into account the maxtasksperchild option, and allowing processes to exit cleanly on their own.

  • Pool: New options max_restart + max_restart_freq

    This means that the supervisor can’t restart processes faster than max_restart’ times per max_restart_freq seconds (like the Erlang supervisor maxR & maxT settings).

    The pool is closed and joined if the max restart frequency is exceeded, where previously it would keep restarting at an unlimited rate, possibly crashing the system.

    The current default value is to stop if it exceeds 100 * process_count restarts in 1 seconds. This may change later.

    It will only count processes with an unsuccessful exit code, this is to take into account the maxtasksperchild setting and code that voluntarily exits.

  • Pool: The WorkerLostError message now includes the exit-code of the process that disappeared.

2.7.3.5 - 2012-05-09

  • Now always cleans up after sys.exc_info() to avoid cyclic references.

  • ExceptionInfo without arguments now defaults to sys.exc_info.

  • Forking can now be disabled using the MULTIPROCESSING_FORKING_DISABLE environment variable.

    Also this envvar is set so that the behavior is inherited after execv.

  • The semaphore cleanup process started when execv is used now sets a useful process name if the setproctitle module is installed.

  • Sets the FORKED_BY_MULTIPROCESSING environment variable if forking is disabled.

2.7.3.4 - 2012-04-27

  • Added billiard.ensure_multiprocessing()

    Raises NotImplementedError if the platform does not support multiprocessing (e.g. Jython).

2.7.3.3 - 2012-04-23

  • PyPy now falls back to using its internal _multiprocessing module, so everything works except for forking_enable(False) (which silently degrades).

  • Fixed Python 2.5 compat. issues.

  • Uses more with statements

  • Merged some of the changes from the Python 3 branch.

2.7.3.2 - 2012-04-20

  • Now installs on PyPy/Jython (but does not work).

2.7.3.1 - 2012-04-20

  • Python 2.5 support added.

2.7.3.0 - 2012-04-20

  • Updated from Python 2.7.3

  • Python 2.4 support removed, now only supports 2.5, 2.6 and 2.7. (may consider py3k support at some point).

  • Pool improvements from Celery.

  • no-execv patch added (http://bugs.python.org/issue8713)

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

billiard-2.7.3.18.tar.gz (125.1 kB view details)

Uploaded Source

Built Distributions

billiard-2.7.3.18-py2.7-macosx-10.6-intel.egg (206.1 kB view details)

Uploaded Source

billiard-2.7.3.18-py2.6-macosx-10.8-intel.egg (206.4 kB view details)

Uploaded Source

File details

Details for the file billiard-2.7.3.18.tar.gz.

File metadata

  • Download URL: billiard-2.7.3.18.tar.gz
  • Upload date:
  • Size: 125.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for billiard-2.7.3.18.tar.gz
Algorithm Hash digest
SHA256 3ec1c17ff0c51ebb03ec3dea4629f81617424c937360ce1e5ba930dea147b659
MD5 ec05c93cb3ce292470567c1f822272aa
BLAKE2b-256 0cf19418735b2fd10935310c062991bb414e8c1d837be76e71b0ad91ed7108c4

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.18-py2.7-macosx-10.6-intel.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.18-py2.7-macosx-10.6-intel.egg
Algorithm Hash digest
SHA256 0211cb33e775000daaa1b794e47d9a381fd0ca521e902b01043a570211917423
MD5 d119cfec669c1612478c9fa14f951be4
BLAKE2b-256 848ccb2cc4c6bac33b5f3d61f59ea9614e3e1b37192e05771b2bf7b971d1fd2b

See more details on using hashes here.

Provenance

File details

Details for the file billiard-2.7.3.18-py2.6-macosx-10.8-intel.egg.

File metadata

File hashes

Hashes for billiard-2.7.3.18-py2.6-macosx-10.8-intel.egg
Algorithm Hash digest
SHA256 0bfaddf08230bb921438297e4537ddefbed0c05393c83d261aae9b566110ebf0
MD5 19f28030eec7da475bf1565544182b84
BLAKE2b-256 948e66ec3478a6a43953094e26e110ec46ad9421b7eec16ad4837d1b32544831

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page