gocept.runner 3.0

Signals

The worker-procss can be terminated by SIGTERM and SIGHUP in a sane way. Write a script to a temporary file:

>>> import sys
>>> runner_path = os.path.abspath(
...     os.path.join(os.path.dirname(__file__), '..', '..'))
>>> fd, script_name = tempfile.mkstemp(suffix='.py')
>>> exchange_fd, exchange_file_name = tempfile.mkstemp()
>>> script = os.fdopen(fd, 'w')
>>> _ = script.write("""\
... import sys
... sys.path[0:0] = %s
... sys.path.insert(0, '%s')
... import gocept.runner
...
... f = open('%s', 'w')
...
... @gocept.runner.appmain(ticks=0.1)
... def worker():
...     f.write("Working.\\n")
...     f.flush()
...
... worker(None, '%s')
... """ % (sys.path, runner_path, exchange_file_name, zope_conf))
>>> script.close()

Call the script and wait for it to produce some output:

>>> import signal
>>> import subprocess
>>> import time
>>> exchange = os.fdopen(exchange_fd, 'r+')
>>> proc = subprocess.Popen(
...     [sys.executable, script_name],
...     stdout=subprocess.PIPE,
...     text=True)
>>> while not exchange.read():
...     time.sleep(0.1)
...     _ = exchange.seek(0, 0)
>>> _ = exchange.seek(0, 0)
>>> print(exchange.read())
Working...

Okay, now kill it:

>>> os.kill(proc.pid, signal.SIGTERM)

Wait for the process to really finish and get the output. The runner logs that it was terminated:

>>> stdout, stderr = proc.communicate()
>>> print(stdout)
------...INFO gocept.runner.runner Received signal 15, terminating...

This also works with SIGHUP:

>>> _ = exchange.truncate(0)
>>> proc = subprocess.Popen(
...     [sys.executable, script_name],
...     stdout=subprocess.PIPE,
...     text=True)
>>> while not exchange.read():
...     time.sleep(0.1)
...     _ = exchange.seek(0, 0)
>>> _ = exchange.seek(0, 0)
>>> print(exchange.read())
Working...

Okay, now kill it:

>>> os.kill(proc.pid, signal.SIGHUP)
>>> stdout, stderr = proc.communicate()
>>> print(stdout)
------...INFO gocept.runner.runner Received signal 1, terminating...

Clean up: >>> exchange.close() >>> os.remove(script_name) >>> os.remove(exchange_file_name)

Setting the principal

It is also prossible to create a main loop which runs in an interaction:

>>> def get_principal():
...     return 'zope.mgr'

>>> import zope.security.management
>>> work_count = 0
>>> def interaction_worker():
...     global work_count
...     work_count += 1
...     if work_count >= 3:
...         raise SystemExit(1)
...     log = logging.getLogger('test')
...     interaction = zope.security.management.getInteraction()
...     principal = interaction.participations[0].principal
...     log.info("Working as %s" % principal.id)
>>> worker = gocept.runner.appmain(ticks=0.1, principal=get_principal)(
...     interaction_worker)

Call the worker now:

>>> worker(None, zope_conf)
------
... INFO test Working as zope.mgr
------
... INFO test Working as zope.mgr

After the worker is run there is no interaction:

>>> zope.security.management.queryInteraction() is None
True

It’s quite common to read the principal from zope.conf. Therefore there is a helper which makes this task easier:

>>> work_count = 0
>>> worker = gocept.runner.appmain(
...     ticks=0.1,
...     principal=gocept.runner.from_config('test', 'test-principal'))(
...     interaction_worker)
>>> worker(None, zope_conf)
------
... INFO test Working as zope.mgr
------
... INFO test Working as zope.mgr

Subsites

It is possible to directly work on sites inside the root. The site must already exist of course, otherwise there will be an error:

>>> worker('a-site', zope_conf)
Traceback (most recent call last):
    ...
KeyError: 'a-site'

Clean up:

>>> import shutil
>>> shutil.rmtree(zodb_path)
>>> os.remove(zope_conf)

Main loop

The main loop loops until it encounters a KeyboardInterrupt or a SystemExit exception and calls the worker once a second.

Define a worker function which exits when it is called the 3rd time:

>>> work_count = 0
>>> def worker():
...     print("Working")
...     global work_count
...     work_count += 1
...     if work_count >= 3:
...         raise SystemExit(1)

Call the main loop:

>>> import gocept.runner.runner
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
Working
Working
Working
>>> work_count
3

During the loop the site is set:

>>> import zope.component.hooks
>>> zope.component.hooks.getSite() is None
True
>>> def worker():
...     print(zope.component.hooks.getSite())
...     raise SystemExit(1)
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
<zope.site.folder.Folder object at 0x...>

After the loop, no site is set again:

>>> zope.component.hooks.getSite() is None
True

When the worker passes without error a transaction is committed:

>>> work_count = 0
>>> def worker():
...     print("Working")
...     global work_count
...     work_count += 1
...     if work_count >= 2:
...         raise SystemExit(1)
...     site = zope.component.hooks.getSite()
...     site.worker_done = 1
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
Working
Working

We have set the attribute worker_done now:

>>> getRootFolder().worker_done
1

When the worker produces an error, the transaction is aborted:

>>> work_count = 0
>>> def worker():
...     global work_count
...     work_count += 1
...     print("Working")
...     site = zope.component.hooks.getSite()
...     site.worker_done += 1
...     if work_count < 3:
...         raise ValueError('hurz')
...     raise SystemExit(1)
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.1, worker)()
Working
Working
Working

We still have the attribute worker_done set to 1:b

>>> getRootFolder().worker_done
1

Controlling sleep time

The worker function can control the sleep time.

Register a log handler so we can observe this:

>>> from io import StringIO
>>> import logging
>>> log = StringIO()
>>> log_handler = logging.StreamHandler(log)
>>> logging.root.addHandler(log_handler)
>>> old_log_level = logging.root.level
>>> logging.root.setLevel(logging.DEBUG)

>>> work_count = 0
>>> def worker():
...     global work_count
...     work_count += 1
...     new_sleep = work_count * 0.1
...     if work_count == 3:
...         print("Will sleep default")
...         return None
...     if work_count > 3:
...         raise SystemExit(1)
...     print("Will sleep %s" % new_sleep)
...     return new_sleep
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.15, worker)()
Will sleep 0.1
Will sleep 0.2
Will sleep default

The real sleep values are in the log:

>>> print(log.getvalue())
new transaction
commit
Sleeping 0.1 seconds
new transaction
commit
Sleeping 0.2 seconds
new transaction
commit
Sleeping 0.15 seconds
new transaction
abort...

When an error occours within the worker, the default sleep time will be used:

>>> _ = log.seek(0)
>>> _ = log.truncate()
>>> work_count = 0
>>> def worker():
...     global work_count
...     work_count += 1
...     if work_count == 1:
...         new_sleep = 0.1
...     elif work_count == 2:
...         print("Failing")
...         raise Exception("Fail!")
...     elif work_count == 3:
...         print("Will sleep default")
...         return None
...     elif work_count > 3:
...         return gocept.runner.Exit
...     print("Will sleep %s" % new_sleep)
...     return new_sleep
>>> gocept.runner.runner.MainLoop(getRootFolder(), 0.15, worker)()
Will sleep 0.1
Failing
Will sleep default

The real sleep values are in the log:

>>> print(log.getvalue())
new transaction
commit
Sleeping 0.1 seconds
new transaction
Error in worker: Exception('Fail!'...)
Traceback (most recent call last):
  ...
Exception: Fail!
abort
Sleeping 0.15 seconds
new transaction
commit
Sleeping 0.15 seconds
new transaction
commit...

Restore old log handler:

>>> logging.root.removeHandler(log_handler)
>>> log.close()
>>> logging.root.setLevel(old_log_level)

3.0 (2023-07-20)

• Drop support for Python 2.7.

• Add support for Python 3.9, 3.10, 3.11.

2.2 (2020-09-14)

• Get rid of ZODB3 dependency.

2.1 (2019-12-02)

• Add support for Python 3.7 and 3.8.

• Migrate to Github.

2.0 (2018-03-14)

• Backwards incompatible change: No longer depend on zope.app.server, and no longer superfluously use its ZConfig schema, which superfluously requires an <accesslog> section that does not make any sense in our context anyway. So <accesslog> is no longer allowed in the ZConfing config file used by this package.

1.0 (2016-01-12)

• No longer depend on zope.app.component.

0.7.1 (2015-01-28)

• Fix faulty release 0.7.0.

0.7.0 (2015-01-28)

• An error message is correctly logged when the worker produces an exception which cannot be represented in us-ascii. This used to cause an exception in the logging module. The cause of those exceptions where quite hard to track.

• Removed dependency on zope.app.security which was neither declared nor necessary.

• Moved project home to <https://bitbucket.org/gocept/gocept.runner/>.

0.6.0 (2011-12-01)

• Added transaction_per_item decorator.

• Use stdlib’s doctest instead of zope.testing.

0.5.3 (2010-04-14)

• Use log.error/log.warning instead of log.exception. Conflict errors are logged as warning now, because they are not really a problem.

0.5.2 (2010-04-14)

• Convert logged exceptions to str.

0.5.1 (2009-10-13)

• Declared dependencies correctly.

0.3.2 (2009-05-21)

• Fixed handling of subsites in appmain.

0.3.1 (2009-05-21)

• Declared namespace package.