Asynchronous tasks add-on for SENAITE
Project description
Queue of asynchronous tasks for SENAITE LIMS
About
This add-on enables asynchronous tasks for SENAITE LIMS, that allows to better handle concurrent actions and processes when the workload is high. Is specially indicated for high-demand instances and for when there are custom processes that take long to complete. Essentially, senaite.queue reduces the chance of transaction commits by handling tasks asynchronously, in an unattended and sequential manner.
Once installed, this add-on enables asynchronous processing of those tasks that usually have a heavier footprint regarding performance, and with highest chance of transaction conflicts:
Assignment of analyses to worksheets
Assignment of worksheet template to a worksheet
Creation of a worksheet by using a worksheet template
Workflow actions (submit, verify, etc.) for analyses assigned to worksheets
Recursive permissions assignment on client contacts creation
This add-on neither provides support for workflow transitions/actions at Sample level nor for Sample creation. However, this add-on can be extended easily to match additional requirements.
Documentation
Contribute
We want contributing to SENAITE.QUEUE to be fun, enjoyable, and educational for anyone, and everyone. This project adheres to the Contributor Covenant.
By participating, you are expected to uphold this code. Please report unacceptable behavior.
Contributions go far beyond pull requests and commits. Although we love giving you the opportunity to put your stamp on SENAITE.QUEUE, we also are thrilled to receive a variety of other contributions.
Please, read Contributing to senaite.queue document.
If you wish to contribute with translations, check the project site on Transifex.
Feedback and support
License
SENAITE.QUEUE Copyright (C) 2019-2020 RIDING BYTES & NARALABS
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2 as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
Release notes
Update from 1.0.1 to 1.0.2
With version 1.0.2, the legacy storage for queued tasks has changed and helper storages (e.g. for Worksheets) are no longer required. IQueued marker interface is no longer used neither. Most of the base code has been refactored keeping in mind the following objectives:
Less complexity: less code, better code
Less chance of transaction commit conflicts
Boost performance: better experience, with no delays
All these changes also makes the add-on easier to extend and maintain. The downside is that old legacy storage is no longer used and therefore, tasks that were queued before the upgrade will be discarded.
Be sure there are no remaining tasks in the queue before the upgrade
If you have your own add-on extending senaite.queue, please review the changes and check if some parts of your add-on require modifications
A queue server has been introduced. Therefore, two zeo clients are recommended: one that acts as the server and at least another one in charge of consuming tasks. Also, this version now depends on three additional packages: requests, senaite.jsonapi and cryptography. Please read the installation instructions and run buildout to download the dependencies.
Installation
Is strongly recommended to have a SENAITE instance setup in ZEO mode, because this add-on is especially useful when a reserved zeo client is used to act as a queue server and at least one additional zeo client for tasks consumption.
In standalone installation, only one CPU / CPU core can be used for processing requests, with a limited number of threads (usually 2). With a ZEO mode setup, the database can be used by multiple zeo clients at the same time, each one using it’s own CPU. See Scalability and ZEO for further information.
Create a new reserved user in SENAITE instance (under /senaite/acl_users). The recommended username is queue_consumer.
This user will be used by the consumer to pop tasks from the queue server in a sequential manner. The consumer will eventually process the task, but acting as the user who initially triggered the process. However, the reserved user responsible of dispatching must have enough privileges. Assign this user to the group “Site Administrator” and/or “Manager”.
First, add this add-on in the eggs section of your buildout configuration file:
[buildout]
...
[instance]
...
eggs =
...
senaite.queue
Then, add a two clients (a consumer and the server) in your buildout configuration:
# Reserved user for dispatching queued tasks
# See https://pypi-hypernode.com/project/senaite.queue
queue-user-name=queue_consumer
queue-user-password=queue_consumer_password
parts =
....
queue_consumer
queue_server
and configure two reserved clients:
[queue_consumer]
# ZEO Client reserved for the consumption of queued tasks
<= client_base
recipe = plone.recipe.zope2instance
http-address = 127.0.0.1:8089
zope-conf-additional =
<clock-server>
method /senaite/queue_consume
period 5
user ${buildout:queue-user-name}
password ${buildout:queue-user-password}
host localhost:8089
</clock-server>
[queue_server]
# ZEO Client reserved to act as the server of the queue
<= client_base
recipe = plone.recipe.zope2instance
http-address = 127.0.0.1:8090
In most scenarios, this configuration is enough. However, senaite.queue supports multi consumers, that can be quite useful for those SENAITE installations that have a very high overload. To add more consumers, add as many zeo client sections as you need with the additional clock-server zope configuration. Do not forget to set the value host correctly to all them, because this value is used by the queue server to identify the consumers when tasks are requested.
The maximum number of concurrent consumers supported by the queue server is 4.
Run bin/buildout afterwards. With this configuration, buildout will download and install the latest published release of senaite.queue from Pypi.
Once buildout finishes, start the clients:
$ sudo -u plone_daemon bin/client1 start
$ sudo -u plone_daemon bin/queue_server start
$ sudo -u plone_daemon bin/queue_client start
Then visit your SENAITE site and login with a user with “Site Administrator” privileges to activate the add-on:
http://localhost:8080/senaite/prefs_install_products_form
Once activated, go to Site Setup > Queue Settings and, in field “Queue Server”, type the url of the zeo client that will act as the server of the queue.
Changelog
1.0.3 (2021-07-23)
#21 Improve the reindex security objects process
Skip guard checks when current thread is a consumer
Make the creation of WS with WST assignment more efficient
Pin cryptography==3.1.1
Fix client’s queue tasks in “queued” status are not updated when “running”
1.0.2 (2020-11-15)
Support for multiple consumers (up to 4 concurrent processes)
Added JSON API endpoints for both queue server and clients
Queue server-client implementation, without the need of annotations
Added PAS plugin for authentication, with symmetric encryption
Delegate the reindex object security to queue when linking contacts to users
#7 Allow to queue generic worflow actions without specific adapter
#7 Redux and better performance
#6 Allow the prioritization of tasks
#5 No actions can be done to worksheets with queued jobs
1.0.1 (2020-02-09)
Allow to manually assign the username to the task to be queued
Support for failed tasks
Notify when the value for max_seconds_unlock is too low
#3 New queue_tasks view with the list of tasks and statistics
#2 Add max_retries setting for failing tasks
#1 Add sample guard to prevent transitions when queued analyses
1.0.0 (2019-11-10)
First version
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.