Lightweight & modular queue and cron system for Django
Project description
Django Lightweight Queue
DLQ is a lightweight & modular queue and cron system for Django. It powers millions of production jobs every day at Thread.
Installation
pip install django-lightweight-queue[redis]
Currently the only production-ready backends are redis-based, so the redis
extra is essentially required. Additional non-redis backed production-ready
backends are great candidates for community contributions.
Basic Usage
import time
from django_lightweight_queue import task
# Define a task
@task()
def long_running_task(first_arg, second_arg):
time.sleep(first_arg * second_arg)
# Request that the task be executed at some point
long_running_task(4, second_arg=9)
See the docstring on the task
decorator
for more details.
Configuration
All automatically picked up configuration options begin with LIGHTWEIGHT_QUEUE_
and can be found in app_settings.py
. They should be placed in the usual Django
settings files, for example:
LIGHTWEIGHT_QUEUE_BACKEND = 'django_lightweight_queue.backends.redis.RedisBackend'
Special Configuration
If desired, specific configuration overrides can be placed in a standalone python file which passed on the command line. This is useful for applying customisations for specific servers.
For example, given a special.py
containing:
LIGHTWEIGHT_QUEUE_REDIS_PORT = 12345
and then running:
$ python manage.py queue_runner --config=special.py
will result in the runner to use the settings from the specified configuration file in preference to settings from the Django environment. Any settings not present in the specified file are inherited from the global configuration.
Backends
There are four built-in backends:
Backend | Type | Description |
---|---|---|
Synchronous | Development | Executes the task inline, without any actual queuing. |
Redis | Production | Executes tasks at-most-once using Redis for storage of the enqueued tasks. |
Reliable Redis | Production | Executes tasks at-least-once using Redis for storage of the enqueued tasks (subject to Redis consistency). Does not guarantee the task completes. |
Debug Web | Debugging | Instead of running jobs it prints the url to a view that can be used to run a task in a transaction which will be rolled back. This is useful for debugging and optimising tasks. |
Running Workers
The queue runner is implemented as a Django management command:
$ python manage.py queue_runner
Workers can be distributed over multiple hosts by telling each runner that it is part of a pool:
$ python manage.py queue_runner --machine 2 --of 4
Alternatively a runner can be told explicitly which configuration to use:
$ python manage.py queue_runner --exact-configuration --config=special.py
When using --exact-configuration
the number of workers is configured exactly,
rather than being treated as the configuration for a pool. Additionally,
exactly-configured runners will not run any cron workers.
Example
Given a Django configuration containing:
LIGHTWEIGHT_QUEUE_WORKERS = {
'queue1': 3,
}
and a special.py
containing:
LIGHTWEIGHT_QUEUE_WORKERS = {
'queue1': 2,
}
Running any of:
$ python manage.py queue_runner --machine 1 --of 3 # or,
$ python manage.py queue_runner --machine 2 --of 3 # or,
$ python manage.py queue_runner --machine 3 --of 3
will result in one worker for queue1
on the current machine, while:
$ python manage.py queue_runner --exact-configuration --config=special.py
will result in two workers on the current machine.
Cron Tasks
DLQ supports the use of a cron-like specification of Django management commands to be run at certain times.
To specify that a management command should be run at a given time, place a
cron.py
file in the root folder of the Django app which defines the command
and which contains a CONFIG
variable:
CONFIG = (
{
'command': 'my_cron_command',
# Day values 1-7 to match datetime.datetime.utcnow().isoweekday()
'days': '*',
'hours': '*',
'minutes': '*',
# Equivalent behaviour to the kwarg to `task` of the same name
'sigkill_on_stop': True,
},
)
Maintainers
This repository was created by Chris Lamb at Thread, and continues to be maintained by the Thread engineering team.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for django-lightweight-queue-3.0.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 577e67e9480c99e06dd2ba4dd6aaa4d3a82e1bb8345b45a71166842eb406aec1 |
|
MD5 | 961dd44fa94d149d0ed08e84fb3aeecc |
|
BLAKE2b-256 | fdfcd396f56bdd52f89a41666b7f8fef060fae54fd4446d5e171b6dbcae05d48 |
Hashes for django_lightweight_queue-3.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5935fd7c1d5b8120b6c659d2e5e5967fcffdf3a8bcb63bc08316976e367779a5 |
|
MD5 | 942f3d67819f5c47b5f0ed791b18c8f8 |
|
BLAKE2b-256 | eb49284acb40cabad7eaa7bf53071481bc8524fdd1d12b6434bc6e5ed8d8399a |