Skip to main content

Lightweight & modular queue and cron system for Django

Project description

Django Lightweight Queue

DLQ is a lightweight & modular queue and cron system for Django. It powers millions of production jobs every day at Thread.

Installation

pip install django-lightweight-queue[redis]

Currently the only production-ready backends are redis-based, so the redis extra is essentially required. Additional non-redis backed production-ready backends are great candidates for community contributions.

Basic Usage

import time
from django_lightweight_queue import task

# Define a task
@task()
def long_running_task(first_arg, second_arg):
    time.sleep(first_arg * second_arg)

# Request that the task be executed at some point
long_running_task(4, second_arg=9)

See the docstring on the task decorator for more details.

Configuration

All automatically picked up configuration options begin with LIGHTWEIGHT_QUEUE_ and can be found in app_settings.py. They should be placed in the usual Django settings files, for example:

LIGHTWEIGHT_QUEUE_BACKEND = 'django_lightweight_queue.backends.redis.RedisBackend'

Special Configuration

If desired, specific configuration overrides can be placed in a standalone python file which passed on the command line. This is useful for applying customisations for specific servers.

For example, given a special.py containing:

LIGHTWEIGHT_QUEUE_REDIS_PORT = 12345

and then running:

$ python manage.py queue_runner --config=special.py

will result in the runner to use the settings from the specified configuration file in preference to settings from the Django environment. Any settings not present in the specified file are inherited from the global configuration.

Backends

There are four built-in backends:

Backend Type Description
Synchronous Development Executes the task inline, without any actual queuing.
Redis Production Executes tasks at-most-once using Redis for storage of the enqueued tasks.
Reliable Redis Production Executes tasks at-least-once using Redis for storage of the enqueued tasks (subject to Redis consistency). Does not guarantee the task completes.
Debug Web Debugging Instead of running jobs it prints the url to a view that can be used to run a task in a transaction which will be rolled back. This is useful for debugging and optimising tasks.

Running Workers

The queue runner is implemented as a Django management command:

$ python manage.py queue_runner

Workers can be distributed over multiple hosts by telling each runner that it is part of a pool:

$ python manage.py queue_runner --machine 2 --of 4

Alternatively a runner can be told explicitly which configuration to use:

$ python manage.py queue_runner --exact-configuration --config=special.py

When using --exact-configuration the number of workers is configured exactly, rather than being treated as the configuration for a pool. Additionally, exactly-configured runners will not run any cron workers.

Example

Given a Django configuration containing:

LIGHTWEIGHT_QUEUE_WORKERS = {
    'queue1': 3,
}

and a special.py containing:

LIGHTWEIGHT_QUEUE_WORKERS = {
    'queue1': 2,
}

Running any of:

$ python manage.py queue_runner --machine 1 --of 3 # or,
$ python manage.py queue_runner --machine 2 --of 3 # or,
$ python manage.py queue_runner --machine 3 --of 3

will result in one worker for queue1 on the current machine, while:

$ python manage.py queue_runner --exact-configuration --config=special.py

will result in two workers on the current machine.

Cron Tasks

DLQ supports the use of a cron-like specification of Django management commands to be run at certain times.

To specify that a management command should be run at a given time, place a cron.py file in the root folder of the Django app which defines the command and which contains a CONFIG variable:

CONFIG = (
    {
        'command': 'my_cron_command',
        # Day values 1-7 to match datetime.datetime.utcnow().isoweekday()
        'days': '*',
        'hours': '*',
        'minutes': '*',
        # Equivalent behaviour to the kwarg to `task` of the same name
        'sigkill_on_stop': True,
    },
)

Maintainers

This repository was created by Chris Lamb at Thread, and continues to be maintained by the Thread engineering team.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django-lightweight-queue-3.0.5.tar.gz (21.0 kB view details)

Uploaded Source

Built Distribution

django_lightweight_queue-3.0.5-py3-none-any.whl (27.8 kB view details)

Uploaded Python 3

File details

Details for the file django-lightweight-queue-3.0.5.tar.gz.

File metadata

  • Download URL: django-lightweight-queue-3.0.5.tar.gz
  • Upload date:
  • Size: 21.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.0.5 CPython/3.5.5 Linux/4.15.0-1052-aws

File hashes

Hashes for django-lightweight-queue-3.0.5.tar.gz
Algorithm Hash digest
SHA256 577e67e9480c99e06dd2ba4dd6aaa4d3a82e1bb8345b45a71166842eb406aec1
MD5 961dd44fa94d149d0ed08e84fb3aeecc
BLAKE2b-256 fdfcd396f56bdd52f89a41666b7f8fef060fae54fd4446d5e171b6dbcae05d48

See more details on using hashes here.

Provenance

File details

Details for the file django_lightweight_queue-3.0.5-py3-none-any.whl.

File metadata

File hashes

Hashes for django_lightweight_queue-3.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 5935fd7c1d5b8120b6c659d2e5e5967fcffdf3a8bcb63bc08316976e367779a5
MD5 942f3d67819f5c47b5f0ed791b18c8f8
BLAKE2b-256 eb49284acb40cabad7eaa7bf53071481bc8524fdd1d12b6434bc6e5ed8d8399a

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page