django-rq-mail 0.1.3.alpha

Architecture

django-rq-mail adds new elements to enjoy Sorted Sets from Redis.

For the purpose of django-rq-mail, it implements the concept of WaitingQueue which delays the processing of a job with a timestamp.

The default behavior of rq is to process jobs via BLPOP which blocks the connection when there are no elements to pop from any of the given queues. With this behavior there is no way to delays the processing of a job and when it’s failing rq pushs it in a failed queue. Of course, you can requeue this job later but there is no fallback mechanism.

In django-rq-mail you can define fallback steps (in seconds) to retry a job until it’s not failing. When a job has been tested on each steps we reintroduce the default behavior of rq on pushing it in the failed queue.

Each steps will create a waiting queue and when a job is failing we take the current timestamp with the delta to retry it in the future.

This mechanism is possible with ZADD which adds a serialized job in the queue with a score and ZREVRANGEBYSCORE to return all the elements in the sorted set with a score between max (current timestamp) and min.

As you may understood, we have dropped the default blocking behavior to replace it by a daemon which is running each seconds.

Installation

1. Either check out the package from GitHub or it pull from a release via PyPI

   pip install django-rq-mail

2. Add ‘rq_mail’ to your INSTALLED_APPS

   INSTALLED_APPS = (
       'rq_mail',
   )

to use the rq_mail command (via Django commandline) shipped by django-rq-mail.

This command is a minimal integration of rq into Django to launch the Dispatcher.

3. Define EMAIL_BACKEND

   EMAIL_BACKEND = 'rq_mail.backends.RqBackend'

4. Define RQ_MAIL_EMAIL_BACKEND the backend used to send your emails, for example

   RQ_MAIL_EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

Logging

RQ 0.3.3 uses standard Python’s logging, this means you can easily configure rqworker’s logging mechanism in django’s settings.py. For example:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'rq_console': {
            'format': '%(asctime)s %(message)s',
            'datefmt': '%H:%M:%S',
        },
    },
    'handlers': {
        'rq_console': {
            'level': 'DEBUG',
            'class': 'rq.utils.ColorizingStreamHandler',
            'formatter': 'rq_console',
            'exclude': ['%(asctime)s'],
        },
    },
    'loggers': {
        'rq.worker': {
            'handlers': ['rq_console'],
            'level': 'DEBUG'
        },
    }
}

Utilisation

Once you have installed it, you can run python manage.py rq_mail from your shell.

Configuration