django-beanstalk-worker

An app for handling deferred and periodic tasks on beanstalk worker machines

• Overview
• Installation and Setup
  • 1: Installed Apps
  • 2: Environment Variables
  • 3: Django Settings
  • 4: URLs's
  • 5: Beanstalk Worker Configuration
• Use
  • Declare a task function
  • Call the task from anywhere in your code
  • Call the task from CRON
  • Call the task from the command line
• Development, Testing and the FakeTaskServer
  • settings.DEBUG
  • Test support
  • settings.BEANSTALK_WORKER

Overview

The core of this app is the task decorator.

When a decorated task function is called instead of running immediately an SQS message is queued. When unqueued by a worker that message will cause the original function to be called.

The arguments may be any combination of JSON serializable types, datetimes and decimals, including keyword arguments. Any other argument types will cause an error.

Queuing of SQS messages happens in transaction on_commit so if you exception out of a transaction any tasks called in that transaction will not be run.

Task functions can also be invoked by cron or a management command as needed.

Installation and Setup

1: Installed Apps

Add beanstalk_worker to installed_apps.

2: Environment Variables

In the Software section of the Beanstalk configuration of your worker environment add the environment variable WORKER and set it to 1

3: Django Settings

Add the following settings to your Django settings:

BEANSTALK_WORKER = bool(os.environ.get("WORKER", False))
BEANSTALK_TASK_SERVICE = "beanstalk_worker.services.TaskService"
BEANSTALK_SQS_URL = <SQS queue URL>
BEANSTALK_SQS_REGION = <Amazon region>

You can find the SQS queue URL in the Worker section of the Beanstalk environment configuration.

For test and development you can omit the SQS settings and should set BEANSTALK_TASK_SERVICE = "beanstalk_worker.services.FakeTaskService"

4: URLs's

In your top level URL's add

if settings.BEANSTALK_WORKER:
    urlpatterns.append(url(r"^tasks/", include("beanstalk_worker.urls")))

This will add the URLS /tasks/task/ and /tasks/cron. You can move the base of these URL's if you do so other instructions on this page will need updating appropriately.

DO NOT include these URL's in a production web server, only the worker.

For test and development you won't have seperate web and worker machines so always include the URL's.

5: Beanstalk Worker Configuration

In the Worker section of the Beanstalk configuration of your worker environment set HTTP path to /tasks/task/ and MIME type to application/json

Use

Declare a task function

from beanstalk_worker import task

@task
def my_task(message="hi"):
    print(message)

Call the task from anywhere in your code

my_task("hello world")

This will queue an SQS message instructing a worker to run the actual function.

The arguments may be any combination of JSON serializable types, datetimes and decimals, including keyword arguments

Call the task from CRON

In cron.yaml add

- name: "my_project.my_app.tasks.my_task"
  url: "/tasks/cron/"
  schedule: "0 0 * * *"

my_project.my_app.tasks.my_task should be replaced with the fully qualified name of your task function.

Arguments are not currently supported for cron.

Call the task from the command line

./manage.py run_task my_project.my_app.tasks task "['hello world']"

my_project.my_app.tasks my_task should be replaced with the fully qualified name of your task function, also note the space between module name and function name.

Development, Testing and the FakeTaskServer

The FakeTaskServer will internally queue tasks but will not run them unless instructed to.

settings.DEBUG

When DEBUG = True is set in the Django settings an additional URL is exposed at /tasks/run_all/ if you poke this URL the FakeTaskService will run all queued tasks. This can be handy for local development.

Test support

In tests you can acquire the running task service instance with from beanstalk_worker import task_service. This class has two helpers clear whcih will discard all queued tasks and run_all which will immediately run all queued tasks.

settings.BEANSTALK_WORKER

While running a task the FakeTaskServer will patch settings.BEANSTALK_WORKER to True. This lets you assert settings.BEANSTALK_WORKER in code that should only ever be run on a worker.