A buildout recipe to install and configure Celery
Project description
Introduction
This recipe installs Celery and creates a celeryconfig.py module with the specified configuration options. It helps managing multiple configurations (e.g. development and production) using buildout.
You can use it in a part like this:
[celery] recipe = collective.recipe.celery broker-transport = sqlakombu.transport.Transport broker-host = sqlite:///celery_broker.db result-backend = database result-dburi = sqlite:///celery_results.db imports = myapp.tasks eggs = kombu-sqlalchemy myapp
Supported options
General options
- eggs
A list of additional eggs you want to make available to Celery. Use this to add additional dependencies such as kombu-sqlalchemy or the module(s) containing your task definitions.
- scripts
Controls which scripts are generated. If the option is omitted, then all scripts will be generated. If no value is given, then script generation is disabled.
- config-path
The location of the directory containing the celeryconfig.py module. By default the config module is created in the part directory. You can use this in other parts to include the config module:
[celery] recipe = collective.recipe.celery [myapp] recipe = zc.recipe.egg eggs = myapp extra-paths = ${celery:config-path}
Celery options
The following configuration options are supported. See Celery documentation for more details.
- broker-transport
The Kombu transport to use. You can use a custom transport class name, or select one of the built-in transports: amqplib, pika, redis, beanstalk, sqlalchemy, django, mongodb, couchdb.
- broker-host
The hostname of the broker.
- broker-port
The port number of the broker.
- broker-user
The username to connect as.
- broker-password
The password to connect with.
- broker-vhost
The virtual host.
- result-backend
The backend used to store task results. Can be one of database, cache, mongodb, redis, tyrant or amqp.
- result-dburi
Connection string for the database result backend.
- imports
A list of modules to import when the celery daemon starts. Specify one module per line.
- celeryd-log-file
The filename where the celery daemon logs messages to.
- celeryd-log-level
The log level, can be one of DEBUG, INFO, WARNING, ERROR or CRITICAL.
- celeryd-concurrency
The number of concurrent worker processes/threads/green threads, executing tasks.
- additional-config
Any additional configuration directives can be added using the additional-config option.
Example:
additional-config = CELERY_TASK_PUBLISH_RETRY=True CELERY_TASK_PUBLISH_RETRY_POLICY={"max_retries": 2, "interval_start": 10, "interval_step": 0, "interval_max": 10}
Changelog
1.0 (2011-08-15)
Initial release. [buchi]
Example usage
We’ll start by creating a buildout that uses the recipe:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = celery ... index = %(server)s/index ... find-links = %(server)s ... ... [celery] ... recipe = collective.recipe.celery ... broker-transport = sqlakombu.transport.Transport ... broker-host = sqlite:///celery_broker.db ... result-backend = database ... result-dburi = sqlite:///celery_results.db ... imports = myapp.tasks ... """% dict(server=link_server))
Running the buildout gives us:
>>> print system(buildout) Installing celery. celery: Creating directory /sample-buildout/parts/celery. celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py. Getting distribution for 'celery'. Got celery 2.3.1. Generated script '/sample-buildout/bin/celeryctl'. Generated script '/sample-buildout/bin/celeryd'. <BLANKLINE>
Check that we have the celery scripts:
>>> ls(sample_buildout, 'bin') - buildout - celeryctl - celeryd
Check that we got a celery config file:
>>> ls(sample_buildout, 'parts', 'celery') - celeryconfig.py
If we run the celeryd script, it prints out the config data:
>>> print(system(join(sample_buildout, 'bin', 'celeryd'))) BROKER_HOST='sqlite:///celery_broker.db' BROKER_TRANSPORT='sqlakombu.transport.Transport' CELERY_IMPORTS=('myapp.tasks',) CELERY_RESULT_BACKEND='database' CELERY_RESULT_DBURI='sqlite:///celery_results.db' <BLANKLINE>
We can include additional eggs using the eggs option:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = celery ... index = %(server)s/index ... find-links = %(server)s ... ... [celery] ... recipe = collective.recipe.celery ... eggs = ... other ... """% dict(server=link_server)) >>> print system(buildout), Uninstalling celery. Installing celery. celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py. Getting distribution for 'other'. Got other 1.0. Generated script '/sample-buildout/bin/celeryctl'. Generated script '/sample-buildout/bin/celeryd'.
We can control which scripts are generated using the scripts option. If no value is given, then script generation is disabled:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = celery ... index = %(server)s/index ... find-links = %(server)s ... ... [celery] ... recipe = collective.recipe.celery ... scripts = ... """% dict(server=link_server)) >>> print system(buildout), Uninstalling celery. Installing celery. celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py. >>> ls(sample_buildout, 'bin') - buildout
Let’s create the celeryd script only:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = celery ... index = %(server)s/index ... find-links = %(server)s ... ... [celery] ... recipe = collective.recipe.celery ... scripts = ... celeryd ... """% dict(server=link_server)) >>> print system(buildout), Uninstalling celery. Installing celery. celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py. Generated script '/sample-buildout/bin/celeryd'. >>> ls(sample_buildout, 'bin') - buildout - celeryd
The supported configuration directives may be of various types including strings, integers and tuples:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = celery ... index = %(server)s/index ... find-links = %(server)s ... ... [celery] ... recipe = collective.recipe.celery ... broker-port = 8080 ... broker-user = guest ... imports = ... myapp.tasks ... other.tasks ... """% dict(server=link_server)) >>> print system(buildout), Uninstalling celery. Installing celery. celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py. Generated script '/sample-buildout/bin/celeryctl'. Generated script '/sample-buildout/bin/celeryd'.
Let’s verify the generated config data:
>>> cat(sample_buildout, 'parts', 'celery', 'celeryconfig.py') BROKER_PORT = 8080 BROKER_USER = 'guest' CELERY_IMPORTS = ('myapp.tasks', 'other.tasks') <BLANKLINE>
The recipe supports a limited set of celery’s configuration directives. Any additional directives can be added using the additional-config option:
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = celery ... index = %(server)s/index ... find-links = %(server)s ... ... [celery] ... recipe = collective.recipe.celery ... additional-config = ... CELERY_TASK_PUBLISH_RETRY = True ... CELERY_TASK_PUBLISH_RETRY_POLICY = {"max_retries": 2, ... "interval_start": 10, ... "interval_step": 0, ... "interval_max": 10} ... """% dict(server=link_server)) >>> print system(buildout), Uninstalling celery. Installing celery. celery: Generated config file /sample-buildout/parts/celery/celeryconfig.py. Generated script '/sample-buildout/bin/celeryctl'. Generated script '/sample-buildout/bin/celeryd'.
Let’s verify the generated config data:
>>> cat(sample_buildout, 'parts', 'celery', 'celeryconfig.py') CELERY_TASK_PUBLISH_RETRY = True CELERY_TASK_PUBLISH_RETRY_POLICY = {"max_retries": 2, "interval_start": 10, "interval_step": 0, "interval_max": 10} <BLANKLINE>
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for collective.recipe.celery-1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbc42e203b3f735520a6f5c2ce7a4e6a3d64ea41917c39ef6b0195ba6c08e0b8 |
|
MD5 | 36aa5e1ea0dc7e730fc906911c3284b2 |
|
BLAKE2b-256 | 17a6a64c016b63f32e7191246d9b8537634221545945331b75b50edf79c4fc81 |