No project description provided
Project description
About tgext.celery
tgext.celery is a TurboGears2 extension that integrates celery into a turbogears application
I tested this extension just with the celery[mongodb]==3.1 bundle of celery and pymongo==3.5.1 using mongodb as both message broker and result backend
integrates both celery beat and celery worker
Installing
tgext.celery can be installed from pypi:
pip install tgext.celery
should just work for most of the users.
Plugging
To enable tgext.celery put inside your application config/app_cfg.py the following:
from tgext.pluggable import plug plug(base_config, 'tgext.celery')
put in your *.ini file the options that you want to pass to your celery application prefixed by `celery.`:
#celery config celery.CELERY_TASK_SERIALIZER = json celery.CELERY_RESULT_SERIALIZER = json celery.CELERY_ACCEPT_CONTENT = json celery.CELERY_TIMEZONE = UTC celery.BROKER_URL = mongodb://localhost:27017/dbname celery.CELERY_RESULT_BACKEND = mongodb://localhost:27017/dbname celery.CELERYD_POOL = celery.concurrency.threads.TaskPool celery.CELERY_INCLUDE = myproject.lib.celery.tasks celery.CELERYD_CONCURRENCY = 3 celery.WITHOUT_GOSSIP = true celery.WITHOUT_HEARTBEAT = true celery.WITHOUT_MINGLE = true
see http://docs.celeryproject.org/en/3.1/configuration.html#configuration for other options
disabling gossip, heartbeat and mingle can be dono only from the celery worker cli, we added them to the configuration. We reccomend to disable them if you aren’t using monitoring tools, because of a bug: https://github.com/celery/celery/issues/1047 that will fill your db with GBs of messages.
you can pass other options (that override the other in the .ini file) when plugging this extension this is convenient because the options in the .ini file aren’t evaluated:
plug( base_config, 'tgext.celery', celery_config={ 'CELERY_MONGODB_BACKEND_SETTINGS': { 'database': 'dbname', }, 'CELERYBEAT_SCHEDULE': { 'delete-unassociated-images-every-12-hours': { 'task': 'delete_unassociated_images', 'schedule': timedelta(hours=12), }, }, }, )
Writing tasks
Remember to set the CELERY_INCLUDE option here’s an example with a task:
from __future__ import absolute_import from myproject import model from tgext.celery.celery import celery_app import logging logger = logging.getLogger(__name__) celery_app.config_from_object(config['celery_configuration_object']) @celery_app.task(name='delete_unassociated_images') def delete_unassociated_images(): logger.info('started') model.Image.query.remove({'post_id': None}) logger.info('finished')
Executing
tgext.celery sets two gearbox commands, you can run celery with:
gearbox celeryworker -c production.ini --logfile=/var/log/circus/myproject_celery_worker_tasks.log
and:
gearbox celerybeat -c production.ini
in a production environment you should put these commands in a circus watcher or supervisord
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file tgext.celery-0.1.2.tar.gz
.
File metadata
- Download URL: tgext.celery-0.1.2.tar.gz
- Upload date:
- Size: 5.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1607547bd7f431646cb1e6f76e4af8f3d4094e61847e57bb51bc2220c942660a |
|
MD5 | 38a44b6bf0c73336eca51527a7b00366 |
|
BLAKE2b-256 | 91f5ac388681f39f95dab412fedb607bd1a1083a5aa6b802eae7d190639abdaf |
File details
Details for the file tgext.celery-0.1.2-py3-none-any.whl
.
File metadata
- Download URL: tgext.celery-0.1.2-py3-none-any.whl
- Upload date:
- Size: 6.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2119ee3a68cab82199d51efbb2c9236bee180d59f6648041f073fc966c10b31c |
|
MD5 | 00229410b473e3ac3a32cdf23380fa46 |
|
BLAKE2b-256 | f0533888f6918f5a04daa4df14eb41d977efa3bf1ca9f968be940b0f54114f5f |