Skip to main content

No project description provided

Project description

About tgext.celery

tgext.celery is a TurboGears2 extension that integrates celery into a turbogears application

I tested this extension just with the celery[mongodb]==3.1 bundle of celery and pymongo==3.5.1 using mongodb as both message broker and result backend

integrates both celery beat and celery worker

Installing

tgext.celery can be installed from pypi:

pip install tgext.celery

should just work for most of the users.

Plugging

To enable tgext.celery put inside your application config/app_cfg.py the following:

from tgext.pluggable import plug
plug(base_config, 'tgext.celery')

put in your *.ini file the options that you want to pass to your celery application prefixed by `celery.`:

#celery config
celery.CELERY_TASK_SERIALIZER = json
celery.CELERY_RESULT_SERIALIZER = json
celery.CELERY_ACCEPT_CONTENT = json
celery.CELERY_TIMEZONE = UTC
celery.BROKER_URL = mongodb://localhost:27017/dbname
celery.CELERY_RESULT_BACKEND = mongodb://localhost:27017/dbname
celery.CELERYD_POOL = celery.concurrency.threads.TaskPool
celery.CELERY_INCLUDE = myproject.lib.celery.tasks
celery.CELERYD_CONCURRENCY = 3
celery.WITHOUT_GOSSIP = true
celery.WITHOUT_HEARTBEAT = true
celery.WITHOUT_MINGLE = true

see http://docs.celeryproject.org/en/3.1/configuration.html#configuration for other options

disabling gossip, heartbeat and mingle can be dono only from the celery worker cli, we added them to the configuration. We reccomend to disable them if you aren’t using monitoring tools, because of a bug: https://github.com/celery/celery/issues/1047 that will fill your db with GBs of messages.

you can pass other options (that override the other in the .ini file) when plugging this extension this is convenient because the options in the .ini file aren’t evaluated:

plug(
    base_config,
    'tgext.celery',
    celery_config={
        'CELERY_MONGODB_BACKEND_SETTINGS': {
            'database': 'dbname',
        },
        'CELERYBEAT_SCHEDULE': {
            'delete-unassociated-images-every-12-hours': {
                'task': 'delete_unassociated_images',
                'schedule': timedelta(hours=12),
            },
        },
    },
)

Writing tasks

Remember to set the CELERY_INCLUDE option here’s an example with a task:

from __future__ import absolute_import

from myproject import model

from tgext.celery.celery import celery_app

import logging

logger = logging.getLogger(__name__)

celery_app.config_from_object(config['celery_configuration_object'])


@celery_app.task(name='delete_unassociated_images')
def delete_unassociated_images():
    logger.info('started')
    model.Image.query.remove({'post_id': None})
    logger.info('finished')

Executing

tgext.celery sets two gearbox commands, you can run celery with:

gearbox celeryworker -c production.ini --logfile=/var/log/circus/myproject_celery_worker_tasks.log

and:

gearbox celerybeat -c production.ini

in a production environment you should put these commands in a circus watcher or supervisord

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tgext.celery-0.1.2.tar.gz (5.7 kB view details)

Uploaded Source

Built Distribution

tgext.celery-0.1.2-py3-none-any.whl (6.8 kB view details)

Uploaded Python 3

File details

Details for the file tgext.celery-0.1.2.tar.gz.

File metadata

  • Download URL: tgext.celery-0.1.2.tar.gz
  • Upload date:
  • Size: 5.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2

File hashes

Hashes for tgext.celery-0.1.2.tar.gz
Algorithm Hash digest
SHA256 1607547bd7f431646cb1e6f76e4af8f3d4094e61847e57bb51bc2220c942660a
MD5 38a44b6bf0c73336eca51527a7b00366
BLAKE2b-256 91f5ac388681f39f95dab412fedb607bd1a1083a5aa6b802eae7d190639abdaf

See more details on using hashes here.

File details

Details for the file tgext.celery-0.1.2-py3-none-any.whl.

File metadata

  • Download URL: tgext.celery-0.1.2-py3-none-any.whl
  • Upload date:
  • Size: 6.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.8.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2

File hashes

Hashes for tgext.celery-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 2119ee3a68cab82199d51efbb2c9236bee180d59f6648041f073fc966c10b31c
MD5 00229410b473e3ac3a32cdf23380fa46
BLAKE2b-256 f0533888f6918f5a04daa4df14eb41d977efa3bf1ca9f968be940b0f54114f5f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page