Postgres Broker for Dramatiq Task Queue
Project description
dramatiq-pg − Postgres Broker for Dramatiq
Dramatiq is a simple task queue implementation for Python3. dramatiq-pg provides a Postgres-based implementation of a dramatiq broker.
Features
- Super simple deployment.
- Uses plain psycopg2. No ORM.
- Stores message payload and results as native JSONb.
- Stores all messages in a single table, in a dedicated schema.
- Uses LISTEN/NOTIFY to keep worker sync. No polling.
- Requeue of failed tasks.
- Delayed task.
- Reliable thanks to Postgres MVCC.
- Self-healing. Old messages are purge from time to time.
- Utility CLI for maintainance : flush, purge, stats, etc.
Note that dramatiq assumes tasks are idempotent. This broker makes the same assumptions for recovering after a crash.
Installation
- Install dramatiq-pg package from PyPI:
$ pip install dramatiq-pg psycopg2-binary
Ensure you have either psycopg2 or psycopg2-binary installed. - Apply dramatiq_pg/schema.sql file in your database:
$ psql -f dramatiq_pg/schema.sql
- Before importing actors, define global broker with a connection
pool:
import dramatiq import psycopg2.pool from dramatiq_pg import PostgresBroker dramatiq.set_broker(PostgresBroker(i)) @dramatiq.actor def myactor(): ...
Now declare/import actors and manage worker just like any dramatiq setup. An example script is available, tested on CI.
The CLI tool dramatiq-pg
allows you to requeue messages, purge old messages
and show stats on the queue. See --help
for details.
Dramatiq-pg documentation is hosted on GitLab and give you more details on deployment and operation of Postgres as a Dramatiq broker.
Support
If you encounter a bug or miss a feature, please open an issue on GitLab with as much information as possible.
dramatiq_pg is available under the PostgreSQL licence.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dramatiq_pg-0.6.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b481f0be14fd0f9ceff229484d390e177725dc47ba02fe95d540ec2f57bcebf7 |
|
MD5 | da76babcb893f7a002aa066b66771e75 |
|
BLAKE2b-256 | 23292ddef79a428bffd4cc7e79a6d1320af54326d89be2fd44864c81b26b0731 |