Distributed Python job queue with asyncio and redis
Project description
SAQ
SAQ (Simple Async Queue) is a simple and performant job queueing framework built on top of asyncio and redis.
It is inspired by ARQ but has several enhancements.
- Avoids polling by leveraging BLMOVE or RPOPLPUSH and NOTIFY
- SAQ has much lower latency than ARQ
- Web interface for monitoring queues and workers
- Heartbeat monitor for abandoned jobs
- More robust failure handling
- Storage of stack traces
- Sweeping stuck jobs
- Handling of cancelled jobs different from failed jobs (machine redeployments)
- Before and after job hooks
Install
# minimal install
pip install saq
# web + hiredis
pip install saq[web,hiredis]
Usage
usage: saq [-h] [--workers WORKERS] [--verbose] [--web] settings
Start Simple Async Queue Worker
positional arguments:
settings Namespaced variable containing worker settings eg: eg module_a.settings
options:
-h, --help show this help message and exit
--workers WORKERS Number of worker processes
--verbose, -v Logging level: 0: ERROR, 1: INFO, 2: DEBUG
--web Start web app
Development
python -m venv env
source env/bin/activate
pip install -e .[dev,web]
docker run -p 6379:6379 redis
./run_checks.sh
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
saq-0.1.0.tar.gz
(33.9 kB
view hashes)
Built Distribution
saq-0.1.0-py3-none-any.whl
(32.6 kB
view hashes)