Distributed Python job queue with asyncio and redis
Project description
SAQ
SAQ (Simple Async Queue) is a simple and performant job queueing framework built on top of asyncio and redis.
It is inspired by ARQ but has several enhancements.
- Avoids polling by leveraging BLMOVE or RPOPLPUSH and NOTIFY
- SAQ has much lower latency than ARQ
- Web interface for monitoring queues and workers
- Heartbeat monitor for abandoned jobs
- More robust failure handling
- Storage of stack traces
- Sweeping stuck jobs
- Handling of cancelled jobs different from failed jobs (machine redeployments)
- Before and after job hooks
Install
# minimal install
pip install saq
# web + hiredis
pip install saq[web,hiredis]
Usage
usage: saq [-h] [--workers WORKERS] [--verbose] [--web] settings
Start Simple Async Queue Worker
positional arguments:
settings Namespaced variable containing worker settings eg: eg module_a.settings
options:
-h, --help show this help message and exit
--workers WORKERS Number of worker processes
--verbose, -v Logging level: 0: ERROR, 1: INFO, 2: DEBUG
--web Start web app
Development
python -m venv env
source env/bin/activate
pip install -e .[dev,web]
docker run -p 6379:6379 redis
./run_checks.sh
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
saq-0.1.0.tar.gz
(33.9 kB
view details)
Built Distribution
saq-0.1.0-py3-none-any.whl
(32.6 kB
view details)
File details
Details for the file saq-0.1.0.tar.gz.
File metadata
- Download URL: saq-0.1.0.tar.gz
- Upload date:
- Size: 33.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | bd904f481953c502828d820d42b39e89f4c9942b6247d882972e5ff2837fa70b |
|
| MD5 | 8699c483b7217af1e5f25489a8f5af08 |
|
| BLAKE2b-256 | 65b30267a531628ad209fdac288ed5e0be95c9e9c3bfe8d48cc7112629081497 |
File details
Details for the file saq-0.1.0-py3-none-any.whl.
File metadata
- Download URL: saq-0.1.0-py3-none-any.whl
- Upload date:
- Size: 32.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.10.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cd1d93dd16ddc68f0ca4adbfc7093e77b920a270e1b109f91f20163b59fc3614 |
|
| MD5 | 8b78f5d84a7ff95987904bae27d393dc |
|
| BLAKE2b-256 | 46e1e4177034f8256c1f9d06fb111403fe9b1fe89517c3633890a4c24697799a |
