Skip to main content

All-in-memory job queue with RESTful interface.

Project description

Why restq?

We wanted to have a simple platform independent solution for managing the coordination and distribution of batched execution across our analysis platforms. restq solved our wants into a system that could:

  • segregate execution based on a category or type (realm),

  • manage priorities of job execution (ordered queues),

  • enqueue, check-out, and expiry time based (almost FIFO) dequeuing of jobs from a queue.

  • status of jobs remaining against arbitrary tag indices.

  • zero configuration for the concepts talked about above.

What’s in restq:

  • An implementation of the execution management system described above.

  • A RESTful web API that exposes complete control over the execution management system.

  • A Python client that seamlessly interfaces the RESTful web API.

For additional tips / tricks with this restq feel free to post a question at the github restq/issues page.

Project hosting provided by github.com.

[mjdorma+restq@gmail.com] and [sptonkin@outlook.com]

Install and run

Simply run the following:

> python setup.py install

or PyPi:

> pip install restq

Getting started

Coding with restq

A simple example on how

> restq web &
> ipython

In [1]: from restq import Realms

In [2]: realms = Realms()

In [3]: realms.test.
realms.test.add
realms.test.bulk_add
realms.test.bulk_flush
realms.test.get_job
realms.test.get_tag_status
realms.test.get_tagged_jobs
realms.test.name
realms.test.pull
realms.test.remove_job
realms.test.remove_tagged_jobs
realms.test.request
realms.test.requester
realms.test.set_default_lease_time
realms.test.set_queue_lease_time
realms.test.status

In [3]: realms.test.add('job 1', 0, 'do the dishes', tags=['house work'])

In [4]: realms.test.add('job 2', 0, 'cut the grass', tags=['house work'])

In [5]: realms.test.add('job 3', 1, 'fix bugs in restq', tags=['devel'])

In [6]: realms.test.add('job 4', 3, 'document restq', tags=['devel'])

In [7]: realms.test.add('job 5', 0, 'go for walk', tags=['sport'])

In [8]: realms.test.status
Out[8]:
{u'queues': {u'0': 4, u'1': 1, u'2': 1, u'3': 1},
 u'total_jobs': 7,
 u'total_tags': 3}

In [9]: jobs = realms.test.pull(count=7)

In [10]: jobs
Out[10]:
{u'job 1': [0, u'do the dishes'],
 u'job 2': [0, u'cut the grass'],
 u'job 3': [1, u'fix bugs in restq'],
 u'job 4': [3, u'document restq'],
 u'job 5': [0, u'go for walk'],
 u'job 6': [0, u'go for walk with dog'],
 u'job 7': [2, u'go for bike ride']}

In [11]: realms.test.get_tag_status('house work')
Out[11]: {u'count': 2}

In [12]: realms.test.get_tagged_jobs('devel')
Out[12]:
{u'job 3': {u'data': u'fix bugs in restq',
  u'queues': [[1, 82.17003393173218]],
  u'tags': [u'devel']},
 u'job 4': {u'data': u'document restq',
  u'queues': [[3, 82.16989994049072]],
  u'tags': [u'devel']}}

Issues

Source code for restq is hosted on GitHub. Please file bug reports with GitHub’s issues system.

Change log

version 0.0.4 (09/06/2013)

  • config and cli implementation

version 0.0.3 (06/06/2013)

  • bulk post & stable error handling

version 0.0.1 (10/04/2013)

  • pre life

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

restq-0.0.4.tar.gz (18.7 kB view details)

Uploaded Source

File details

Details for the file restq-0.0.4.tar.gz.

File metadata

  • Download URL: restq-0.0.4.tar.gz
  • Upload date:
  • Size: 18.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for restq-0.0.4.tar.gz
Algorithm Hash digest
SHA256 f4a574892745d08d6ac261d89628521c3e9e30f09c2efebc88c64e914af7fbce
MD5 61c9bdd717e338ebf7f58dcd80cf589b
BLAKE2b-256 16869f0d734ce11f24f972041ae3def9d244cc5e35f59a485d287a4d2ddf149a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page