Skip to main content

Robust, high-volume, message based communication made easy

Project description

kiwiPy

kiwiPy Coveralls Github Actions Latest Version https://img.shields.io/pypi/pyversions/kiwipy.svg https://img.shields.io/pypi/l/kiwipy.svg https://joss.theoj.org/papers/10.21105/joss.02351/status.svg

kiwiPy is a library that makes remote messaging using RabbitMQ (and possibly other message brokers) EASY. It was designed to support high-throughput workflows in big-data and computational science settings and is currently used by AiiDA for computational materials research around the world. That said, kiwiPy is entirely general and can be used anywhere where high-throughput and robust messaging are needed.

Here’s what you get:

  • RPC

  • Broadcast (with filters)

  • Task queue messages

Let’s dive in, with some examples taken from the rmq tutorial. To see more detail head over to the documentation.

RPC

The client:

import kiwipy

with kiwipy.connect('amqp://localhost') as comm:
    # Send an RPC message
    print(" [x] Requesting fib(30)")
    response = comm.rpc_send('fib', 30).result()
    print((" [.] Got %r" % response))

(rmq_rpc_client.py source)

The server:

import threading
import kiwipy

def fib(comm, num):
    if num == 0:
        return 0
    if num == 1:
        return 1

    return fib(comm, num - 1) + fib(comm, num - 2)

with kiwipy.connect('amqp://127.0.0.1') as comm:
    # Register an RPC subscriber with the name 'fib'
    comm.add_rpc_subscriber(fib, 'fib')
    # Now wait indefinitely for fibonacci calls
    threading.Event().wait()

(rmq_rpc_server.py source)

Worker

Create a new task:

import sys
import kiwipy

message = ' '.join(sys.argv[1:]) or "Hello World!"

with rmq.connect('amqp://localhost') as comm:
    comm.task_send(message)

(rmq_new_task.py source)

And the worker:

import time
import threading
import kiwipy

print(' [*] Waiting for messages. To exit press CTRL+C')


def callback(_comm, task):
    print((" [x] Received %r" % task))
    time.sleep(task.count(b'.'))
    print(" [x] Done")


try:
    with kiwipy.connect('amqp://localhost') as comm:
        comm.add_task_subscriber(callback)
        threading.Event().wait()
except KeyboardInterrupt:
    pass

(rmq_worker.py source)

Citing

If you use kiwiPy directly or indirectly (e.g. by using AiiDA) then please cite:

Uhrin, M., & Huber, S. P. (2020). kiwiPy : Robust , high-volume , messaging for big-data and computational science workflows, 5, 4–6. http://doi.org/10.21105/joss.02351

This helps us to keep making community software.

Versioning

This software follows Semantic Versioning

Contributing

Want a new feature? Found a bug? Want to contribute more documentation or a translation perhaps?

Help is always welcome, get started with the contributing guide.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kiwipy-0.7.1.tar.gz (44.4 kB view details)

Uploaded Source

Built Distribution

kiwipy-0.7.1-py3-none-any.whl (27.8 kB view details)

Uploaded Python 3

File details

Details for the file kiwipy-0.7.1.tar.gz.

File metadata

  • Download URL: kiwipy-0.7.1.tar.gz
  • Upload date:
  • Size: 44.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.8

File hashes

Hashes for kiwipy-0.7.1.tar.gz
Algorithm Hash digest
SHA256 5651b2317053ec47271a90e39c8d609f09d58c5828cf44a69db09d2b89878dbd
MD5 ed714ef7db8061a6d98c601a30ea92d0
BLAKE2b-256 c03cdb92fe989584305d0ce504bba3f32d3004782d23a72c3113988c0a37565a

See more details on using hashes here.

Provenance

File details

Details for the file kiwipy-0.7.1-py3-none-any.whl.

File metadata

  • Download URL: kiwipy-0.7.1-py3-none-any.whl
  • Upload date:
  • Size: 27.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.7.8

File hashes

Hashes for kiwipy-0.7.1-py3-none-any.whl
Algorithm Hash digest
SHA256 8f58a7d6ed9871842187afcc877a2fdef874ba0f33489fc824b3bb624ef71545
MD5 3f6b67f81c9db40c8eafaf2270ed967b
BLAKE2b-256 e94ea4cc906ed9e5748936db1c0bb8faf1dfa0550cf96f53844f4fab8aa9f372

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page