Callosum

An RPC Transport Library

It provides an asynchronous multi-channel order-preserving message and data streaming transport for upper RPC layers (e.g., Apache Thrift) by wrapping lower transport implementations (e.g., ZeroMQ).

It aims to follow the latest coding style and conventions in Python asyncio.

Corpus callosum is a bundle of neuron fibers that connects two cerebral hemispheres of a human brain.

Prerequisite

Python 3.11 or higher.

Features

• RPC
  • Native timeout and cancellation support
  • Explicit server-to-client error propagation including stringified tracebacks
  • Order preserving based on user-defined keys while keeping executions asynchronous
• Streaming
  • Broadcast & shared pipelines
• Optional client authentication and encrypted communication
  • Currently supported for only ZeroMQ with its CURVE library
• Optional message compression using snappy
• Replacible and combinable lower/upper layers (ZeroMQ/Redis + JSON/msgpack/Thrift)

Planned features

• Managed streaming (with acks)
• Tunneling to bundle other channels and generic network traffic in a single connection
• Bidirectional RPC
• Chunked transfer of large messages

Installation

To install the core:

$ pip install -U pip setuptools
$ pip install callosum

You may add extra dependencies like:

$ pip install 'callosum[zeromq,redis,thrift,snappy]'

Examples

Please check out the examples directory.

Development

Create a virtual environment or an isolated Python environment using your favorite tool.

Inside it, run editable installation as follows:

$ pip install -U pip setuptools
$ pip install -U -r requirements/dev.txt

Changes

1.0.3 (2024-03-04)

Fixes

• Relax yarl version requirements to avoid potential dependency conflicts in the downstream Backend.AI platform (#28)

1.0.2 (2024-02-29)

Fixes

• Stabilize ZMQ RPC auth and malformed packet handling (#27)

Miscellaneous

• Update dependencies including pyzmq ('>=23' → '>=25.1.1' to support Python 3.12) and development tools including ruff to replace the black formatter (#26)

v1.0.1 (2023-09-18)

Fixes

• Prevent leaking secret keys in the logs and allow infinite timeouts on connection pings (#24)

v1.0.0 (2023-09-14)

Breaking change

• Now it requires Python 3.11 to work!

Features

• Implement the full version of secure, encrypted RPC communication based on ZeroMQ's ZAP protocol using CURVE keypairs (#21)

v0.9.10 (2022-02-17)

Fixes

• Fix pyzmq attribute error on Ubuntu 20.04 at aarch64, which is built using older libzeromq without some socket monitoring event constants, by loading the constant declarations dynamically (#20)

v0.9.9 (2021-10-05)

Fixes

• Add explicit task_done() call to the output queue of rpc.Peer instances to avoid potential missing wakeup on joined coroutines (#16)

v0.9.8 (2021-10-05)

Features

• lower.zeromq: Add a transport option to attach monitors for logging underlying socket events (#17)

Fixes

• Improve zsock_opts handling when explicitly specified by the library users, as previously it caused argument errors in binders and connectors (#18)

v0.9.7 (2020-12-22)

Features

• Provide repr() of exceptions in RPC user/internal errors for better application-level error logging (#15)

v0.9.6 (2020-06-05)

Features

• upper.rpc: Support server-side cancellation and propagation to clients by adding the CANCELLED RPC message type (#14)

Deprecations

• Drop use of aiojobs in favor of native semaphores for limiting the task concurrency and less clutters on job scheduling semantics (#14)

Fixes

• Stability updates for the RPC layer: (#14)
  • Fix wrong message sequence calcuation with SEQ_BITS and clarify the roles of cleanup() and cancel() methods in the schedulers.
  • Now we use the exit-ordered scheduler by default.

v0.9.5 (2020-05-12)

Fixes

• lower.zeromq: Use destroy() for zmq context termination to improve stability and shutdown open socekts cleanly (#13)

v0.9.4 (2020-04-10)

Fixes

• Fix a race condition due to overlapping RPC message sequence IDs by separating server/client message sequence IDs (#12)

Miscellaneous

• Adopt towncrier for changelog management (#11)

v0.9.3 (2020-03-20)

• FIX: Mitigate bogus KeyError when cleaning up task futures in RPC servers that has caused event loop spinning.

v0.9.2 (2020-02-28)

• MAINTENANCE: Update dependencies and only specify the minimum versions since Callosum is a library.

v0.9.1 (2020-01-05)

• FIX: wrong typing of RPCMessage.body field

• IMPROVE: Add debug_rpc option to rpc.Peer for logging exceptions in RPC scheduler and user-defined handlers explicitly.

• Update dependencies and remove unused ones.

v0.9.0 (2019-12-06)

• First public release with a working RPC based on ZeroMQ DEALER/ROUTER sockets.

2018-05-02

• Started the project.