Messaging module of Project Thoth
Project description
This provides a library called thoth-messaging used in project Thoth. It is a basic module to encapsule all messaging (here it is Kafka via Faust) primitives.
Message Factory
For the purposes of Thoth we have a few of our custom messages defined within the messaging module. However, we wanted to allow individuals to be able to use our module without having to add their own messages first. For testing and development purposes you can use message_factory(…) which allows you to create arbitrary messages using thoth-messaging using the following syntax:
message_factory(t_name=<str>, message_contents=<Tuple[str, str]>, [num_partitions=<int>], [replication_factor=<int>])bracketed arguments are optional.
Development and Testing
For development and testing it is very useful to have a local instance of Kafka running on your machine
We provide a docker-compose file to get you up and running quickly with a basic Kafka server; this file is based on Single Zookeeper/Multiple Kafka<https://github.com/simplesteph/kafka-stack-docker-compose#single-zookeeper–multiple-kafka.
In order to start Zookeeper as well as the Kafka Servers simply run $ podman-compose up or $ docker-compose up, choose the appropriate option based on the system which you are using.
Once you have Kafka up and running you should be ready to begin coding your own messaging producers and consumers. The interface between Kafka and Python is handled by a library called Faust<https://faust.readthedocs.io/en/latest/>_. Faust’s documentation will be extremely helpful to you when you are developing your own applications. If you would like examples of producers and consumers from Team Thoth, look at the following two repositories, investigator <https://github.com/thoth-station/investigator>_ and package-update<https://github.com/thoth-station/package-update-job>_.
You may find it useful to use console producers and consumers while testing your, to create one simply attach a bash shell to one of your Kafka Servers by running: $ podman exec -it messaging_kafka1_1 bash, your container names should be the same as given here, if not, run $ podman ps and choose the correct container. These containers have all Kafka binaries in appropriate places so you can simply run $ kafka-console-consumer, $ kafka-console-producer, or any other kafka command that you may find useful.
example: ` $ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning `
Note Data is not persistent. Once pods are deleted so is the data associated with them.
Note Faust producers and consumers can’t be run by calling $ python producer.py, instead they are Faust specific applications, in order to run them you need to call faust -A <filename> <function> [options]
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file thoth-messaging-0.6.4.tar.gz.
File metadata
- Download URL: thoth-messaging-0.6.4.tar.gz
- Upload date:
- Size: 6.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/39.2.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 98649c0cbfc8be17c006c6e2f21b34cc9dcf6b9e46cbae336815a08db57af50b |
|
| MD5 | cd6f1e40adda0d480b8ce8ed717ee4f2 |
|
| BLAKE2b-256 | bd5f99037e705b1c54cfc9e5d7b35cef37086a1b498dd2540739271dc6d7fb8d |
File details
Details for the file thoth_messaging-0.6.4-py3-none-any.whl.
File metadata
- Download URL: thoth_messaging-0.6.4-py3-none-any.whl
- Upload date:
- Size: 15.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/39.2.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | c5149106122dd341cdead67b64c2e749e08711bf4dd4f93d1b4ad3962fc02e9b |
|
| MD5 | ae1630332cfce2ab4cf4ce09b635fa43 |
|
| BLAKE2b-256 | e5aedc632e778a1d3c1abf647af32d44611bbcb593a6aec3e68b59d9d6f20fd3 |
