Skip to main content

Messaging module of Project Thoth

Project description

GitHub tag (latest by date) PyPI - Module Version PyPI - License PyPI - Downloads Quay - Build

This provides a library called thoth-messaging used in project Thoth. It is a basic module to encapsule all messaging (here it is Kafka via Faust) primitives.

Type Hinting With MyPy

This module uses pydantic for type hinting and enforcing a regular schema in messaging. If you are using mypy to check your code please add the following to your mypy configuration file:

[mypy]
plugins = pydantic.mypy

If you are creating an instance of a pydantic model in your own module you should directly use the MessageContents class within the associate message file. Using foo_bar_message.model will only type hint for BaseMessageContents.

Development and Testing

For development and testing it is very useful to have a local instance of Kafka running on your machine

We provide a docker-compose file to get you up and running quickly with a basic Kafka server; this file is based on Single Zookeeper/Multiple Kafka.

In order to start Zookeeper as well as the Kafka Servers simply run $ podman-compose up or $ docker-compose up, choose the appropriate option based on the system which you are using.

Once you have Kafka up and running you should be ready to begin coding your own messaging producers and consumers. The interface between Kafka and Python is handled by a library called Confluent Kafka. Faust’s documentation will be extremely helpful to you when you are developing your own applications. If you would like examples of producers and consumers from Team Thoth, look at the following two repositories, investigator and package-update.

You may find it useful to use console producers and consumers while testing your, to create one simply attach a bash shell to one of your Kafka Servers by running: $ podman exec -it messaging_kafka1_1 bash, your container names should be the same as given here, if not, run $ podman ps and choose the correct container. These containers have all Kafka binaries in appropriate places so you can simply run $ kafka-console-consumer, $ kafka-console-producer, or any other kafka command that you may find useful.

example:

kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

You can test sending a message via the CLI using a file as -

example:

pipenv shell
python cli.py --message-file messages_to_be_sent.json

Note Data is not persistent. Once pods are deleted so is the data associated with them.

Pitfalls

Some schemas in this library are defined as Dict[str, Any]. This usually does not accurately reflect the actual schema required. These schemas can be purposefully vague because they are defined elsewhere and importing them from the requisite libraries couples thoth-messaging version too closely to other components in the best case, and introduces circular dependencies in the worst case. In short, thoth-messaging is naive and can only enforce schemas which it has defined.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

thoth-messaging-0.16.2.tar.gz (24.7 kB view details)

Uploaded Source

Built Distribution

thoth_messaging-0.16.2-py3-none-any.whl (41.2 kB view details)

Uploaded Python 3

File details

Details for the file thoth-messaging-0.16.2.tar.gz.

File metadata

  • Download URL: thoth-messaging-0.16.2.tar.gz
  • Upload date:
  • Size: 24.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.8

File hashes

Hashes for thoth-messaging-0.16.2.tar.gz
Algorithm Hash digest
SHA256 6a814ed282dd914c22ca54069c6869ede2405ecccfc02f0eb349661a0f3cc279
MD5 14214bb7b1c6b106441d964631b39bff
BLAKE2b-256 a50b163054d34aef4c43af4b25f5fd81fa3d0f8726803e7c7c9fc2f7665b59da

See more details on using hashes here.

File details

Details for the file thoth_messaging-0.16.2-py3-none-any.whl.

File metadata

  • Download URL: thoth_messaging-0.16.2-py3-none-any.whl
  • Upload date:
  • Size: 41.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.8

File hashes

Hashes for thoth_messaging-0.16.2-py3-none-any.whl
Algorithm Hash digest
SHA256 113a7874c1e2f50bd1dfec0063d24a4f0a3ce0fadde612db26bb5d358e381cd8
MD5 6a0fc012e7221e1cd6504de7b6fb6c77
BLAKE2b-256 22211f6b6fa571b569a050f95152ced472f332b2b80d1fac85148e628eb196c4

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page