Kafka integration for bluesky.
Project description
Kafka integration for bluesky.
Free software: 3-clause BSD license
Features
BlueskyConsumer
MongoConsumer
Publisher
RemoteDispatcher
Release History
v0.3.0 (2020-09-03)
added BlueskyConsumer
added MongoConsumer
added supervisor configuration file for mongo_normalized_consumer.py
rewrote RemoteDispatcher to use BlueskyConsumer
changed default serialization method to MessagePack
v0.4.0 (2021-04-09)
added continue_polling parameter to BlueskyConsumer.start()
added utils.py
added BlueskyKafkaException
split tests into multiple files
create and clean up topics for each test
simplified produce/consume tests to run in one process
configured live logging in pytest.ini
switched from travis to github for continuous integration
Test
Install docker and docker-compose.
Start a Kafka server:
$ cd bluesky-kafka/scripts $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
$ cd bluesky-kafka $ pytest
Optionally increase logging output to the console by specifying a logging level:
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
Create a conda environment:
$ conda create -n consumers python=3.8 $ conda activate consumers
Install packages:
$ pip install bluesky-kafka supervisor
Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3 $ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bluesky_kafka-0.4.0-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2ea4bf50ee4b206520c6209dde0f7d9fe52a4ff3449c414ec776df0a1627185a |
|
| MD5 | 00b0408958c20893979425d0599497a7 |
|
| BLAKE2b-256 | 6e246883cbaf4046c629783e4afd693b68e3c47dec3b6c557771377d4b72a16e |
