Kafka integration for bluesky.
Project description
Kafka integration for bluesky.
Free software: 3-clause BSD license
Features
BlueskyConsumer
MongoConsumer
Publisher
RemoteDispatcher
Release History
v0.5.0 (2021-08-09)
added timeout parameter to utils.get_cluster_metadata() and utils.list_topics()
put release history in reverse chronological order
v0.4.0 (2021-04-09)
added continue_polling parameter to BlueskyConsumer.start()
added utils.py
added BlueskyKafkaException
split tests into multiple files
create and clean up topics for each test
simplified produce/consume tests to run in one process
configured live logging in pytest.ini
switched from travis to github for continuous integration
v0.3.0 (2020-09-03)
added BlueskyConsumer
added MongoConsumer
added supervisor configuration file for mongo_normalized_consumer.py
rewrote RemoteDispatcher to use BlueskyConsumer
changed default serialization method to MessagePack
Test
Install docker and docker-compose.
Start a Kafka server:
$ cd bluesky-kafka/scripts $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
$ cd bluesky-kafka $ pytest
Optionally increase logging output to the console by specifying a logging level:
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
Create a conda environment:
$ conda create -n consumers python=3.8 $ conda activate consumers
Install packages:
$ pip install bluesky-kafka supervisor
Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3 $ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bluesky_kafka-0.5.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c3e6458e1d948203b32d89c0301e144d7dbb57c149564755ba630813ebc3d5e |
|
MD5 | ab5225cf519ae267cdc6b3c3f7dc40fe |
|
BLAKE2b-256 | aa2db27fc5f96aaa78cfbe5010fc0474e99d3a6b2fcc7536c3e6fd7bf8d94041 |