Kafka integration for bluesky.
Project description
Kafka integration for bluesky.
Free software: 3-clause BSD license
Features
BlueskyConsumer
MongoConsumer
Publisher
RemoteDispatcher
Test
Install docker and docker-compose.
Start a Kafka server:
$ cd bluesky-kafka/scripts $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
$ cd bluesky-kafka $ pytest
Optionally increase logging output to the console by specifying a logging level:
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
Create a conda environment:
$ conda create -n consumers python=3.8 $ conda activate consumers
Install packages:
$ pip install bluesky-kafka supervisor
Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3 $ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bluesky_kafka-0.9.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 945f158cfd707372bdf6b24ff4e0c2e560e7de676d0d08905969563dcda169f6 |
|
MD5 | bb7196a46266ec22549021758dc89d20 |
|
BLAKE2b-256 | d7188267b7bc349daf1448569b6c5dd021c341672295db52a36911e0fb10fba1 |