Kafka integration for bluesky.
Project description
Kafka integration for bluesky.
Free software: 3-clause BSD license
Features
BlueskyConsumer
MongoConsumer
Publisher
RemoteDispatcher
Test
Install docker and docker-compose.
Start a Kafka server:
$ cd bluesky-kafka/scripts $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
$ cd bluesky-kafka $ pytest
Optionally increase logging output to the console by specifying a logging level:
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
Create a conda environment:
$ conda create -n consumers python=3.8 $ conda activate consumers
Install packages:
$ pip install bluesky-kafka supervisor
Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3 $ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file bluesky-kafka-0.6.0.tar.gz
.
File metadata
- Download URL: bluesky-kafka-0.6.0.tar.gz
- Upload date:
- Size: 39.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | beab97dead2597eb475124d024b2c9a81fc7ba035eb87d7c08d8934c707cbe4e |
|
MD5 | 2ab6b079b60e1970b7696f67acaafb29 |
|
BLAKE2b-256 | a9887ec9bfe4c8a27ecb6d23c799fc784baae697efe4c727dbe7d0abc4435fc9 |
Provenance
File details
Details for the file bluesky_kafka-0.6.0-py3-none-any.whl
.
File metadata
- Download URL: bluesky_kafka-0.6.0-py3-none-any.whl
- Upload date:
- Size: 27.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c15813c108a0121d3b0155d01ec07d9578a67e6adf4b4e8ac45d599ae05cd33 |
|
MD5 | 7ce5224784a86fb00b9eb82634470d25 |
|
BLAKE2b-256 | 9ac107866132ecf8ab926b79ab699cf9abdce49dc3c4f22e25588fb6cd31fa46 |