Kafka integration for bluesky.
Project description
Kafka integration for bluesky.
Free software: 3-clause BSD license
Features
BlueskyConsumer
MongoConsumer
Publisher
RemoteDispatcher
Release History
v0.3.0 (2020-09-03)
added BlueskyConsumer
added MongoConsumer
added supervisor configuration file for mongo_normalized_consumer.py
rewrote RemoteDispatcher to use BlueskyConsumer
changed default serialization method to MessagePack
v0.4.0 (2021-04-09)
added continue_polling parameter to BlueskyConsumer.start()
added utils.py
added BlueskyKafkaException
split tests into multiple files
create and clean up topics for each test
simplified produce/consume tests to run in one process
configured live logging in pytest.ini
switched from travis to github for continuous integration
Test
Install docker and docker-compose.
Start a Kafka server:
$ cd bluesky-kafka/scripts $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
$ cd bluesky-kafka $ pytest
Optionally increase logging output to the console by specifying a logging level:
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
Create a conda environment:
$ conda create -n consumers python=3.8 $ conda activate consumers
Install packages:
$ pip install bluesky-kafka supervisor
Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3 $ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file bluesky-kafka-0.4.0.tar.gz.
File metadata
- Download URL: bluesky-kafka-0.4.0.tar.gz
- Upload date:
- Size: 35.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 55d8d2e231b851471d031a1473a0dad2f52f5cff0f6c7a7387809d9f91c911f5 |
|
| MD5 | 7954e3b94858d306f533f616398cef03 |
|
| BLAKE2b-256 | 95bc1a109b0bd2986b91eff9eadd41bc4eaf832db1e1d7013772dfb69aaed832 |
Provenance
File details
Details for the file bluesky_kafka-0.4.0-py3-none-any.whl.
File metadata
- Download URL: bluesky_kafka-0.4.0-py3-none-any.whl
- Upload date:
- Size: 21.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.7.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2ea4bf50ee4b206520c6209dde0f7d9fe52a4ff3449c414ec776df0a1627185a |
|
| MD5 | 00b0408958c20893979425d0599497a7 |
|
| BLAKE2b-256 | 6e246883cbaf4046c629783e4afd693b68e3c47dec3b6c557771377d4b72a16e |
