Skip to main content

Kafka integration for bluesky.

Project description

https://github.com/bluesky/bluesky-kafka/actions/workflows/tests.yml/badge.svg https://img.shields.io/pypi/v/bluesky-kafka.svg

Kafka integration for bluesky.

  • Free software: 3-clause BSD license

Features

  • BlueskyConsumer

  • MongoConsumer

  • Publisher

  • RemoteDispatcher

Test

Install docker and docker-compose.

Start a Kafka server:

$ cd bluesky-kafka/scripts
$ sudo docker-compose -f bitnami-kafka-docker-compose.yml up

Run tests:

$ cd bluesky-kafka
$ pytest

Optionally increase logging output to the console by specifying a logging level:

$ pytest --log-cli-level=DEBUG

Run a Mongo Consumer Group

Create a conda environment:

$ conda create -n consumers python=3.8
$ conda activate consumers

Install packages:

$ pip install bluesky-kafka supervisor

Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.

$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3
$ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"

Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.

Start the consumer processes:

$ supervisord -c bluesky_kafka/supervisor/supervisord.conf

Monitor the consumer processes:

$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bluesky-kafka-0.6.0.tar.gz (39.9 kB view details)

Uploaded Source

Built Distribution

bluesky_kafka-0.6.0-py3-none-any.whl (27.6 kB view details)

Uploaded Python 3

File details

Details for the file bluesky-kafka-0.6.0.tar.gz.

File metadata

  • Download URL: bluesky-kafka-0.6.0.tar.gz
  • Upload date:
  • Size: 39.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12

File hashes

Hashes for bluesky-kafka-0.6.0.tar.gz
Algorithm Hash digest
SHA256 beab97dead2597eb475124d024b2c9a81fc7ba035eb87d7c08d8934c707cbe4e
MD5 2ab6b079b60e1970b7696f67acaafb29
BLAKE2b-256 a9887ec9bfe4c8a27ecb6d23c799fc784baae697efe4c727dbe7d0abc4435fc9

See more details on using hashes here.

Provenance

File details

Details for the file bluesky_kafka-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: bluesky_kafka-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 27.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12

File hashes

Hashes for bluesky_kafka-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 9c15813c108a0121d3b0155d01ec07d9578a67e6adf4b4e8ac45d599ae05cd33
MD5 7ce5224784a86fb00b9eb82634470d25
BLAKE2b-256 9ac107866132ecf8ab926b79ab699cf9abdce49dc3c4f22e25588fb6cd31fa46

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page