Skip to main content

Python client for managing Kafka connectors..

Project description

#####################
kafka-connect-manager
#####################

|Build| |Docker|

A Python client to configure and create connectors using the `Kafka Connect API <https://docs.confluent.io/current/connect/references/restapi.html>`_.

Overview
========

Kafka-connect-manager helps to configure and create Kafka connectors. It can select Kafka topics using a regular expression and exclude topics added to a exclusion list. The auto-update option dynamically updates the connector if topics are created or deleted in Kafka.

Kafka-connect-manager can validate the connector configuration and return the validation output in case of errors.

Kafka-connect-manager provides a command line interface (CLI) to interact with the `Kafka Connect API <https://docs.confluent.io/current/connect/references/restapi.html>`_. Using the CLI you can manage existing connectors or create a new one.

To deploy a supported connector, you can use the `kafka-connect-manager Helm chart <https://lsst-sqre.github.io/charts/>`_.

Supported connectors
--------------------

* `Lenses InfluxDB Sink <https://docs.lenses.io/connectors/sink/influx.html>`_
* `Amazon S3 Sink <https://docs.confluent.io/current/connect/kafka-connect-s3>`_

See `the docs <https://kafka-connect-manager.lsst.io>`_ for more information.


.. |Build| image:: https://github.com/lsst-sqre/kafka-connect-manager/workflows/CI/badge.svg
:alt: GitHub Actions
:scale: 100%
:target: https://github.com/lsst-sqre/kafka-connect-manager/actions

.. |Docker| image:: https://img.shields.io/docker/v/lsstsqre/kafkaconnect?sort=date
:alt: Docker Hub repository
:scale: 100%
:target: https://hub.docker.com/repository/docker/lsstsqre/kafkaconnect

##########
Change log
##########

0.8.2 (2021-01-25)
==================

* Update ``cp-kafka-connect`` image with new version of the InfluxDB Sink connector. This version bumps the ``influxdb-java`` dependency from version 2.9 to 2.21. In parrticular 2.16 introduced a fix to skip fields with ``NaN`` and ``Infinity`` values when writing to InfluxDB.
* Reorganize developer and user guides.
* Add documentation in the user guide on how to run the InfluxDB Sink connector locally.
* Update dependencies

0.8.1 (2020-10-18)
==================

* Fix bug preventing to read InfluxDB password from the environment
* Update ``cp-kafka-connect`` image with Confluent Platform 5.5.2
* Update dependencies

0.8.0 (2020-08-05)
==================

* Use data classes for the application and connector configuration.
* Plugin like organization, to support new connectors add a cli and a config file.
* Add support to the Amazon S3 Sink connector

0.7.2 (2020-03-31)
==================

* Add support to the InfluxDB Sink Connector.
* Add --timestamp option to select the timestamp field to use in the InfluxDB Sink connector.
* Fix Header Converter Class configuration setting.
* Fix ``tasks.max`` configuration setting name.
* Add connector ``name`` configuration setting to support multiple connectors of the same class.
* Handle empty list of topics properly.


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kafkaconnect-0.8.2.tar.gz (59.9 kB view details)

Uploaded Source

Built Distribution

kafkaconnect-0.8.2-py3-none-any.whl (17.5 kB view details)

Uploaded Python 3

File details

Details for the file kafkaconnect-0.8.2.tar.gz.

File metadata

  • Download URL: kafkaconnect-0.8.2.tar.gz
  • Upload date:
  • Size: 59.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/51.1.2 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.7

File hashes

Hashes for kafkaconnect-0.8.2.tar.gz
Algorithm Hash digest
SHA256 f5aa084f4700364cded23d80053310d88882dd6dc8f356ef1dbc7a8fee3d1129
MD5 f86af559425f3b9e5e2bf87917bff860
BLAKE2b-256 d39f76525263437c6e08d1bafef9957bfcd8557087089e9ed0a9439402703900

See more details on using hashes here.

File details

Details for the file kafkaconnect-0.8.2-py3-none-any.whl.

File metadata

  • Download URL: kafkaconnect-0.8.2-py3-none-any.whl
  • Upload date:
  • Size: 17.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/51.1.2 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.7

File hashes

Hashes for kafkaconnect-0.8.2-py3-none-any.whl
Algorithm Hash digest
SHA256 21dcb1c79d98e3895c7f996bf8808d60230c3964bad45f4fbade9f98f2cb9b6f
MD5 b7fdd6ed0bd6f7ef6729ba81af69013c
BLAKE2b-256 22343f7e5403b2d125d881e45350a8f012dbff0e9304d641f31dfdff3999412a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page