Python client for managing Kafka connectors..
Project description
#####################
kafka-connect-manager
#####################
|Build| |Docker|
A Python client to configure and create connectors using the `Kafka Connect API <https://docs.confluent.io/current/connect/references/restapi.html>`_.
Overview
========
Kafka-connect-manager helps to configure and create Kafka connectors. It can select Kafka topics using a regular expression and exclude topics added to a exclusion list. The auto-update option dynamically updates the connector if topics are created or deleted in Kafka.
Kafka-connect-manager can validate the connector configuration and return the validation output in case of errors.
Kafka-connect-manager provides a command line interface (CLI) to interact with the `Kafka Connect API <https://docs.confluent.io/current/connect/references/restapi.html>`_. Using the CLI you can manage existing connectors or create a new one.
To deploy a supported connector, you can use the `kafka-connect-manager Helm chart <https://lsst-sqre.github.io/charts/>`_.
Supported connectors
--------------------
* `Lenses InfluxDB Sink <https://docs.lenses.io/connectors/sink/influx.html>`_
* `Amazon S3 Sink <https://docs.confluent.io/current/connect/kafka-connect-s3>`_
See `the docs <https://kafka-connect-manager.lsst.io>`_ for more information.
.. |Build| image:: https://github.com/lsst-sqre/kafka-connect-manager/workflows/CI/badge.svg
:alt: GitHub Actions
:scale: 100%
:target: https://github.com/lsst-sqre/kafka-connect-manager/actions
.. |Docker| image:: https://img.shields.io/docker/v/lsstsqre/kafkaconnect?sort=date
:alt: Docker Hub repository
:scale: 100%
:target: https://hub.docker.com/repository/docker/lsstsqre/kafkaconnect
##########
Change log
##########
0.8.2 (2021-01-25)
==================
* Update ``cp-kafka-connect`` image with new version of the InfluxDB Sink connector. This version bumps the ``influxdb-java`` dependency from version 2.9 to 2.21. In parrticular 2.16 introduced a fix to skip fields with ``NaN`` and ``Infinity`` values when writing to InfluxDB.
* Reorganize developer and user guides.
* Add documentation in the user guide on how to run the InfluxDB Sink connector locally.
* Update dependencies
0.8.1 (2020-10-18)
==================
* Fix bug preventing to read InfluxDB password from the environment
* Update ``cp-kafka-connect`` image with Confluent Platform 5.5.2
* Update dependencies
0.8.0 (2020-08-05)
==================
* Use data classes for the application and connector configuration.
* Plugin like organization, to support new connectors add a cli and a config file.
* Add support to the Amazon S3 Sink connector
0.7.2 (2020-03-31)
==================
* Add support to the InfluxDB Sink Connector.
* Add --timestamp option to select the timestamp field to use in the InfluxDB Sink connector.
* Fix Header Converter Class configuration setting.
* Fix ``tasks.max`` configuration setting name.
* Add connector ``name`` configuration setting to support multiple connectors of the same class.
* Handle empty list of topics properly.
kafka-connect-manager
#####################
|Build| |Docker|
A Python client to configure and create connectors using the `Kafka Connect API <https://docs.confluent.io/current/connect/references/restapi.html>`_.
Overview
========
Kafka-connect-manager helps to configure and create Kafka connectors. It can select Kafka topics using a regular expression and exclude topics added to a exclusion list. The auto-update option dynamically updates the connector if topics are created or deleted in Kafka.
Kafka-connect-manager can validate the connector configuration and return the validation output in case of errors.
Kafka-connect-manager provides a command line interface (CLI) to interact with the `Kafka Connect API <https://docs.confluent.io/current/connect/references/restapi.html>`_. Using the CLI you can manage existing connectors or create a new one.
To deploy a supported connector, you can use the `kafka-connect-manager Helm chart <https://lsst-sqre.github.io/charts/>`_.
Supported connectors
--------------------
* `Lenses InfluxDB Sink <https://docs.lenses.io/connectors/sink/influx.html>`_
* `Amazon S3 Sink <https://docs.confluent.io/current/connect/kafka-connect-s3>`_
See `the docs <https://kafka-connect-manager.lsst.io>`_ for more information.
.. |Build| image:: https://github.com/lsst-sqre/kafka-connect-manager/workflows/CI/badge.svg
:alt: GitHub Actions
:scale: 100%
:target: https://github.com/lsst-sqre/kafka-connect-manager/actions
.. |Docker| image:: https://img.shields.io/docker/v/lsstsqre/kafkaconnect?sort=date
:alt: Docker Hub repository
:scale: 100%
:target: https://hub.docker.com/repository/docker/lsstsqre/kafkaconnect
##########
Change log
##########
0.8.2 (2021-01-25)
==================
* Update ``cp-kafka-connect`` image with new version of the InfluxDB Sink connector. This version bumps the ``influxdb-java`` dependency from version 2.9 to 2.21. In parrticular 2.16 introduced a fix to skip fields with ``NaN`` and ``Infinity`` values when writing to InfluxDB.
* Reorganize developer and user guides.
* Add documentation in the user guide on how to run the InfluxDB Sink connector locally.
* Update dependencies
0.8.1 (2020-10-18)
==================
* Fix bug preventing to read InfluxDB password from the environment
* Update ``cp-kafka-connect`` image with Confluent Platform 5.5.2
* Update dependencies
0.8.0 (2020-08-05)
==================
* Use data classes for the application and connector configuration.
* Plugin like organization, to support new connectors add a cli and a config file.
* Add support to the Amazon S3 Sink connector
0.7.2 (2020-03-31)
==================
* Add support to the InfluxDB Sink Connector.
* Add --timestamp option to select the timestamp field to use in the InfluxDB Sink connector.
* Fix Header Converter Class configuration setting.
* Fix ``tasks.max`` configuration setting name.
* Add connector ``name`` configuration setting to support multiple connectors of the same class.
* Handle empty list of topics properly.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kafkaconnect-0.8.2.tar.gz
(59.9 kB
view details)
Built Distribution
File details
Details for the file kafkaconnect-0.8.2.tar.gz
.
File metadata
- Download URL: kafkaconnect-0.8.2.tar.gz
- Upload date:
- Size: 59.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/51.1.2 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5aa084f4700364cded23d80053310d88882dd6dc8f356ef1dbc7a8fee3d1129 |
|
MD5 | f86af559425f3b9e5e2bf87917bff860 |
|
BLAKE2b-256 | d39f76525263437c6e08d1bafef9957bfcd8557087089e9ed0a9439402703900 |
File details
Details for the file kafkaconnect-0.8.2-py3-none-any.whl
.
File metadata
- Download URL: kafkaconnect-0.8.2-py3-none-any.whl
- Upload date:
- Size: 17.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/51.1.2 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21dcb1c79d98e3895c7f996bf8808d60230c3964bad45f4fbade9f98f2cb9b6f |
|
MD5 | b7fdd6ed0bd6f7ef6729ba81af69013c |
|
BLAKE2b-256 | 22343f7e5403b2d125d881e45350a8f012dbff0e9304d641f31dfdff3999412a |