kafka-aggregator 0.2.0

################
kafka-aggregator
################

|Build| |Docker|

A Kafka aggregator based on the `Faust <https://faust.readthedocs.io/en/latest/index.html>`_ Python Stream Processing library.

kafka-aggregator development is based on the `Safir <https://safir.lsst.io>`__ application template.


Overview
========

kafka-aggregator uses `Faust's windowing feature <https://faust.readthedocs.io/en/latest/userguide/tables.html#windowing>`_ to aggregate a stream of messages from Kafka.

kafka-aggregator implements a Faust agent, a "stream processor", that adds messages from a source topic into a Faust table. The table is configured as a tumbling window with a size, representing the window duration (time interval) and an expiration time, which specifies the duration for which the data allocated to each window will be stored. Every time a window expires, a callback function is called to aggregate the messages allocated to that window. The size of the window controls the frequency of the aggregated stream.

kafka-aggregator uses `faust-avro <https://github.com/masterysystems/faust-avro>`_ to add Avro serialization and Schema Registry support to Faust. faust-avro can parse Faust models into Avro Schemas.

See `the docs <https://kafka-aggregator.lsst.io/>`_ for more information.

.. |Build| image:: https://github.com/lsst-sqre/kafka-aggregator/workflows/CI/badge.svg
:alt: GitHub Actions
:scale: 100%
:target: https://github.com/lsst-sqre/kafka-aggregator/actions

.. |Docker| image:: https://img.shields.io/docker/v/lsstsqre/kafkaaggregator?sort=date
:alt: Docker Hub repository
:scale: 100%
:target: https://hub.docker.com/repository/docker/lsstsqre/kafkaaggregator

##########
Change log
##########

0.2.0 (2020-08-14)
==================

* Add first and third quartiles (``q1`` and ``q3``) to the list of summary statistics computed by the aggregator.
* Ability to configure the list of summary statistics to be computed.
* Pinned top-level requeriments.
* Add Kafka Connect to the docker-compose setup.
* Use only one Schema Registry by default to simplify local execution.
* First release to PyPI.


0.1.0 (2020-07-13)
==================

Initial release of kafka-aggregator with the following features:

* Use Faust windowing feature to aggregate a stream of messages.
* Use Faust-avro to add Avro serialization and Schema Registry support to Faust.
* Support to an internal Schema Registry to store schemas for the aggreated topics (optional).
* Create aggregation topic schemas from the source topic schemas and from the list of summary statistics to be computed.
* Ability to create Faust records dynamically from aggregation topic schemas.
* Ability to auto-generate code for the Faust agents (stream processors).
* Compute summary statistics for numeric fields: ``min()``, ``mean()``, ``median()``, ``stdev()``, ``max()``.
* Add example module to initialize a number of source topics in kafka, control the number of fields in each topic, and produce messages for those topics at a given frequency.
* Use Kafdrop to inspect messages from source and aggregated topics.
* Add kafka-aggregator documentation site.