Data Aggregation and Transformation component for Monasca
Project description
Team and repository tags
Monasca Transform
monasca-transform is a data driven aggregation engine which collects, groups and aggregates existing individual Monasca metrics according to business requirements and publishes new transformed (derived) metrics to the Monasca Kafka queue.
Since the new transformed metrics are published as any other metric in Monasca, alarms can be set and triggered on the transformed metric.
Monasca Transform uses Apache Spark to aggregate data. Apache Spark is a highly scalable, fast, in-memory, fault tolerant and parallel data processing framework. All monasca-transform components are implemented in Python and use Spark’s PySpark Python API to interact with Spark.
Monasca Transform does transformation and aggregation of incoming metrics in two phases.
In the first phase spark streaming application is set to retrieve in data from kafka at a configurable stream interval (default stream_inteval is 10 minutes) and write the data aggregated for stream interval to pre_hourly_metrics topic in kafka.
In the second phase, which is kicked off every hour, all metrics in metrics_pre_hourly topic in Kafka are aggregated again, this time over a larger interval of an hour. These hourly aggregated metrics published to metrics topic in kafka.
Use Cases handled by Monasca Transform
Please refer to Problem Description section on the Monasca/Transform wiki
Operation
Please refer to How Monasca Transform Operates section on the Monasca/Transform wiki
Architecture
Please refer to Architecture and Logical processing data flow sections on the Monasca/Transform wiki
To set up the development environment
The monasca-transform uses DevStack as a common dev environment. See the README.md in the devstack directory for details on how to include monasca-transform in a DevStack deployment.
Generic aggregation components
Monasca Transform uses a set of generic aggregation components which can be assembled in to an aggregation pipeline.
Please refer to the generic-aggregation-components document for information on list of generic aggregation components available.
Create a new aggregation pipeline example
Generic aggregation components make it easy to build new aggregation pipelines for different Monasca metrics.
This create a new aggregation pipeline example shows how to create pre_transform_specs and transform_specs to create an aggregation pipeline for a new set of Monasca metrics, while leveraging existing set of generic aggregation components.
Original proposal and blueprint
Original proposal: Monasca/Transform-proposal
Blueprint: monasca-transform blueprint
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file monasca_transform-0.18.0.tar.gz.
File metadata
- Download URL: monasca_transform-0.18.0.tar.gz
- Upload date:
- Size: 142.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.5.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 6cb8bfbe8e68475b50564fb207328ac1e552c979df1318ff71ac7182127d315e |
|
| MD5 | 831bddce2f25a358e914b381f83d8018 |
|
| BLAKE2b-256 | 1c3b94244a318dd01032447f97c86e79de7e8c39bdb0aee19d471a175db9e66f |
Provenance
File details
Details for the file monasca_transform-0.18.0-py2.py3-none-any.whl.
File metadata
- Download URL: monasca_transform-0.18.0-py2.py3-none-any.whl
- Upload date:
- Size: 78.1 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.5.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | db1d237f7b749551f7c1b0128524cfa542a9d1ffa20805d2b95cf9e0b72cf58e |
|
| MD5 | 1fa729c46650a6447329b64829f100f3 |
|
| BLAKE2b-256 | 636df69975280ca2648bbc648c2f8cbfb97a4d0269390701c2192022ca1baf74 |
