* set the maximum line length to 100
* cleaned up the codes for pep8
Change-Id: Iab260a4e77584aae31c0596f39146dd5092b807a
Signed-off-by: Amir Mofakhar <amofakhar@op5.com>
There is a desire to use Monasca Transform to aggregate
kubernetes metrics. This change is a start in that
direction. The transformation specs in the tests
folder now include some representative aggregations
of some kubernetes metrics.
This commit also includes some changes to get
first_attempt_at_spark_test.py working again
after being moved from the unit test folder to the
functional test folder.
Change-Id: I038ecaf42e67d5c994980991232a2a8428a4f4e3
Breaking down the aggregation into two stages.
The first stage aggregates raw metrics frequently and is
implemented as a Spark Streaming job which
aggregates metrics at a configurable time interval
(defaults to 10 minutes) and writes the intermediate
aggregated data, or instance usage data
to new "metrics_pre_hourly" kafka topic.
The second stage is implemented
as a batch job using Spark Streaming createRDD
direct stream batch API, which is triggered by the
first stage only when first stage runs at the
top of the hour.
Also enhanced kafka offsets table to keep track
of offsets from two stages along with streaming
batch time, last time version row got updated
and revision number. By default it should keep
last 10 revisions to the offsets for each
application.
Change-Id: Ib2bf7df6b32ca27c89442a23283a89fea802d146
The monasca-transform is a new component in Monasca that
aggregates and transforms metrics.
monasca-transform is a Spark based data driven aggregation
engine which collects, groups and aggregates existing individual
Monasca metrics according to business requirements and publishes
new transformed (derived) metrics to the Monasca Kafka queue.
Since the new transformed metrics are published as any other
metric in Monasca, alarms can be set and triggered on the
transformed metric, just like any other metric.
Co-Authored-By: Flint Calvin <flint.calvin@hp.com>
Co-Authored-By: David Charles Kennedy <david.c.kennedy@hpe.com>
Co-Authored-By: Ashwin Agate <ashwin.agate@hp.com>
Implements: blueprint monasca-transform
Change-Id: I0e67ac7a4c9a5627ddaf698855df086d55a52d26