openstack
/
monasca-transform
Code Issues Proposed changes
DEPRECATED, Monasca Transform and Aggregation Engine
12 Commits 2 Branches 34 Tags 2.5 MiB
Go to file
Ashwin Agate 00b874a6b3 Two stage transformation 
Breaking down the aggregation into two stages.

The first stage aggregates raw metrics frequently and is
implemented as a Spark Streaming job which
aggregates metrics at a configurable time interval
(defaults to 10 minutes) and writes the intermediate
aggregated data, or instance usage data
to new "metrics_pre_hourly" kafka topic.

The second stage is implemented
as a batch job using Spark Streaming createRDD
direct stream batch API, which is triggered by the
first stage only when first stage runs at the
top of the hour.

Also enhanced kafka offsets table to keep track
of offsets from two stages along with streaming
batch time, last time version row got updated
and revision number. By default it should keep
last 10 revisions to the offsets for each
application.

Change-Id: Ib2bf7df6b32ca27c89442a23283a89fea802d146
 		2016-06-28 13:47:50 +00:00
devstack Two stage transformation 2016-06-28 13:47:50 +00:00
etc Two stage transformation 2016-06-28 13:47:50 +00:00
monasca_transform Two stage transformation 2016-06-28 13:47:50 +00:00
scripts Removed spark-events from config 2016-06-20 10:15:02 +01:00
tests Two stage transformation 2016-06-28 13:47:50 +00:00
tools/vagrant Two stage transformation 2016-06-28 13:47:50 +00:00
.gitreview Added .gitreview 2016-04-08 07:58:56 +00:00
LICENSE monasca-transform initial commit 2016-05-26 00:10:37 +00:00
README.md monasca-transform initial commit 2016-05-26 00:10:37 +00:00
requirements.txt monasca-transform initial commit 2016-05-26 00:10:37 +00:00
setup.cfg Two stage transformation 2016-06-28 13:47:50 +00:00
setup.py monasca-transform initial commit 2016-05-26 00:10:37 +00:00
test-requirements.txt monasca-transform initial commit 2016-05-26 00:10:37 +00:00
tox.ini Two stage transformation 2016-06-28 13:47:50 +00:00

README.md

Monasca Transform

##To set up the development environment

The monasca-transform dev environment uses devstack so see the README in the devstack directory.