Check periodically if host continues to be a
leader once elected. Failure to check
might lead to a situation where the host
has lost leadership but is not aware of the
situation.
If the host is no longer the leader then
stand down as a leader, stop any spark-submit
processes running on the node and reset state
in the transform thread.
Removed --supervise option when invoking
spark-submit to turn off built in driver
management.
Added some hardening to better catch exceptions
in main transform service thread and also
periodic leader check function so that the
threads don't die when they encounter
an unhandled exception.
Change-Id: If2e13e3ed6cb30b3d7fa5f1b440c4c39b87692be
kafka_python 0.9.5 was moved to monasca common
Upstream community wants to move to
newer version of kafka python which has number of
performance problems.
See https://review.openstack.org/#/c/420579/
and
https://review.openstack.org/#/c/424840/
Monasca transform
uses kafka python library to write aggregated
metrics to kafka as well as read offset information
in case of hourly aggregation. Since long term
plan is to move to pykafka in the future we will
have to investigate if that functionality
is available.
Change-Id: I831c9e259b3d7b92fb2834193034e15b62c80c37
In I530bd05641f39d94e5a543dcbb219a38f08a1460 it's been requested that the
requirements bot sync global requirements to monasca-transform.
This will fail as tox is listed in test-requirements.txt. This change
remove tox from test-requirements as manually runs the bot against this
repo. meaning that once the change merges requirements syncs will be
automatic and simpler
Change-Id: I1c5e5c288b86bd44c1346a26079bf8bd9b7b9167
Needed-By: I530bd05641f39d94e5a543dcbb219a38f08a1460
Moved what were called unit tests to the functional directory as there is
more work to be done to make them self supporting.
Added sphinx to the test-requirements along with testr deps.
Adjusted the tox.ini to suit.
Added doc source tree with the minimal set of files to allow sphinx
success.
Change-Id: Ic64c2ccd01778bb0d1142d12ca43b2346bdc7aa7
The version can create issues with the build strategy used for monasca.
Removing the version brings this into line with other monasca projects.
Change-Id: I141e13ec1942e98f96f6f236aa6119205d86c7f1
Two stage aggregation refactored to use kafka-python 0.9.5
as this is the version we are limited to by openstack.
Change-Id: I20c4dc58727432c1336c5cfdb37768a24e578eb0
The log file was being duplicated at monasca-transform.log and
monasca_transform.log. Fixed this to be set simply at
monasca-transform.log.
Change-Id: I6a63737c569b06a271e11b880675edadfbdcc250
The monasca-transform is a new component in Monasca that
aggregates and transforms metrics.
monasca-transform is a Spark based data driven aggregation
engine which collects, groups and aggregates existing individual
Monasca metrics according to business requirements and publishes
new transformed (derived) metrics to the Monasca Kafka queue.
Since the new transformed metrics are published as any other
metric in Monasca, alarms can be set and triggered on the
transformed metric, just like any other metric.
Co-Authored-By: Flint Calvin <flint.calvin@hp.com>
Co-Authored-By: David Charles Kennedy <david.c.kennedy@hpe.com>
Co-Authored-By: Ashwin Agate <ashwin.agate@hp.com>
Implements: blueprint monasca-transform
Change-Id: I0e67ac7a4c9a5627ddaf698855df086d55a52d26
Witek Bedyk