This commit updates hacking version to 1.1.x and fixes related
pep8 issues.
Also added pycodestyle in test-requirements
Story: 2004930
Task: 29318
Co-Authored-By: Akhil Jain <akhil.jain@india.nec.com>
Change-Id: Id3ad30d23b902ee6f7277f7ec20d7d523df232f6
* set the maximum line length to 100
* cleaned up the codes for pep8
Change-Id: Iab260a4e77584aae31c0596f39146dd5092b807a
Signed-off-by: Amir Mofakhar <amofakhar@op5.com>
With this change pre hourly processor which does the
hourly aggregation (second stage) and writes the
final aggregated metrics to metris topic in kafka
now accounts for any early arriving metrics.
This change along with two previous changes
to pre hourly processor that added
1.) configurable late metrics slack time
(https://review.openstack.org/#/c/394497/), and
2.) batch filtering
(https://review.openstack.org/#/c/363100/)
will make sure all late arriving or early
arriving metrics for an hour are aggregated
appropriately.
Also made improvement in MySQL offset to call
delete excess revisions only once.
Change-Id: I919cddf343821fe52ad6a1d4170362311f84c0e4
kafka_python 0.9.5 was moved to monasca common
Upstream community wants to move to
newer version of kafka python which has number of
performance problems.
See https://review.openstack.org/#/c/420579/
and
https://review.openstack.org/#/c/424840/
Monasca transform
uses kafka python library to write aggregated
metrics to kafka as well as read offset information
in case of hourly aggregation. Since long term
plan is to move to pykafka in the future we will
have to investigate if that functionality
is available.
Change-Id: I831c9e259b3d7b92fb2834193034e15b62c80c37
Fixed a bug where the hourly agregation would run at every iteration
if the hour is zero (midnight) because zero is falsey.
Change-Id: I9652f02aea30f3ddb6f154db716aa4057455be06
Pre Hourly processor fails if offsets recorded in
kafka_offsets table no longer exist in kafka.
This change deletes the offsets from kafka_offsets
table, so that the pre hourly processor can resume
processing with the next run.
Change-Id: I017c271e630fdf6de05a73b3bfcb14f5ed18615f
Added configuration option to allow the pre-hourly transformation to be
done at a specified period past the hour. This includes a check to
ensure that if not done yet for the hour but overdue processing is done
at the earliest time.
Change-Id: I8882f3089ca748ce435b4e9a92196a72a0a8e63f
Made changes such that debug-level log entries are written to
the application log noting which aggregated metrics are submitted
during pre-hourly and hourly processing.
Change-Id: I64c6a18233614fe680aa0b084570ee7885f316e5
entries for caching levels. Also changed the calculate_rate
component to use values from instance usage if available (rather
than using 'all').
Change-Id: Ibdbc8d57c2566de76051c9277f9c75225546d4d7
Two stage aggregation refactored to use kafka-python 0.9.5
as this is the version we are limited to by openstack.
Change-Id: I20c4dc58727432c1336c5cfdb37768a24e578eb0
Breaking down the aggregation into two stages.
The first stage aggregates raw metrics frequently and is
implemented as a Spark Streaming job which
aggregates metrics at a configurable time interval
(defaults to 10 minutes) and writes the intermediate
aggregated data, or instance usage data
to new "metrics_pre_hourly" kafka topic.
The second stage is implemented
as a batch job using Spark Streaming createRDD
direct stream batch API, which is triggered by the
first stage only when first stage runs at the
top of the hour.
Also enhanced kafka offsets table to keep track
of offsets from two stages along with streaming
batch time, last time version row got updated
and revision number. By default it should keep
last 10 revisions to the offsets for each
application.
Change-Id: Ib2bf7df6b32ca27c89442a23283a89fea802d146