Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I5868501da8cfc975c62e63a106f0fcae2384112a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
New releases of oslo.config support a 'mutable' parameter to Opts.
oslo.service provides an option here Icec3e664f3fe72614e373b2938e8dee53cf8bc5e
allows services to tell oslo.service they want mutate_config_files to be
called by passing a parameter.
This commit is to use the same. This allows monasca_transform to benefit from
I1e7a69de169cc85f4c09954b2f46ce2da7106d90, where the 'debug' option
(owned by oslo.log) is made mutable. we should be able to turn debug
logging on and off by changing the config.
tc goal:
https://governance.openstack.org/tc/goals/rocky/enable-mutable-configuration.html
Change-Id: I86571df78014a810ffa881ceceeddfc5193c9ca5
Check periodically if host continues to be a
leader once elected. Failure to check
might lead to a situation where the host
has lost leadership but is not aware of the
situation.
If the host is no longer the leader then
stand down as a leader, stop any spark-submit
processes running on the node and reset state
in the transform thread.
Removed --supervise option when invoking
spark-submit to turn off built in driver
management.
Added some hardening to better catch exceptions
in main transform service thread and also
periodic leader check function so that the
threads don't die when they encounter
an unhandled exception.
Change-Id: If2e13e3ed6cb30b3d7fa5f1b440c4c39b87692be
Changing devstack environment vagrant box and
also rename the devstack VM to 'devstack'
from 'pg-tips'
Also fixing all the tests that were broken when
they were moved from tests/unit to tests/functional
with this review
https://review.openstack.org/#/c/400237/
Update devstack README with a section called
Development workflow for monasca-transform with
steps developers can take to develop and run
tests.
Change-Id: I11678148ba2bcb96eb3e2a522176683dc8bca30a
Fixed a bug where the hourly agregation would run at every iteration
if the hour is zero (midnight) because zero is falsey.
Change-Id: I9652f02aea30f3ddb6f154db716aa4057455be06
Added configuration option to allow the pre-hourly transformation to be
done at a specified period past the hour. This includes a check to
ensure that if not done yet for the hour but overdue processing is done
at the earliest time.
Change-Id: I8882f3089ca748ce435b4e9a92196a72a0a8e63f
Moved what were called unit tests to the functional directory as there is
more work to be done to make them self supporting.
Added sphinx to the test-requirements along with testr deps.
Adjusted the tox.ini to suit.
Added doc source tree with the minimal set of files to allow sphinx
success.
Change-Id: Ic64c2ccd01778bb0d1142d12ca43b2346bdc7aa7
This needs to be the admin project id so for devstack this needs to be written
to the configuration file once the users/projects etc are created and
identifiable.
Add a similar process to the refresh script.
Correct the configuration property name to 'project' rather than using the old
nomencature 'tenant'.
Change-Id: Ib9970ffacf5ee0f7f006722038a1db8024c1385e
Made changes such that debug-level log entries are written to
the application log noting which aggregated metrics are submitted
during pre-hourly and hourly processing.
Change-Id: I64c6a18233614fe680aa0b084570ee7885f316e5
Removed the calls to the ceiling function on utilization
metrics aggregation such that they now are exact values (i.e.,
not rounded up to the next integral value).
Change-Id: I9813b94acb051f6754da2d559090318010f86e57
correctly based on swiftlm.diskusage.host.val.avail (instead of
incorrectly being based on swiftlm.diskusage.host.val.size).
Change-Id: If17853e166c050cefbf390791a8696ce520fca96
we transitioned to the upstream Monasca-Transform OpenStack repo.
Specifically, the missing aggregations were those for the
nova.vm.cpu.total_allocated and nova.vm.mem.total_allocated_mb
source metrics.
This set of changes also includes the resolution of a couple
pre-existing pep8 errors.
Change-Id: I84bf19b674aeadcd0d27799a887d0b89d0381550
Add properties to conf file to allow configuration of
SSL for the database connection. Done for both the python
and java connection strings.
Change-Id: I4c3d25c3f8f12eae801a6a818bf4ac7acd93d2dc
Breaking down the aggregation into two stages.
The first stage aggregates raw metrics frequently and is
implemented as a Spark Streaming job which
aggregates metrics at a configurable time interval
(defaults to 10 minutes) and writes the intermediate
aggregated data, or instance usage data
to new "metrics_pre_hourly" kafka topic.
The second stage is implemented
as a batch job using Spark Streaming createRDD
direct stream batch API, which is triggered by the
first stage only when first stage runs at the
top of the hour.
Also enhanced kafka offsets table to keep track
of offsets from two stages along with streaming
batch time, last time version row got updated
and revision number. By default it should keep
last 10 revisions to the offsets for each
application.
Change-Id: Ib2bf7df6b32ca27c89442a23283a89fea802d146
swiftlm.diskusage.host.val.size. Also renamed disk.allocation to
vm.disk.allocation and resolved a problem with resource_id not
being found for certain aggregations.
Change-Id: Iad82d149e7a04ed1e0ecfe936b90acfff1dca13e
apache download source to use the archive site to ensure that
the dependency package does not disappear. Also brought the
vagrant environment inline with monasca-api (i.e., use the
same values for private network, add substitution for kafka
brokers ip address to the conf). Also parameterised
dependency sources (i.e., added settings to parameterise the
maven and apache repositories for the devstack plugin).
Change-Id: If9f0e2ed16bbfcd62152d29e5c7c86f5d555f9aa
The monasca-transform is a new component in Monasca that
aggregates and transforms metrics.
monasca-transform is a Spark based data driven aggregation
engine which collects, groups and aggregates existing individual
Monasca metrics according to business requirements and publishes
new transformed (derived) metrics to the Monasca Kafka queue.
Since the new transformed metrics are published as any other
metric in Monasca, alarms can be set and triggered on the
transformed metric, just like any other metric.
Co-Authored-By: Flint Calvin <flint.calvin@hp.com>
Co-Authored-By: David Charles Kennedy <david.c.kennedy@hpe.com>
Co-Authored-By: Ashwin Agate <ashwin.agate@hp.com>
Implements: blueprint monasca-transform
Change-Id: I0e67ac7a4c9a5627ddaf698855df086d55a52d26
Witek Bedyk