Commit Graph

22 Commits

Author SHA1 Message Date
Witek Bedyk 811acd76c9 Remove project content on master branch
This is step 2b of repository deprecation process as described in [1].
Project deprecation has been anounced here [2].

[1] https://docs.openstack.org/project-team-guide/repository.html#step-2b-remove-project-content
[2] http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016814.html

Depends-On: https://review.opendev.org/751983
Change-Id: I83bb2821d64a4dddd569ff9939aa78d271834f08
2020-09-15 10:12:44 +02:00
zhulingjie ef6f94d3a6 Replace git.openstack.org URLs with opendev.org URLs
Change-Id: I1d8a8c0eecf7be5b92cc7855987a74291c3b726d
2019-04-24 16:50:05 +00:00
Ian Wienand c3e87aea34 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Iefd089227b889e88b6053bd752f0e9f0434b6b72
2019-03-24 20:34:16 +00:00
Ashwin Agate 1454ade2dc Update refresh monasca transfom script to systemd
devstack switched to systemd to manage processes
see https://review.openstack.org/#/c/448323 and
https://review.openstack.org/#/c/460062.
If USE_SCREEN variable is set to False, use
systemctl to stop and start monasca-transform
process instead.

Change-Id: I32882e55e1ce235f125439de72c5e3416c828eae
2017-06-01 15:02:03 -07:00
Ashwin Agate 5c15a99a5c Enhanced refresh monasca transform script
refresh_monasca_script.sh is useful for
development in a devstack environment.

Enhanced the script to
* start and stop monasca-transform process
  running in a screen session
* added more hardening to catch for errors
  and exit if any command fails.
* added debugging statements help track
  down any errors when the script is run.

Change-Id: Idab02d555eed192d8242c870017955b935532c3d
2017-04-19 17:16:50 -07:00
Ashwin Agate f99a3faf68 Fix development environment and functional tests
Changing devstack environment vagrant box and
also rename the devstack VM to 'devstack'
from 'pg-tips'

Also fixing all the tests that were broken when
they were moved from tests/unit to tests/functional
with this review
https://review.openstack.org/#/c/400237/

Update devstack README with a section called
Development workflow for monasca-transform with
steps developers can take to develop and run
tests.

Change-Id: I11678148ba2bcb96eb3e2a522176683dc8bca30a
2017-04-11 12:13:25 -07:00
Flint Calvin d8f283c378 Started adding kubernetes metrics aggregation
There is a desire to use Monasca Transform to aggregate
kubernetes metrics.  This change is a start in that
direction.  The transformation specs in the tests
folder now include some representative aggregations
of some kubernetes metrics.

This commit also includes some changes to get
first_attempt_at_spark_test.py working again
after being moved from the unit test folder to the
functional test folder.

Change-Id: I038ecaf42e67d5c994980991232a2a8428a4f4e3
2017-03-02 15:38:25 +00:00
Joseph Davis f34176819d Adjust proxy port parsing to only return number.
A simple robustness tweak to remove any trailing character,
like a / in "http://web-proxy.example.com:8080/", from
the parsed proxy port.

Change-Id: Iac75f8ae431b923b86dbd5835b1c48ca090e14e7
2017-01-13 08:56:38 -08:00
David C Kennedy 88e04a3096 Fix ci
The devstack plugin carries usage of `sudo -u` which doesn't seem to
work in the ci environment.  Replace it with sudo followed by
appropriate permissions changes.
Use ${DEST} instead of literal /opt/stack to fit with gate usage.
Enabled monasca-api plugin in the settings and the required monasca
services along with zookeeper.

Change-Id: I6effede4ac9a2faf1c44eff9cd96bbf9c924d703
2016-12-15 07:55:17 +00:00
David C Kennedy 903c9fe5bf Addressed proxy issues for maven in vagrant
Change-Id: Ia349bf07cf354cfa2a1efe19b85e918c22c63732
2016-12-14 21:11:06 +00:00
David C Kennedy 0a73dc9110 Refactored vagrant environment for xenial
Change-Id: I6cf6452f68bb143710ece4e80e2b216a07eaf07b
2016-11-25 10:52:55 +00:00
David C Kennedy fd6a79c6ec Fixed vagrant provision script
Add sudo to command removing monasca-transform directory for renewal.
Without this it fails as the .git directories are permissioned elsehow
from the default user.

Change-Id: I4c9d3ff7ed1c2b841be82a66ec594033eec2af15
2016-11-22 11:06:06 +00:00
David C Kennedy 8a6e619f72 Populate the project id for kafka publish
This needs to be the admin project id so for devstack this needs to be written
to the configuration file once the users/projects etc are created and
identifiable.
Add a similar process to the refresh script.
Correct the configuration property name to 'project' rather than using the old
nomencature 'tenant'.

Change-Id: Ib9970ffacf5ee0f7f006722038a1db8024c1385e
2016-11-11 15:24:22 +00:00
David C Kennedy 30b9540a6c Fixed the refresh script
The refresh script, usable in development to update the code executed in
monasca-transform from the developers local copy was broken.  When the refresh
was carried out the script copied a fresh configuration file into place,
wiping the changes made by the devstack plugin during deployment.  This
resulted in an incorrect ip address being used for the kafka broker.

Change-Id: I606dad9f6c8a59006e37560f5bb5fdceb51266d4
2016-11-08 15:47:27 +00:00
David C Kennedy 8ac3250aef Remove metric filters from transform-spec
The metric filter allows metrics to be filtered before aggregation,
for example to exclude metrics from certain environments or nodes
from being included where they are outside the scope of data
aggregation.  This is a powerful feature but not appropriate for
all scenarios (e.g. devstack).  So default the filter to empty.

Change-Id: Icb790a0ec41133bfac54244aae8782a5cc665186
2016-11-02 15:58:07 +00:00
Tomasz Trębski 698856b383 Use database password from variable
There is a variable that tells what is the database (mysql)
password. However plugin.sh is using a hardcoded password.
Commits provides using DATABASE_PASSWORD variable (the same
one as devstack is using) + defines a variable for m-transform
user - MONASCA_TRANSFORM_DB_PASSWORD

Change-Id: I9fc8296ef31b22564f2cf1536e51ab3abc8c9dc9
2016-10-25 14:55:47 +01:00
Michael Dong 1afd0823ea fix no_proxy bug in tools/vagrant/Vagrantfile
Closes-Bug:#1598317

Change-Id: I926d770a541ff6a5eefae905f5b4fe0dd09e4efc
2016-07-01 15:59:59 -07:00
Ashwin Agate 00b874a6b3 Two stage transformation
Breaking down the aggregation into two stages.

The first stage aggregates raw metrics frequently and is
implemented as a Spark Streaming job which
aggregates metrics at a configurable time interval
(defaults to 10 minutes) and writes the intermediate
aggregated data, or instance usage data
to new "metrics_pre_hourly" kafka topic.

The second stage is implemented
as a batch job using Spark Streaming createRDD
direct stream batch API, which is triggered by the
first stage only when first stage runs at the
top of the hour.

Also enhanced kafka offsets table to keep track
of offsets from two stages along with streaming
batch time, last time version row got updated
and revision number. By default it should keep
last 10 revisions to the offsets for each
application.

Change-Id: Ib2bf7df6b32ca27c89442a23283a89fea802d146
2016-06-28 13:47:50 +00:00
David C Kennedy 633aff4279 Vagrant to use local branch for devstack
By default devstack was cloning master when setting up devstack.
This change has the devstack plugin clone the local working branch so
that uncommitted changes can be tested.

Change-Id: If963ea1d703cd9d4d96665e75db08834cf473b33
2016-06-20 10:08:39 +01:00
David C Kennedy 72f240b926 Removed patch for monasca-api plugin
The patch allowing us to disable other monasca components is no
longer necessary so remove it from our vagrant process.

Change-Id: I490b557de05ce6696d35783d7cc55db9d0044fbd
2016-06-14 10:32:27 +01:00
Flint Calvin e4ade60711 Implemented aggregation for disk.allocation. Also set the
apache download source to use the archive site to ensure that
the dependency package does not disappear.  Also brought the
vagrant environment inline with monasca-api (i.e., use the
same values for private network, add substitution for kafka
brokers ip address to the conf).  Also parameterised
dependency sources (i.e., added settings to parameterise the
maven and apache repositories for the devstack plugin).

Change-Id: If9f0e2ed16bbfcd62152d29e5c7c86f5d555f9aa
2016-06-01 15:56:26 +00:00
Ashwin Agate 8f61dd95a9 monasca-transform initial commit
The monasca-transform is a new component in Monasca that
aggregates and transforms metrics.

monasca-transform is a Spark based data driven aggregation
engine which collects, groups and aggregates existing individual
Monasca metrics according to business requirements and publishes
new transformed (derived) metrics to the Monasca Kafka queue.

Since the new transformed metrics are published as any other
metric in Monasca, alarms can be set and triggered on the
transformed metric, just like any other metric.

Co-Authored-By: Flint Calvin <flint.calvin@hp.com>
Co-Authored-By: David Charles Kennedy <david.c.kennedy@hpe.com>
Co-Authored-By: Ashwin Agate <ashwin.agate@hp.com>

Implements: blueprint monasca-transform

Change-Id: I0e67ac7a4c9a5627ddaf698855df086d55a52d26
2016-05-26 00:10:37 +00:00