Monasca Database Persister

Go to file

Joe Keen 3964f2aae2 Adapting persister because of changes to monasca common Change-Id: I1a10e26340dcbd4dd634b32a52f554d073bfb946		2015-11-09 10:50:50 -07:00
common	Change repositories from stackforge to openstack	2015-10-19 09:25:20 +05:30
java	Merge "Validate metricsString does not exceed max length"	2015-10-20 17:38:08 +00:00
monasca_persister	Adapting persister because of changes to monasca common	2015-11-09 10:50:50 -07:00
openstack	Python persister batches data points with common name	2015-02-10 12:42:12 -07:00
.gitignore	Add virtenv dir and contents to .gitignore	2015-08-31 08:32:11 -06:00
.gitreview	Update .gitreview for new namespace	2015-10-17 22:30:59 +00:00
LICENSE	Added copyright header and LICENSE file.	2014-05-01 12:45:08 -06:00
README.md	Change repositories from stackforge to openstack	2015-10-19 09:25:20 +05:30
pom.xml	Change repositories from stackforge to openstack	2015-10-19 09:25:20 +05:30
requirements.txt	Modifying Persister to use common kafka interface	2015-10-30 12:24:36 -06:00
run_maven.sh	target link was being created incorrectly	2015-08-20 10:53:50 -06:00
setup.cfg	Change repositories from stackforge to openstack	2015-10-19 09:25:20 +05:30
setup.py	Add files for pypi.	2014-09-30 15:21:50 -06:00
test-requirements.txt	Always store value_meta	2015-06-02 07:24:18 -06:00
tox.ini	Increase throughput of persister	2014-10-02 11:40:18 -06:00

README.md

monasca-persister

The Monitoring Persister consumes metrics and alarm state transitions from the Message Queue and stores them in the Metrics and Alarms database.

Although the Persister isn't primarly a Web service it uses DropWizard, https://dropwizard.github.io/dropwizard/, which provides a nice Web appliction framework to expose an http endpoint that provides an interface through which metrics about the Persister can be queried as well as health status.

The basic design of the Persister is to have one Kafka consumer publish to a Disruptor, https://github.com/LMAX-Exchange/disruptor, that has output processors. The output processors use prepared batch statements to write to the Metrics and Alarms database.

The number of output processors/threads in the Persister can be specified to scale to more messages. To horizontally scale and provide fault-tolerance any number of Persisters can be started as consumers from the the Message Queue.

Build

Requires monasca-common from https://github.com/openstack/monasca-common. Download and build following instructions in its README.md. Then build monasca-persister by:

mvn clean package

Configuration

A sample configuration file is available in java/src/deb/etc/persister-config.yml-sample.

A second configuration file is provided in java/src/main/resources/persister-config.yml for use with the vagrant "mini-mon" development environment.

TODO

Purge metrics on shutdown
Add more robust offset management in Kafka. Currently, the offset is advanced as each message is read. If the Persister stops after the metric has been read and prior to it being committed to the Metrics and Alarms database, the metric will be lost.
Add better handling of SQL exceptions.
Complete health check.
Specify and document the names of the metrics that are available for monitoring of the Persister.
Document the yaml configuration parameters.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.