openstack
/
monasca-persister
Code Issues Proposed changes
Monasca Database Persister
256 Commits 8 Branches 27 Tags 5.7 MiB
Java 49.4%
Python 45%
Shell 3.9%
Jinja 1.1%
Dockerfile 0.6%
Go to file
Deklan Dieterly 452c020f63 Enhance fail fast on error functionality 
Make persister fail faster on errors.
Use one executor service for all perister threads.
Shutdown all threads in thread executor service on any error.
Catch java.lang.Throwable to catch java.lang.Error as well as java.lang.Exception.

Change-Id: I0dc421cf6bb4ab3f52c47e97b7f396483283b561
 		2015-08-19 08:37:34 -06:00
common Use Maven 3 for build 2015-03-27 12:04:07 -06:00
java Enhance fail fast on error functionality 2015-08-19 08:37:34 -06:00
monasca_persister Convert empty string to empty JSON for sub alarms 2015-06-25 14:57:53 -06:00
openstack Python persister batches data points with common name 2015-02-10 12:42:12 -07:00
.gitignore Exclude all .pyc files from git 2015-05-12 10:52:54 -06:00
.gitreview Get initial build to work. 2014-07-15 15:29:35 -06:00
LICENSE Added copyright header and LICENSE file. 2014-05-01 12:45:08 -06:00
README.md Use monasca everywhere 2014-08-27 21:55:22 -06:00
pom.xml Bump to version 1.1 2015-04-27 13:33:29 -06:00
requirements.txt Convert empty string to empty JSON for sub alarms 2015-06-25 14:57:53 -06:00
run_maven.sh Use Maven 3 for build 2015-03-27 12:04:07 -06:00
setup.cfg Specified all needed subpackages 2014-10-16 16:18:32 -06:00
setup.py Add files for pypi. 2014-09-30 15:21:50 -06:00
test-requirements.txt Always store value_meta 2015-06-02 07:24:18 -06:00
tox.ini Increase throughput of persister 2014-10-02 11:40:18 -06:00

README.md

monasca-persister

The Monitoring Persister consumes metrics and alarm state transitions from the Message Queue and stores them in the Metrics and Alarms database.

Although the Persister isn't primarly a Web service it uses DropWizard, https://dropwizard.github.io/dropwizard/, which provides a nice Web appliction framework to expose an http endpoint that provides an interface through which metrics about the Persister can be queried as well as health status.

The basic design of the Persister is to have one Kafka consumer publish to a Disruptor, https://github.com/LMAX-Exchange/disruptor, that has output processors. The output processors use prepared batch statements to write to the Metrics and Alarms database.

The number of output processors/threads in the Persister can be specified to scale to more messages. To horizontally scale and provide fault-tolerance any number of Persisters can be started as consumers from the the Message Queue.

Build

Requires monasca-common from https://github.com/stackforge/monasca-common. Download and build following instructions in its README.md. Then build monasca-persister by:

mvn clean package

TODO

  • Purge metrics on shutdown
  • Add more robust offset management in Kafka. Currently, the offset is advanced as each message is read. If the Persister stops after the metric has been read and prior to it being committed to the Metrics and Alarms database, the metric will be lost.
  • Add better handling of SQL exceptions.
  • Complete health check.
  • Specify and document the names of the metrics that are available for monitoring of the Persister.
  • Document the yaml configuration parameters.

License

Copyright (c) 2014 Hewlett-Packard Development Company, L.P.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.