This patch adds a custom decoder to deserialize bulk metrics.
The deserialization of bulk_metric message is configurable per Logger and
is disabled by default.
Currently, only the HTTP metrics are deserialized (Logger: aggregated_http_metrics_filter)
To deserialize several Loggers, a list of Loggers separated by space can
be configured for the sandbox.
blueprint: keystone-performance-monitoring
Change-Id: I08e8c816d889f4b0917d766b000fb25330eff174
This use case is not really supported because Nagios configuration is
too dependent on the LMA Collector plugin and in practice, no one
chooses this option.
DocImpact
Change-Id: Ia09efb40f476c1daec51530e2c0fb16bc6f99393
This change separates the processing of the logs/notifications and
metric/alerting into 2 dedicated hekad processes, these services are
named 'log_collector' and 'metric_collector'.
Both services are managed by Pacemaker on controller nodes and by Upstart on
other nodes.
All metrics computed by log_collector (HTTP response times and creation time
for instances and volumes) are sent directly to the metric_collector via TCP.
Elasticsearch output (log_collector) uses full_action='block' and the
TCP output uses full_action='drop'.
All outputs of metric_collector (InfluxDB, HTTP and TCP) use
full_action='drop'.
The buffer size configurations are:
* metric_collector:
- influxdb-output buffer size is increased to 1Gb.
- aggregator-output (tcp) buffer size is decreased to 256Mb (vs 1Gb).
- nagios outputs (x3) buffer size are decreased to 1Mb.
* log_collector:
- elasticsearch-output buffer size is decreased to 256Mb (vs 1Gb).
- tcp-output buffer size is set to 256Mb.
Implements: blueprint separate-lma-collector-pipelines
Fixes-bug: #1566748
Change-Id: Ieadb93b89f81e944e21cf8e5a65f4d683fd0ffb8
And decrease the max_retries from 3 to 2 to stay in the 50 seconds window.
This change allows to retrieve large number of objects and also avoids to
overload the system by performing 3 'zombies' requests every 50 seconds
without any metrics collected.
Partial-bug: #1554502
Change-Id: I60a7611bc82598831538da01245b87fb29a15c44
This change introduces the fuel_lma_collector Puppet module. This new
module contains all the code which was in the lma_collector module and
relies on the way Fuel uses Puppet like the use of Hiera.
Change-Id: Ibfe97e745742b3f9718f873a8d564ed87476f334
Implements: blueprint lma-without-fuel
This change aims to be more compatible with
https://github.com/voxpupuli/puppet-collectd/pull/391, where the
module.conf.erb template does not add quotes around strings.
With this change the adapt_collectd_python_plugin_config function no
longer adds quotes, and lma_collector::collectd::python users are
responsible for adding quotes around strings.
In this way, when a new version of puppet-collectd is released, we
can just remove the adapt_collectd_python_plugin_config function and
things should just work without further changes.
Change-Id: I74501d4fc5a5ed097312975d56fc47ef587de837
This adds a Ruby function to work-around a limitation in the
collectd::plugin::python::module defined type.
See https://github.com/voxpupuli/puppet-collectd/issues/390.
Change-Id: I76a38c8b6bfdd041ebf2a077f4a6151ea889e074
This adds an lma_collector::collectd::openstack define that, when
declared, configures collectd to collect statistics from an OpenStack
service endpoint.
Usage example:
lma_collector::collectd::openstack { 'nova':
keystone_url => 'http://example.com/keystone'
}
Works for 'nova', 'cinder', 'glance', 'keystone', and 'neutron'.
Change-Id: I200a1b3de3b2385292e7b17489621382dc2b2e62
This adds an lma_collector::collectd::python define to use to
configure collectd Python plugins.
This define will eventually replace
lma_collector::collectd::python_script.
This define is not documented in the README.md file as it won't be
exposed to module users. Instead, wrapper classes/defines will be
added.
Usage example:
lma_collector::collectd::python { 'rabbitmq_info': }
This will configure a collectd Python plugin for the rabbitmq_info.py
Python script.
Change-Id: I997360166c2f121e1b2690578c8e5e92e1b9ad0e
Partially Implements: blueprint lma-without-fuel
Using "/" as the separator in the Heka logstreamer differentiator
prevents the logstreamer from creating a journal file. This is
because the logstreamer uses the Logger name produced by the
differentiator as the journal file name.
This change fixes the problem by changing the differentiator separator
from "/" to "_".
Change-Id: Iaf838f0e5df370df24f0d9d63a9bf13e37aa4bbe
Closes-Bug: #1530326
Notes about this change:
* The nova, neutron, cinder, glance, heat, keystone, horizon and murano logs
are now read from /var/log/<service> (where <service> is the OpenStack
service name). So we no longer rely on rsyslog for these services.
* We make an exception for Swift which only writes its logs to syslog. So we
still assume Rsyslog and the Fuel-defined Rsyslog log pattern for Swift.
* The "heka" user is now part of the "keystone" group. This is for Heka to be
able to read the Keystone logs from /var/log/keystone.
* We used to read Keystone logs from two different places, namely
/var/log/keystone-all.log and /var/log/user.log. This is no longer the case.
The Keystone logs are read from the files in /var/log/keystone.
Things to do next (with separate changes):
* Correctly parse Tracebacks (which are multi-line).
* Correctly parse other multi-line logs (Command: logs from
neutron.agent.linux.utils for example).
* Add an Heka message field to store the Python module name.
* Document the new defined types in the module's README.md file.
* Move patterns in openstack_log.lua to patterns.lua and add unit-tests.
* Read Keystone HTTP logs from /var/log/apache2/keystone_wsgi_*_access.log to
still be able to generate HTTP metrics for Keystone.
Change-Id: I667376030bff054e8ce9921d92f202e06f71269f
Implements: blueprint lma-without-fuel
Closes-Bug: #1520330
Related-Bug: #1525247
* remove useless suffix ".status"
* use Lua module msg_interpolate to get service name
* rename Puppet resource s/nagios/gse_nagios/
* add unit test
Co-Authored-By: Swann Croiset <scroiset@mirantis.com>
Co-Authored-By: Simon Pasquier <spasquier@mirantis.com>
Depends-On: Ideeaffd87a0a52375900da51129822ec65c6024c
Change-Id: Ifbb93522b341228afe6eec665030bc0c2efdbae5
This change modifies the implementation of the GSE filters. The main
differences are:
- level-1 dependencies define now the members of a cluster and the
status of a cluster is defined by the highest severity among all
members.
- level-2 dependencies are now known as 'hints', they define
relationships between clusters (eg, Nova depends on Keystone) but
have no influence on the status of a cluster.
Change-Id: I58bd79463de78b04b9bad92d02e3fb0da4bacdf4
With stlib <= 4.9 this test doesn't pass. Will be reactivated with the
update of stdlib > 4.9
see comment in lma_collector/manifests/hiera_data.pp
Change-Id: I2e3a8b30c61fdb34dfa9166c6f12dca6d566e1e4
This change modifies the Hiera configuration to add a datasource for
the LMA collector (located at /etc/hiera/override/lma_collector.yaml').
It also modifies the Puppet manifests to use hiera_hash() instead of
hiera() when looking up hashes: this ensures that hashes are merged with
the 'deeper-merge' strategy. Otherwise the 'lma_collector' key in
/etc/astute.yaml would be hidden.
Change-Id: Ia9fd3505aae05f63dfae99fc4bac11e831b63133
This change configures the 3 GSE cluster filters:
- one filter for the service clusters (nova-api, ...) that consumes
AFD service events.
- one filter for the node clusters (controller, ...) that consumes
AFD node events.
- one filter for the meta clusters (nova, ...) that consumes events
emitted by the 2 previous filters.
Change-Id: I25d21c4f3ad2ea57175faecdfbe4b93561f621d8
Implements: blueprint alerting-lma-collector
Andreas Jaeger