x
/
fuel-plugin-lma-collector
Code Issues Proposed changes

Separate the (L)og of the LMA collector 

This change separates the processing of the logs/notifications and
metric/alerting into 2 dedicated hekad processes, these services are
named 'log_collector' and 'metric_collector'.

Both services are managed by Pacemaker on controller nodes and by Upstart on
other nodes.

All metrics computed by log_collector (HTTP response times and creation time
for instances and volumes) are sent directly to the metric_collector via TCP.
Elasticsearch output (log_collector) uses full_action='block' and the
TCP output uses full_action='drop'.

All outputs of metric_collector (InfluxDB, HTTP and TCP) use
full_action='drop'.

The buffer size configurations are:
* metric_collector:
  - influxdb-output buffer size is increased to 1Gb.
  - aggregator-output (tcp) buffer size is decreased to 256Mb (vs 1Gb).
  - nagios outputs (x3) buffer size are decreased to 1Mb.
* log_collector:
  - elasticsearch-output buffer size is decreased to 256Mb (vs 1Gb).
  - tcp-output buffer size is set to 256Mb.

Implements: blueprint separate-lma-collector-pipelines
Fixes-bug: #1566748

Change-Id: Ieadb93b89f81e944e21cf8e5a65f4d683fd0ffb8
This commit is contained in:
Swann Croiset 2016-03-27 22:46:52 +02:00
parent 3808d9f233
commit ebac150f8a
52 changed files with 838 additions and 446 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
deployment_scripts/puppet/manifests/aggregator.pp
View File

@ -26,10 +26,13 @@ $aggregator_port = 5565
$check_port = 5566
if $is_controller {
# On controllers make sure the LMA service is configured
# with the "pacemaker" provider
# On controllers make sure the Log and Metric collector services are
# configured with the "pacemaker" provider
include lma_collector::params
Service<| title == $lma_collector::params::service_name |> {
Service<| title == $lma_collector::params::log_service_name |> {
provider => 'pacemaker'
}
Service<| title == $lma_collector::params::metric_service_name |> {
provider => 'pacemaker'
}
}

144
deployment_scripts/puppet/manifests/base.pp
View File

@ -93,17 +93,27 @@ if $is_controller {
# On controller nodes the log parsing can generate a lot of http_metrics
# which can block heka (idle packs). It was observed that a poolsize set to 200
# solves the issue.
$poolsize = 200
$log_poolsize = 200
} else {
# For other nodes, the poolsize is set to 100 (the Heka default value)
$poolsize = 100
$log_poolsize = 100
}
class { 'lma_collector':
tags => $tags,
tags => $tags,
}
lma_collector::heka { 'log_collector':
user => $heka_user,
groups => $additional_groups,
poolsize => $poolsize,
poolsize => $log_poolsize,
require => Class['lma_collector'],
}
lma_collector::heka { 'metric_collector':
user => $heka_user,
groups => $additional_groups,
require => Class['lma_collector'],
}
# On controller nodes the LMA collector service is managed by Pacemaker, so we
@ -111,18 +121,21 @@ class { 'lma_collector':
# the "pacemaker" service provider
if $is_controller {
# TODO(all): remove this include from the manifest
include lma_collector::params
$service_name = $lma_collector::params::service_name
$config_dir = $lma_collector::params::config_dir
$log_service_name = $lma_collector::params::log_service_name
$metric_service_name = $lma_collector::params::metric_service_name
$log_config_dir = $lma_collector::params::log_config_dir
$metric_config_dir = $lma_collector::params::metric_config_dir
$rabbitmq_resource = 'master_p_rabbitmq-server'
if $fuel_version < 9.0 {
pacemaker_wrappers::service { $service_name:
pacemaker_wrappers::service { $log_service_name:
ensure => present,
prefix => false,
primitive_class => 'ocf',
primitive_type => 'ocf-lma_collector',
primitive_type => 'ocf-log_collector',
complex_type => 'clone',
use_handler => false,
ms_metadata => {
@ -133,9 +146,10 @@ if $is_controller {
'failure-timeout' => '120',
},
parameters => {
'config' => $config_dir,
'log_file' => "/var/log/${service_name}.log",
'user' => $heka_user,
'service_name' => $log_service_name,
'config' => $log_config_dir,
'log_file' => "/var/log/${log_service_name}.log",
'user' => $heka_user,
},
operations => {
'monitor' => {
@ -152,34 +166,68 @@ if $is_controller {
ocf_script_file => 'lma_collector/ocf-lma_collector',
}
cs_rsc_colocation { "${service_name}-with-rabbitmq":
cs_rsc_colocation { "${log_service_name}-with-rabbitmq":
ensure => present,
alias => $service_name,
primitives => ["clone_${service_name}", $rabbitmq_resource],
alias => $log_service_name,
primitives => ["clone_${log_service_name}", $rabbitmq_resource],
score => 0,
require => Pacemaker_wrappers::Service[$service_name],
require => Pacemaker_wrappers::Service[$log_service_name],
}
cs_rsc_order { "${service_name}-after-rabbitmq":
cs_rsc_order { "${log_service_name}-after-rabbitmq":
ensure => present,
alias => $service_name,
alias => $log_service_name,
first => $rabbitmq_resource,
second => "clone_${service_name}",
second => "clone_${log_service_name}",
# Heka cannot start if RabbitMQ isn't ready to accept connections. But
# once it is initialized, it can recover from a RabbitMQ outage. This is
# why we set score to 0 (interleave) meaning that the collector should
# start once RabbitMQ is active but a restart of RabbitMQ
# won't trigger a restart of the LMA collector.
score => 0,
require => Cs_rsc_colocation[$service_name],
require => Cs_rsc_colocation[$log_service_name],
before => Class['lma_collector'],
}
pacemaker_wrappers::service { $metric_service_name:
ensure => present,
prefix => false,
primitive_class => 'ocf',
primitive_type => 'ocf-metric_collector',
complex_type => 'clone',
use_handler => false,
ms_metadata => {
# The resource can start at any time
'interleave' => false,
'migration-threshold' => '3',
'failure-timeout' => '120',
},
parameters => {
'service_name' => $metric_service_name,
'config' => $metric_config_dir,
'log_file' => "/var/log/${metric_service_name}.log",
'user' => $heka_user,
},
operations => {
'monitor' => {
'timeout' => '10',
},
'start' => {
'timeout' => '30',
},
'stop' => {
'timeout' => '30',
},
},
ocf_script_file => 'lma_collector/ocf-lma_collector',
}
} else {
pacemaker::service { $service_name:
pacemaker::service { $log_service_name:
ensure => present,
prefix => false,
primitive_class => 'ocf',
primitive_type => 'ocf-lma_collector',
primitive_type => 'ocf-log_collector',
use_handler => false,
complex_type => 'clone',
complex_metadata => {
@ -190,9 +238,10 @@ if $is_controller {
'failure-timeout' => '120',
},
parameters => {
'config' => $config_dir,
'log_file' => "/var/log/${service_name}.log",
'user' => $heka_user,
'service_name' => $log_service_name,
'config' => $log_config_dir,
'log_file' => "/var/log/${log_service_name}.log",
'user' => $heka_user,
},
operations => {
'monitor' => {
@ -209,17 +258,51 @@ if $is_controller {
ocf_script_file => 'lma_collector/ocf-lma_collector',
}
pcmk_colocation { "${service_name}-with-rabbitmq":
pcmk_colocation { "${log_service_name}-with-rabbitmq":
ensure => present,
alias => $service_name,
alias => $log_service_name,
first => $rabbitmq_resource,
second => "clone_${service_name}",
second => "clone_${log_service_name}",
score => 0,
require => Pacemaker::Service[$service_name],
require => Pacemaker::Service[$log_service_name],
}
pacemaker::service { $metric_service_name:
ensure => present,
prefix => false,
primitive_class => 'ocf',
primitive_type => 'ocf-metric_collector',
use_handler => false,
complex_type => 'clone',
complex_metadata => {
# The resource can start at any time
'interleave' => false,
'migration-threshold' => '3',
'failure-timeout' => '120',
},
parameters => {
'service_name' => $metric_service_name,
'config' => $metric_config_dir,
'log_file' => "/var/log/${metric_service_name}.log",
'user' => $heka_user,
},
operations => {
'monitor' => {
'timeout' => '10',
},
'start' => {
'timeout' => '30',
},
'stop' => {
'timeout' => '30',
},
},
ocf_script_file => 'lma_collector/ocf-lma_collector',
}
}
}
$influxdb_mode = $lma_collector['influxdb_mode']
if $elasticsearch_mode != 'disabled' {
class { 'lma_collector::logs::system':
require => Class['lma_collector'],
@ -238,7 +321,6 @@ if $elasticsearch_mode != 'disabled' {
}
}
$influxdb_mode = $lma_collector['influxdb_mode']
case $influxdb_mode {
'remote','local': {
if $influxdb_mode == 'remote' {
@ -281,10 +363,6 @@ case $influxdb_mode {
require => Class['lma_collector'],
}
class { 'lma_collector::metrics::heka_monitoring':
require => Class['lma_collector']
}
}
'disabled': {
# Nothing to do

9
deployment_scripts/puppet/manifests/cinder.pp
View File

@ -22,10 +22,13 @@ $roles = node_roles(hiera('nodes'), hiera('uid'))
$is_controller = member($roles, 'controller') or member($roles, 'primary-controller')
if $is_controller {
# On controllers make sure the LMA service is configured
# with the "pacemaker" provider
# On controllers make sure the Log and Metric collector services are
# configured with the "pacemaker" provider
include lma_collector::params
Service<| title == $lma_collector::params::service_name |> {
Service<| title == $lma_collector::params::log_service_name |> {
provider => 'pacemaker'
}
Service<| title == $lma_collector::params::metric_service_name |> {
provider => 'pacemaker'
}
}

9
deployment_scripts/puppet/manifests/configure_afd_filters.pp
View File

@ -26,10 +26,13 @@ if $alarms_definitions == undef {
}
if $is_controller {
# On controllers make sure the LMA service is configured
# with the "pacemaker" provider
# On controllers make sure the Log and Metric collector services are
# configured with the "pacemaker" provider
include lma_collector::params
Service<| title == $lma_collector::params::service_name |> {
Service<| title == $lma_collector::params::log_service_name |> {
provider => 'pacemaker'
}
Service<| title == $lma_collector::params::metric_service_name |> {
provider => 'pacemaker'
}
}

8
deployment_scripts/puppet/manifests/controller.pp
View File

@ -44,9 +44,13 @@ else {
$rabbitmq_user = 'nova'
}
# Make sure the LMA service is configured with the "pacemaker" provider
# Make sure the Log and Metric collector services are configured with the
# "pacemaker" provider
include lma_collector::params
Service<| title == $lma_collector::params::service_name |> {
Service<| title == $lma_collector::params::log_service_name |> {
provider => 'pacemaker'
}
Service<| title == $lma_collector::params::metric_service_name |> {
provider => 'pacemaker'
}

144
deployment_scripts/puppet/modules/heka/manifests/init.pp
View File

@ -12,15 +12,12 @@
# License for the specific language governing permissions and limitations
# under the License.
#
# == Class: heka
# == Define: heka
#
# Install and configure the core of the Heka service.
#
# === Parameters
#
# [*service_name*]
# The name of the service daemon (default: 'hekad').
#
# [*config_dir*]
# The directory where to store the configuration (default: '/etc/hekad').
#
@ -66,61 +63,115 @@
#
# Copyright 2015 Mirantis Inc, unless otherwise noted.
#
class heka (
$service_name = $heka::params::service_name,
$config_dir = $heka::params::config_dir,
$user = $heka::params::user,
$additional_groups = $heka::params::additional_groups,
$hostname = $heka::params::hostname,
$maxprocs = $heka::params::maxprocs,
$max_message_size = $heka::params::max_message_size,
$max_process_inject = $heka::params::max_process_inject,
$max_timer_inject = $heka::params::max_timer_inject,
define heka (
$config_dir = undef,
$user = undef,
$additional_groups = undef,
$hostname = undef,
$maxprocs = undef,
$max_message_size = undef,
$max_process_inject = undef,
$max_timer_inject = undef,
$poolsize = undef,
$pre_script = undef,
$internal_statistics = $heka::params::internal_statistics,
) inherits heka::params {
$internal_statistics = undef,
) {
include heka::params
if $poolsize {
validate_integer($poolsize)
}
$service_name = $title
if $user {
$heka_user = $user
} else {
$heka_user = $heka::params::user
}
if $config_dir {
$_config_dir = $config_dir
} else {
$_config_dir = $heka::params::config_dir
}
$run_as_root = $heka_user == 'root'
if $run_as_root {
$_run_as_root = $run_as_root
} else {
$_run_as_root = $heka::params::run_as_root
}
if $additional_groups {
$_additional_groups = $additional_groups
} else {
$_additional_groups = $heka::params::additional_groups
}
if $hostname {
$_hostname = $hostname
} else {
$_hostname = $heka::params::hostname
}
if $maxprocs {
$_maxprocs = $maxprocs
} else {
$_maxprocs = $heka::params::maxprocs
}
if $max_message_size {
$_max_message_size = $max_message_size
} else {
$_max_message_size = $heka::params::max_message_size
}
if $max_process_inject {
$_max_process_inject = $max_process_inject
} else {
$_max_process_inject = $heka::params::max_process_inject
}
if $max_timer_inject {
$_max_timer_inject = $max_timer_inject
} else {
$_max_timer_inject = $heka::params::max_timer_inject
}
$hekad_wrapper = "/usr/local/bin/${service_name}_wrapper"
$base_dir = "/var/cache/${service_name}"
$log_file = "/var/log/${service_name}.log"
$heka_user = $user
$run_as_root = $heka_user == 'root'
package { $heka::params::package_name:
ensure => latest,
alias => 'heka',
}
if $::osfamily == 'Debian' {
# Starting from Heka 0.10.0, the Debian package provides a SysV init
# script so we need to stop the service and disable it.
exec { 'stop_heka_daemon':
command => '/etc/init.d/heka stop',
onlyif => '/usr/bin/test -f /etc/init.d/heka',
require => Package['heka'],
notify => Exec['disable_heka_daemon']
if ! defined(Package[$heka::params::package_name]) {
package { $heka::params::package_name:
ensure => latest,
alias => 'heka',
}
exec { 'disable_heka_daemon':
command => '/usr/sbin/update-rc.d heka disable',
refreshonly => true,
if $::osfamily == 'Debian' {
# Starting from Heka 0.10.0, the Debian package provides a SysV init
# script so we need to stop the service and disable it.
exec { 'stop_heka_daemon':
command => '/etc/init.d/heka stop',
onlyif => '/usr/bin/test -f /etc/init.d/heka',
require => Package['heka'],
notify => Exec['disable_heka_daemon']
}
exec { 'disable_heka_daemon':
command => '/usr/sbin/update-rc.d heka disable',
refreshonly => true,
}
}
}
# This Puppet User resource is used by other manifests even if the hekad
# process runs as 'root'.
user { $heka_user:
shell => $heka::params::nologin_bin,
home => $base_dir,
system => true,
groups => $additional_groups,
alias => 'heka',
before => Package['heka'],
if ! defined(User[$heka_user]) {
user { $heka_user:
shell => $heka::params::nologin_bin,
home => $base_dir,
system => true,
groups => $_additional_groups,
alias => 'heka',
before => Package['heka'],
}
}
file { $base_dir:
@ -131,7 +182,7 @@ class heka (
require => [User[$heka_user], Package['heka']],
}
file { $config_dir:
file { $_config_dir:
ensure => directory,
owner => $heka_user,
group => $heka_user,
@ -183,6 +234,7 @@ class heka (
group => 'root',
mode => '0755',
content => template('heka/hekad_wrapper.erb'),
require => Package['heka'],
}
case $::osfamily {
@ -191,7 +243,7 @@ class heka (
ensure => present,
content => template('heka/hekad.upstart.conf.erb'),
notify => Service[$service_name],
alias => 'heka_init_script',
alias => "${service_name}_heka_init_script",
require => File[$hekad_wrapper],
}
}
@ -202,7 +254,7 @@ class heka (
content => template('heka/hekad.initd.erb'),
mode => '0755',
notify => Service[$service_name],
alias => 'heka_init_script',
alias => "${service_name}_heka_init_script",
require => File[$hekad_wrapper],
}
}
@ -211,13 +263,13 @@ class heka (
}
}
file { "${config_dir}/global.toml":
file { "${_config_dir}/global.toml":
ensure => present,
content => template('heka/global.toml.erb'),
mode => '0600',
owner => $heka_user,
group => $heka_user,
require => File[$config_dir],
require => File[$_config_dir],
notify => Service[$service_name],
}

2
deployment_scripts/puppet/modules/heka/manifests/output/elasticsearch.pp
View File

@ -22,7 +22,7 @@ define heka::output::elasticsearch (
$flush_count = 10,
$use_buffering = true,
$max_buffer_size = 1024 * 1024 * 1024, # 1GiB
$queue_full_action = 'drop',
$queue_full_action = 'block',
$max_file_size = undef,
$ensure = present,
) {

3
deployment_scripts/puppet/modules/heka/manifests/params.pp
View File

@ -14,7 +14,6 @@
#
class heka::params {
$package_name = 'heka'
$service_name = 'hekad'
$user = 'heka'
$additional_groups = []
@ -25,7 +24,7 @@ class heka::params {
$max_timer_inject = undef
$internal_statistics = false
$config_dir = "/etc/${service_name}"
$config_dir = '/etc/hekad'
$share_dir = '/usr/share/heka'
$lua_modules_dir = '/usr/share/heka/lua_modules'

9
deployment_scripts/puppet/modules/heka/spec/classes/heka_spec.rb → deployment_scripts/puppet/modules/heka/spec/defines/heka_spec.rb
View File

@ -20,8 +20,9 @@ describe 'heka' do
end
describe 'with defaults' do
let(:title) { :log}
it { is_expected.to contain_user('heka') }
it { is_expected.to contain_file('/etc/init/hekad.conf') \
it { is_expected.to contain_file('/etc/init/log.conf') \
.with_content(/--chuid heka/) }
end
@ -29,9 +30,11 @@ describe 'heka' do
let(:params) do
{:user => 'root'}
end
let(:title) { :foo}
it { is_expected.to contain_user('root') }
it { is_expected.to contain_file('/etc/init/hekad.conf') }
it { is_expected.not_to contain_file('/etc/init/hekad.conf') \
it { is_expected.to contain_file('/etc/init/foo.conf') }
it { is_expected.not_to contain_file('/etc/init/foo.conf') \
.with_content(/--chuid/) }
end
end

18
deployment_scripts/puppet/modules/heka/templates/global.toml.erb
View File

@ -1,17 +1,17 @@
[hekad]
maxprocs=<%= @maxprocs %>
maxprocs=<%= @_maxprocs %>
base_dir="<%= @base_dir %>"
<% if @hostname %>
hostname="<%= @hostname %>"
<% if @_hostname %>
hostname="<%= @_hostname %>"
<% end %>
<% if @max_message_size -%>
max_message_size = <%= @max_message_size %>
<% if @_max_message_size -%>
max_message_size = <%= @_max_message_size %>
<% end -%>
<% if @max_process_inject -%>
max_process_inject = <%= @max_process_inject %>
<% if @_max_process_inject -%>
max_process_inject = <%= @_max_process_inject %>
<% end -%>
<% if @max_timer_inject -%>
max_timer_inject = <%= @max_timer_inject %>
<% if @_max_timer_inject -%>
max_timer_inject = <%= @_max_timer_inject %>
<% end -%>
<% if @poolsize -%>
poolsize = <%= @poolsize %>

4
deployment_scripts/puppet/modules/heka/templates/hekad.initd.erb
View File

@ -26,12 +26,12 @@ HEKA_USER=root
start() {
[ -x $exec ] || exit 5
[ -f $CONF_FILE ] || exit 6
<% unless @run_as_root -%>
<% unless @_run_as_root -%>
touch <%= @log_file %>
chown <%= @heka_user %>:<%= @heka_user %> <%= @log_file %>
<% end -%>
echo -n $"Starting $prog: "
daemonize -p $pidfile -e <%= @log_file %> <%= @run_as_root ? "" : "-u #{ @heka_user }" %> -l $lockfile $exec
daemonize -p $pidfile -e <%= @log_file %> <%= @_run_as_root ? "" : "-u #{ @heka_user }" %> -l $lockfile $exec
retval=$?
[ $retval -eq 0 ] && success || failure
echo

4
deployment_scripts/puppet/modules/heka/templates/hekad.upstart.conf.erb
View File

@ -7,7 +7,7 @@ stop on runlevel [!2345]
respawn
<% unless @run_as_root -%>
<% unless @_run_as_root -%>
pre-start script
touch <%= @log_file %>
chown <%= @heka_user %>:<%= @heka_user %> <%= @log_file %>
@ -17,5 +17,5 @@ end script
script
# https://bugs.launchpad.net/lma-toolchain/+bug/1543289
ulimit -n 102400
exec start-stop-daemon --start <%= @run_as_root ? "" : " --chuid #{ @heka_user }" %> --exec <%= @hekad_wrapper %> 2>><%= @log_file %>
exec start-stop-daemon --start <%= @_run_as_root ? "" : " --chuid #{ @heka_user }" %> --exec <%= @hekad_wrapper %> 2>><%= @log_file %>
end script

34
deployment_scripts/puppet/modules/lma_collector/README.md
View File

@ -24,10 +24,18 @@ The following versions of Heka and collectd are known to work for LMA:
### Setup
To install and configure the main components, declare the `lma_collector`
class:
class and `lma_collector::heka` defines:
```puppet
class { 'lma_collector': }
lma_collector::heka { 'log_collector':
require => Class['lma_collector'],
}
lma_collector::heka { 'metric_collector':
require => Class['lma_collector'],
}
```
This installs Heka and configures it with Heka plugins necessary for LMA.
@ -464,14 +472,24 @@ Private Classes:
#### Class: `lma_collector`
Main class. Install and configure the main components of the LMA collector.
Install the common Lua modules used by LMA collectors.
##### Parameters
* `tags`: *Optional*. Fields added to Heka messages. Valid options: a hash. Default: `{}`.
#### Define: `lma_collector::heka`
Main Define. Install and configure the Log and Metric collector.
The title must be either `log_collector` or `metric_collector`.
##### Parameters
* `user`: *Optional*. User the Heka service is run as. You may have to use `'root'` on some systems for the Heka service to be able to access log files, run additional commands, ... Valid options: a string. Default: `'heka'`.
* `groups`: *Optional*. Additional groups to add to the user running the Heka service. Ignored if the Heka service is run as "root". Valid options: an array of strings. Default: `['syslog', 'adm']`.
* `poolsize`: *Optional*. The pool size of maximum messages that can exist (default: 100).
* `heka_monitoring`: *Optional*. Enable the hekad plugins monitoring by configuring
the Heka dashboard and a filter plugin. (default: true, valid option: boolean).
#### Class: `lma_collector::elasticsearch`
@ -865,15 +883,19 @@ Declare this class to configure the GSE cluster policies on the aggregator node.
in the [Cluster Policies](http://fuel-plugin-lma-collector.readthedocs.org/en/latest/alarms.html#cluster-policies)
documentation. Valid options: a hash.
#### Class: `lma_collector::metrics::heka_monitoring`
#### Class: `lma_collector::metrics::heka_monitoring`
Declare this class to collect metrics for the Heka service itself.
Declare this class to collect metrics for the Heka services themselves.
##### Parameters
* `dashboard_address`: *Optional*. The address the Heka dashboard listens on.
* `dashboard_address`: *Optional*. The address Heka dashboards listen on.
Valid options: a string. Default: `127.0.0.1`.
* `dashboard_port`: *Optional*. The port the Heka dashboard listens on.
* `metric_dashboard_port`: *Optional*. The port the Heka dashboard of
metric collector listens on.
Valid options: a string. Default: `4353`.
* `log_dashboard_port`: *Optional*. The port the Heka dashboard of
log collector listens on.
Valid options: a string. Default: `4352`.
#### Class: `lma_collector::metrics::service_heartbeat`

81
deployment_scripts/puppet/modules/lma_collector/files/ocf-lma_collector
View File

@ -1,5 +1,5 @@
#!/bin/bash
# Copyright 2015 Mirantis, Inc.
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
@ -19,7 +19,6 @@
# OCF_RESKEY_binary
# OCF_RESKEY_config
# OCF_RESKEY_log_file
# OCF_RESKEY_pid
# OCF_RESKEY_user
# OCF_RESKEY_watchdog_file
# OCF_RESKEY_watchdog_timeout
@ -33,20 +32,12 @@
# Fill in some defaults if no values are specified
SERVICE_NAME="LMA collector"
OCF_RESKEY_binary_default="/usr/bin/hekad"
OCF_RESKEY_config_default="/etc/lma_collector"
OCF_RESKEY_log_file_default="/var/log/lma_collector.log"
OCF_RESKEY_pid_default="${HA_RSCTMP}/${__SCRIPT_NAME}/${__SCRIPT_NAME}.pid"
OCF_RESKEY_user_default="root"
OCF_RESKEY_watchdog_file_default=
OCF_RESKEY_watchdog_timeout_default=20
: ${OCF_RESKEY_binary=${OCF_RESKEY_binary_default}}
: ${OCF_RESKEY_config=${OCF_RESKEY_config_default}}
: ${OCF_RESKEY_log_file=${OCF_RESKEY_log_file_default}}
: ${OCF_RESKEY_pid=${OCF_RESKEY_pid_default}}
: ${OCF_RESKEY_user=${OCF_RESKEY_user_default}}
: ${OCF_RESKEY_watchdog_file=${OCF_RESKEY_watchdog_file_default}}
: ${OCF_RESKEY_watchdog_timeout=${OCF_RESKEY_watchdog_timeout_default}}
@ -57,14 +48,14 @@ usage() {
cat <<UEND
usage: $0 (start|stop|validate-all|meta-data|status|monitor)
$0 manages the ${SERVICE_NAME} process as an HA resource
$0 manages the collector process as an HA resource
The 'start' operation starts the ${SERVICE_NAME}
The 'stop' operation stops the ${SERVICE_NAME}
The 'start' operation starts the collector
The 'stop' operation stops the collector
The 'validate-all' operation reports whether the parameters are valid
The 'meta-data' operation reports this RA's meta-data information
The 'status' operation reports whether the ${SERVICE_NAME} is running
The 'monitor' operation reports whether the ${SERVICE_NAME} is running
The 'status' operation reports whether the collector is running
The 'monitor' operation reports whether the collector is running
UEND
}
@ -79,9 +70,17 @@ meta_data() {
<longdesc lang="en">
Manages the LMA collector daemon as a Pacemaker Resource.
</longdesc>
<shortdesc lang="en">Manages LMA collector</shortdesc>
<shortdesc lang="en">Manages Log or Metric collector</shortdesc>
<parameters>
<parameter name="service_name" unique="0" required="1">
<longdesc lang="en">
Name of the collector service.
</longdesc>
<shortdesc lang="en">Collector service name</shortdesc>
<content type="string" />
</parameter>
<parameter name="binary" unique="0" required="0">
<longdesc lang="en">
Path of the LMA collector binary file that will be run.
@ -90,28 +89,20 @@ Path of the LMA collector binary file that will be run.
<content type="string" default="${OCF_RESKEY_binary_default}" />
</parameter>
<parameter name="config" unique="0" required="0">
<parameter name="config" unique="0" required="1">
<longdesc lang="en">
Path to the LMA collector configuration file or directory
</longdesc>
<shortdesc lang="en">LMA collector configuration</shortdesc>
<content type="string" default="${OCF_RESKEY_config_default}" />
<content type="string" />
</parameter>
<parameter name="log_file" unique="0" required="0">
<parameter name="log_file" unique="0" required="1">
<longdesc lang="en">
Path to the LMA collector log file
</longdesc>
<shortdesc lang="en">LMA collector log file</shortdesc>
<content type="string" default="${OCF_RESKEY_log_file_default}" />
</parameter>
<parameter name="pid" unique="0" required="0">
<longdesc lang="en">
The pid file to use for the LMA collector service
</longdesc>
<shortdesc lang="en">LMA collector pid file</shortdesc>
<content type="string" default="${OCF_RESKEY_pid_default}" />
<content type="string" />
</parameter>
<parameter name="user" unique="0" required="0">
@ -159,6 +150,8 @@ END
service_validate() {
local rc
PID_FILE="${HA_RSCTMP}/${__SCRIPT_NAME}/${OCF_RESKEY_service_name}.pid"
check_binary "$OCF_RESKEY_binary"
if [[ ! -f $OCF_RESKEY_config && ! -d $OCF_RESKEY_config ]]; then
@ -182,7 +175,7 @@ service_status() {
# check and make PID file dir
local PID_DIR
PID_DIR=$( dirname "${OCF_RESKEY_pid}" )
PID_DIR=$( dirname "${PID_FILE}" )
if [ ! -d "${PID_DIR}" ] ; then
ocf_log debug "Create pid file dir: ${PID_DIR} and chown to ${OCF_RESKEY_user}"
mkdir -p "${PID_DIR}"
@ -190,11 +183,11 @@ service_status() {
chmod 755 "${PID_DIR}"
fi
if [ ! -f "$OCF_RESKEY_pid" ]; then
if [ ! -f "$PID_FILE" ]; then
ocf_log info "LMA collector is not running"
return "$OCF_NOT_RUNNING"
else
pid=$(cat "$OCF_RESKEY_pid")
pid=$(cat "$PID_FILE")
fi
if [ -n "${pid}" ]; then
@ -205,7 +198,7 @@ service_status() {
return "$OCF_NOT_RUNNING"
fi
else
ocf_log err "PID file ${OCF_RESKEY_pid} is empty!"
ocf_log err "PID file ${PID_FILE} is empty!"
return "$OCF_ERR_GENERIC"
fi
@ -241,7 +234,7 @@ service_start() {
service_monitor
rc=$?
if [ $rc -eq "$OCF_SUCCESS" ]; then
ocf_log info "${SERVICE_NAME} is already running"
ocf_log info "${OCF_RESKEY_service_name} is already running"
return "$OCF_SUCCESS"
fi
@ -249,7 +242,7 @@ service_start() {
ulimit -n 102400
su "${OCF_RESKEY_user}" -s /bin/sh -c "${OCF_RESKEY_binary} \
-config=${OCF_RESKEY_config} >> $OCF_RESKEY_log_file 2>&1"' & echo $!' > "$OCF_RESKEY_pid"
-config=${OCF_RESKEY_config} >> $OCF_RESKEY_log_file 2>&1"' & echo $!' > "$PID_FILE"
# Spin waiting for the server to come up
while true; do
@ -257,13 +250,13 @@ service_start() {
rc=$?
[ $rc -eq "$OCF_SUCCESS" ] && break
if [ $rc -ne "$OCF_NOT_RUNNING" ]; then
ocf_log err "${SERVICE_NAME} start failed"
ocf_log err "${OCF_RESKEY_service_name} start failed"
exit "$OCF_ERR_GENERIC"
fi
sleep 3
done
ocf_log info "${SERVICE_NAME} started"
ocf_log info "${OCF_RESKEY_service_name} started"
return "$OCF_SUCCESS"
}
@ -274,16 +267,16 @@ service_stop() {
service_monitor
rc=$?
if [ $rc -eq "$OCF_NOT_RUNNING" ]; then
ocf_log info "${SERVICE_NAME} is already stopped"
ocf_log info "${OCF_RESKEY_service_name} is already stopped"
return "$OCF_SUCCESS"
fi
# Try SIGTERM
pid=$(cat "$OCF_RESKEY_pid")
pid=$(cat "$PID_FILE")
ocf_run kill -s TERM "$pid"
rc=$?
if [ $rc -ne 0 ]; then
ocf_log err "${SERVICE_NAME} couldn't be stopped"
ocf_log err "${OCF_RESKEY_service_name} couldn't be stopped"
exit "$OCF_ERR_GENERIC"
fi
@ -301,21 +294,21 @@ service_stop() {
fi
count=$(( count + 1))
sleep 1
ocf_log debug "${SERVICE_NAME} still hasn't stopped yet. Waiting ..."
ocf_log debug "${OCF_RESKEY_service_name} still hasn't stopped yet. Waiting ..."
done
service_monitor
rc=$?
if [ "${rc}" -ne "${OCF_NOT_RUNNING}" ]; then
# SIGTERM didn't help either, try SIGKILL
ocf_log info "${SERVICE_NAME} failed to stop after ${shutdown_timeout}s using SIGTERM. Trying SIGKILL ..."
ocf_log info "${OCF_RESKEY_service_name} failed to stop after ${shutdown_timeout}s using SIGTERM. Trying SIGKILL ..."
ocf_run kill -s KILL "${pid}"
fi
ocf_log info "${SERVICE_NAME} stopped"
ocf_log info "${OCF_RESKEY_service_name} stopped"
ocf_log debug "Delete pid file: ${OCF_RESKEY_pid} with content $(cat "${OCF_RESKEY_pid}")"
rm -f "${OCF_RESKEY_pid}"
ocf_log debug "Delete pid file: ${PID_FILE} with content $(cat "${PID_FILE}")"
rm -f "${PID_FILE}"
return "${OCF_SUCCESS}"
}

8
deployment_scripts/puppet/modules/lma_collector/manifests/afd/api.pp
View File

@ -18,19 +18,19 @@ class lma_collector::afd::api () {
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::filter::sandbox { 'afd_api_backends':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/afd_api_backends.lua",
message_matcher => '(Type == \'metric\' || Type == \'heka.sandbox.metric\') && Fields[name] == \'haproxy_backend_servers\'',
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::filter::sandbox { 'afd_api_endpoints':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/afd_api_endpoints.lua",
message_matcher => join(['(Type == \'metric\' || Type == \'heka.sandbox.metric\') &&',
' (Fields[name] =~ /^openstack.*check_api$/ || Fields[name] == \'http_check\')'], ''),
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}

4
deployment_scripts/puppet/modules/lma_collector/manifests/afd/workers.pp
View File

@ -23,10 +23,10 @@ class lma_collector::afd::workers () {
], '')
heka::filter::sandbox { 'afd_workers':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/afd_workers.lua",
message_matcher => $metrics_matcher,
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}

8
deployment_scripts/puppet/modules/lma_collector/manifests/afd_filter.pp
View File

@ -22,7 +22,7 @@ define lma_collector::afd_filter (
$message_matcher,
) {
include lma_collector::params
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -34,12 +34,12 @@ define lma_collector::afd_filter (
file { $afd_filename:
ensure => present,
content => template('lma_collector/lma_alarms.lua.erb'),
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
# Create the confguration file for Heka
heka::filter::sandbox { "afd_${type}_${cluster_name}_${logical_name}":
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/afd.lua",
message_matcher => "(Type == \'metric\' || Type == \'heka.sandbox.metric\') && (${message_matcher})",
ticker_interval => 10,
@ -52,7 +52,7 @@ define lma_collector::afd_filter (
},
module_directory => $lua_modules_dir,
require => File[$afd_filename],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}

35
deployment_scripts/puppet/modules/lma_collector/manifests/afd_nagios.pp
View File

@ -22,7 +22,7 @@ define lma_collector::afd_nagios(
$message_type = 'afd_node_metric',
){
include lma_collector::params
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -33,29 +33,30 @@ define lma_collector::afd_nagios(
$config = {'nagios_host' => $hostname, 'service_template' => $service_template}
heka::encoder::sandbox { "nagios_afd_${title}":
ensure => $ensure,
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/encoders/status_nagios.lua",
config => $config,
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::output::http { "nagios_afd_${title}":
ensure => $ensure,
config_dir => $lma_collector::params::config_dir,
url => $url,
message_matcher => "Fields[${lma_collector::params::aggregator_flag}] == NIL && Type == 'heka.sandbox.${message_type}'",
username => $user,
password => $password,
encoder => "nagios_afd_${title}",
timeout => $lma_collector::params::nagios_timeout,
headers => {
ensure => $ensure,
config_dir => $lma_collector::params::metric_config_dir,
url => $url,
message_matcher => "Fields[${lma_collector::params::aggregator_flag}] == NIL && Type == 'heka.sandbox.${message_type}'",
username => $user,
password => $password,
encoder => "nagios_afd_${title}",
timeout => $lma_collector::params::nagios_timeout,
headers => {
'Content-Type' => 'application/x-www-form-urlencoded'
},
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_tiny_size,
max_file_size => $lma_collector::params::buffering_max_file_tiny_size,
require => Heka::Encoder::Sandbox["nagios_afd_${title}"],
notify => Class['lma_collector::service'],
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size_for_nagios,
max_file_size => $lma_collector::params::buffering_max_file_size_for_nagios,
queue_full_action => $lma_collector::params::queue_full_action_for_nagios,
require => Heka::Encoder::Sandbox["nagios_afd_${title}"],
notify => Class['lma_collector::service::metric'],
}
}

21
deployment_scripts/puppet/modules/lma_collector/manifests/aggregator/client.pp
View File

@ -16,7 +16,7 @@ class lma_collector::aggregator::client (
$port = $lma_collector::params::aggregator_port,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::metric
if $address == undef {
fail('address parameter should be defined!')
@ -24,16 +24,15 @@ class lma_collector::aggregator::client (
validate_string($address)
$config_dir = $lma_collector::params::config_dir
heka::output::tcp { 'aggregator':
config_dir => $config_dir,
address => $address,
port => $port,
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size,
max_file_size => $lma_collector::params::buffering_max_file_size,
message_matcher => $lma_collector::params::aggregator_client_message_matcher,
notify => Class['lma_collector::service'],
config_dir => $lma_collector::params::metric_config_dir,
address => $address,
port => $port,
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size_for_aggregator,
max_file_size => $lma_collector::params::buffering_max_file_size_for_aggregator,
message_matcher => $lma_collector::params::aggregator_client_message_matcher,
queue_full_action => $lma_collector::params::queue_full_action_for_aggregator,
notify => Class['lma_collector::service::metric'],
}
}

24
deployment_scripts/puppet/modules/lma_collector/manifests/aggregator/server.pp
View File

@ -17,55 +17,57 @@ class lma_collector::aggregator::server (
$listen_port = $lma_collector::params::aggregator_port,
$http_check_port = undef,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
validate_string($listen_address)
validate_integer($listen_port)
$config_dir = $lma_collector::params::metric_config_dir
heka::decoder::scribbler { 'aggregator_flag':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
config => {
"${lma_collector::params::aggregator_flag}" => 'present',
},
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::decoder::multidecoder { 'aggregator':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
subs => ['ProtobufDecoder', 'aggregator_flag_decoder'],
log_sub_errors => true,
cascade_strategy => 'all',
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::input::tcp { 'aggregator':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
address => $listen_address,
port => $listen_port,
decoder => 'aggregator',
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
if $http_check_port {
heka::decoder::sandbox { 'http-check':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/noop.lua" ,
config => {
msg_type => 'lma.http-check',
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::input::httplisten { 'http-check':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
address => $listen_address,
port => $http_check_port,
decoder => 'http-check',
require => Heka::Decoder::Sandbox['http-check'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}
}

11
deployment_scripts/puppet/modules/lma_collector/manifests/collectd/base.pp
View File

@ -20,7 +20,8 @@ class lma_collector::collectd::base (
$hostname = undef,
$purge = false,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::metric
$type_directory = "${lma_collector::params::plugins_dir}/collectd_types/"
$type_files = suffix(prefix($lma_collector::params::collectd_types, $type_directory), '.db')
@ -118,21 +119,21 @@ class lma_collector::collectd::base (
$real_hostname = $::hostname
}
heka::decoder::sandbox { 'collectd':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/collectd.lua" ,
config => {
hostname => $real_hostname
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::input::httplisten { 'collectd':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
address => '127.0.0.1',
port => $port,
decoder => 'collectd',
require => Heka::Decoder::Sandbox['collectd'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}

25
deployment_scripts/puppet/modules/lma_collector/manifests/elasticsearch.pp
View File

@ -16,27 +16,28 @@ class lma_collector::elasticsearch (
$server = $lma_collector::params::elasticsearch_server,
$port = $lma_collector::params::elasticsearch_port,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::log
validate_string($server)
heka::encoder::es_json { 'elasticsearch':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
index => '%{Type}-%{%Y.%m.%d}',
es_index_from_timestamp => true,
fields => $lma_collector::params::elasticsearch_fields,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::output::elasticsearch { 'elasticsearch':
config_dir => $lma_collector::params::config_dir,
server => $server,
port => $port,
message_matcher => 'Type == \'log\' || Type == \'notification\'',
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size,
max_file_size => $lma_collector::params::buffering_max_file_size,
require => Heka::Encoder::Es_json['elasticsearch'],
notify => Class['lma_collector::service'],
config_dir => $lma_collector::params::log_config_dir,
server => $server,
port => $port,
message_matcher => 'Type == \'log\' || Type == \'notification\'',
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size_for_log,
max_file_size => $lma_collector::params::buffering_max_file_size_for_log,
queue_full_action => $lma_collector::params::queue_full_action_for_log,
require => Heka::Encoder::Es_json['elasticsearch'],
notify => Class['lma_collector::service::log'],
}
}

6
deployment_scripts/puppet/modules/lma_collector/manifests/gse_cluster_filter.pp
View File

@ -24,7 +24,7 @@ define lma_collector::gse_cluster_filter (
$ensure = present,
) {
include lma_collector::params
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -51,7 +51,7 @@ define lma_collector::gse_cluster_filter (
], '')
heka::filter::sandbox { "gse_${title}":
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/gse_cluster_filter.lua",
message_matcher => $message_matcher,
ticker_interval => 1,
@ -69,7 +69,7 @@ define lma_collector::gse_cluster_filter (
},
module_directory => $lua_modules_dir,
require => File[$topology_file],
notify => Class['lma_collector::service']
notify => Class['lma_collector::service::metric']
}
file { $topology_file:

35
deployment_scripts/puppet/modules/lma_collector/manifests/gse_nagios.pp
View File

@ -23,7 +23,7 @@ define lma_collector::gse_nagios (
$virtual_hostname = undef,
) {
include lma_collector::params
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -44,29 +44,30 @@ define lma_collector::gse_nagios (
$config = {'nagios_host' => $_nagios_host, 'service_template' => $service_template}
heka::encoder::sandbox { "nagios_gse_${title}":
ensure => $ensure,
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/encoders/status_nagios.lua",
config => $config,
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::output::http { "nagios_gse_${title}":
ensure => $ensure,
config_dir => $lma_collector::params::config_dir,
url => $url,
message_matcher => "Type == 'heka.sandbox.${message_type}'",
username => $user,
password => $password,
encoder => "nagios_gse_${title}",
timeout => $lma_collector::params::nagios_timeout,
headers => {
ensure => $ensure,
config_dir => $lma_collector::params::metric_config_dir,
url => $url,
message_matcher => "Type == 'heka.sandbox.${message_type}'",
username => $user,
password => $password,
encoder => "nagios_gse_${title}",
timeout => $lma_collector::params::nagios_timeout,
headers => {
'Content-Type' => 'application/x-www-form-urlencoded'
},
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_tiny_size,
max_file_size => $lma_collector::params::buffering_max_file_tiny_size,
require => Heka::Encoder::Sandbox["nagios_gse_${title}"],
notify => Class['lma_collector::service'],
use_buffering => $lma_collector::params::buffering_enabled,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size_for_nagios,
max_file_size => $lma_collector::params::buffering_max_file_size_for_nagios,
queue_full_action => $lma_collector::params::queue_full_action_for_nagios,
require => Heka::Encoder::Sandbox["nagios_gse_${title}"],
notify => Class['lma_collector::service::metric'],
}
}

4
deployment_scripts/puppet/modules/lma_collector/manifests/gse_policies.pp
View File

@ -15,7 +15,7 @@ class lma_collector::gse_policies (
$policies
) {
include lma_collector::params
include lma_collector::service
include lma_collector::service::metric
validate_hash($policies)
@ -25,6 +25,6 @@ class lma_collector::gse_policies (
ensure => present,
path => $gse_policies_path,
content => template('lma_collector/gse_policies.lua.erb'),
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}

121
deployment_scripts/puppet/modules/lma_collector/manifests/heka.pp Normal file
View File

@ -0,0 +1,121 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# == Define: lma_collector::heka
#
# The lma_collector::heka resource installs and configures heka service
#
# === Parameters
#
# === Examples
#
# === Authors
#
# Simon Pasquier <spasquier@mirantis.com>
# Swann Croiset <scroiset@mirantis.com>
#
# === Copyright
#
# Copyright 2016 Mirantis Inc., unless otherwise noted.
#
define lma_collector::heka (
$user = 'heka',
$groups = [],
$heka_monitoring = true,
$poolsize = 100,
) {
include lma_collector::params
validate_array($groups)
validate_bool($heka_monitoring)
validate_integer($poolsize)
if ! member(['log_collector', 'metric_collector'], $title){
fail('lma_collector::heka title must be either "log_collector" or "metric_collector"')
}
$lua_modules_dir = $lma_collector::params::lua_modules_dir
$additional_groups = $user ? {
'root' => [],
default => union($lma_collector::params::groups, $groups),
}
if $title == 'metric_collector' {
$config_dir = $lma_collector::params::metric_config_dir
$service_class = 'lma_collector::service::metric'
$dashboard_port = $lma_collector::params::metric_dashboard_port
heka::input::tcp { 'metric':
config_dir => $config_dir,
address => $lma_collector::params::metric_input_address,
port => $lma_collector::params::metric_input_port,
decoder => 'ProtobufDecoder',
require => ::Heka[$title],
notify => Class[$service_class],
}
} elsif $title == 'log_collector' {
$config_dir = $lma_collector::params::log_config_dir
$service_class = 'lma_collector::service::log'
$dashboard_port = $lma_collector::params::log_dashboard_port
heka::output::tcp { 'metric':
config_dir => $config_dir,
address => $lma_collector::params::metric_input_address,
port => $lma_collector::params::metric_input_port,
message_matcher => '(Type == \'metric\' || Type == \'heka.sandbox.metric\' || Type == \'heka.sandbox.bulk_metric\')',
max_buffer_size => $lma_collector::params::buffering_max_buffer_log_metric_size,
max_file_size => $lma_collector::params::buffering_max_file_log_metric_size,
queue_full_action => $lma_collector::params::queue_full_action_log_metric,
require => ::Heka[$title],
notify => Class[$service_class],
}
}
::heka { $title:
config_dir => $config_dir,
user => $user,
additional_groups => $additional_groups,
hostname => $::hostname,
max_message_size => $lma_collector::params::hekad_max_message_size,
max_process_inject => $lma_collector::params::hekad_max_process_inject,
max_timer_inject => $lma_collector::params::hekad_max_timer_inject,
poolsize => $poolsize,
}
# Heka self-monitoring
if $heka_monitoring {
heka::filter::sandbox { "heka_monitoring_${title}":
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/heka_monitoring.lua",
message_matcher => "Type == 'heka.all-report'",
require => ::Heka[$title],
module_directory => $lua_modules_dir,
notify => Class[$service_class],
}
# Dashboard is required to enable monitoring messages
heka::output::dashboard { "dashboard_${title}":
config_dir => $config_dir,
dashboard_address => $lma_collector::params::dashboard_address,
dashboard_port => $dashboard_port,
require => ::Heka[$title],
notify => Class[$service_class],
}
}
}

39
deployment_scripts/puppet/modules/lma_collector/manifests/influxdb.pp
View File

@ -21,7 +21,7 @@ class lma_collector::influxdb (
$tag_fields = $lma_collector::params::influxdb_tag_fields,
$time_precision = $lma_collector::params::influxdb_time_precision,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -32,7 +32,7 @@ class lma_collector::influxdb (
validate_array($tag_fields)
heka::filter::sandbox { 'influxdb_accumulator':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/influxdb_accumulator.lua",
message_matcher => $lma_collector::params::influxdb_message_matcher,
ticker_interval => 1,
@ -46,39 +46,40 @@ class lma_collector::influxdb (
# access to the tenant name and user name for services
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::filter::sandbox { 'influxdb_annotation':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/influxdb_annotation.lua",
message_matcher => 'Type == \'heka.sandbox.gse_cluster_metric\'',
config => {
serie_name => $lma_collector::params::annotations_serie_name
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::encoder::payload { 'influxdb':
config_dir => $lma_collector::params::config_dir,
notify => Class['lma_collector::service'],
config_dir => $lma_collector::params::metric_config_dir,
notify => Class['lma_collector::service::metric'],
}
heka::output::http { 'influxdb':
config_dir => $lma_collector::params::config_dir,
url => "http://${server}:${port}/write?db=${database}&precision=${time_precision}",
message_matcher => 'Fields[payload_type] == \'txt\' && Fields[payload_name] == \'influxdb\'',
username => $user,
password => $password,
timeout => $lma_collector::params::influxdb_timeout,
headers => {
config_dir => $lma_collector::params::metric_config_dir,
url => "http://${server}:${port}/write?db=${database}&precision=${time_precision}",
message_matcher => 'Fields[payload_type] == \'txt\' && Fields[payload_name] == \'influxdb\'',
username => $user,
password => $password,
timeout => $lma_collector::params::influxdb_timeout,
headers => {
'Content-Type' => 'application/x-www-form-urlencoded'
},
use_buffering => $lma_collector::params::buffering_enabled,
max_file_size => $lma_collector::params::buffering_max_file_size,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size,
require => Heka::Encoder::Payload['influxdb'],
notify => Class['lma_collector::service'],
use_buffering => $lma_collector::params::buffering_enabled,
max_file_size => $lma_collector::params::buffering_max_file_size_for_metric,
max_buffer_size => $lma_collector::params::buffering_max_buffer_size_for_metric,
queue_full_action => $lma_collector::params::queue_full_action_for_metric,
require => Heka::Encoder::Payload['influxdb'],
notify => Class['lma_collector::service::metric'],
}
}

59
deployment_scripts/puppet/modules/lma_collector/manifests/init.pp
View File

@ -14,8 +14,8 @@
#
# == Class: lma_collector
#
# The lma_collector class is able to install the common components for running
# the Logging, Monitoring and Alerting collector service.
# The lma_collector class installs the common Lua modules used by
# the Logging, Monitoring and Alerting collector services.
#
# === Parameters
#
@ -24,57 +24,36 @@
# === Authors
#
# Simon Pasquier <spasquier@mirantis.com>
# Swann Croiset <scroiset@mirantis.com>
#
# === Copyright
#
# Copyright 2015 Mirantis Inc., unless otherwise noted.
# Copyright 2016 Mirantis Inc., unless otherwise noted.
#
class lma_collector (
$tags = $lma_collector::params::tags,
$user = undef,
$groups = [],
$poolsize = 100,
) inherits lma_collector::params {
$tags = {},
) {
include lma_collector::params
validate_hash($tags)
validate_integer($poolsize)
$service_name = $lma_collector::params::service_name
$config_dir = $lma_collector::params::config_dir
$plugins_dir = $lma_collector::params::plugins_dir
$lua_modules_dir = $lma_collector::params::lua_modules_dir
$additional_groups = $user ? {
'root' => [],
default => union($lma_collector::params::groups, $groups),
}
class { 'heka':
service_name => $service_name,
config_dir => $config_dir,
user => $user,
additional_groups => $additional_groups,
hostname => $::hostname,
internal_statistics => false,
max_message_size => $lma_collector::params::hekad_max_message_size,
max_process_inject => $lma_collector::params::hekad_max_process_inject,
max_timer_inject => $lma_collector::params::hekad_max_timer_inject,
poolsize => $poolsize,
}
file { $lua_modules_dir:
ensure => directory,
source => 'puppet:///modules/lma_collector/plugins/common',
recurse => remote,
notify => Class['lma_collector::service'],
require => File[$plugins_dir]
notify => [Class['lma_collector::service::metric'],
Class['lma_collector::service::log']],
}
file { "${lua_modules_dir}/extra_fields.lua":
ensure => present,
content => template('lma_collector/extra_fields.lua.erb'),
require => File[$lua_modules_dir],
notify => Class['lma_collector::service'],
notify => [Class['lma_collector::service::metric'],
Class['lma_collector::service::log']],
}
file { $plugins_dir:
@ -85,7 +64,8 @@ class lma_collector (
ensure => directory,
source => 'puppet:///modules/lma_collector/plugins/decoders',
recurse => remote,
notify => Class['lma_collector::service'],
notify => [Class['lma_collector::service::metric'],
Class['lma_collector::service::log']],
require => File[$plugins_dir]
}
@ -93,7 +73,8 @@ class lma_collector (
ensure => directory,
source => 'puppet:///modules/lma_collector/plugins/filters',
recurse => remote,
notify => Class['lma_collector::service'],
notify => [Class['lma_collector::service::metric'],
Class['lma_collector::service::log']],
require => File[$plugins_dir]
}
@ -101,7 +82,8 @@ class lma_collector (
ensure => directory,
source => 'puppet:///modules/lma_collector/plugins/encoders',
recurse => remote,
notify => Class['lma_collector::service'],
notify => [Class['lma_collector::service::metric'],
Class['lma_collector::service::log']],
require => File[$plugins_dir]
}
@ -109,7 +91,8 @@ class lma_collector (
ensure => directory,
source => 'puppet:///modules/lma_collector/plugins/outputs',
recurse => remote,
notify => Class['lma_collector::service'],
notify => [Class['lma_collector::service::metric'],
Class['lma_collector::service::log']],
require => File[$plugins_dir]
}
@ -118,8 +101,4 @@ class lma_collector (
ensure => present,
}
}
class { 'lma_collector::service':
require => Class['heka'],
}
}

6
deployment_scripts/puppet/modules/lma_collector/manifests/logs/counter.pp
View File

@ -17,12 +17,12 @@ class lma_collector::logs::counter (
$interval = 60,
) {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::filter::sandbox { 'logs_counter':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/logs_counter.lua",
message_matcher => 'Type == \'log\' && Logger =~ /^openstack\\./',
ticker_interval => 1,
@ -32,6 +32,6 @@ class lma_collector::logs::counter (
hostname => $hostname,
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

7
deployment_scripts/puppet/modules/lma_collector/manifests/logs/http_metrics.pp
View File

@ -13,16 +13,17 @@
# under the License.
#
class lma_collector::logs::http_metrics {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::filter::sandbox { 'http_metrics':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/http_metrics.lua",
message_matcher => 'Type == \'log\' && Fields[http_response_time] != NIL',
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

10
deployment_scripts/puppet/modules/lma_collector/manifests/logs/keystone_wsgi.pp
View File

@ -19,28 +19,28 @@
class lma_collector::logs::keystone_wsgi (
$log_directory = $lma_collector::params::apache_log_directory,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::log
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::decoder::sandbox { 'keystone_wsgi':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/keystone_wsgi_log.lua",
config => {
apache_log_pattern => $lma_collector::params::apache_log_pattern,
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::input::logstreamer { 'keystone_wsgi':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
decoder => 'keystone_wsgi',
log_directory => $log_directory,
file_match => 'keystone_wsgi_(?P<Service>.+)_access\.log\.?(?P<Seq>\d*)$',
differentiator => "['keystone-wsgi-', 'Service']",
priority => '["^Seq"]',
require => Heka::Decoder::Sandbox['keystone_wsgi'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

13
deployment_scripts/puppet/modules/lma_collector/manifests/logs/libvirt.pp
View File

@ -15,8 +15,11 @@
# Class lma_collector::logs::libvirt
class lma_collector::logs::libvirt {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$config_dir = $lma_collector::params::log_config_dir
$libvirt_dir = '/var/log/libvirt'
$libvirt_log = 'libvirtd.log'
@ -42,18 +45,18 @@ class lma_collector::logs::libvirt {
}
heka::decoder::sandbox { 'libvirt':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/libvirt_log.lua",
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::input::logstreamer { 'libvirt':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
log_directory => $libvirt_dir,
file_match => $libvirt_log,
decoder => 'libvirt',
differentiator => '["libvirt"]',
require => Heka::Decoder::Sandbox['libvirt'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

12
deployment_scripts/puppet/modules/lma_collector/manifests/logs/mysql.pp
View File

@ -14,26 +14,28 @@
#
class lma_collector::logs::mysql {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$config_dir = $lma_collector::params::log_config_dir
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::decoder::sandbox { 'mysql':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/mysql_log.lua" ,
config => {
syslog_pattern => $lma_collector::params::syslog_pattern
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::input::logstreamer { 'mysql':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
decoder => 'mysql',
file_match => 'mysqld\.log$',
differentiator => '[\'mysql\']',
require => Heka::Decoder::Sandbox['mysql'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

6
deployment_scripts/puppet/modules/lma_collector/manifests/logs/openstack.pp
View File

@ -30,11 +30,11 @@ define lma_collector::logs::openstack (
# Note: $log_directory could be made configurable in the future.
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
include lma_collector::logs::openstack_decoder_splitter
heka::input::logstreamer { $title:
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
log_directory => "/var/log/${title}",
decoder => 'openstack',
splitter => 'openstack',
@ -42,6 +42,6 @@ define lma_collector::logs::openstack (
differentiator => "['${title}', '_', 'Service']",
priority => '["^Seq"]',
require => Class['lma_collector::logs::openstack_decoder_splitter'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

10
deployment_scripts/puppet/modules/lma_collector/manifests/logs/openstack_decoder_splitter.pp
View File

@ -21,20 +21,20 @@
#
class lma_collector::logs::openstack_decoder_splitter {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::decoder::sandbox { 'openstack':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/openstack_log.lua",
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::splitter::token { 'openstack':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::log_config_dir,
delimiter => '\n',
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

13
deployment_scripts/puppet/modules/lma_collector/manifests/logs/ovs.pp
View File

@ -13,25 +13,28 @@
# under the License.
#
class lma_collector::logs::ovs {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$config_dir = $lma_collector::params::log_config_dir
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::decoder::sandbox { 'ovs':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/ovs_log.lua",
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::input::logstreamer { 'ovs':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
decoder => 'ovs',
log_directory => '/var/log/openvswitch',
file_match => '(?P<Service>ovs\-vswitchd|ovsdb\-server)\.log$',
differentiator => '[ \'Service\' ]',
require => Heka::Decoder::Sandbox['ovs'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

12
deployment_scripts/puppet/modules/lma_collector/manifests/logs/pacemaker.pp
View File

@ -14,28 +14,30 @@
#
class lma_collector::logs::pacemaker {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$config_dir = $lma_collector::params::log_config_dir
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::decoder::sandbox { 'pacemaker':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/pacemaker_log.lua",
config => {
syslog_pattern => $lma_collector::params::syslog_pattern,
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
# Use the default splitter 'TokenSplitter' with 'newline' delimiter,
# because Pacemaker may log messages with and without the <PRI> preamble.
heka::input::logstreamer { 'pacemaker':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
decoder => 'pacemaker',
file_match => 'pacemaker\.log$',
differentiator => '[ \'pacemaker\' ]',
require => Heka::Decoder::Sandbox['pacemaker'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

16
deployment_scripts/puppet/modules/lma_collector/manifests/logs/rabbitmq.pp
View File

@ -14,32 +14,34 @@
#
class lma_collector::logs::rabbitmq {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$config_dir = $lma_collector::params::log_config_dir
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::decoder::sandbox { 'rabbitmq':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/rabbitmq.lua" ,
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::splitter::regex { 'rabbitmq':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
delimiter => '\n\n(=[^=]+====)',
delimiter_eol => false,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::input::logstreamer { 'rabbitmq':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
decoder => 'rabbitmq',
splitter => 'rabbitmq',
log_directory => '/var/log/rabbitmq',
file_match => 'rabbit@(?P<Node>.+)\.log$',
differentiator => '["rabbitmq.", "Node"]',
require => [Heka::Decoder::Sandbox['rabbitmq'], Heka::Splitter::Regex['rabbitmq']],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

11
deployment_scripts/puppet/modules/lma_collector/manifests/logs/swift.pp
View File

@ -49,30 +49,31 @@ class lma_collector::logs::swift (
$priority = undef,
$log_directory = $lma_collector::params::log_directory,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::log
$lua_modules_dir = $lma_collector::params::lua_modules_dir
$config_dir = $lma_collector::params::log_config_dir
# Note: syslog_pattern could be made configurable in the future.
heka::decoder::sandbox { 'swift':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/generic_syslog.lua",
config => {
syslog_pattern => $lma_collector::params::syslog_pattern,
fallback_syslog_pattern => $lma_collector::params::fallback_syslog_pattern
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::input::logstreamer { 'swift':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
decoder => 'swift',
log_directory => $log_directory,
file_match => $file_match,
differentiator => '[\'openstack.swift\']',
priority => $priority,
require => Heka::Decoder::Sandbox['swift'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

12
deployment_scripts/puppet/modules/lma_collector/manifests/logs/system.pp
View File

@ -14,27 +14,29 @@
#
class lma_collector::logs::system {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$config_dir = $lma_collector::params::log_config_dir
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::decoder::sandbox { 'system':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/generic_syslog.lua" ,
config => {
syslog_pattern => $lma_collector::params::syslog_pattern,
fallback_syslog_pattern => $lma_collector::params::fallback_syslog_pattern
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
heka::input::logstreamer { 'system':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
decoder => 'system',
file_match => '(?P<Service>daemon\.log|cron\.log|haproxy\.log|kern\.log|auth\.log|syslog|messages|debug)',
differentiator => '[ \'system.\', \'Service\' ]',
require => Heka::Decoder::Sandbox['system'],
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

38
deployment_scripts/puppet/modules/lma_collector/manifests/metrics/heka_monitoring.pp
View File

@ -1,38 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
class lma_collector::metrics::heka_monitoring (
$dashboard_address = $lma_collector::params::dashboard_address,
$dashboard_port = $lma_collector::params::dashboard_port,
){
include lma_collector::service
$lua_modules_dir = $lma_collector::params::lua_modules_dir
heka::filter::sandbox { 'heka_monitoring':
config_dir => $lma_collector::params::config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/heka_monitoring.lua",
message_matcher => "Type == 'heka.all-report'",
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
}
# Dashboard is required to enable monitoring messages
heka::output::dashboard { 'dashboard':
config_dir => $lma_collector::params::config_dir,
dashboard_address => $dashboard_address,
dashboard_port => $dashboard_port,
notify => Class['lma_collector::service'],
}
}

6
deployment_scripts/puppet/modules/lma_collector/manifests/metrics/service_heartbeat.pp
View File

@ -16,7 +16,7 @@ class lma_collector::metrics::service_heartbeat (
$services,
$timeout = $lma_collector::params::heartbeat_timeout,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -24,7 +24,7 @@ class lma_collector::metrics::service_heartbeat (
if (size($services) > 0) {
heka::filter::sandbox { 'service_heartbeat':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/service_heartbeat.lua",
message_matcher => join(['Fields[name] =~ /^', join(sort($services), '|'), '/'], ''),
ticker_interval => 10,
@ -32,7 +32,7 @@ class lma_collector::metrics::service_heartbeat (
timeout => $timeout,
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}
}

11
deployment_scripts/puppet/modules/lma_collector/manifests/notifications/input.pp
View File

@ -20,7 +20,7 @@ class lma_collector::notifications::input (
$port = $lma_collector::params::rabbitmq_port,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::log
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -35,14 +35,15 @@ class lma_collector::notifications::input (
# case it doesn't exist yet.
$exchange = 'nova'
$config_dir = $lma_collector::params::log_config_dir
heka::decoder::sandbox { 'notification':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/decoders/notification.lua" ,
config => {
include_full_notification => false
},
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
create_resources(
@ -62,7 +63,7 @@ class lma_collector::notifications::input (
},
},
{
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
decoder => 'notification',
user => $user,
password => $password,
@ -73,7 +74,7 @@ class lma_collector::notifications::input (
exchange_auto_delete => false,
queue_auto_delete => false,
exchange_type => 'topic',
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
)
}

12
deployment_scripts/puppet/modules/lma_collector/manifests/notifications/metrics.pp
View File

@ -14,25 +14,27 @@
#
class lma_collector::notifications::metrics {
include lma_collector::params
include lma_collector::service
include lma_collector::service::log
$config_dir = $lma_collector::params::log_config_dir
$lua_modules_dir = $lma_collector::params::lua_modules_dir
# Filter to compute resource's creation time metric
heka::filter::sandbox { 'resource_creation_time':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/resource_creation_time.lua",
message_matcher => 'Type == \'notification\' && Fields[event_type] =~ /^(compute.instance|volume).create.end$/',
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
# Filter to compute the instance state change metric
heka::filter::sandbox { 'instance_state':
config_dir => $lma_collector::params::config_dir,
config_dir => $config_dir,
filename => "${lma_collector::params::plugins_dir}/filters/instance_state.lua",
message_matcher => 'Type == \'notification\' && Fields[event_type] == \'compute.instance.update\' && Fields[state] != NIL',
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::log'],
}
}

44
deployment_scripts/puppet/modules/lma_collector/manifests/params.pp
View File

@ -13,10 +13,13 @@
# under the License.
#
class lma_collector::params {
$service_name = 'lma_collector'
$config_dir = "/etc/${service_name}"
$plugins_dir = "/usr/share/${service_name}"
$metric_service_name = 'metric_collector'
$log_service_name = 'log_collector'
$metric_config_dir = "/etc/${metric_service_name}"
$log_config_dir = "/etc/${log_service_name}"
$lua_modules_dir = '/usr/share/lma_collector_modules'
# Lua plugins are shared across log and metric collectors
$plugins_dir = '/usr/share/lma_collector'
$apt_config_file = '/etc/apt/apt.conf.d/99norecommends'
@ -24,7 +27,12 @@ class lma_collector::params {
# Address and port of the Heka dashboard for health reports.
$dashboard_address = '127.0.0.1'
$dashboard_port = '4352'
$log_dashboard_port = '4352'
$metric_dashboard_port = '4353'
# Address and port of the metric input
$metric_input_address = '127.0.0.1'
$metric_input_port = 5567
$aggregator_address = '127.0.0.1'
$aggregator_port = 5565
@ -35,8 +43,10 @@ class lma_collector::params {
'Type =~ /^heka\\.sandbox\\.afd.*metric$/'
], '')
$watchdog_file = "/tmp/${service_name}.watchdog"
$watchdog_payload_name = "${service_name}.watchdog"
$log_watchdog_file = "/tmp/${log_service_name}.watchdog"
$log_watchdog_payload_name = "${log_service_name}.watchdog"
$metric_watchdog_file = "/tmp/${metric_service_name}.watchdog"
$metric_watchdog_payload_name = "${metric_service_name}.watchdog"
$watchdog_interval = 1
$watchdog_timeout = 10 * $watchdog_interval
@ -70,11 +80,25 @@ class lma_collector::params {
# https://bugs.launchpad.net/lma-toolchain/+bug/1548093
$hekad_max_message_size = 256 * 1024
$buffering_max_file_size = 128 * 1024 * 1024
$buffering_max_buffer_size = 1024 * 1024 * 1024
$buffering_max_file_size_for_metric = 128 * 1024 * 1024
$buffering_max_buffer_size_for_metric = 1536 * 1024 * 1024
$queue_full_action_for_metric = 'drop'
$buffering_max_file_tiny_size = 1 * 1024 * 1024
$buffering_max_buffer_tiny_size = 2 * 1024 * 1024
$buffering_max_file_size_for_aggregator = 64 * 1024 * 1024
$buffering_max_buffer_size_for_aggregator = 256 * 1024 * 1024
$queue_full_action_for_aggregator = 'drop'
$buffering_max_file_size_for_log = 64 * 1024 * 1024
$buffering_max_buffer_size_for_log = 256 * 1024 * 1024
$queue_full_action_for_log = 'block'
$buffering_max_file_log_metric_size = 64 * 1024 * 1024
$buffering_max_buffer_log_metric_size = 256 * 1024 * 1024
$queue_full_action_for_log_metric = 'drop'
$buffering_max_file_size_for_nagios = 512 * 1024
$buffering_max_buffer_size_for_nagios = 1 * 1024 * 1024
$queue_full_action_for_nagios = 'drop'
# Heka's default value is 1
$hekad_max_process_inject = 1

33
deployment_scripts/puppet/modules/lma_collector/manifests/service/log.pp Normal file
View File

@ -0,0 +1,33 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# Class: lma_collector::service::log
#
# Manages the Log and Notification collector daemon
#
# Sample Usage:
#
# sometype { 'foo':
# notify => Class['lma_collector::service::log'],
# }
#
#
class lma_collector::service::log {
include lma_collector::params
service { $::lma_collector::params::log_service_name:
ensure => 'running',
enable => true,
}
}

12
deployment_scripts/puppet/modules/lma_collector/manifests/service.pp → deployment_scripts/puppet/modules/lma_collector/manifests/service/metric.pp
View File

@ -12,24 +12,22 @@
# License for the specific language governing permissions and limitations
# under the License.
#
# Class: lma_collector::service
# Class: lma_collector::service:metric
#
# Manages the LMA collector daemon
# Manages the Metric collector daemon
#
# Sample Usage:
#
# sometype { 'foo':
# notify => Class['lma_collector::service'],
# notify => Class['lma_collector::service::metric'],
# }
#
#
class lma_collector::service {
class lma_collector::service::metric {
include lma_collector::params
service { $lma_collector::params::service_name:
service { $::lma_collector::params::metric_service_name:
ensure => 'running',
enable => true,
}
}

10
deployment_scripts/puppet/modules/lma_collector/manifests/smtp_alert.pp
View File

@ -24,7 +24,7 @@ class lma_collector::smtp_alert (
$ensure = present,
) inherits lma_collector::params {
include lma_collector::service
include lma_collector::service::metric
$lua_modules_dir = $lma_collector::params::lua_modules_dir
@ -39,14 +39,14 @@ class lma_collector::smtp_alert (
}
heka::encoder::sandbox { 'smtp_alert':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
filename => "${lma_collector::params::plugins_dir}/encoders/status_smtp.lua",
module_directory => $lua_modules_dir,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
heka::output::smtp { 'smtp_alert':
config_dir => $lma_collector::params::config_dir,
config_dir => $lma_collector::params::metric_config_dir,
send_from => $send_from,
send_to => $send_to,
message_matcher => 'Type == \'heka.sandbox.gse_cluster_metric\' || Type == \'heka.sandbox.gse_node_cluster_metric\'',
@ -57,6 +57,6 @@ class lma_collector::smtp_alert (
user => $user,
password => $password,
send_interval => $send_interval,
notify => Class['lma_collector::service'],
notify => Class['lma_collector::service::metric'],
}
}

16
deployment_scripts/puppet/modules/lma_collector/spec/classes/lma_collector_service_spec.rb
View File

@ -13,14 +13,26 @@
# under the License.
require 'spec_helper'
describe 'lma_collector::service' do
describe 'lma_collector::service::log' do
let(:facts) do
{:kernel => 'Linux', :operatingsystem => 'Ubuntu',
:osfamily => 'Debian'}
end
describe 'default params' do
it { is_expected.to contain_service('lma_collector') \
it { is_expected.to contain_service('log_collector') \
.with({'ensure' => 'running', 'enable' => true}) }
end
end
describe 'lma_collector::service::metric' do
let(:facts) do
{:kernel => 'Linux', :operatingsystem => 'Ubuntu',
:osfamily => 'Debian'}
end
describe 'default params' do
it { is_expected.to contain_service('metric_collector') \
.with({'ensure' => 'running', 'enable' => true}) }
end
end

2
deployment_scripts/puppet/modules/lma_collector/spec/classes/lma_collector_spec.rb
View File

@ -20,6 +20,6 @@ describe 'lma_collector' do
end
describe 'with defaults' do
it { is_expected.to contain_class('Heka') }
it { is_expected.to contain_file('/usr/share/lma_collector_modules/extra_fields.lua') }
end
end

72
deployment_scripts/puppet/modules/lma_collector/spec/defines/lma_collector_heka_spec.rb Normal file
View File

@ -0,0 +1,72 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'spec_helper'
describe 'lma_collector::heka' do
let(:facts) do
{:kernel => 'Linux', :operatingsystem => 'Ubuntu',
:osfamily => 'Debian'}
end
describe 'log_collector with default' do
let(:title) { :log_collector}
it {
should contain_heka('log_collector').with(
'user' => 'heka',
'poolsize' => 100,
)
should contain_heka__output__tcp('metric')
should contain_heka__filter__sandbox('heka_monitoring_log_collector')
should contain_heka__output__dashboard('dashboard_log_collector' )
}
end
describe 'metric_collector with default' do
let(:title) { :metric_collector}
it {
should contain_heka('metric_collector').with(
'user' => 'heka',
'poolsize' => 100,
)
should contain_heka__input__tcp('metric')
should contain_heka__filter__sandbox('heka_monitoring_metric_collector')
should contain_heka__output__dashboard('dashboard_metric_collector' )
}
end
describe 'with an invalid title' do
let(:title) { :invalidname}
it do
expect {
is_expected.to compile
}.to raise_error(/title must be either/)
end
end
describe 'metric_collector with no self-monitoring and poolsize' do
let(:title) { :metric_collector}
let(:params) do
{ :heka_monitoring => false,
:poolsize => 42,
:user => 'foo',
}
end
it {
should contain_heka('metric_collector').with(
'user' => 'foo',
'poolsize' => 42,
)
should contain_heka__input__tcp('metric')
is_expected.to_not contain_heka__filter__sandbox('heka_monitoring_metric_collector')
is_expected.to_not contain_heka__output__dashboard('dashboard_metric_collector' )
}
end
end

22
doc/user/source/configuration.rst
View File

@ -71,10 +71,11 @@ Plugin verification
-------------------
Once the OpenStack environment is ready, you may want to check that both
the 'collectd' and 'hekad' processes of the LMA Collector are running on the OpenStack nodes::
the 'collectd' and 'hekad' processes are running on the OpenStack nodes::
[root@node-1 ~]# pidof hekad
5568
5569
[root@node-1 ~]# pidof collectd
5684
@ -85,23 +86,28 @@ Troubleshooting
If you see no data in the Kibana and/or Grafana dashboards, use the instructions below to troubleshoot the problem:
1. Check if the LMA Collector service is up and running::
1. Check if LMA Collector services are up and running::
# On the controller node(s)
[root@node-1 ~]# crm resource status lma_collector
[root@node-1 ~]# crm resource status metric_collector
[root@node-1 ~]# crm resource status log_collector
# On non controller nodes
[root@node-1 ~]# status lma_collector
[root@node-2 ~]# status log_collector
[root@node-2 ~]# status metric_collector
2. If the LMA Collector is down, restart it::
2. If one of the LMA Collectors is down, restart it::
# On the controller node(s)
[root@node-1 ~]# crm resource start lma_collector
[root@node-1 ~]# crm resource start log_collector
[root@node-1 ~]# crm resource start metric_collector
# On non controller nodes
[root@node-1 ~]# start lma_collector
[root@node-2 ~]# start log_collector
[root@node-2 ~]# start metric_collector
3. Look for errors in the LMA Collector log file (located at /var/log/lma_collector.log) on the different nodes.
3. Look for errors in the LMA Collector log file (located at /var/log/log_collector.log and /var/log/metric_collector.log)
on the different nodes.
4. Look for errors in the collectd log file (located at /var/log/collectd.log) on the different nodes.