Remove notification agent support from the plugin

Change-Id: I3f3332efb59d27f08368c5d0571cdbd9750b96b5
This commit is contained in:
Nadya Shakhat 2016-06-15 14:58:05 +03:00
parent 4fec0439df
commit f7a306221a
3 changed files with 34 additions and 39 deletions

View File

@ -140,7 +140,6 @@ class redis::main (
'coordination/backend_url' : value => redis_backend_url($redis_hosts, $redis_sentinel_port, $timeout, $master_name);
'coordination/heartbeat' : value => '1.0';
'coordination/check_watchers' : value => $timeout;
'notification/workload_partitioning': value => true
}
service { 'ceilometer-agent-central':

View File

@ -4,14 +4,13 @@ Ceilometer Redis plugin
Ceilometer Redis Plugin aims to install Redis to MOS environment and provide a coordination mechanism for
`Ceilometer agents <https://ceilometer.readthedocs.org/en/latest/architecture.html>`_ and Alarm Evaluator
through the `tooz library <http://docs.openstack.org/developer/tooz/>`_ with a `Redis backend <http://redis.io>`_
The plugin supports coordination for the following Ceilometer services: central agent, notification agent
and alarm-evaluator. Each of these services are running on every controller after the plugin
is installed. All of them are joined into the corresponding coordination group (one coordination group
per each service). It differs from the default configuration when there should be only one central agent
and alarm-evaluator per cloud. The plugin also configures redis-server under pacemaker to monitor its process.
The plugin configures `redis-sentinel <http://redis.io/topics/sentinel>`_ to monitor the state of the redis
cluster, to elect new master during failovers, to forward ceilometer services to new elected redis master,
to organize sync between redis nodes.
The plugin supports coordination for the following Ceilometer services: central agent and alarm-evaluator.
Each of these services are running on every controller after the plugin is installed. All of them are joined
into the corresponding coordination group (one coordination group per each service). It differs from the default
configuration when there should be only one central agent and alarm-evaluator per cloud. The plugin also configures
redis-server under pacemaker to monitor its process. The plugin configures `redis-sentinel <http://redis.io/topics/sentinel>`_
to monitor the state of the redis cluster, to elect new master during failovers, to forward ceilometer services to new
elected redis master, to organize sync between redis nodes.
Central agent
@ -32,23 +31,6 @@ If there are several alarm evaluators and no coordination enabled, all of them w
every configurable time interval. The alarm sets for evaluators should be disjoint. So, coordination is responsible
for providing the set of alarms to evaluate to each alarm-evaluator in the cloud.
Notification agent
------------------
Before Liberty, there was no need to coordinate Ceilometer notification agents. But starting from Liberty, samples
transformations started to be handled not by compute/central agents as it was before, but by a notification agent.
Some of Ceilometer transformers have a local cache where it is stored the data from previously processed samples.
For example, "cpu_util" metric are obtained from two consecutive Samples with "cpu" metric: one is subtracted from
another and divided to an amount of cpu (this information is stored in Sample's metadata).
Thus, it should be guaranteed that all the Samples which should be transformed by one transformer, will go to the
same notification agent. If some of the samples go to another, the cache cannot be shared and some data will be lost.
To handle this process properly, IPC queues was introduced - inter process communication queues in message bus
(RabbitMQ). With coordination enabled, each notification agent has two set of listeners: for main queues and for IPC
queues. All notification agents listen to _all_ the main queues (where we have all messages from OpenStack services
and polling-based messages from central/compute agents) and re-publish messages to _all_ IPC queues. Coordination
starts to work at this point: every notification agent in the cloud has it's own set of IPC queues to listen to. Thus,
we can be sure that local cache on each notification agent contains all the previous data required for transformation.
Requirements
------------
@ -69,8 +51,31 @@ Limitations
This requirement is mandatory because Redis needs an odd number of nodes to
choose the master successfully.
* In MOS 8.0, there are no transformers configured by default. The plugin doesn't add any of them into
ceilometer's pipeline.yaml. Thus, you need to configure it manually if you want to use transformers.
If you don't need this feature, it is recommended to disable coordination for the notification agents.
* Before Liberty, there was no need to coordinate Ceilometer notification agents. Starting from Liberty, samples
transformations started to be handled not by compute/central agents as it was before, but by a notification agent.
Some of Ceilometer transformers have a local cache where they store the data from the previously processed samples.
For example, "cpu_util" metric are obtained from two consecutive Samples with "cpu" metric: one is subtracted from
another and divided by an amount of cpu (this information is stored in Sample's metadata).
Thus, it should be guaranteed that all the Samples which should be transformed by one transformer, will go to the
same notification agent. If some of the samples go to another, the cache cannot be shared and some data will be lost.
To handle this process properly, IPC queues was introduced - inter process communication queues in message bus
(RabbitMQ). With coordination enabled, each notification agent has two set of listeners: for main queues and for IPC
queues. All notification agents listen to _all_ the main queues (where we have all messages from OpenStack services
and polling-based messages from central/compute agents) and re-publish messages to _all_ IPC queues. Coordination
starts to work at this point: every notification agent in the cloud has it's own set of IPC queues to listen to. Thus,
we can be sure that local cache on each notification agent contains all the previous data required for transformation.
After some investigations, some performance issues were found with IPC approach. That's why in In MOS 8.0 all basic
transformations (cpu_util, disk.read.requests.rate, disk.write.requests.rate, disk.read.bytes.rate, disk.write.bytes.rate,
disk.device.read.requests.rate, disk.device.read.bytes.rate, disk.device.write.bytes.rate, network.incoming.bytes.rate,
network.outgoing.bytes.rate, network.incoming.packets.rate, network.outgoing.packets.rate) were moved back to compute
nodes, i.e. for the basic set of transformations there is no need to run notification agent in coordination mode.
That's the reason why the plugin does't support coordination for notification agents, although it is possible to configure
notification agents to run in coordination mode manually. Anyway, it is not recommended.
If you have any custom transformers, you need to be sure that they are cache-less, i.e. are based only on
``unit_conversion`` transformer or the ``arithmetic`` transformer. If it's not the case, you may consider the following
options: run only one notification agent in the cloud or install this plugin and do all configuration manually.

View File

@ -2,7 +2,7 @@ User Guide
==========
Once the Ceilometer Redis plugin plugin has been installed (following :ref:`Installation Guide`), you can
create *OpenStack* environments with Ceilometer whose Central agents, Notification agent and Alarm evaluator
create *OpenStack* environments with Ceilometer whose Central agents and Alarm evaluator
work in workload_partitioned mode.
Ceilometer installation
@ -110,15 +110,6 @@ How to check that plugin works
.... 2015-11-05T10:26:26 |
.... 2015-11-05T10:26:17 |
#. For the notification agent: Check that IPC queues are created and have consumers:
ubuntu@ubuntu:/opt/stack/ceilometer$ sudo rabbitmqctl list_queues name messages consumers | grep ceilo
ceilometer-pipe-meter_source:meter_sink-0.sample 0 1
ceilometer-pipe-meter_source:meter_sink-1.sample 0 1
ceilometer-pipe-meter_source:meter_sink-2.sample 0 1
ceilometer-pipe-meter_source:meter_sink-3.sample 0 1
ceilometer-pipe-meter_source:meter_sink-4.sample 0 1
By default, you should see 10 queues in this list. Every queue should have one and only one consumer.
#. For the alarm evaluator, it is possible to see that everything works as expected only from the logs. Grep the
line "extract_my_subset". There should be different "My subset: [" results on each alarm evaluator instance.