summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorNadya Shakhat <nprivalova@mirantis.com>2016-06-15 14:58:05 +0300
committerNadya Shakhat <nprivalova@mirantis.com>2016-06-21 11:23:37 +0300
commitf7a306221ae99b2ae750fa9f15f5d11d59ed1c57 (patch)
tree28132940390af7246946971c78ce700a79a9b59a
parent4fec0439df45a7e6df0ac5367f186c89dc646a5d (diff)
Remove notification agent support from the plugin
Notes
Notes (review): Code-Review+2: Nadya Shakhat <nprivalova@mirantis.com> Workflow+1: Nadya Shakhat <nprivalova@mirantis.com> Verified+2: Jenkins Submitted-by: Jenkins Submitted-at: Tue, 21 Jun 2016 15:04:51 +0000 Reviewed-on: https://review.openstack.org/329891 Project: openstack/fuel-plugin-ceilometer-redis Branch: refs/heads/master
-rw-r--r--deployment_scripts/puppet/modules/redis/manifests/main.pp1
-rw-r--r--doc/source/description.rst61
-rw-r--r--doc/source/guide.rst11
3 files changed, 34 insertions, 39 deletions
diff --git a/deployment_scripts/puppet/modules/redis/manifests/main.pp b/deployment_scripts/puppet/modules/redis/manifests/main.pp
index 6ddef59..c820760 100644
--- a/deployment_scripts/puppet/modules/redis/manifests/main.pp
+++ b/deployment_scripts/puppet/modules/redis/manifests/main.pp
@@ -140,7 +140,6 @@ class redis::main (
140 'coordination/backend_url' : value => redis_backend_url($redis_hosts, $redis_sentinel_port, $timeout, $master_name); 140 'coordination/backend_url' : value => redis_backend_url($redis_hosts, $redis_sentinel_port, $timeout, $master_name);
141 'coordination/heartbeat' : value => '1.0'; 141 'coordination/heartbeat' : value => '1.0';
142 'coordination/check_watchers' : value => $timeout; 142 'coordination/check_watchers' : value => $timeout;
143 'notification/workload_partitioning': value => true
144 } 143 }
145 144
146 service { 'ceilometer-agent-central': 145 service { 'ceilometer-agent-central':
diff --git a/doc/source/description.rst b/doc/source/description.rst
index 1ea9019..316d29d 100644
--- a/doc/source/description.rst
+++ b/doc/source/description.rst
@@ -4,14 +4,13 @@ Ceilometer Redis plugin
4Ceilometer Redis Plugin aims to install Redis to MOS environment and provide a coordination mechanism for 4Ceilometer Redis Plugin aims to install Redis to MOS environment and provide a coordination mechanism for
5`Ceilometer agents <https://ceilometer.readthedocs.org/en/latest/architecture.html>`_ and Alarm Evaluator 5`Ceilometer agents <https://ceilometer.readthedocs.org/en/latest/architecture.html>`_ and Alarm Evaluator
6through the `tooz library <http://docs.openstack.org/developer/tooz/>`_ with a `Redis backend <http://redis.io>`_ 6through the `tooz library <http://docs.openstack.org/developer/tooz/>`_ with a `Redis backend <http://redis.io>`_
7The plugin supports coordination for the following Ceilometer services: central agent, notification agent 7The plugin supports coordination for the following Ceilometer services: central agent and alarm-evaluator.
8and alarm-evaluator. Each of these services are running on every controller after the plugin 8Each of these services are running on every controller after the plugin is installed. All of them are joined
9is installed. All of them are joined into the corresponding coordination group (one coordination group 9into the corresponding coordination group (one coordination group per each service). It differs from the default
10per each service). It differs from the default configuration when there should be only one central agent 10configuration when there should be only one central agent and alarm-evaluator per cloud. The plugin also configures
11and alarm-evaluator per cloud. The plugin also configures redis-server under pacemaker to monitor its process. 11redis-server under pacemaker to monitor its process. The plugin configures `redis-sentinel <http://redis.io/topics/sentinel>`_
12The plugin configures `redis-sentinel <http://redis.io/topics/sentinel>`_ to monitor the state of the redis 12to monitor the state of the redis cluster, to elect new master during failovers, to forward ceilometer services to new
13cluster, to elect new master during failovers, to forward ceilometer services to new elected redis master, 13elected redis master, to organize sync between redis nodes.
14to organize sync between redis nodes.
15 14
16 15
17Central agent 16Central agent
@@ -32,23 +31,6 @@ If there are several alarm evaluators and no coordination enabled, all of them w
32every configurable time interval. The alarm sets for evaluators should be disjoint. So, coordination is responsible 31every configurable time interval. The alarm sets for evaluators should be disjoint. So, coordination is responsible
33for providing the set of alarms to evaluate to each alarm-evaluator in the cloud. 32for providing the set of alarms to evaluate to each alarm-evaluator in the cloud.
34 33
35Notification agent
36------------------
37Before Liberty, there was no need to coordinate Ceilometer notification agents. But starting from Liberty, samples
38transformations started to be handled not by compute/central agents as it was before, but by a notification agent.
39Some of Ceilometer transformers have a local cache where it is stored the data from previously processed samples.
40For example, "cpu_util" metric are obtained from two consecutive Samples with "cpu" metric: one is subtracted from
41another and divided to an amount of cpu (this information is stored in Sample's metadata).
42Thus, it should be guaranteed that all the Samples which should be transformed by one transformer, will go to the
43same notification agent. If some of the samples go to another, the cache cannot be shared and some data will be lost.
44
45To handle this process properly, IPC queues was introduced - inter process communication queues in message bus
46(RabbitMQ). With coordination enabled, each notification agent has two set of listeners: for main queues and for IPC
47queues. All notification agents listen to _all_ the main queues (where we have all messages from OpenStack services
48and polling-based messages from central/compute agents) and re-publish messages to _all_ IPC queues. Coordination
49starts to work at this point: every notification agent in the cloud has it's own set of IPC queues to listen to. Thus,
50we can be sure that local cache on each notification agent contains all the previous data required for transformation.
51
52 34
53Requirements 35Requirements
54------------ 36------------
@@ -69,8 +51,31 @@ Limitations
69 This requirement is mandatory because Redis needs an odd number of nodes to 51 This requirement is mandatory because Redis needs an odd number of nodes to
70 choose the master successfully. 52 choose the master successfully.
71 53
72* In MOS 8.0, there are no transformers configured by default. The plugin doesn't add any of them into 54* Before Liberty, there was no need to coordinate Ceilometer notification agents. Starting from Liberty, samples
73 ceilometer's pipeline.yaml. Thus, you need to configure it manually if you want to use transformers. 55 transformations started to be handled not by compute/central agents as it was before, but by a notification agent.
74 If you don't need this feature, it is recommended to disable coordination for the notification agents. 56 Some of Ceilometer transformers have a local cache where they store the data from the previously processed samples.
57 For example, "cpu_util" metric are obtained from two consecutive Samples with "cpu" metric: one is subtracted from
58 another and divided by an amount of cpu (this information is stored in Sample's metadata).
59 Thus, it should be guaranteed that all the Samples which should be transformed by one transformer, will go to the
60 same notification agent. If some of the samples go to another, the cache cannot be shared and some data will be lost.
61
62 To handle this process properly, IPC queues was introduced - inter process communication queues in message bus
63 (RabbitMQ). With coordination enabled, each notification agent has two set of listeners: for main queues and for IPC
64 queues. All notification agents listen to _all_ the main queues (where we have all messages from OpenStack services
65 and polling-based messages from central/compute agents) and re-publish messages to _all_ IPC queues. Coordination
66 starts to work at this point: every notification agent in the cloud has it's own set of IPC queues to listen to. Thus,
67 we can be sure that local cache on each notification agent contains all the previous data required for transformation.
68
69 After some investigations, some performance issues were found with IPC approach. That's why in In MOS 8.0 all basic
70 transformations (cpu_util, disk.read.requests.rate, disk.write.requests.rate, disk.read.bytes.rate, disk.write.bytes.rate,
71 disk.device.read.requests.rate, disk.device.read.bytes.rate, disk.device.write.bytes.rate, network.incoming.bytes.rate,
72 network.outgoing.bytes.rate, network.incoming.packets.rate, network.outgoing.packets.rate) were moved back to compute
73 nodes, i.e. for the basic set of transformations there is no need to run notification agent in coordination mode.
74 That's the reason why the plugin does't support coordination for notification agents, although it is possible to configure
75 notification agents to run in coordination mode manually. Anyway, it is not recommended.
76
77 If you have any custom transformers, you need to be sure that they are cache-less, i.e. are based only on
78 ``unit_conversion`` transformer or the ``arithmetic`` transformer. If it's not the case, you may consider the following
79 options: run only one notification agent in the cloud or install this plugin and do all configuration manually.
75 80
76 81
diff --git a/doc/source/guide.rst b/doc/source/guide.rst
index bbf089e..76ca453 100644
--- a/doc/source/guide.rst
+++ b/doc/source/guide.rst
@@ -2,7 +2,7 @@ User Guide
2========== 2==========
3 3
4Once the Ceilometer Redis plugin plugin has been installed (following :ref:`Installation Guide`), you can 4Once the Ceilometer Redis plugin plugin has been installed (following :ref:`Installation Guide`), you can
5create *OpenStack* environments with Ceilometer whose Central agents, Notification agent and Alarm evaluator 5create *OpenStack* environments with Ceilometer whose Central agents and Alarm evaluator
6work in workload_partitioned mode. 6work in workload_partitioned mode.
7 7
8Ceilometer installation 8Ceilometer installation
@@ -110,15 +110,6 @@ How to check that plugin works
110 .... 2015-11-05T10:26:26 | 110 .... 2015-11-05T10:26:26 |
111 .... 2015-11-05T10:26:17 | 111 .... 2015-11-05T10:26:17 |
112 112
113#. For the notification agent: Check that IPC queues are created and have consumers:
114 ubuntu@ubuntu:/opt/stack/ceilometer$ sudo rabbitmqctl list_queues name messages consumers | grep ceilo
115 ceilometer-pipe-meter_source:meter_sink-0.sample 0 1
116 ceilometer-pipe-meter_source:meter_sink-1.sample 0 1
117 ceilometer-pipe-meter_source:meter_sink-2.sample 0 1
118 ceilometer-pipe-meter_source:meter_sink-3.sample 0 1
119 ceilometer-pipe-meter_source:meter_sink-4.sample 0 1
120
121 By default, you should see 10 queues in this list. Every queue should have one and only one consumer.
122 113
123#. For the alarm evaluator, it is possible to see that everything works as expected only from the logs. Grep the 114#. For the alarm evaluator, it is possible to see that everything works as expected only from the logs. Grep the
124 line "extract_my_subset". There should be different "My subset: [" results on each alarm evaluator instance. 115 line "extract_my_subset". There should be different "My subset: [" results on each alarm evaluator instance.