diff --git a/doc/source/conf.py b/doc/source/conf.py index 9909511..f30d55f 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -4,10 +4,10 @@ extensions = [] templates_path = ['_templates'] source_suffix = '.rst' master_doc = 'index' -project = u'The Redis Plugin for Ceilometer documentation' +project = u'The Ceilometer Redis Plugin' copyright = u'2016, Mirantis Inc.' version = '0.1' -release = '0.1.0' +release = '0.1.2' exclude_patterns = [ ] pygments_style = 'sphinx' @@ -16,15 +16,15 @@ htmlhelp_basename = 'RedisPlugindoc' latex_elements = { } latex_documents = [ - ('index', 'RedisPlugindoc.tex', u'The Redis Plugin for Ceilometer documentation', + ('index', 'RedisPlugindoc.tex', u'The Ceilometer Redis Plugin', u'Mirantis Inc.', 'manual'), ] man_pages = [ - ('index', 'redisplugin', u'The Redis Plugin for Ceilometer documentation', + ('index', 'redisplugin', u'The Ceilometer Redis Plugin', [u'Mirantis Inc.'], 1) ] texinfo_documents = [ - ('index', 'RedisPlugin', u'The Redis Plugin for Ceilometer documentation', + ('index', 'RedisPlugin', u'The Ceilometer Redis Plugin', u'Mirantis Inc.', 'RedisPlugin', 'One line description of project.', 'Miscellaneous'), ] diff --git a/doc/source/description.rst b/doc/source/description.rst index 316d29d..3c3d115 100644 --- a/doc/source/description.rst +++ b/doc/source/description.rst @@ -1,45 +1,100 @@ -Ceilometer Redis plugin -======================= +Overview +======== -Ceilometer Redis Plugin aims to install Redis to MOS environment and provide a coordination mechanism for -`Ceilometer agents `_ and Alarm Evaluator -through the `tooz library `_ with a `Redis backend `_ -The plugin supports coordination for the following Ceilometer services: central agent and alarm-evaluator. -Each of these services are running on every controller after the plugin is installed. All of them are joined -into the corresponding coordination group (one coordination group per each service). It differs from the default -configuration when there should be only one central agent and alarm-evaluator per cloud. The plugin also configures -redis-server under pacemaker to monitor its process. The plugin configures `redis-sentinel `_ -to monitor the state of the redis cluster, to elect new master during failovers, to forward ceilometer services to new -elected redis master, to organize sync between redis nodes. +The *Ceilometer Redis Plugin* installs `Redis `_ and +the `Tooz library `_, in a +Mirantis OpenStack (MOS) environment deployed by Fuel. +Both Redis and the Tooz library should be installed on all the controller +nodes of the environment. +The *Ceilometer Redis Plugin* is used to provide coordination mechanisms to +enable the horizontal scaling of the Ceilometer services. Using the plugin, +the Ceilometer services are joined into a so-called **coordination group**, +which allows for resources and alarms sharding. +There is one coordination group per service type. -Central agent -------------- -Ceilometer Central agent is responsible for polling all OpenStack resources except Nova's (Nova resources, -i.e. vms, are polled by a Compute agent). Without coordination enabled, only one Central agent should be running -per cloud. The reason is that all the central agents have the same set of OpenStack resources to poll every -configurable time interval. If coordination is not enabled, each OpenStack resource will be polled as many times -as many instances of Central agents are running in a cloud. -Thus, coordination provides a disjoint set of OpenStack resources to poll for every Central agent running on the -cloud to avoid polling one resource several times. +Please refer to the `Telemetry architecture +`_ +documentation for more information about the Ceilometer services. -Alarm evaluator ---------------- -Ceilometer alarm evaluator service is responsible for Ceilometer alarm evaluation. -By default, in MOS there is only one alarm evaluator per cloud. The reason is the same as for a central agent. -If there are several alarm evaluators and no coordination enabled, all of them will evaluate the same set of alarms -every configurable time interval. The alarm sets for evaluators should be disjoint. So, coordination is responsible -for providing the set of alarms to evaluate to each alarm-evaluator in the cloud. +In MOS 7.0 and MOS 8.0, the *Ceilometer Redis Plugin* enables coordination +for both: + * The **ceilometer-agent-central service**. + + The ceilometer-agent-central service is responsible for polling all the OpenStack resources, + excepted those of Nova, like the VM instances, that are polled by the **ceilometer-agent-compute**. + Without coordination, there can be only one ceilometer-agent-central running at a time. + This is because, by default, the ceilometer-agent-central works with an entire set of resources. + As such, running multiple ceilometer-agent-central without coordination would poll the entire + set of resources as many times as the number of agents running on the controller nodes every + polling interval. This is obviously not a proper way to scale out the ceilometer-agent-central. + To cope with this problem, the coordination mechanism provided + by the *Ceilometer Redis Plugin* allows distributing the polling workload + across multiple instances of the ceilometer-agent-central using disjoint sets + of resources. + + * The **ceilometer-alarm-evaluator service**. + + The **ceilometer-alarm-evaluator** service is responsible for evaluating the Ceilometer alarms. + By default, there is only one ceilometer-alarm-evaluator running per environment. + Without coordination, there can be only one ceilometer-alarm-evaluator running at a time. + This is because, as for the ceilometer-agent-central, the ceilometer-alarm-evaluator works + with an entire set of alarms. Running multiple ceilometer-alarm-evaluator + without coordination would evaluate all the alarms as many times as the number of evaluators + running on the controller nodes every evaluation interval. To cope with this problem, + the coordination mechanism provided by the *Ceilometer Redis Plugin* allows distributing + the alarms evaluation workload across multiple instances of the ceilometer-alarm-evaluator + using disjoint sets of alarms. + +Please note that with MOS 8.0, the *Ceilometer Redis Plugin* doesn't provide support +(out-of-the-box) for the coordination of the **ceilometer-agent-notification** service because +it is not needed for the most common samples transformations. + +.. note:: Before Liberty, the transformation of the samples was handled by the + **ceilometer-agent-compute** and the **ceilometer-agent-central** services. + In Liberty, the transformation of the samples was moved + to the **ceilometer-agent-notification** service, but after thorough performance analysis + of Ceilometer at scale, we discovered that this change has a bad impact on performance. + In MOS 8.0, the transformations for the following list of measurements were moved back + to the ceilometer-agent-compute service. + + * cpu_util + * disk.read.requests.rate + * disk.write.requests.rate + * disk.read.bytes.rate + * disk.write.bytes.rate + * disk.device.read.requests.rate + * disk.device.read.bytes.rate + * disk.device.write.bytes.rate + * network.incoming.bytes.rate + * network.outgoing.bytes.rate + * network.incoming.packets.rate + * network.outgoing.packets.rate + + As a result, in MOS 8.0, there is no need to run the ceilometer-agent-notification + in coordination mode unless you need to maintain the transformation of custom samples that + are not listed above. In this case, it is possible to enable coordination for the + ceilometer-agent-notification service manually event though, it is not recommended + for performance reasons. + +In addition to the above, the *Ceilometer Redis Plugin* configures *Pacemaker* +and `redis-sentinel `_ +to enable **high availability** of the Redis cluster. Redis clustering includes: + + * Monitoring the state of the **redis-server** processes + * Elect a new redis-server master during a failover + * Connect Ceilometer to the elected redis-server + * Organize the synchronization between Redis nodes Requirements ------------ ======================= ================ -Requirement Version/Comment +Requirements Version/Comment ======================= ================ -Fuel 7.0, 8.0 -tooz <0.14.0,>=0.13.1 +MOS 7.0, 8.0 +Tooz <0.14.0,>=0.13.1 ======================= ================ .. _limitations: @@ -47,35 +102,13 @@ tooz <0.14.0,>=0.13.1 Limitations ----------- -* The plugin works correctly only on clouds with odd numbers of controllers. - This requirement is mandatory because Redis needs an odd number of nodes to - choose the master successfully. - -* Before Liberty, there was no need to coordinate Ceilometer notification agents. Starting from Liberty, samples - transformations started to be handled not by compute/central agents as it was before, but by a notification agent. - Some of Ceilometer transformers have a local cache where they store the data from the previously processed samples. - For example, "cpu_util" metric are obtained from two consecutive Samples with "cpu" metric: one is subtracted from - another and divided by an amount of cpu (this information is stored in Sample's metadata). - Thus, it should be guaranteed that all the Samples which should be transformed by one transformer, will go to the - same notification agent. If some of the samples go to another, the cache cannot be shared and some data will be lost. - - To handle this process properly, IPC queues was introduced - inter process communication queues in message bus - (RabbitMQ). With coordination enabled, each notification agent has two set of listeners: for main queues and for IPC - queues. All notification agents listen to _all_ the main queues (where we have all messages from OpenStack services - and polling-based messages from central/compute agents) and re-publish messages to _all_ IPC queues. Coordination - starts to work at this point: every notification agent in the cloud has it's own set of IPC queues to listen to. Thus, - we can be sure that local cache on each notification agent contains all the previous data required for transformation. - - After some investigations, some performance issues were found with IPC approach. That's why in In MOS 8.0 all basic - transformations (cpu_util, disk.read.requests.rate, disk.write.requests.rate, disk.read.bytes.rate, disk.write.bytes.rate, - disk.device.read.requests.rate, disk.device.read.bytes.rate, disk.device.write.bytes.rate, network.incoming.bytes.rate, - network.outgoing.bytes.rate, network.incoming.packets.rate, network.outgoing.packets.rate) were moved back to compute - nodes, i.e. for the basic set of transformations there is no need to run notification agent in coordination mode. - That's the reason why the plugin does't support coordination for notification agents, although it is possible to configure - notification agents to run in coordination mode manually. Anyway, it is not recommended. - - If you have any custom transformers, you need to be sure that they are cache-less, i.e. are based only on - ``unit_conversion`` transformer or the ``arithmetic`` transformer. If it's not the case, you may consider the following - options: run only one notification agent in the cloud or install this plugin and do all configuration manually. - +* The *Ceilometer Redis Plugin* requires to install on an odd number of controller + nodes to work properly. This is because, Redis clustering requires an odd number of nodes + to avoid the split brain effect when electing a master. +* If you have any custom transformers you need to ensure that they are cache-less. + If transformation is cache-less, then there is no need to enable the coordination. + That is, based only on the ``unit_conversion`` transformer or the ``arithmetic`` transformers. + Otherwise, you will have to consider running only one instance of the ceilometer-agent-notification + service in your MOS environment or install the *Ceilometer Redis Plugin* and do all the + configuration manually. \ No newline at end of file diff --git a/doc/source/guide.rst b/doc/source/guide.rst index 76ca453..6a179e1 100644 --- a/doc/source/guide.rst +++ b/doc/source/guide.rst @@ -1,115 +1,109 @@ User Guide ========== -Once the Ceilometer Redis plugin plugin has been installed (following :ref:`Installation Guide`), you can -create *OpenStack* environments with Ceilometer whose Central agents and Alarm evaluator -work in workload_partitioned mode. +Once the *Ceilometer Redis Plugin* is installed following the instructions of +the :ref:`Installation Guide`, you can create a Mirantis OpenStack (MOS) environment +with Ceilometer whose **ceilometer-agent-central** and **ceilometer-alarm-evaluator** +services will work in **workload partitioned** mode. +This plugin was created to enable the scale-out of these Ceilometer services. +It is useless and **shouldn't be used if Ceilometer is not installed**. -Ceilometer installation ------------------------ +Plugin Configuration +-------------------- -This plugin was created to provide partitioning for Ceilometer services. So its -usage is senseless without Ceilometer installed. -So, you will need to `create a new OpenStack environment `_ -with `Ceilometer `_ using the Fuel UI Wizard. +To use the *Ceilometer Redis Plugin*, you need to `create a new MOS environment +`_ +with the `Telemetry service +`_ +(a.k.a Ceilometer) enabled and follow these steps using the *Fuel UI Wizard*. +1. Make sure that the plugin is properly installed on the Fuel Master node. -Plugin configuration in MOS 8.0 -------------------------------- - -#. First of all, make sure that plugin was successfully installed. Go to the *Plugins* tab. You should see the following: - .. image:: images/redis-plugin-on8.0.png - :width: 100% - -#. The next step is enable the plugin. Go to *Environments* tab and - select the *Redis plugin for Ceilometer* checkbox: - - .. image:: images/redis-plugin-8.0.png - :width: 100% - -#. When adding nodes to environment and assigning roles to them `_, please consider using odd number of controllers as mentioned in :ref:`Limitations`. - -#. Finish - `environment configuration `_ - -#. Run `network verification check `_. - -#. Press "Deploy button" to once you are done with environment configuration. - -Plugin configuration in MOS 7.0 -------------------------------- - -#. First of all, make sure that plugin was successfully installed. - Go to the *Plugins* tab. You should see the following: + On Mos 8.0 .. image:: images/redis-plugin.png :width: 100% -#. The next step is enable the plugin. Go to *Environments* tab and - select the *Redis plugin for Ceilometer* checkbox: + On Mos 7.0 + + .. image:: images/redis-plugin-on8-0.png + :width: 100% + +2. Enable the plugin. + + Go to the *Environments* tab and select the *Redis plugin for Ceilometer* checkbox: + + On Mos 8.0 + + .. image:: images/redis-plugin-8-0.png + :width: 100% + + On Mos 7.0 .. image:: images/redis-plugin-on.png :width: 100% -#. When - `adding nodes to environment and assigning roles to them in MOS 7.0 `_, please consider using odd number of controllers as mentioned in :ref:`Limitations`. +3. Add nodes to your environment to which you will assign the **controller role**. -#. Finish - `environment configuration for MOS 7.0 `_ + .. note:: When `adding nodes + `_ + to the environment and `assign or change a role + `_, + do not forget to use an odd number of controllers as mentioned in :ref:`Limitations` section. -#. Run `network verification check for MOS 7.0 `_. +4. `Verify your network configuration + `_. -#. Press `Deploy button `_ to once you are done with environment configuration. +5. `Deploy your changes + `_ + once you are done with the configuration of your environment. +Plugin Verification +------------------- +#. Check that the ceilometer-agent-central and ceilometer-alarm-evaluator services are running + on each controller. -How to check that plugin works ------------------------------- -#. Check that ceilometer-agent-central and ceilometer-alarm-evaluator services are running - on each controller. Run ``pcs resource`` and you should see the following in the output:: + Run ``pcs resource``. You should see the following in the output:: - Clone Set: clone_p_ceilometer-agent-central [p_ceilometer-agent-central] - Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ] + Clone Set: clone_p_ceilometer-agent-central [p_ceilometer-agent-central] + Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ] - Clone Set: clone_p_ceilometer-alarm-evaluator [p_ceilometer-alarm-evaluator] - Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ] + Clone Set: clone_p_ceilometer-alarm-evaluator [p_ceilometer-alarm-evaluator] + Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ] - ``Started`` list should contain all controllers. + The *Started* list should contain all controllers. -#. For the central agent: check that samples are not duplicated. For this purpose you may choose - any metric collected by central agent. All these metrics may be found here - `Measurements `_ . - You may choose any section *except* OpenStack Compute and then select metric with 'Pollster' Origin. - For example, let's choose storage.objects. +#. For the ceilometer-agent-central, check that the samples are not duplicated. + For this check you may choose any metric collected by the ceilometer-agent-central. + All the Ceilometer metrics can be found in + `Measurements `_ . + You may choose any section excepted *OpenStack Compute* and then select a metric with *Pollster Origin*. + For example, let's choose *storage.objects*. - Plugin works *correctly* if you see one sample for each resource every polling_interval (1 minute in this example):: + The plugin **works correctly** if you see one sample for each resource type every + *polling interval* (1 minute in this example):: - root@node-2:~# ceilometer sample-list -m storage.objects -l 10| grep storage.objects - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object | 2015-11-05T10:32:27 | - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object | 2015-11-05T10:31:29 | + root@node-2:~# ceilometer sample-list -m storage.objects -l 10| grep storage.objects + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:32:27 | + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:31:29 | - + The plugin **works incorrectly** if there are duplicates. In this example, the plugin works + incorectly because there are three samples for the same resource type every *polling interval*:: - Plugin works *incorrectly* if there are duplications. In this example is seen that every - ``polling_interval`` there are 3 samples about one resource:: + root@node-2:~# ceilometer sample-list -m storage.objects -l 20| grep storage.objects + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:27:37 | + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:27:26 | + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:27:17 | + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:26:38 | + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:26:26 | + | 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:26:17 | - root@node-2:~# ceilometer sample-list -m storage.objects -l 20| grep storage.objects - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....| - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....| - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....| - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....| - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....| - | 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....| +#. For the alarm evaluator, it is possible to see that everything works as expected + only from the logs:: - .... 2015-11-05T10:27:37 | - .... 2015-11-05T10:27:26 | - .... 2015-11-05T10:27:17 | - .... 2015-11-05T10:26:38 | - .... 2015-11-05T10:26:26 | - .... 2015-11-05T10:26:17 | + # grep extract_my_subset /var/log/ceilometer/ceilometer-alarm-evaluator.log - -#. For the alarm evaluator, it is possible to see that everything works as expected only from the logs. Grep the - line "extract_my_subset". There should be different "My subset: [" results on each alarm evaluator instance. + There should be different *My subset: [* results for the ceilometer-alarm-evaluator instances. diff --git a/doc/source/images/redis-plugin-8.0.png b/doc/source/images/redis-plugin-8-0.png similarity index 100% rename from doc/source/images/redis-plugin-8.0.png rename to doc/source/images/redis-plugin-8-0.png diff --git a/doc/source/images/redis-plugin-on8.0.png b/doc/source/images/redis-plugin-on8-0.png similarity index 100% rename from doc/source/images/redis-plugin-on8.0.png rename to doc/source/images/redis-plugin-on8-0.png diff --git a/doc/source/index.rst b/doc/source/index.rst index fe347f9..68724fb 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,13 +1,13 @@ -======================================================================== +===================================================== Welcome to the Ceilometer Redis Plugin Documentation! -======================================================================== +===================================================== .. toctree:: :maxdepth: 2 description - guide installation + guide Indices and Tables ================== diff --git a/doc/source/installation.rst b/doc/source/installation.rst index 53f3d6e..aca0aa4 100644 --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -6,33 +6,30 @@ Installation Guide Install the Plugin ------------------ -To install the Redis plugin: +To install the *Ceilometer Redis Plugin*, you need to follow these steps. -#. Please refer to limitations before you proceed. +#. Please refer to the :ref:`limitations` section before you proceed. -#. Download the Redis plugin from the +#. Download the plugin from the `Fuel Plugins Catalog `_. -#. Move the plugin's rpm to the - `Fuel Master node `_ with secure copy (scp):: +#. Copy the plugin's RPM file to the + `Fuel Master node + `_ + with secure copy (scp):: - scp fuel-plugin-ceilometer-redis/ceilometer-redis-1.0-1.0.0-1.noarch.rpm / - root@::/tmp + # scp fuel-plugin-ceilometer-redis/ceilometer-redis-1.0-1.0.0-1.noarch.rpm / + root@::/tmp +#. Log into the Fuel Master node and install the plugin:: -#. Log into the Fuel Master node and install the Ceilometer Redis plugin:: - - ssh root@: - cd /tmp - fuel plugins --install ceilometer-redis-1.0-1.0.0-1.noarch.rpm - + # ssh root@: + [root@fuel-master ~]# cd /tmp + [root@fuel-master ~]# fuel plugins --install ceilometer-redis-1.0-1.0.0-1.noarch.rpm #. Verify that the plugin is installed correctly:: - [root@fuel-master ~]# fuel plugins list - id | name | version | package_version - ---|------------------|---------------|---------------- - 4 | ceilometer-redis | 1.0.2 | 3.0.0 - - - + [root@fuel-master ~]# fuel plugins list + id | name | version | package_version + ---|------------------|---------------|---------------- + 4 | ceilometer-redis | 1.0.2 | 3.0.0