Redis Plugin Doc final review for MOS 8 & 7

Change-Id: I422b1fd4c6f6c763c0fb622ce8dd0afba00a4429
This commit is contained in:
Patrick Petit 2016-06-24 18:44:30 +02:00 committed by Nadya Privalova
parent f7a306221a
commit 5e6601c98f
7 changed files with 195 additions and 171 deletions

View File

@ -4,10 +4,10 @@ extensions = []
templates_path = ['_templates']
source_suffix = '.rst'
master_doc = 'index'
project = u'The Redis Plugin for Ceilometer documentation'
project = u'The Ceilometer Redis Plugin'
copyright = u'2016, Mirantis Inc.'
version = '0.1'
release = '0.1.0'
release = '0.1.2'
exclude_patterns = [
]
pygments_style = 'sphinx'
@ -16,15 +16,15 @@ htmlhelp_basename = 'RedisPlugindoc'
latex_elements = {
}
latex_documents = [
('index', 'RedisPlugindoc.tex', u'The Redis Plugin for Ceilometer documentation',
('index', 'RedisPlugindoc.tex', u'The Ceilometer Redis Plugin',
u'Mirantis Inc.', 'manual'),
]
man_pages = [
('index', 'redisplugin', u'The Redis Plugin for Ceilometer documentation',
('index', 'redisplugin', u'The Ceilometer Redis Plugin',
[u'Mirantis Inc.'], 1)
]
texinfo_documents = [
('index', 'RedisPlugin', u'The Redis Plugin for Ceilometer documentation',
('index', 'RedisPlugin', u'The Ceilometer Redis Plugin',
u'Mirantis Inc.', 'RedisPlugin', 'One line description of project.',
'Miscellaneous'),
]

View File

@ -1,45 +1,100 @@
Ceilometer Redis plugin
=======================
Overview
========
Ceilometer Redis Plugin aims to install Redis to MOS environment and provide a coordination mechanism for
`Ceilometer agents <https://ceilometer.readthedocs.org/en/latest/architecture.html>`_ and Alarm Evaluator
through the `tooz library <http://docs.openstack.org/developer/tooz/>`_ with a `Redis backend <http://redis.io>`_
The plugin supports coordination for the following Ceilometer services: central agent and alarm-evaluator.
Each of these services are running on every controller after the plugin is installed. All of them are joined
into the corresponding coordination group (one coordination group per each service). It differs from the default
configuration when there should be only one central agent and alarm-evaluator per cloud. The plugin also configures
redis-server under pacemaker to monitor its process. The plugin configures `redis-sentinel <http://redis.io/topics/sentinel>`_
to monitor the state of the redis cluster, to elect new master during failovers, to forward ceilometer services to new
elected redis master, to organize sync between redis nodes.
The *Ceilometer Redis Plugin* installs `Redis <http://redis.io>`_ and
the `Tooz library <http://docs.openstack.org/developer/tooz/>`_, in a
Mirantis OpenStack (MOS) environment deployed by Fuel.
Both Redis and the Tooz library should be installed on all the controller
nodes of the environment.
The *Ceilometer Redis Plugin* is used to provide coordination mechanisms to
enable the horizontal scaling of the Ceilometer services. Using the plugin,
the Ceilometer services are joined into a so-called **coordination group**,
which allows for resources and alarms sharding.
There is one coordination group per service type.
Central agent
-------------
Ceilometer Central agent is responsible for polling all OpenStack resources except Nova's (Nova resources,
i.e. vms, are polled by a Compute agent). Without coordination enabled, only one Central agent should be running
per cloud. The reason is that all the central agents have the same set of OpenStack resources to poll every
configurable time interval. If coordination is not enabled, each OpenStack resource will be polled as many times
as many instances of Central agents are running in a cloud.
Thus, coordination provides a disjoint set of OpenStack resources to poll for every Central agent running on the
cloud to avoid polling one resource several times.
Please refer to the `Telemetry architecture
<http://docs.openstack.org/admin-guide/telemetry-system-architecture.html>`_
documentation for more information about the Ceilometer services.
Alarm evaluator
---------------
Ceilometer alarm evaluator service is responsible for Ceilometer alarm evaluation.
By default, in MOS there is only one alarm evaluator per cloud. The reason is the same as for a central agent.
If there are several alarm evaluators and no coordination enabled, all of them will evaluate the same set of alarms
every configurable time interval. The alarm sets for evaluators should be disjoint. So, coordination is responsible
for providing the set of alarms to evaluate to each alarm-evaluator in the cloud.
In MOS 7.0 and MOS 8.0, the *Ceilometer Redis Plugin* enables coordination
for both:
* The **ceilometer-agent-central service**.
The ceilometer-agent-central service is responsible for polling all the OpenStack resources,
excepted those of Nova, like the VM instances, that are polled by the **ceilometer-agent-compute**.
Without coordination, there can be only one ceilometer-agent-central running at a time.
This is because, by default, the ceilometer-agent-central works with an entire set of resources.
As such, running multiple ceilometer-agent-central without coordination would poll the entire
set of resources as many times as the number of agents running on the controller nodes every
polling interval. This is obviously not a proper way to scale out the ceilometer-agent-central.
To cope with this problem, the coordination mechanism provided
by the *Ceilometer Redis Plugin* allows distributing the polling workload
across multiple instances of the ceilometer-agent-central using disjoint sets
of resources.
* The **ceilometer-alarm-evaluator service**.
The **ceilometer-alarm-evaluator** service is responsible for evaluating the Ceilometer alarms.
By default, there is only one ceilometer-alarm-evaluator running per environment.
Without coordination, there can be only one ceilometer-alarm-evaluator running at a time.
This is because, as for the ceilometer-agent-central, the ceilometer-alarm-evaluator works
with an entire set of alarms. Running multiple ceilometer-alarm-evaluator
without coordination would evaluate all the alarms as many times as the number of evaluators
running on the controller nodes every evaluation interval. To cope with this problem,
the coordination mechanism provided by the *Ceilometer Redis Plugin* allows distributing
the alarms evaluation workload across multiple instances of the ceilometer-alarm-evaluator
using disjoint sets of alarms.
Please note that with MOS 8.0, the *Ceilometer Redis Plugin* doesn't provide support
(out-of-the-box) for the coordination of the **ceilometer-agent-notification** service because
it is not needed for the most common samples transformations.
.. note:: Before Liberty, the transformation of the samples was handled by the
**ceilometer-agent-compute** and the **ceilometer-agent-central** services.
In Liberty, the transformation of the samples was moved
to the **ceilometer-agent-notification** service, but after thorough performance analysis
of Ceilometer at scale, we discovered that this change has a bad impact on performance.
In MOS 8.0, the transformations for the following list of measurements were moved back
to the ceilometer-agent-compute service.
* cpu_util
* disk.read.requests.rate
* disk.write.requests.rate
* disk.read.bytes.rate
* disk.write.bytes.rate
* disk.device.read.requests.rate
* disk.device.read.bytes.rate
* disk.device.write.bytes.rate
* network.incoming.bytes.rate
* network.outgoing.bytes.rate
* network.incoming.packets.rate
* network.outgoing.packets.rate
As a result, in MOS 8.0, there is no need to run the ceilometer-agent-notification
in coordination mode unless you need to maintain the transformation of custom samples that
are not listed above. In this case, it is possible to enable coordination for the
ceilometer-agent-notification service manually event though, it is not recommended
for performance reasons.
In addition to the above, the *Ceilometer Redis Plugin* configures *Pacemaker*
and `redis-sentinel <http://redis.io/topics/sentinel>`_
to enable **high availability** of the Redis cluster. Redis clustering includes:
* Monitoring the state of the **redis-server** processes
* Elect a new redis-server master during a failover
* Connect Ceilometer to the elected redis-server
* Organize the synchronization between Redis nodes
Requirements
------------
======================= ================
Requirement Version/Comment
Requirements Version/Comment
======================= ================
Fuel 7.0, 8.0
tooz <0.14.0,>=0.13.1
MOS 7.0, 8.0
Tooz <0.14.0,>=0.13.1
======================= ================
.. _limitations:
@ -47,35 +102,13 @@ tooz <0.14.0,>=0.13.1
Limitations
-----------
* The plugin works correctly only on clouds with odd numbers of controllers.
This requirement is mandatory because Redis needs an odd number of nodes to
choose the master successfully.
* Before Liberty, there was no need to coordinate Ceilometer notification agents. Starting from Liberty, samples
transformations started to be handled not by compute/central agents as it was before, but by a notification agent.
Some of Ceilometer transformers have a local cache where they store the data from the previously processed samples.
For example, "cpu_util" metric are obtained from two consecutive Samples with "cpu" metric: one is subtracted from
another and divided by an amount of cpu (this information is stored in Sample's metadata).
Thus, it should be guaranteed that all the Samples which should be transformed by one transformer, will go to the
same notification agent. If some of the samples go to another, the cache cannot be shared and some data will be lost.
To handle this process properly, IPC queues was introduced - inter process communication queues in message bus
(RabbitMQ). With coordination enabled, each notification agent has two set of listeners: for main queues and for IPC
queues. All notification agents listen to _all_ the main queues (where we have all messages from OpenStack services
and polling-based messages from central/compute agents) and re-publish messages to _all_ IPC queues. Coordination
starts to work at this point: every notification agent in the cloud has it's own set of IPC queues to listen to. Thus,
we can be sure that local cache on each notification agent contains all the previous data required for transformation.
After some investigations, some performance issues were found with IPC approach. That's why in In MOS 8.0 all basic
transformations (cpu_util, disk.read.requests.rate, disk.write.requests.rate, disk.read.bytes.rate, disk.write.bytes.rate,
disk.device.read.requests.rate, disk.device.read.bytes.rate, disk.device.write.bytes.rate, network.incoming.bytes.rate,
network.outgoing.bytes.rate, network.incoming.packets.rate, network.outgoing.packets.rate) were moved back to compute
nodes, i.e. for the basic set of transformations there is no need to run notification agent in coordination mode.
That's the reason why the plugin does't support coordination for notification agents, although it is possible to configure
notification agents to run in coordination mode manually. Anyway, it is not recommended.
If you have any custom transformers, you need to be sure that they are cache-less, i.e. are based only on
``unit_conversion`` transformer or the ``arithmetic`` transformer. If it's not the case, you may consider the following
options: run only one notification agent in the cloud or install this plugin and do all configuration manually.
* The *Ceilometer Redis Plugin* requires to install on an odd number of controller
nodes to work properly. This is because, Redis clustering requires an odd number of nodes
to avoid the split brain effect when electing a master.
* If you have any custom transformers you need to ensure that they are cache-less.
If transformation is cache-less, then there is no need to enable the coordination.
That is, based only on the ``unit_conversion`` transformer or the ``arithmetic`` transformers.
Otherwise, you will have to consider running only one instance of the ceilometer-agent-notification
service in your MOS environment or install the *Ceilometer Redis Plugin* and do all the
configuration manually.

View File

@ -1,115 +1,109 @@
User Guide
==========
Once the Ceilometer Redis plugin plugin has been installed (following :ref:`Installation Guide`), you can
create *OpenStack* environments with Ceilometer whose Central agents and Alarm evaluator
work in workload_partitioned mode.
Once the *Ceilometer Redis Plugin* is installed following the instructions of
the :ref:`Installation Guide`, you can create a Mirantis OpenStack (MOS) environment
with Ceilometer whose **ceilometer-agent-central** and **ceilometer-alarm-evaluator**
services will work in **workload partitioned** mode.
This plugin was created to enable the scale-out of these Ceilometer services.
It is useless and **shouldn't be used if Ceilometer is not installed**.
Ceilometer installation
-----------------------
Plugin Configuration
--------------------
This plugin was created to provide partitioning for Ceilometer services. So its
usage is senseless without Ceilometer installed.
So, you will need to `create a new OpenStack environment <https://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#create-a-new-openstack-environment>`_
with `Ceilometer <https://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#related-projects>`_ using the Fuel UI Wizard.
To use the *Ceilometer Redis Plugin*, you need to `create a new MOS environment
<http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-user-guide/create-environment.html>`_
with the `Telemetry service
<http://docs.openstack.org/admin-guide/telemetry.html>`_
(a.k.a Ceilometer) enabled and follow these steps using the *Fuel UI Wizard*.
1. Make sure that the plugin is properly installed on the Fuel Master node.
Plugin configuration in MOS 8.0
-------------------------------
#. First of all, make sure that plugin was successfully installed.
Go to the *Plugins* tab. You should see the following:
.. image:: images/redis-plugin-on8.0.png
:width: 100%
#. The next step is enable the plugin. Go to *Environments* tab and
select the *Redis plugin for Ceilometer* checkbox:
.. image:: images/redis-plugin-8.0.png
:width: 100%
#. When adding nodes to environment and assigning roles to them <https://docs.mirantis.com/openstack/fuel/fuel-8.0/operations.html#adding-redeploying-and-replacing-nodes>`_, please consider using odd number of controllers as mentioned in :ref:`Limitations`.
#. Finish
`environment configuration <https://docs.mirantis.com/openstack/fuel/fuel-8.0/mos-planning-guide.html#fuel-reference-architecture-overview>`_
#. Run `network verification check <https://docs.mirantis.com/openstack/fuel/fuel-8.0/operations.html#network-issues>`_.
#. Press "Deploy button" to once you are done with environment configuration.
Plugin configuration in MOS 7.0
-------------------------------
#. First of all, make sure that plugin was successfully installed.
Go to the *Plugins* tab. You should see the following:
On Mos 8.0
.. image:: images/redis-plugin.png
:width: 100%
#. The next step is enable the plugin. Go to *Environments* tab and
select the *Redis plugin for Ceilometer* checkbox:
On Mos 7.0
.. image:: images/redis-plugin-on8-0.png
:width: 100%
2. Enable the plugin.
Go to the *Environments* tab and select the *Redis plugin for Ceilometer* checkbox:
On Mos 8.0
.. image:: images/redis-plugin-8-0.png
:width: 100%
On Mos 7.0
.. image:: images/redis-plugin-on.png
:width: 100%
#. When
`adding nodes to environment and assigning roles to them in MOS 7.0 <https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#add-nodes-ug>`_, please consider using odd number of controllers as mentioned in :ref:`Limitations`.
3. Add nodes to your environment to which you will assign the **controller role**.
#. Finish
`environment configuration for MOS 7.0 <https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#configure-your-environment>`_
.. note:: When `adding nodes
<http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-user-guide/configure-environment/add-nodes.html>`_
to the environment and `assign or change a role
<http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-user-guide/configure-environment/change-roles.html>`_,
do not forget to use an odd number of controllers as mentioned in :ref:`Limitations` section.
#. Run `network verification check for MOS 7.0 <https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#verify-networks>`_.
4. `Verify your network configuration
<http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-user-guide/configure-environment/verify-networks.html>`_.
#. Press `Deploy button <https://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#deploy-changes>`_ to once you are done with environment configuration.
5. `Deploy your changes
<http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-user-guide/deploy-environment.html>`_
once you are done with the configuration of your environment.
Plugin Verification
-------------------
#. Check that the ceilometer-agent-central and ceilometer-alarm-evaluator services are running
on each controller.
How to check that plugin works
------------------------------
#. Check that ceilometer-agent-central and ceilometer-alarm-evaluator services are running
on each controller. Run ``pcs resource`` and you should see the following in the output::
Run ``pcs resource``. You should see the following in the output::
Clone Set: clone_p_ceilometer-agent-central [p_ceilometer-agent-central]
Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ]
Clone Set: clone_p_ceilometer-agent-central [p_ceilometer-agent-central]
Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ]
Clone Set: clone_p_ceilometer-alarm-evaluator [p_ceilometer-alarm-evaluator]
Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ]
Clone Set: clone_p_ceilometer-alarm-evaluator [p_ceilometer-alarm-evaluator]
Started: [ node-21.domain.tld node-27.domain.tld node-33.domain.tld ]
``Started`` list should contain all controllers.
The *Started* list should contain all controllers.
#. For the central agent: check that samples are not duplicated. For this purpose you may choose
any metric collected by central agent. All these metrics may be found here
`Measurements <http://docs.openstack.org/admin-guide-cloud/telemetry-measurements.html>`_ .
You may choose any section *except* OpenStack Compute and then select metric with 'Pollster' Origin.
For example, let's choose storage.objects.
#. For the ceilometer-agent-central, check that the samples are not duplicated.
For this check you may choose any metric collected by the ceilometer-agent-central.
All the Ceilometer metrics can be found in
`Measurements <http://docs.openstack.org/admin-guide/telemetry-measurements.html>`_ .
You may choose any section excepted *OpenStack Compute* and then select a metric with *Pollster Origin*.
For example, let's choose *storage.objects*.
Plugin works *correctly* if you see one sample for each resource every polling_interval (1 minute in this example)::
The plugin **works correctly** if you see one sample for each resource type every
*polling interval* (1 minute in this example)::
root@node-2:~# ceilometer sample-list -m storage.objects -l 10| grep storage.objects
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object | 2015-11-05T10:32:27 |
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object | 2015-11-05T10:31:29 |
root@node-2:~# ceilometer sample-list -m storage.objects -l 10| grep storage.objects
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:32:27 |
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:31:29 |
The plugin **works incorrectly** if there are duplicates. In this example, the plugin works
incorectly because there are three samples for the same resource type every *polling interval*::
Plugin works *incorrectly* if there are duplications. In this example is seen that every
``polling_interval`` there are 3 samples about one resource::
root@node-2:~# ceilometer sample-list -m storage.objects -l 20| grep storage.objects
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:27:37 |
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:27:26 |
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:27:17 |
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:26:38 |
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:26:26 |
| 65e486c7... | storage.objects | gauge | 0.0 | object | 2015-11-05T10:26:17 |
root@node-2:~# ceilometer sample-list -m storage.objects -l 20| grep storage.objects
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....|
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....|
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....|
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....|
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....|
| 65e486c734394d3ea321ae72639ebe91 | storage.objects | gauge | 0.0 | object ....|
#. For the alarm evaluator, it is possible to see that everything works as expected
only from the logs::
.... 2015-11-05T10:27:37 |
.... 2015-11-05T10:27:26 |
.... 2015-11-05T10:27:17 |
.... 2015-11-05T10:26:38 |
.... 2015-11-05T10:26:26 |
.... 2015-11-05T10:26:17 |
# grep extract_my_subset /var/log/ceilometer/ceilometer-alarm-evaluator.log
#. For the alarm evaluator, it is possible to see that everything works as expected only from the logs. Grep the
line "extract_my_subset". There should be different "My subset: [" results on each alarm evaluator instance.
There should be different *My subset: [* results for the ceilometer-alarm-evaluator instances.

View File

Before

Width:  |  Height:  |  Size: 82 KiB

After

Width:  |  Height:  |  Size: 82 KiB

View File

Before

Width:  |  Height:  |  Size: 69 KiB

After

Width:  |  Height:  |  Size: 69 KiB

View File

@ -1,13 +1,13 @@
========================================================================
=====================================================
Welcome to the Ceilometer Redis Plugin Documentation!
========================================================================
=====================================================
.. toctree::
:maxdepth: 2
description
guide
installation
guide
Indices and Tables
==================

View File

@ -6,33 +6,30 @@ Installation Guide
Install the Plugin
------------------
To install the Redis plugin:
To install the *Ceilometer Redis Plugin*, you need to follow these steps.
#. Please refer to limitations before you proceed.
#. Please refer to the :ref:`limitations` section before you proceed.
#. Download the Redis plugin from the
#. Download the plugin from the
`Fuel Plugins Catalog <https://www.mirantis.com/products/openstack-drivers-and-plugins/fuel-plugins/>`_.
#. Move the plugin's rpm to the
`Fuel Master node <https://docs.mirantis.com/openstack/fuel/fuel-8.0/quickstart-guide.html#quickstart-guide>`_ with secure copy (scp)::
#. Copy the plugin's RPM file to the
`Fuel Master node
<http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/intro/intro_fuel_intro.html>`_
with secure copy (scp)::
scp fuel-plugin-ceilometer-redis/ceilometer-redis-1.0-1.0.0-1.noarch.rpm /
root@:<the_Fuel_Master_node_IP address>:/tmp
# scp fuel-plugin-ceilometer-redis/ceilometer-redis-1.0-1.0.0-1.noarch.rpm /
root@:<the_Fuel_Master_node_IP address>:/tmp
#. Log into the Fuel Master node and install the plugin::
#. Log into the Fuel Master node and install the Ceilometer Redis plugin::
ssh root@:<the_Fuel_Master_node_IP address>
cd /tmp
fuel plugins --install ceilometer-redis-1.0-1.0.0-1.noarch.rpm
# ssh root@:<the_Fuel_Master_node_IP address>
[root@fuel-master ~]# cd /tmp
[root@fuel-master ~]# fuel plugins --install ceilometer-redis-1.0-1.0.0-1.noarch.rpm
#. Verify that the plugin is installed correctly::
[root@fuel-master ~]# fuel plugins list
id | name | version | package_version
---|------------------|---------------|----------------
4 | ceilometer-redis | 1.0.2 | 3.0.0
[root@fuel-master ~]# fuel plugins list
id | name | version | package_version
---|------------------|---------------|----------------
4 | ceilometer-redis | 1.0.2 | 3.0.0