Updated the user section
Fixed couple spellings
Moved the dashboard switching explanation at the begining
of the section.
Removed blank lines in lists
Formatted the list correctly this time.
Change-Id: Icb0122821af81479db397de1cc734091ec3c1178
(cherry picked from commit 92bc84d1c6
)
Before Width: | Height: | Size: 25 KiB After Width: | Height: | Size: 54 KiB |
Before Width: | Height: | Size: 353 KiB After Width: | Height: | Size: 69 KiB |
Before Width: | Height: | Size: 54 KiB After Width: | Height: | Size: 46 KiB |
Before Width: | Height: | Size: 105 KiB After Width: | Height: | Size: 199 KiB |
After Width: | Height: | Size: 202 KiB |
|
@ -10,52 +10,71 @@ Plugin configuration
|
||||||
|
|
||||||
To configure your plugin, you need to follow these steps:
|
To configure your plugin, you need to follow these steps:
|
||||||
|
|
||||||
1. `Create a new environment <http://docs.mirantis.com/openstack/fuel/fuel-7.0/user-guide.html#launch-wizard-to-create-new-environment>`_
|
1. `Create a new environment <http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#launch-wizard-to-create-new-environment>`_
|
||||||
with the Fuel web user interface.
|
from the Fuel web user interface.
|
||||||
|
|
||||||
#. Click on the Settings tab of the Fuel web UI.
|
#. Click the **Settings** tab and select the **Other** category.
|
||||||
|
|
||||||
#. Select the 'Other' section in the left column.
|
#. Scroll down through the settings until you find the **Elasticsearch-Kibana Server
|
||||||
The Elasticsearch-Kibana Plugin settings screen should appear as shown below.
|
Plugin** section. You should see a page like this.
|
||||||
|
|
||||||
.. image:: ../images/elastic_kibana_settings.png
|
.. image:: ../images/elastic_kibana_settings.png
|
||||||
:width: 800
|
:width: 800
|
||||||
:align: center
|
:align: center
|
||||||
|
|
||||||
4. Select the Elasticsearch-Kibana Plugin checkbox and fill-in the required fields.
|
#. Check the *Elasticsearch-Kibana Server Plugin* box and fill-in the required fields
|
||||||
|
as indicated below.
|
||||||
|
|
||||||
a. Specify the data retention period in number of days.
|
a. Specify the number of days of retention for your data.
|
||||||
b. Specify the JVM heap size for Elastisearch. See configuration recommendations below.
|
b. Specify the JVM heap size for Elastisearch. See configuration recommendations below.
|
||||||
|
|
||||||
.. note:: By default, 1GB of heap memory is allocated to the Elasticsearch process.
|
.. note:: By default, 1GB of heap memory is allocated to the Elasticsearch process.
|
||||||
This value is too small to run Elasticsearch for anything else than local testing.
|
This value is too small to run Elasticsearch for anything else than local testing.
|
||||||
To run Elasticsearch in production you need to allocate at least 4 GB of memory
|
To run Elasticsearch in production you need to allocate at least 4 GB of memory
|
||||||
but it is recommended to allocate 50% of the available memory up to 32 GB maximum.
|
but it is recommended to allocate 50% of the available memory up to 32 GB maximum.
|
||||||
|
If you set a value that is greater than the memory size, Elasticsearch won't start.
|
||||||
|
Keep in mind also to reserve enough memory for the operating system and the other services.
|
||||||
|
|
||||||
If you set a value that is greater than the memory size, Elasticsearch won't start.
|
At this point, you can choose to edit advanced settings or let the plugin
|
||||||
Keep in mind also to reserve enough memory for the operating system and the other services.
|
apply sane defaults for you. The advanced settings are used to specify the clustering
|
||||||
|
parameters when the *Elasticsearch-Kibana Server Plugin* is installed on more than one node.
|
||||||
|
To manually configure those advanced settings, check the *Advanced settings* box and fill-in
|
||||||
|
the required parameters.
|
||||||
|
|
||||||
5. Assign the *Elasticsearch Kibana* role to 1 node (up to 5 nodes) as shown in the figure below.
|
#. When you are done with the settings, scroll down to the bottom of the page and click
|
||||||
|
the **Save Settings** button.
|
||||||
|
|
||||||
|
#. Click the *Nodes* tab and assign the *Elasticsearch_Kibana* role to nodes as shown
|
||||||
|
in the figure below. You can see in this example that the *Elasticsearch_Kibana* role
|
||||||
|
is assigned to three different nodes along with the *Infrastructure_Alerting* role
|
||||||
|
and the *InfluxDB_Grafana* role. This means that the three plugins of the LMA toolchain
|
||||||
|
can be installed on the same nodes.
|
||||||
|
|
||||||
|
.. image:: ../images/elastic_kibana_role.png
|
||||||
|
:width: 800
|
||||||
|
:align: center
|
||||||
|
|
||||||
.. image:: ../images/elastic_kibana_role.png
|
.. note:: You can assign the *Elasticsearch_Kibana* role up to five nodes.
|
||||||
:width: 800
|
The Elasticsearch clustering for high availability requires that you assign
|
||||||
:align: center
|
the *Elasticsearch_Kibana* role to at least three nodes. Note also that
|
||||||
|
is possible to add or remove a node with the *Elasticsearch_Kibana* role after deployment.
|
||||||
|
|
||||||
6. Adjust the disk configuration if necessary (see the `Fuel User Guide
|
#. Clik on **Apply Changes**
|
||||||
|
|
||||||
|
#. Adjust the disk configuration if necessary (see the `Fuel User Guide
|
||||||
<http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#disk-partitioning>`_
|
<http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#disk-partitioning>`_
|
||||||
for details). By default, the Elasticsearch-Kibana Plugin allocates:
|
for details). By default, the Elasticsearch-Kibana Plugin allocates:
|
||||||
|
|
||||||
- 20% of the first available disk for the operating system by honoring a range of 15GB minimum and 50GB maximum.
|
- 20% of the first available disk for the operating system by honoring a range of 15GB minimum and 50GB maximum.
|
||||||
- 10GB for */var/log*.
|
- 10GB for */var/log*.
|
||||||
- At least 30 GB for the Elasticsearch database in */opt/es-data*.
|
- At least 30 GB for the Elasticsearch database in */opt/es-data*.
|
||||||
|
|
||||||
7. `Configure your environment <http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#configure-your-environment>`_
|
#. `Configure your environment <http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#configure-your-environment>`_
|
||||||
as needed.
|
as needed.
|
||||||
|
|
||||||
#. `Verify the networks <http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#verify-networks>`_ on the Networks tab of the Fuel web UI.
|
#. `Verify the networks <http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#verify-networks>`_ on the Networks tab of the Fuel web UI.
|
||||||
|
|
||||||
#. `Deploy <http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#deploy-changes>`_ your changes.
|
#. And finally, `Deploy <http://docs.mirantis.com/openstack/fuel/fuel-8.0/user-guide.html#deploy-changes>`_ your changes.
|
||||||
|
|
||||||
.. _plugin_install_verification:
|
.. _plugin_install_verification:
|
||||||
|
|
||||||
|
@ -66,29 +85,76 @@ Be aware, that depending on the number of nodes and deployment setup,
|
||||||
deploying a Mirantis OpenStack environment can typically take anything
|
deploying a Mirantis OpenStack environment can typically take anything
|
||||||
from 30 minutes to several hours. But once your deployment is complete,
|
from 30 minutes to several hours. But once your deployment is complete,
|
||||||
you should see a deployment success notification message with
|
you should see a deployment success notification message with
|
||||||
a link to the Kibana dashboard as shown in the picture below:
|
a link to the Kibana dashboard as shown in the figure below:
|
||||||
|
|
||||||
.. image:: ../images/deploy_notif.png
|
.. image:: ../images/deploy_notif.png
|
||||||
:align: center
|
:align: center
|
||||||
:width: 800
|
:width: 800
|
||||||
|
|
||||||
|
Verifying Elasticsearch
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
You should verify that the Elasticsearch cluster is running properly.
|
||||||
|
To do that, you need first to retrieve the Elasticsearch cluster VIP address.
|
||||||
|
Here is how to proceed.
|
||||||
|
|
||||||
|
#. On the Fuel Master node, find the IP address of a node where the Elasticsearch
|
||||||
|
server is installed using the following command::
|
||||||
|
|
||||||
|
[root@fuel ~]# fuel nodes
|
||||||
|
id | status | name | cluster | ip | mac | roles |
|
||||||
|
---|----------|------------------|---------|------------|-------------------|---------------------------|
|
||||||
|
1 | ready | Untitled (fa:87) | 1 | 10.109.0.8 | 64:18:ef:86:fa:87 | elasticsearch_kibana, ... |
|
||||||
|
2 | ready | Untitled (12:aa) | 1 | 10.109.0.3 | 64:5f:c6:88:12:aa | elasticsearch_kibana, ... |
|
||||||
|
3 | ready | Untitled (4e:6e) | 1 | 10.109.0.7 | 64:ca:bf:a4:4e:6e | elasticsearch_kibana, ... |
|
||||||
|
|
||||||
|
|
||||||
|
#. Then `ssh` to anyone of these nodes (ex. *node-1*) and type the command::
|
||||||
|
|
||||||
|
root@node-1:~# hiera lma::elasticsearch::vip
|
||||||
|
10.109.1.5
|
||||||
|
|
||||||
|
This tells you that the VIP address of your Elasticsearch cluster is *10.109.1.5*.
|
||||||
|
|
||||||
|
#. With that VIP address type the command::
|
||||||
|
|
||||||
|
curl http://10.109.1.5:9200/
|
||||||
|
|
||||||
|
The output should look like something like this::
|
||||||
|
|
||||||
|
{
|
||||||
|
"status" : 200,
|
||||||
|
"name" : "node-3.test.domain.local_es-01",
|
||||||
|
"cluster_name" : "lma",
|
||||||
|
"version" : {
|
||||||
|
"number" : "1.7.4",
|
||||||
|
"build_hash" : "0d3159b9fc8bc8e367c5c40c09c2a57c0032b32e",
|
||||||
|
"build_timestamp" : "2015-12-15T11:25:18Z",
|
||||||
|
"build_snapshot" : false,
|
||||||
|
"lucene_version" : "4.10.4"
|
||||||
|
},
|
||||||
|
"tagline" : "You Know, for Search"
|
||||||
|
}
|
||||||
|
|
||||||
|
Verifying Kibana
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
From the Fuel web UI **Dashboard** view, click on the **Kibana** link,
|
||||||
|
you should be directed to the *Logs Dashboard* as shown in the figure below.
|
||||||
|
|
||||||
|
.. image:: ../images/kibana_logs_dash.png
|
||||||
|
:align: center
|
||||||
|
:width: 800
|
||||||
|
|
||||||
Dashboards management
|
Dashboards management
|
||||||
---------------------
|
---------------------
|
||||||
|
|
||||||
The Elasticsearch-Kibana plugin comes with two pre-configured dashboards:
|
The *Elasticsearch-Kibana Server Plugin* comes with two predefined dashboards:
|
||||||
|
|
||||||
- The *Logs Dashboard* that is the Kibana Home Dashboard for viewing the log messages.
|
- The *Logs Dashboard* that is the Kibana Home Dashboard for viewing the log messages.
|
||||||
- The *Notifications Dashboard* for viewing the OpenStack notifications if you enabled
|
- The *Notifications Dashboard* for viewing the OpenStack notifications if you enabled
|
||||||
this option in the LMA Collector settings.
|
this option in the LMA Collector settings.
|
||||||
|
|
||||||
By default, you will be redirected to the *Logs Dashboard*.
|
|
||||||
|
|
||||||
Each dashboard provides a single pane of glass for visualizing and searching
|
|
||||||
all the logs and notifications of your OpenStack cluster.
|
|
||||||
Note that in the LMA Collector settings, it is possible to tag the logs by
|
|
||||||
environment name so that you can distinguish which logs (and notifications)
|
|
||||||
belong to what environment.
|
|
||||||
|
|
||||||
You can switch from one dashboard to another by clicking on the top-right *Load*
|
You can switch from one dashboard to another by clicking on the top-right *Load*
|
||||||
icon in the toolbar to select the requested dashboard from the list, as shown below.
|
icon in the toolbar to select the requested dashboard from the list, as shown below.
|
||||||
|
|
||||||
|
@ -96,32 +162,30 @@ icon in the toolbar to select the requested dashboard from the list, as shown be
|
||||||
:align: center
|
:align: center
|
||||||
:width: 800
|
:width: 800
|
||||||
|
|
||||||
Pointing your browser to the URL *http://10.20.0.4:80/* you should see the Logs Dashboard:
|
Each dashboard provides a single pane of glass for visualizing and searching
|
||||||
|
all the logs and notifications of your OpenStack environment.
|
||||||
.. image:: ../images/kibana_logs_dash.png
|
Note that in the LMA Collector settings, it is possible to tag the logs by
|
||||||
:align: center
|
environment name so that you can distinguish which logs (and notifications)
|
||||||
:width: 800
|
belong to which environment.
|
||||||
|
|
||||||
As you can see, the Kibana dashboard for logs is divided into four main sections:
|
As you can see, the Kibana dashboard for logs is divided into four main sections:
|
||||||
|
|
||||||
|
.. image:: ../images/kibana_logs_sections.png
|
||||||
|
:align: center
|
||||||
|
:width: 800
|
||||||
|
|
||||||
1. A time-picker control that lets you choose the time period you want
|
1. A time-picker control that lets you choose the time period you want
|
||||||
to select and refresh frequency.
|
to select and refresh frequency.
|
||||||
|
|
||||||
2. A query and filter section where all the filters are displayed.
|
#. A query and filter section where all the filters are displayed.
|
||||||
|
#. A log analytics row which contains six panels to visualize:
|
||||||
3. A log analytics row which contains four panels to visualize:
|
|
||||||
|
|
||||||
a. The number of log messages for the chosen time period.
|
a. The number of log messages for the chosen time period.
|
||||||
|
#. The top 10 hosts filter.
|
||||||
b. The top 10 hosts filter.
|
#. The top 10 log sources.
|
||||||
|
#. The number of log messages grouped by severity.
|
||||||
c. The top 10 log sources.
|
#. The top 10 programs.
|
||||||
|
#. The number of log messages grouped by role.
|
||||||
d. The number of log messages grouped by severity.
|
|
||||||
|
|
||||||
e. The top 10 programs.
|
|
||||||
|
|
||||||
d. The number of log messages grouped by role.
|
|
||||||
|
|
||||||
4. A table of log messages sorted in reverse chronological order.
|
4. A table of log messages sorted in reverse chronological order.
|
||||||
|
|
||||||
|
@ -180,68 +244,46 @@ in *ERROR* versus those that are not as shown below.
|
||||||
Troubleshooting
|
Troubleshooting
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
If you cannot access the Kibana interface or you get no data in dashboards,
|
If you cannot access the Kibana dashboard or you get no data in the dashboard,
|
||||||
follow these troubleshoot tips.
|
follow these troubleshoot tips.
|
||||||
|
|
||||||
1. First, check that Elasticsearch is running properly using *curl*::
|
1. First, check that the LMA Collector is running properly by following the
|
||||||
|
LMA Collector troubleshooting instructions in the
|
||||||
|
`LMA Collector Fuel Plugin User Guide <http://fuel-plugin-lma-collector.readthedocs.org/>`_.
|
||||||
|
|
||||||
curl http://$HOST:9200/
|
#. Check that the nodes are able to connect to the Elasticsearch cluster via the VIP address
|
||||||
|
on port *9200* as explained in the `Verifying Elasticsearch` section above.
|
||||||
|
|
||||||
Where *HOST* is the same IP address as the Kibana dashboard.
|
#. On anyone of the *Elasticsearch_Kibana* role nodes, check the status of the VIP address
|
||||||
The output should look like something like this::
|
and HAProxy resources in the Pacemaker cluster::
|
||||||
|
|
||||||
{
|
root@node-1:~# crm resource status vip__es_vip_mgmt
|
||||||
"status" : 200,
|
resource vip__es_vip_mgmt is running on: node-1.test.domain.local
|
||||||
"name" : "node-10.test.domain.local_es-01",
|
|
||||||
"cluster_name" : "lma",
|
|
||||||
"version" : {
|
|
||||||
"number" : "1.7.4",
|
|
||||||
"build_hash" : "0d3159b9fc8bc8e367c5c40c09c2a57c0032b32e",
|
|
||||||
"build_timestamp" : "2015-12-15T11:25:18Z",
|
|
||||||
"build_snapshot" : false,
|
|
||||||
"lucene_version" : "4.10.4"
|
|
||||||
},
|
|
||||||
"tagline" : "You Know, for Search"
|
|
||||||
}
|
|
||||||
|
|
||||||
2. Check the status of the VIP and HAProxy resources in the Pacemaker cluster::
|
root@node-1:~# crm resource status p_haproxy
|
||||||
|
resource p_haproxy is running on: node-1.test.domain.local
|
||||||
|
|
||||||
# On one of the elasticsearch-kibana node
|
#. If the VIP or HAProxy resources are down, restart them::
|
||||||
root@node-10:~# crm resource status vip__es_vip_mgmt
|
|
||||||
resource vip__es_vip_mgmt is running on: node-10.test.domain.local
|
|
||||||
|
|
||||||
root@node-10:~# crm resource status p_haproxy
|
root@node-1:~# crm resource start vip__es_vip_mgmt
|
||||||
resource p_haproxy is running on: node-10.test.domain.local
|
root@node-1:~# crm resource start p_haproxy
|
||||||
|
|
||||||
3. If the VIP or HAProxy resources are down, restart them::
|
#. Check that the Elasticsearch server is up and running::
|
||||||
|
|
||||||
# On one of the elasticsearch-kibana node
|
|
||||||
root@node-10:~# crm resource start vip__es_vip_mgmt
|
|
||||||
root@node-10:~# crm resource start p_haproxy
|
|
||||||
|
|
||||||
4. Check that the Elasticsearch server is up and running::
|
|
||||||
|
|
||||||
# On both CentOS and Ubuntu
|
# On both CentOS and Ubuntu
|
||||||
[root@node-13 ~]# /etc/init.d/elasticsearch-es-01 status
|
[root@node-1 ~]# /etc/init.d/elasticsearch-es-01 status
|
||||||
|
|
||||||
5. If Elasticsearch is down, start it::
|
#. If Elasticsearch is down, restart it::
|
||||||
|
|
||||||
# On both CentOS and Ubuntu
|
# On both CentOS and Ubuntu
|
||||||
[root@node-13 ~]# /etc/init.d/elasticsearch-es-01 start
|
[root@node-1 ~]# /etc/init.d/elasticsearch-es-01 start
|
||||||
|
|
||||||
6. Check if nginx is up and running::
|
#. Check if nginx is up and running::
|
||||||
|
|
||||||
# On both CentOS and Ubuntu
|
# On both CentOS and Ubuntu
|
||||||
[root@node-13 ~]# /etc/init.d/nginx status
|
[root@node-1 ~]# /etc/init.d/nginx status
|
||||||
|
|
||||||
7. If nginx is down, start it::
|
#. If nginx is down, restart it::
|
||||||
|
|
||||||
# On both CentOS and Ubuntu
|
# On both CentOS and Ubuntu
|
||||||
[root@node-13 ~]# /etc/init.d/nginx start
|
[root@node-1 ~]# /etc/init.d/nginx start
|
||||||
|
|
||||||
8. Check that the LMA Collector is running properly on nodes by following the
|
|
||||||
troubleshooting instructions of the
|
|
||||||
`LMA Collector Fuel Plugin User Guide <http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/configuration.html#troubleshooting>`_.
|
|
||||||
|
|
||||||
9. Check if the nodes are able to connect to the Elasticsearch cluster through
|
|
||||||
the VIP address on port *9200*.
|
|
||||||
|
|