Merge "Example setup for gnocchi with ceph and redis"

This commit is contained in:
Zuul 2018-10-26 10:55:44 +00:00 committed by Gerrit Code Review
commit f8cb114869
2 changed files with 178 additions and 0 deletions

View File

@ -31,3 +31,4 @@ For in-depth technical information, see the
ceph/ceilometer.rst
security/index.rst
source-overrides/index.rst
prod/gnocchi_redis.rst

View File

@ -0,0 +1,177 @@
==============================================
Telemetry with Gnocchi, Ceph and Redis example
==============================================
The default openstack-ansible installation configures gnocchi to use a file as
storage backend. When you already have a pre-installed ceph, you can use this
as backend for gnocchi. This documentation will guide you how to set up
gnocchi to use your ceph as storage backend.
Ceph as metric storage
~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
gnocchi_storage_driver: ceph
You have to add some pip packages to your gnocchi setup:
.. code-block:: yaml
gnocchi_pip_packages:
- cryptography
- gnocchiclient
# this is what we want:
# - "gnocchi[mysql,ceph,ceph_alternative_lib,redis]"
# but as there is no librados >=12.2 pip package we have to first install ceph without alternative support
# after adding the ceph repo to gnocchi container, python-rados>=12.2.0 is installed and linked automatically
# and gnocchi will automatically take up the features present in the used rados lib.
- "gnocchi[mysql,ceph,redis]"
- keystonemiddleware
- python-memcached
But when your setup grows, gnocchi might slow down or block your ceph
installation. You might experience slow requests and stuck PGs in your Ceph.
As this might have multiple causes, take a look at the presentations linked
in the `Performance Tests for Gnocchi`_ section. They also include various
parameters which you might tune.
Redis as measure storage
~~~~~~~~~~~~~~~~~~~~~~~~
One solution to possible performance problems is to use an incoming measure
storage for your gnocchi installation. The `supported storage systems`_ are:
* File (default)
* Ceph (preferred)
* OpenStack Swift
* Amazon S3
* Redis
.. _supported storage systems: https://gnocchi.xyz/intro.html#incoming-and-storage-drivers
When your Swift installation uses Ceph as backend, the only one left for this
setup is Redis.
So first of all setup a redis server/cluster, e.g. with this `ansible role`_.
Next, you have to configure Gnocchi with OpenStack-Ansible to use the Redis
Cluster as incoming storage:
.. _ansible role: https://github.com/DavidWittman/ansible-redis
.. code-block:: yaml
gnocchi_conf_overrides:
incoming:
driver: redis
redis_url: redis://{{ hostvars[groups['redis-master'][0]]['ansible_default_ipv4']['address'] }}:{{ hostvars[groups['redis-master'][0]]['redis_sentinel_port'] }}?sentinel=master01{% for host in groups['redis-slave'] %}&sentinel_fallback={{ hostvars[host]['ansible_default_ipv4']['address'] }}:{{ hostvars[host]['redis_sentinel_port'] }}{% endfor %}
You also have to install additional pip/distro packages to use the redis
cluster:
.. code-block:: yaml
gnocchi_distro_packages:
- apache2
- apache2-utils
- libapache2-mod-wsgi
- git
- build-essential
- python-dev
- libpq-dev
- python-rados
# additional package for python redis client
- python-redis
.. code-block:: yaml
gnocchi_pip_packages:
- cryptography
- gnocchiclient
# this is what we want:
# - "gnocchi[mysql,ceph,ceph_alternative_lib,redis]"
# but as there is no librados >=12.2 pip package we have to first install ceph without alternative support
# after adding the ceph repo to gnocchi container, python-rados>=12.2.0 is installed and linked automatically
# and gnocchi will automatically take up the features present in the used rados lib.
- "gnocchi[mysql,ceph,redis]"
- keystonemiddleware
- python-memcached
- redis
.. note::
A word of caution: the name of the Ceph alternative lib implementation (ceph_alternative_lib) varies between Gnocchi versions.
Zookeeper for coordination
~~~~~~~~~~~~~~~~~~~~~~~~~~
When you deployed Gnocchi on multiple servers to distribute the work,
add Zookeeper as coordination backend. To setup Zookeeper, you can use
`this ansible role`_.
.. _this ansible role: https://github.com/openstack/ansible-role-zookeeper.git
Create containers for Zookeeper:
.. code-block:: yaml
## conf.d
zookeeper_hosts:
{% for server in groups['control_nodes'] %}
{{ server }}:
ip: {{ hostvars[server]['ansible_default_ipv4']['address'] }}
{% endfor%}
.. code-block:: yaml
## env.d
component_skel:
zookeeper_server:
belongs_to:
- zookeeper_all
container_skel:
zookeeper_container:
belongs_to:
- infra_containers
- shared-infra_containers
contains:
- zookeeper_server
properties:
service_name: zookeeper
Now you can set up Zookeeper as coordination backend for Gnocchi:
.. code-block:: yaml
gnocchi_coordination_url: "zookeeper://{% for host in groups['zookeeper_all'] %}{{ hostvars[host]['container_address'] }}:2181{% if not loop.last %},{% endif %}{% endfor %}"
You also have to install additional packages:
.. code-block:: yaml
gnocchi_pip_packages:
- cryptography
- gnocchiclient
# this is what we want:
# - "gnocchi[mysql,ceph,ceph_alternative_lib,redis]"
# but as there is no librados >=12.2 pip package we have to first install ceph without alternative support
# after adding the ceph repo to gnocchi container, python-rados>=12.2.0 is installed and linked automatically
# and gnocchi will automatically take up the features present in the used rados lib.
- "gnocchi[mysql,ceph,redis]"
- keystonemiddleware
- python-memcached
- redis
# addiitional pip packages needed for zookeeper coordination backend
- tooz
- lz4
- kazoo
Performance Tests for Gnocchi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For more ideas how to tune your Gnocchi stack, take a look at these
presentations:
* https://docs.openstack.org/developer/performance-docs/test_results/telemetry_gnocchi_with_ceph/index.html
* https://julien.danjou.info/talks/OpenStack_Telemetry_and_the_10000_Instances.pdf