Set a log retention policy for OpenSearch

We previously used ElasticSearch Curator for managing log
retention. Now that we have moved to OpenSearch, we can use
the Index State Management (ISM) plugin which is bundled with
OpenSearch.

This change adds support for automating the configuration of
the ISM plugin via the OpenSearch API. By default, it has
similar behaviour to the previous ElasticSearch Curator
default policy.

Closes-Bug: #2047037

Change-Id: I5c6d938f2bc380f1575ee4f16fe17c6dca37dcba
This commit is contained in:
Doug Szumski 2022-12-07 14:50:21 +00:00 committed by Bartosz Bezak
parent e50c99d1e1
commit 5e5a2dca09
6 changed files with 183 additions and 0 deletions

View File

@ -58,6 +58,54 @@ opensearch_cluster_name: "kolla_logging"
opensearch_heap_size: "1g"
opensearch_java_opts: "{% if opensearch_heap_size %}-Xms{{ opensearch_heap_size }} -Xmx{{ opensearch_heap_size }}{% endif %} -Dlog4j2.formatMsgNoLookups=true"
opensearch_apply_log_retention_policy: true
# Duration after which an index is staged for deletion. This is implemented
# by closing the index. Whilst in this state the index contributes negligible
# load on the cluster and may be manually re-opened if required.
# NOTE: We carry over legacy settings from ElasticSearch Curator if they
# are set. This may be removed in a later release.
opensearch_soft_retention_period_days: "{{ elasticsearch_curator_soft_retention_period_days | default(30) }}"
# Duration after which an index is permanently erased from the cluster.
opensearch_hard_retention_period_days: "{{ elasticsearch_curator_hard_retention_period_days | default(60) }}"
opensearch_retention_policy: |
policy:
description: Retention policy for OpenStack logs
error_notification:
default_state: open
states:
- name: open
actions: []
transitions:
- state_name: close
conditions:
min_index_age: "{{ opensearch_soft_retention_period_days }}d"
- name: close
actions:
- retry:
count: 3
backoff: exponential
delay: 1m
close: {}
transitions:
- state_name: delete
conditions:
min_index_age: "{{ opensearch_hard_retention_period_days }}d"
- name: delete
actions:
- retry:
count: 3
backoff: exponential
delay: 1m
delete: {}
transitions: []
ism_template:
- index_patterns:
- "{{ opensearch_log_index_prefix }}-*"
priority: 1
####################
# Keystone
####################

View File

@ -10,3 +10,6 @@
- name: Flush handlers
meta: flush_handlers
- include_tasks: post-config.yml
when: opensearch_apply_log_retention_policy | bool

View File

@ -0,0 +1,65 @@
---
- name: Wait for OpenSearch to become ready
become: true
kolla_toolbox:
container_engine: "{{ kolla_container_engine }}"
module_name: uri
module_args:
url: "{{ opensearch_internal_endpoint }}/_cluster/stats"
status_code: 200
register: result
until: result.get('status') == 200
retries: 30
delay: 2
run_once: true
- name: Check if a log retention policy exists
become: true
kolla_toolbox:
container_engine: "{{ kolla_container_engine }}"
module_name: uri
module_args:
url: "{{ opensearch_internal_endpoint }}/_plugins/_ism/policies/retention"
method: GET
status_code: 200, 404
return_content: yes
register: opensearch_retention_policy_check
delegate_to: "{{ groups['opensearch'][0] }}"
run_once: true
- name: Create new log retention policy
become: true
kolla_toolbox:
container_engine: "{{ kolla_container_engine }}"
module_name: uri
module_args:
url: "{{ opensearch_internal_endpoint }}/_plugins/_ism/policies/retention"
method: PUT
status_code: 201
return_content: yes
body: "{{ opensearch_retention_policy | from_yaml | to_json }}"
body_format: json
register: opensearch_retention_policy_create
delegate_to: "{{ groups['opensearch'][0] }}"
run_once: true
changed_when: opensearch_retention_policy_create.status == 201
when: opensearch_retention_policy_check.status == 404
- name: Apply retention policy to existing indicies
become: true
vars:
opensearch_set_policy_body: {"policy_id": "retention"}
kolla_toolbox:
container_engine: "{{ kolla_container_engine }}"
module_name: uri
module_args:
url: "{{ opensearch_internal_endpoint }}/_plugins/_ism/add/{{ opensearch_log_index_prefix }}-*"
method: POST
status_code: 200
return_content: yes
body: "{{ opensearch_set_policy_body | to_json }}"
body_format: json
delegate_to: "{{ groups['opensearch'][0] }}"
run_once: true
changed_when: opensearch_retention_policy_create.status == 201
when: opensearch_retention_policy_check.status == 404

View File

@ -46,3 +46,6 @@
- name: Flush handlers
meta: flush_handlers
- include_tasks: post-config.yml
when: opensearch_apply_log_retention_policy | bool

View File

@ -34,6 +34,50 @@ By default OpenSearch is deployed on port ``9200``.
``opensearch`` to store the data of OpenSearch. The path can be set via
the variable ``opensearch_datadir_volume``.
Applying log retention policies
-------------------------------
To stop your disks filling up, the Index State Management plugin for
OpenSearch can be used to define log retention policies. A default
retention policy is applied to all indicies which match the
``opensearch_log_index_prefix``. This policy first closes old indicies,
and then eventually deletes them. It can be customised via the following
variables:
- ``opensearch_apply_log_retention_policy``
- ``opensearch_soft_retention_period_days``
- ``opensearch_hard_retention_period_days``
By default the soft and hard retention periods are 30 and 60 days
respectively. If you are upgrading from ElasticSearch, and have previously
configured ``elasticsearch_curator_soft_retention_period_days`` or
``elasticsearch_curator_hard_retention_period_days``, those variables will
be used instead of the defaults. You should migrate your configuration to
use the new variable names before the Caracal release.
Advanced users may wish to customise the retention policy, which
is possible by overriding ``opensearch_retention_policy`` with
a valid policy. See the `Index Management plugin documentation <https://opensearch.org/docs/latest/im-plugin/index/>`__
for further details.
Updating log retention policies
-------------------------------
By design, Kolla Ansible will NOT update an existing retention
policy in OpenSearch. This is to prevent policy changes that may have
been made via the OpenSearch Dashboards UI, or external tooling,
from being wiped out.
There are three options for modifying an existing policy:
1. Via the OpenSearch Dashboards UI. See the `Index Management plugin documentation <https://opensearch.org/docs/latest/im-plugin/index/>`__
for further details.
2. Via the OpenSearch API using external tooling.
3. By manually removing the existing policy via the OpenSearch Dashboards
UI (or API), before re-applying the updated policy with Kolla Ansible.
OpenSearch Dashboards
~~~~~~~~~~~~~~~~~~~~~

View File

@ -0,0 +1,20 @@
---
features:
- |
Set a log retention policy for OpenSearch via Index State Management (ISM).
`Documentation
<https://docs.openstack.org/kolla-ansible/latest/reference/logging-and-monitoring/central-logging-guide.html#applying-log-retention-policies>`__.
fixes:
- |
Added log retention in OpenSearch, previously handled by Elasticsearch
Curator, now using Index State Management (ISM) OpenSearch bundled plugin.
`LP#2047037 <https://bugs.launchpad.net/kolla-ansible/+bug/2047037>`__.
upgrade:
- |
Added log retention in OpenSearch, previously handled by Elasticsearch
Curator. By default the soft and hard retention periods are 30 and 60 days
respectively. If you are upgrading from Elasticsearch, and have previously
configured ``elasticsearch_curator_soft_retention_period_days`` or
``elasticsearch_curator_hard_retention_period_days``, those variables will
be used instead of the defaults. You should migrate your configuration
to use the new variable names before the Caracal release.