Commit Graph

226 Commits

Author SHA1 Message Date
Stephen Taylor 2fd438b4b1 Update Ceph images to patched 18.2.2 and restore debian-reef repo
This change updates the Ceph images to 18.2.2 images patched with a
fix for https://tracker.ceph.com/issues/63684. It also reverts the
package repository in the deployment scripts to use the debian-reef
directory on download.ceph.com instead of debian-18.2.1. The issue
with the repo that prompted the previous change to debian-18.2.1
has been resolved and the more generic debian-reef directory may
now be used again.

Change-Id: I85be0cfa73f752019fc3689887dbfd36cec3f6b2
2024-03-12 13:45:42 -06:00
Stephen Taylor f641f34b00 [ceph] Update Ceph images to Jammy and Reef 18.2.1
This change updates all Ceph images in openstack-helm-infra to
ubuntu_jammy_18.2.1-1-20240130.

Change-Id: I16d9897bc5f8ca410059a5f53cc637eb8033ba47
2024-01-30 07:58:03 -07:00
astebenkova 98f9438ba7 [elasticsearch-exporter] Update to the latest v1.7.0
The current version of the exporter is outdated, switch to the upstream
+ rename --es.snapshots to --collector.snapshots (v1.7.0) and
  --es.cluster_settings to --collector.clustersettings (v1.6.0)

Change-Id: I4b496d859a4764fbec3271817391667a53286acd
2024-01-18 17:23:24 +02:00
Ritchie, Frank (fr801x) 7167b9bf31 Update curator for es v8
This PS is to update es curator for elasticsearch v8. Curator 5.x
is not compatible with es v8.

Changes are needed for config.yml:

https://github.com/elastic/curator#new-client-configuration

No changes are required for the actions file.

Change-Id: I6968e22c7ae5f630e1342f47feee0c2c494b767f
2023-12-12 11:16:14 -05:00
Ritchie, Frank (fr801x) e36b5d6dab Make curator path configurable
Some es curator images do not use /usr/bin/curator for the executable. This PS
makes the path configurable via values.yaml.

Change-Id: I640e0f4928683810ef0b4a6d4dbac9bdf865aa2a
2023-12-05 17:11:15 -05:00
Vladimir Kozhukalov 7f783dba51 Update elasticsearch chart to work with Rook Ceph
When using Rook for managing Ceph we can use
Rook CRDs to create S3 buckets and users.

This PR adds bucket claim template to the
elasticsearch chart. Rook creates a bucket for
a bucket claim and also creates a secret
containing the credentials to get access to this
bucket. So we also add a snippet to expose
these credentials via environment variables to
containers where they are needed.

Change-Id: Ic5cd35a5c64a914af97d2b3cfec21dbe399c0f14
2023-11-26 19:34:42 -06:00
Stephen Taylor 5e5a52cc04 Update Rook to 1.12.5 and Ceph to 18.2.0
This change updates Rook to the 1.12.5 release and Ceph to the
18.2.0 (Reef) release.

Change-Id: I546780ce33b6965aa699f1578d1db9790dc4e002
2023-10-13 12:58:56 -06:00
Vladimir Kozhukalov ae91cf3fc3 Use deploy-env role for all deployment jobs
To make it easier to maintain the jobs all experimental
jobs (those which are not run in check and gate pipelines)
are moved to a separate file. They will be revised later
to use the same deploy-env role.

Also many charts use Openstack images for testing this
PR adds 2023.1 Ubuntu Focal overrides for all these charts.

Change-Id: I4a6fb998c7eb1026b3c05ddd69f62531137b6e51
2023-09-22 15:02:07 -05:00
Leontii Istomin 4a74ff2ba9 Upgrade ElasticSearch and Kibana to v8.9.0
Change-Id: I5ce965a2abf40bad14f0a8a505c8f3000f110d37
2023-08-24 11:09:19 -05:00
Stephen Taylor 45b492bcf7 [ceph] Update Ceph to 17.2.6
This change updates the openstack-helm-infra charts to use 17.2.6
Quincy images based on Focal.

See https://review.opendev.org/c/openstack/openstack-helm-images/+/881217

Change-Id: Ibb89435ae22f6d634846755e8121facd13d5d331
2023-05-09 12:25:07 +00:00
Ruslan Aliev c4a9e8b03d Add configurable liveness probe for elasticsearch client
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I46e1382123ce4497e3f8e414a83fe0861f0cf43b
2023-04-07 15:12:34 -05:00
Stephen Taylor fc92933346 [ceph] Update all Ceph images to Focal
This change updates all Ceph image references to use Focal images
for all charts in openstack-helm-infra.

Change-Id: I759d3bdcf1ff332413e14e367d702c3b4ec0de44
2023-03-16 16:39:37 -06:00
Brian Haley f31cfb2ef9 support image registries with authentication
Based on spec in openstack-helm repo,
support-OCI-image-registry-with-authentication-turned-on.rst

Each Helm chart can configure an OCI image registry and
credentials to use. A Kubernetes secret is then created with these
info. Service Accounts then specify an imagePullSecret specifying
the Secret with creds for the registry. Then any pod using one
of these ServiceAccounts may pull images from an authenticated
container registry.

Change-Id: Iebda4c7a861aa13db921328776b20c14ba346269
2022-07-20 14:28:47 -05:00
Ritchie, Frank (fr801x) 4f0f5155e7 Set default python
Python needs to be set to python2 before checking for python3 to prevent
certain test framework errors.

Change-Id: Ifd1ed35160338688d3c723c055ca75cd999e46e0
2022-06-27 17:58:20 +00:00
Schubert Anselme 753a32c33d
Migrate CronJob resources to batch/v1 and PodDisruptionBudget resources to policy/v1
This change updates the following charts to migrate CronJob resources to the batch/v1 API version, available since v1.21. [0]
and to migrate PodDisruptionBudget to the policy/v1 API version, also available since v1.21. [1]

This also uplift ingress controller to 1.1.3

- ceph-client (CronJob)
- cert-rotation (CronJob)
- elasticsearch (CronJob)
- mariadb (CronJob & PodDisruptionBudget)
- postgresql (CronJob)

0: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#cronjob-v125
1: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#poddisruptionbudget-v125

Change-Id: Ia6189b98a86b3f7575dc4678bb3a0cce69562c93
2022-05-10 15:12:53 -04:00
Gage Hugo 09d8d190ef Update default image value in elasticsearch
This change updates the default image value in the elasticsearch
chart from newton to wallaby for the one image that utilizes a
heat image.

Change-Id: Ia94cfb62a6602dcaf465c2c314ee75d24cff4286
2022-04-27 12:39:48 -05:00
Phil Sphicas dbf841c09c Annotate ES master/data sts with S3 secret hash
To ensure that a Helm upgrade with changed S3 credentials results in a
restart of the elasticsearch-master and elasticsearch-data pods, add an
annotation with the hash of the S3 secret.

Change-Id: Id30e5749a378167b9c2c14a155bc6ca236d78516
2022-04-27 08:37:06 -07:00
Ritchie, Frank (fr801x) 3ce8d71483 Use python3 when present
Some newer images include python3 but not python. This change will
alias python to python3 when the executable is found.

Change-Id: I752a265c67887b6e6b2389bf4009bdbf8e2aed09
2022-03-31 13:33:52 -05:00
Phil Sphicas 03e7fedb2b Fix elasticsearch-data shutdown
The shutdown script for the elasticsearch-data container uses a trap
handler to run the steps outlined in the rolling restart procedure [0].
However, when trying to kill the elasticsearch process (step 3), the
script sends the TERM signal to itself.

The traps are handled recursively, causing the entire termination grace
period to be exhausted before the pod is finally removed.

This change updates the trap handler to terminate the child process(es)
instead, and wait for their completion.

0: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/restart-cluster.html

Change-Id: I0c92ea5cce345cff951f044026a2179dcbd5a3e2
2022-03-16 16:04:15 -07:00
Phil Sphicas c3da3a6f79 Fix elasticsearch cronjob rendering
The pod security context for the elasticsearch cron jobs is in the wrong
location, causing an error when installing or upgrading the chart.

    ValidationError(CronJob.spec.jobTemplate.spec):
        unknown field "securityContext" in io.k8s.api.batch.v1.JobSpec

This change fixes the rendering.

Change-Id: I0e04b1ba27113d4b7aeefa2035b2b29c45be455a
2022-03-16 15:58:31 -07:00
Gage Hugo 22e50a5569 Update htk requirements
This change updates the helm-toolkit path in each chart as part
of the move to helm v3. This is due to a lack of helm serve.

Change-Id: I011e282616bf0b5a5c72c1db185c70d8c721695e
2021-10-06 01:02:28 +00:00
Sean Eagan b1a247e7f5 Helm 3 - Fix Job labels
If labels are not specified on a Job, kubernetes defaults them
to include the labels of their underlying Pod template. Helm 3
injects metadata into all resources [0] including a
`app.kubernetes.io/managed-by: Helm` label. Thus when kubernetes
sees a Job's labels they are no longer empty and thus do not get
defaulted to the underlying Pod template's labels. This is a
problem since Job labels are depended on by
- Armada pre-upgrade delete hooks
- Armada wait logic configurations
- kubernetes-entrypoint dependencies

Thus for each Job template this adds labels matching the
underlying Pod template to retain the same labels that were
present with Helm 2.

[0]: https://github.com/helm/helm/pull/7649

Change-Id: I3b6b25fcc6a1af4d56f3e2b335615074e2f04b6d
2021-09-30 16:01:31 -05:00
Lo, Chi (cl566n) 09dfafbd6b Enable TLS path between Curator and Elasticsearch
Elasticsearch is TLS enabled.  Curator needs to be configured to use
cacert when communicating with Elasticsearch.

Change-Id: Ia78458516d6c8f975e478d85643dc4436b70b87c
2021-08-11 18:28:05 +00:00
Lo, Chi (cl566n) 830df06628 Enable TLS path between Prometheus-elasticsearch-exporter and Elasticsearch
Elasticsearch is TLS enabled.  Prometheus-elasticsearch-exporter
needs to be configured to use cacert when communicating with Elasticsearch.

Change-Id: I4a87226fed541777df78733f3650363859ff01b8
2021-08-06 10:02:18 -07:00
aw4825 ff2d317064 Removed additional checks from Elasicsearch Helm test
This test (create and remove test index) already validates that elasticsearch is working correctly. Removed additional check for repo verification for external service like S3 as this seems out of scope since this can be configured differently and causes test to fail.

Change-Id: Ic9328b204c82bdf0e328370d7060a265210c9e8a
2021-06-16 13:34:02 -05:00
Thiago Brito 5a0ba49d50 Prepending library/ to docker official images
This will ease mirroring capabilities for the docker official images.

Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I0f9177b0b83e4fad599ae0c3f3820202bf1d450d
2021-06-02 15:04:38 -03:00
Lo, Chi (cl566n) f7fde88b6e Remove env variable from s3 bucket job
Remove the TLS_OPTION env from helm-toolkit s3-bucket job. There
can be different option for tls connection, depending on whether
the rgw server is local or remote. This change allows the
create-s3-bucket script to customize its connection argument
which can be pulled from values.yaml.

Change-Id: I2a34c1698e02cd71905bc6ef66f4aefcd5e25e44
2021-05-14 15:12:15 -07:00
Lo, Chi (cl566n) fd4bf57211 Enable TLS for Elasticsearch
The change enables:

(1) TLS for the Elasticsearch transport networking layer. The
    transport networking layer is used for internal communication
    between nodes in a cluster.

(2) TLS path between Elasticsearch and Ceph-rgw host.

Change-Id: Ifb6cb5db19bc5db2c8cb914f6a5887cf3d0f9434
2021-05-03 19:39:32 -07:00
Lo, Chi (cl566n) 9a719e2a18 Enable TLS between Elasticsearch and Kibana
This change enables TLS between Elasticsearch and Kibana
data path. Note that TLS terminates at apache-proxy container
of the Elasticsearch-client pod, not directly to port 9200 of
elasticsearch-client container.

Since all data traffic goes through apache-proxy container,
fluentd output to Elasticsearch are configured to have TLS
enabled as well.

In additon, other Elasticsearch pods that communicate with
Elasticsearch-client endpoint are modified to provide
the cacert option with curl.

Change-Id: I3373c0c350b30c175be4a34d25a403b9caf74294
2021-04-25 09:07:33 -07:00
Pai, Radhika (rp592h) ed8c3fac88 [Update] ES helm-test script updated
This ps removes the test_api_object_creation function as the api_objects map is now more
dynamic with ability to create, delete etc.
This function throws error when it does a GET on the objects that first
needs to be created(PUT).
This function is no longer relevant with the updated create-templates
job which is more robust.

Change-Id: I9f37c86ae9ca4bf32c417880926b6a3c3e78cb8a
2021-04-20 10:40:04 -05:00
Steven Fitzpatrick 38e6023351 Elasticsearch: Add configurable backoffLimit to templates job
This change allows us to control the backofflimit for this job

Change-Id: I9c3ccc0842a0e5c31b7838576648dae966b15a6e
2021-04-16 18:01:47 +00:00
Pai, Radhika (rp592h) dbb20c786d [fix] Update the ES curator config
The curator actions in the configmap gets set to
null which is causing error when redering any actions downstream.
Adding the {} should resolve this issue.

Change-Id: I8c337ee1f089c13f75cb7a9997a7bf6f04246160
2021-04-14 14:35:00 -05:00
Steven Fitzpatrick d3c6069be3 Elasticsearch: Make templates job more robust
This change primarily changes the type of the api_objects yaml structure
to a map, which allows for additional objects to be added by values
overrides (Arrays/Lists are not mutable like this)

Also, in the previous change, some scripts in HTK were modified, while
other were copied over to the Elasticsearch chart. To simplify the chart's
structure, this change also moves the create_s3_bucket script to Elasticsearch,
and reverts the changes in HTK.

Those HTK scripts are no longer referenced by osh charts, and could be candidates
for removal if that chart needed to be pruned

Change-Id: I7d8d7ef28223948437450dcb64bd03f2975ad54d
2021-04-12 18:40:11 +00:00
Steven Fitzpatrick 6de864110e Elasticsearch S3 Update
This change updates how the Elasticsearch chart handles
S3 configuration and snapshot repository registration.

This allows for
  - Multiple snapshot destinations to be configued
  - Repositories to use a specific placement target
  - Management of multiple account credentials

Change-Id: I12de918adc5964a4ded46f6f6cd3fa94c7235112
2021-04-06 15:12:34 +00:00
Steven Fitzpatrick 4fb159f7a3 Elasticsearch Disable Curator in Gate & Chart Defaults
Since chart v0.1.3 SLM policies have been supported, but we still
run curator in the gate, and its manifest toggles still default to
true

Change-Id: I5d8a29ae78fa4f93cb71bdf6c7d1ab3254c31325
2021-03-22 21:16:59 +00:00
Mohammed Naser 737f5610e3 Pin a few Java configuration values to 8-13
The newer versions of ElasticSearch use Java 15 which has dropped
some of those options, we can keep backwards compatibility by
pinning to certain versions[1].

[1]: https://discuss.elastic.co/t/elasticsearch-wont-start-after-7-9-1-to-7-9-2-upgrade/249878/2

Change-Id: Iaa29bc202d9eb9c5eda3040b38596f0524a0c453
2021-03-10 17:23:36 -05:00
Phil Sphicas b11fa5509b Fix elasticsearch-master rendering error
Update the elasticsearch-master statefulset to use the correct
helm-toolkit snippet for the update strategy.

Change-Id: Ifd07a13cc63f1ba610a3f70052ec64be9db3b09c
2021-02-23 20:17:18 +00:00
Steven Fitzpatrick 0ab71ae35c Elasticsearch: Make templates job more generic
This change updates the logic in our create-elasticsearch-templates
job to support creation of a variety of different API objects.

Change-Id: I380a55b93e7aabb606e713c21d71a383fef78b3f
2021-02-03 22:40:19 +00:00
Graham Steffaniak c1241918c2 Add elasticsearch ILM functionality
Add functionality to delete indexes older than 14 days. ILM api
will handle deleting indexes.

Change-Id: I22c02af78b6ce979d0c70b420c106917b0fc5a4e
2021-01-21 09:02:57 -06:00
Graham Steffaniak fcb4681cb1 Add elasticsearch snapshot policy template for SLM
ADD: new snapshot policy template job which creates templates for
        ES SLM manager to snapshot indicies instead of curator.

Change-Id: I629d30691d6d3f77646bde7d4838056b117ce091
2020-12-29 15:55:53 +00:00
Steven Fitzpatrick 6c05fee08d Elasticsearch: Update to 7.6.2 image
Change-Id: Ic0f5b6c802938ca91726210c43f81d2c73969575
2020-12-14 20:29:16 +00:00
Andrii Ostapenko 1532958c80
Change helm-toolkit dependency version to ">= 0.1.0"
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0

Change-Id: I15950b735b4f8566bc0018fe4f4ea9ba729235fc
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-24 12:19:28 -05:00
Mohammed Naser c7a45f166f Run chart-testing on all charts
Added chart lint in zuul CI to enhance the stability for charts.
Fixed some lint errors in the current charts.

Change-Id: I9df4024c7ccf8b3510e665fc07ba0f38871fcbdb
2020-09-11 18:02:38 +03:00
Steven Fitzpatrick 68cd0027d1 Fluentd & Elasticsaerch: Use the latest openstackhelm image tag
Also, removed an unnecessary image reference from the fluentd chart

Change-Id: Ic9ce88f5ddc5096b2eed2ed2286bc73fe6dd5e73
2020-07-22 16:35:16 -05:00
Zuul 89cdcf7771 Merge "Remove the Elasticsearch Wait job from the chart" 2020-07-17 09:59:12 +00:00
KHIYANI, RAHUL (rk0850) 9cfb1f8509 Add missing security-context for elasticsearch-data and elasticsearch-master
This also implements security-context template to add readOnly-fs flag

Change-Id: Iaeea66dad34a2616c0620eafacc53574ed79a7b5
2020-07-16 19:54:43 +00:00
Steven Fitzpatrick 3257ed1db8 Remove the Elasticsearch Wait job from the chart
The elastic-cluster-wait job was meant to serve as a dependency check
for a couple of other jobs, such that when this wait job was complete
the other jobs could procede successfully. This goal can be achieved
by using our HTK init container's dependency check however.

The two jobs that waited on this wait job just need to use the
elasticsearch API, which is available once the `elasticsearch-logging`
service has endpoints.

Change-Id: I87e1c1fe3d61680a73701d48f85e5c48c11b6325
2020-07-16 10:16:23 -05:00
Steven Fitzpatrick 083c9498c6 Elasticsearch: Improve logging in cluster wait
The cluster wait function can sometimes receive an invalid response,
and this would "pass" the status check condition. This change
prints the response to make it more clear what occured, and changes
the condition to explicitly wait for a "yellow" or "green" status.

Change-Id: Ifd1267a5fa19acbc6bc8bba65b1ba41409a584a3
2020-07-13 16:28:14 -05:00
Zuul 84426374b6 Merge "Elasticsearch - Cluster Wait Function Improvements" 2020-07-13 16:06:28 +00:00
Steven Fitzpatrick 57b1f3905b Elasticsearch - Cluster Wait Function Improvements
This change modifies the cluster wait function to
check the cluster health status explicitly.

Once a status of at least "yellow" has been reached,
the Elasticsearch cluster should be able to facilitate
the API calls required by the other jobs of this chart.

Change-Id: I2660422a8e8122186d648042f5422ca9a82d23c7
2020-07-10 15:01:30 -05:00