This change updates the Ceph images to 18.2.2 images patched with a
fix for https://tracker.ceph.com/issues/63684. It also reverts the
package repository in the deployment scripts to use the debian-reef
directory on download.ceph.com instead of debian-18.2.1. The issue
with the repo that prompted the previous change to debian-18.2.1
has been resolved and the more generic debian-reef directory may
now be used again.
Change-Id: I85be0cfa73f752019fc3689887dbfd36cec3f6b2
The current version of the exporter is outdated, switch to the upstream
+ rename --es.snapshots to --collector.snapshots (v1.7.0) and
--es.cluster_settings to --collector.clustersettings (v1.6.0)
Change-Id: I4b496d859a4764fbec3271817391667a53286acd
This PS is to update es curator for elasticsearch v8. Curator 5.x
is not compatible with es v8.
Changes are needed for config.yml:
https://github.com/elastic/curator#new-client-configuration
No changes are required for the actions file.
Change-Id: I6968e22c7ae5f630e1342f47feee0c2c494b767f
Some es curator images do not use /usr/bin/curator for the executable. This PS
makes the path configurable via values.yaml.
Change-Id: I640e0f4928683810ef0b4a6d4dbac9bdf865aa2a
When using Rook for managing Ceph we can use
Rook CRDs to create S3 buckets and users.
This PR adds bucket claim template to the
elasticsearch chart. Rook creates a bucket for
a bucket claim and also creates a secret
containing the credentials to get access to this
bucket. So we also add a snippet to expose
these credentials via environment variables to
containers where they are needed.
Change-Id: Ic5cd35a5c64a914af97d2b3cfec21dbe399c0f14
To make it easier to maintain the jobs all experimental
jobs (those which are not run in check and gate pipelines)
are moved to a separate file. They will be revised later
to use the same deploy-env role.
Also many charts use Openstack images for testing this
PR adds 2023.1 Ubuntu Focal overrides for all these charts.
Change-Id: I4a6fb998c7eb1026b3c05ddd69f62531137b6e51
This change updates all Ceph image references to use Focal images
for all charts in openstack-helm-infra.
Change-Id: I759d3bdcf1ff332413e14e367d702c3b4ec0de44
Based on spec in openstack-helm repo,
support-OCI-image-registry-with-authentication-turned-on.rst
Each Helm chart can configure an OCI image registry and
credentials to use. A Kubernetes secret is then created with these
info. Service Accounts then specify an imagePullSecret specifying
the Secret with creds for the registry. Then any pod using one
of these ServiceAccounts may pull images from an authenticated
container registry.
Change-Id: Iebda4c7a861aa13db921328776b20c14ba346269
Python needs to be set to python2 before checking for python3 to prevent
certain test framework errors.
Change-Id: Ifd1ed35160338688d3c723c055ca75cd999e46e0
This change updates the default image value in the elasticsearch
chart from newton to wallaby for the one image that utilizes a
heat image.
Change-Id: Ia94cfb62a6602dcaf465c2c314ee75d24cff4286
To ensure that a Helm upgrade with changed S3 credentials results in a
restart of the elasticsearch-master and elasticsearch-data pods, add an
annotation with the hash of the S3 secret.
Change-Id: Id30e5749a378167b9c2c14a155bc6ca236d78516
Some newer images include python3 but not python. This change will
alias python to python3 when the executable is found.
Change-Id: I752a265c67887b6e6b2389bf4009bdbf8e2aed09
The shutdown script for the elasticsearch-data container uses a trap
handler to run the steps outlined in the rolling restart procedure [0].
However, when trying to kill the elasticsearch process (step 3), the
script sends the TERM signal to itself.
The traps are handled recursively, causing the entire termination grace
period to be exhausted before the pod is finally removed.
This change updates the trap handler to terminate the child process(es)
instead, and wait for their completion.
0: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/restart-cluster.html
Change-Id: I0c92ea5cce345cff951f044026a2179dcbd5a3e2
The pod security context for the elasticsearch cron jobs is in the wrong
location, causing an error when installing or upgrading the chart.
ValidationError(CronJob.spec.jobTemplate.spec):
unknown field "securityContext" in io.k8s.api.batch.v1.JobSpec
This change fixes the rendering.
Change-Id: I0e04b1ba27113d4b7aeefa2035b2b29c45be455a
This change updates the helm-toolkit path in each chart as part
of the move to helm v3. This is due to a lack of helm serve.
Change-Id: I011e282616bf0b5a5c72c1db185c70d8c721695e
If labels are not specified on a Job, kubernetes defaults them
to include the labels of their underlying Pod template. Helm 3
injects metadata into all resources [0] including a
`app.kubernetes.io/managed-by: Helm` label. Thus when kubernetes
sees a Job's labels they are no longer empty and thus do not get
defaulted to the underlying Pod template's labels. This is a
problem since Job labels are depended on by
- Armada pre-upgrade delete hooks
- Armada wait logic configurations
- kubernetes-entrypoint dependencies
Thus for each Job template this adds labels matching the
underlying Pod template to retain the same labels that were
present with Helm 2.
[0]: https://github.com/helm/helm/pull/7649
Change-Id: I3b6b25fcc6a1af4d56f3e2b335615074e2f04b6d
Elasticsearch is TLS enabled. Curator needs to be configured to use
cacert when communicating with Elasticsearch.
Change-Id: Ia78458516d6c8f975e478d85643dc4436b70b87c
Elasticsearch is TLS enabled. Prometheus-elasticsearch-exporter
needs to be configured to use cacert when communicating with Elasticsearch.
Change-Id: I4a87226fed541777df78733f3650363859ff01b8
This test (create and remove test index) already validates that elasticsearch is working correctly. Removed additional check for repo verification for external service like S3 as this seems out of scope since this can be configured differently and causes test to fail.
Change-Id: Ic9328b204c82bdf0e328370d7060a265210c9e8a
This will ease mirroring capabilities for the docker official images.
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I0f9177b0b83e4fad599ae0c3f3820202bf1d450d
Remove the TLS_OPTION env from helm-toolkit s3-bucket job. There
can be different option for tls connection, depending on whether
the rgw server is local or remote. This change allows the
create-s3-bucket script to customize its connection argument
which can be pulled from values.yaml.
Change-Id: I2a34c1698e02cd71905bc6ef66f4aefcd5e25e44
The change enables:
(1) TLS for the Elasticsearch transport networking layer. The
transport networking layer is used for internal communication
between nodes in a cluster.
(2) TLS path between Elasticsearch and Ceph-rgw host.
Change-Id: Ifb6cb5db19bc5db2c8cb914f6a5887cf3d0f9434
This change enables TLS between Elasticsearch and Kibana
data path. Note that TLS terminates at apache-proxy container
of the Elasticsearch-client pod, not directly to port 9200 of
elasticsearch-client container.
Since all data traffic goes through apache-proxy container,
fluentd output to Elasticsearch are configured to have TLS
enabled as well.
In additon, other Elasticsearch pods that communicate with
Elasticsearch-client endpoint are modified to provide
the cacert option with curl.
Change-Id: I3373c0c350b30c175be4a34d25a403b9caf74294
This ps removes the test_api_object_creation function as the api_objects map is now more
dynamic with ability to create, delete etc.
This function throws error when it does a GET on the objects that first
needs to be created(PUT).
This function is no longer relevant with the updated create-templates
job which is more robust.
Change-Id: I9f37c86ae9ca4bf32c417880926b6a3c3e78cb8a
The curator actions in the configmap gets set to
null which is causing error when redering any actions downstream.
Adding the {} should resolve this issue.
Change-Id: I8c337ee1f089c13f75cb7a9997a7bf6f04246160
This change primarily changes the type of the api_objects yaml structure
to a map, which allows for additional objects to be added by values
overrides (Arrays/Lists are not mutable like this)
Also, in the previous change, some scripts in HTK were modified, while
other were copied over to the Elasticsearch chart. To simplify the chart's
structure, this change also moves the create_s3_bucket script to Elasticsearch,
and reverts the changes in HTK.
Those HTK scripts are no longer referenced by osh charts, and could be candidates
for removal if that chart needed to be pruned
Change-Id: I7d8d7ef28223948437450dcb64bd03f2975ad54d
This change updates how the Elasticsearch chart handles
S3 configuration and snapshot repository registration.
This allows for
- Multiple snapshot destinations to be configued
- Repositories to use a specific placement target
- Management of multiple account credentials
Change-Id: I12de918adc5964a4ded46f6f6cd3fa94c7235112
Since chart v0.1.3 SLM policies have been supported, but we still
run curator in the gate, and its manifest toggles still default to
true
Change-Id: I5d8a29ae78fa4f93cb71bdf6c7d1ab3254c31325
Update the elasticsearch-master statefulset to use the correct
helm-toolkit snippet for the update strategy.
Change-Id: Ifd07a13cc63f1ba610a3f70052ec64be9db3b09c
This change updates the logic in our create-elasticsearch-templates
job to support creation of a variety of different API objects.
Change-Id: I380a55b93e7aabb606e713c21d71a383fef78b3f
ADD: new snapshot policy template job which creates templates for
ES SLM manager to snapshot indicies instead of curator.
Change-Id: I629d30691d6d3f77646bde7d4838056b117ce091
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0
Change-Id: I15950b735b4f8566bc0018fe4f4ea9ba729235fc
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
Added chart lint in zuul CI to enhance the stability for charts.
Fixed some lint errors in the current charts.
Change-Id: I9df4024c7ccf8b3510e665fc07ba0f38871fcbdb
The elastic-cluster-wait job was meant to serve as a dependency check
for a couple of other jobs, such that when this wait job was complete
the other jobs could procede successfully. This goal can be achieved
by using our HTK init container's dependency check however.
The two jobs that waited on this wait job just need to use the
elasticsearch API, which is available once the `elasticsearch-logging`
service has endpoints.
Change-Id: I87e1c1fe3d61680a73701d48f85e5c48c11b6325
The cluster wait function can sometimes receive an invalid response,
and this would "pass" the status check condition. This change
prints the response to make it more clear what occured, and changes
the condition to explicitly wait for a "yellow" or "green" status.
Change-Id: Ifd1267a5fa19acbc6bc8bba65b1ba41409a584a3
This change modifies the cluster wait function to
check the cluster health status explicitly.
Once a status of at least "yellow" has been reached,
the Elasticsearch cluster should be able to facilitate
the API calls required by the other jobs of this chart.
Change-Id: I2660422a8e8122186d648042f5422ca9a82d23c7