Commit Graph

89 Commits

Author SHA1 Message Date
Vladimir Kozhukalov ae91cf3fc3 Use deploy-env role for all deployment jobs
To make it easier to maintain the jobs all experimental
jobs (those which are not run in check and gate pipelines)
are moved to a separate file. They will be revised later
to use the same deploy-env role.

Also many charts use Openstack images for testing this
PR adds 2023.1 Ubuntu Focal overrides for all these charts.

Change-Id: I4a6fb998c7eb1026b3c05ddd69f62531137b6e51
2023-09-22 15:02:07 -05:00
Leontii Istomin 4a74ff2ba9 Upgrade ElasticSearch and Kibana to v8.9.0
Change-Id: I5ce965a2abf40bad14f0a8a505c8f3000f110d37
2023-08-24 11:09:19 -05:00
Alexey Terekhin 2dcd38e4b0 Update kibana index pattern creation
This change updates the kibana indices creation to repeatedly make
call attempts until we get a 200 response back.

Change-Id: Id0f012bda83913fc66c4ce105de97496043e487c
2022-08-04 15:36:13 +00:00
Brian Haley f31cfb2ef9 support image registries with authentication
Based on spec in openstack-helm repo,
support-OCI-image-registry-with-authentication-turned-on.rst

Each Helm chart can configure an OCI image registry and
credentials to use. A Kubernetes secret is then created with these
info. Service Accounts then specify an imagePullSecret specifying
the Secret with creds for the registry. Then any pod using one
of these ServiceAccounts may pull images from an authenticated
container registry.

Change-Id: Iebda4c7a861aa13db921328776b20c14ba346269
2022-07-20 14:28:47 -05:00
Gage Hugo 34b3a013e4 Update kibana image default values
This change updates the default image values for the kibana chart
to move the heat images from newton to wallaby.

Change-Id: Ic991664c2f18354fae3f8b21aee028bad4716987
2022-04-27 12:19:16 -05:00
Chi Lo fa2c1e0b55 Revert "Remove Kibana indices before pod start up"
This reverts commit 122dcef629.
https://review.opendev.org/c/openstack/openstack-helm-infra/+/805246

The changes from the above patchset is a result of upgrading
Elasticsearch and Kibana images to v7.14.  This image has been
reverted back to v7.9.2.  As such, these changes are no longer
correct.

Change-Id: I44e9993002cbf1d2c4f5cb23d340b01bad521427
2021-10-21 15:42:02 -07:00
Gage Hugo 22e50a5569 Update htk requirements
This change updates the helm-toolkit path in each chart as part
of the move to helm v3. This is due to a lack of helm serve.

Change-Id: I011e282616bf0b5a5c72c1db185c70d8c721695e
2021-10-06 01:02:28 +00:00
Sean Eagan b1a247e7f5 Helm 3 - Fix Job labels
If labels are not specified on a Job, kubernetes defaults them
to include the labels of their underlying Pod template. Helm 3
injects metadata into all resources [0] including a
`app.kubernetes.io/managed-by: Helm` label. Thus when kubernetes
sees a Job's labels they are no longer empty and thus do not get
defaulted to the underlying Pod template's labels. This is a
problem since Job labels are depended on by
- Armada pre-upgrade delete hooks
- Armada wait logic configurations
- kubernetes-entrypoint dependencies

Thus for each Job template this adds labels matching the
underlying Pod template to retain the same labels that were
present with Helm 2.

[0]: https://github.com/helm/helm/pull/7649

Change-Id: I3b6b25fcc6a1af4d56f3e2b335615074e2f04b6d
2021-09-30 16:01:31 -05:00
Lo, Chi (cl566n) 122dcef629 Remove Kibana indices before pod start up
The ps removes kibana indices from elasticsearch when a pod
comes up. It also removes the source code in values.yaml for
the flush job since it is not needed at this point.

Change-Id: Icb0376fed4872308b26e608d5be0fbac504d802d
2021-08-23 21:31:39 +00:00
Thiago Brito 5a0ba49d50 Prepending library/ to docker official images
This will ease mirroring capabilities for the docker official images.

Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: I0f9177b0b83e4fad599ae0c3f3820202bf1d450d
2021-06-02 15:04:38 -03:00
Lo, Chi (cl566n) 181cbf5599 Secure ingress path for Grafana and Kibana
The change enables TLS for the ingress path of
Grafana and Kibana.

Change-Id: I1bca5a3d78421873bff275d315ec0cca6682a498
2021-05-12 08:50:28 -07:00
Lo, Chi (cl566n) 9a719e2a18 Enable TLS between Elasticsearch and Kibana
This change enables TLS between Elasticsearch and Kibana
data path. Note that TLS terminates at apache-proxy container
of the Elasticsearch-client pod, not directly to port 9200 of
elasticsearch-client container.

Since all data traffic goes through apache-proxy container,
fluentd output to Elasticsearch are configured to have TLS
enabled as well.

In additon, other Elasticsearch pods that communicate with
Elasticsearch-client endpoint are modified to provide
the cacert option with curl.

Change-Id: I3373c0c350b30c175be4a34d25a403b9caf74294
2021-04-25 09:07:33 -07:00
Mohammed Naser 3fee13c5cd Stop using fsGroup inside container securityContext
fsGroup is not supported inside the container securityContext,
only inside the pod.  This drops a configuration that is not
valid and makes things deployable.

Change-Id: I956a1de107768c3fadc704722db83eb661cd25d2
2021-03-10 16:51:16 -05:00
Andrii Ostapenko 1532958c80
Change helm-toolkit dependency version to ">= 0.1.0"
Since we introduced chart version check in gates, requirements are not
satisfied with strict check of 0.1.0

Change-Id: I15950b735b4f8566bc0018fe4f4ea9ba729235fc
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-24 12:19:28 -05:00
Mohammed Naser c7a45f166f Run chart-testing on all charts
Added chart lint in zuul CI to enhance the stability for charts.
Fixed some lint errors in the current charts.

Change-Id: I9df4024c7ccf8b3510e665fc07ba0f38871fcbdb
2020-09-11 18:02:38 +03:00
Andrii Ostapenko 824f168efc Undo octal-values restriction together with corresponding code
Unrestrict octal values rule since benefits of file modes readability
exceed possible issues with yaml 1.2 adoption in future k8s versions.
These issues will be addressed when/if they occur.

Also ensure osh-infra is a required project for lint job, that matters
when running job against another project.

Change-Id: Ic5e327cf40c4b09c90738baff56419a6cef132da
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-07-07 15:42:53 +00:00
Andrii Ostapenko 83e27e600c Enable key-duplicates and octal-values yamllint checks
With corresponding code changes.

Change-Id: I11cde8971b3effbb6eb2b69a7d31ecf12140434e
2020-06-17 13:14:30 -05:00
Andrii Ostapenko dfb32ccf60 Enable yamllint rules for templates
- braces
- brackets
- colons
- commas
- comments
- comments-indentation
- document-start
- hyphens
- indentation

With corresponding code changes.

Also idempotency fix for lint script.

Change-Id: Ibe5281cbb4ad7970e92f3d1f921abb1efc89dc3b
2020-06-17 13:13:53 -05:00
Andrii Ostapenko 8f24a74bc7 Introduces templates linting
This commit rewrites lint job to make template linting available.
Currently yamllint is run in warning mode against all templates
rendered with default values. Duplicates detected and issues will be
addressed in subsequent commits.

Also all y*ml files are added for linting and corresponding code changes
are made. For non-templates warning rules are disabled to improve
readability. Chart and requirements yamls are also modified in the name
of consistency.

Change-Id: Ife6727c5721a00c65902340d95b7edb0a9c77365
2020-06-11 23:29:42 -05:00
Radhika Pai e81583ac3e [update] kibana : add install hook and dependencies
The flush-kibana-metadata job was causing issue in loading the kibana
dashboard due to conflict in order this is run. Adding dependencies to avoid
running jobs simultaneously.

Change-Id: If5a2564a8b6a16fb0dbd6a93f2e6e02d91f394dc
2020-06-03 21:37:03 +00:00
Andrii Ostapenko 731a6b4cfa Enable yamllint checks
- document-end
- document-start
- empty-lines
- hyphens
- indentation
- key-duplicates
- new-line-at-end-of-file
- new-lines
- octal-values

with corresponding code adjustment.

Change-Id: I92d6aa20df82aa0fe198f8ccd535cfcaf613f43a
2020-05-29 19:49:05 +00:00
Steven Fitzpatrick ff291b5abb Kibana - Add hook to delete .kibana indices
This hook is enabled for post-delete and pre-upgrade triggers.
The indices deleted by this hook are Kibana's meta indices
  - .kibana
  - .kibana_1
  - .kibana_2
  etc

This is done to get around https://github.com/elastic/kibana/issues/58388
which sometimes prevents Kibana deployments from upgrading successfully.

Change-Id: I99ccc7de20c6dadb5154e4bb714dfd302a694a78
2020-05-21 11:22:28 -05:00
Radhika Pai e966ae6ba8 Kibana: Add support for arbitrary object definitions via
overrides

This allows for customizing the
indexes required by different deployment targets instead of
assuming all indexes are common for every type of deployment.

Change-Id: Iae9a35462400f7c8612ee7d0b49bfd6a20d3120c
2020-05-19 09:11:40 -05:00
Zuul e53d28718d Merge "Remove OSH Authors copyright" 2020-05-12 20:00:38 +00:00
diwakar thyagaraj 53b5fda1c6 Enable Apparmor to Kibana Completed pods
Change-Id: Idf408846f6a6f4350ce5c78247338cfebb280e38
Signed-off-by: diwakar thyagaraj <diwakar.chitoor.thyagaraj@att.com>
2020-05-08 23:07:52 +00:00
Gage Hugo d14d826b26 Remove OSH Authors copyright
The current copyright refers to a non-existent group
"openstack helm authors" with often out-of-date references that
are confusing when adding a new file to the repo.

This change removes all references to this copyright by the
non-existent group and any blank lines underneath.

Change-Id: I1882738cf9757c5350a8533876fd37b5920b5235
2020-05-07 02:11:15 +00:00
dt241s@att.com d59b6e5944 Enable Apparmor to Kibana
Also added new apparmor zuul gates jobs for Logging, as initial 
apparmor is  getting timeout.

Change-Id: Iea0a5055238d75f401caf9ddb0ddd9985a091aab
2020-03-14 04:37:32 +00:00
Fitzpatrick, Steven (sf280x) 53991041ab Actually add Kibana Liveness Probe
The patch submitted last week mistakenly added a liveness probe
for the apache sidecar container instead of the failing Kibana
container.

Change-Id: I61a979099f5c387a8256788ceab2f91e45d17838
2020-03-04 13:27:55 -06:00
Steven Fitzpatrick 371b1cbe89 Add Liveness Probe to Kibana Deployment
This change adds a liveness probe to the Kibana deployment spec.
If multiple kibana replicas are deployed simultaniously they race
to update the .kibana index in Elasticsearch, which sometimes
results in a pod to stall without starting it's http server.

Change-Id: Ib685d738ced59df66ff3501749316a01b5cacf79
2020-02-26 16:12:40 +00:00
Steve Wilkerson 2d3c9575ff Elasticsearch/Kibana: Update version to 7.1.0
This updates the Elasticsearch and Kibana charts to deploy
version 7.1.0. This move required significant changes to both
charts, including: changing elasticsearch masters to a statefulset
to utilize reliable dns names for the discovery process, config
updates to reflect deprecated/updated/removed values, use the
kibana saved objects api for managing index patterns and setting
the default index, and updating the elasticsearch entrypoint
scripts to reflect the use of elastic-keystore for storing s3
credentials instead of defining them in the configuration file

Change-Id: I270d905f266fc15492e47d8376714ba80603e66d
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-12-03 07:43:29 -06:00
Bjoern Teipel b500d69591 Fxing lint errors for Helm 2.16
This commit fixes helm lint errors when linting against
the recent helm version.

Change-Id: I2a940ad1cea406ba923519cd5be188ee1bc409aa
2019-11-12 11:28:22 -06:00
Steve Wilkerson b50fae62a4 Update kubernetes-entrypoint image reference
This updates the kubernetes-entrypoint image reference to consume
the publicly available kubernetes-entrypoint image that is built
and maintained under the airshipit namespace, as the stackanetes
image is no longer actively maintained

Change-Id: I5bfdc156ae228ab16da57569ac6b05a9a125cb6a
Signed-off-by: Steve Wilkerson <sw5822@att.com>
2019-10-18 18:20:11 +00:00
caoyuan 040edeb79a Replace git.openstack.org URLs with opendev.org URLs
Change-Id: I0e3af4a3385f5b2a7705bc19b775863b16c2e08e
2019-05-31 01:52:10 +00:00
Jean-Philippe Evrard 5f5e988fb3 Point to OSH-images images
We now have a process for OSH-images image building,
using Zuul, so we should point the images by default to those
images, instead of pointing to stale images.

Without this, the osh-images build process is completely not
in use (and completely opaque to deployers), and updating the
osh-images process or patching its code has no impact on OSH.

This should fix it.

Change-Id: Ic00bd98c151669dc2485cd88e0e8c2ab05445959
2019-05-17 08:17:32 +00:00
Zuul b69584bd65 Merge "Expose Anti-Affinity Weight Setting" 2019-05-16 17:17:03 +00:00
RAHUL KHIYANI 366357d893 Kibana: set read-only-fs
This PS permits read-only filesystems to back the containers by setting
the default to true

Additionally /run is uniformly applied across all long running pods
as a memory backed emptydir

Change-Id: Ia7344e2c8caa1f25101bf30445cdfe277f89c143
2019-05-15 20:19:59 +00:00
Roy Tang (rt7380) 85bd731562 Expose Anti-Affinity Weight Setting
This ps exposes the anti-affinity weight value, including
default, that will be consumed by the updated htk function.

Change-Id: Id8eb303674764ef8b0664f62040723aaf77e0a54
2019-05-14 17:04:52 -05:00
Meg Heisler e1f2a3cf78 Fix broken network policy check/gate
This adds a basic egress policy to the charts run by the
network-policy check. A change was recently merged requiring
the eggress tag to be in the chart but did not add it, this
addresses that

Change-Id: I60669c9351db7854cba8c69723eb783a966d2a56
2019-05-10 05:55:22 +00:00
RAHUL KHIYANI cd99469454 Kibana: Fix security context
This PS fixes the use of the security context macros for the
Kibana chart.

Change-Id: Iaad821ac3df7e42eb52ba2f274fe47e4847d30af
2019-04-23 04:32:41 +00:00
Pete Birley 2abf62ff4d OSH-Infra: Add emptydirs for tmp
This PS adds emptydirs backing the /tmp directory in pods, which
is required in most cases for full operation when using a read only
filesystem backing the container.

Additionally some yaml indent issues are resolved.

Change-Id: I8b7f1614da059783254aa6efc09facf23fca3cad
Signed-off-by: Pete Birley <pete@port.direct>
2019-04-20 20:50:59 +00:00
Rahul Khiyani 5e1ecd9840 Revert "readOnlyRootFilesystem: true for kibana chart"
This reverts commit 244f177ecb.

removing readOnlyRootFilesystem flag since pods are running to "crashLoopBackOff" state by implementing HTK functionality

when we have set the readOnly flag at pod without HTK functionality the changes were not effected. That is why it passed the gate.

Change-Id: I6920956b881fa358a37003d21a7b76602e2ac61c
2019-04-20 03:36:25 +00:00
Steve Wilkerson 2e8c96a623 Elasticsearch, Fluent-logging, Kibana Ingress Policy
This adds ingress network policy for the fluent-logging, kibana
and Elasticsearch charts. This leverages the helm-toolkit template
that was used in openstack-helm for the openstack services

Change-Id: I2a89b62f1002851346e9a25de40113078e9c518f
2019-04-16 19:44:46 +00:00
Steve Wilkerson 84f30ec103 Add release-annotation to pod spec, add missing annotations
This adds the release-annotation to the pod spec for the charts in
openstack-helm-infra. This also adds missing configmap annotations
to charts in openstack-helm-infra

Change-Id: Ie23f0c16a7a21d3929e98928db2bbcef69ae6490
2019-03-21 09:10:48 -05:00
Zuul ff84ab86dc Merge "Update ingress controller image, ingress cookie annotations" 2019-03-08 20:32:07 +00:00
Rahul Khiyani 244f177ecb readOnlyRootFilesystem: true for kibana chart
Fix for adding readOnlyRootFilesystem flag at pod
level

Change-Id: Ie2ce8bf66ac1194a319154b58c2980d1260fffe0
2019-03-08 05:24:19 +00:00
Steve Wilkerson 3413dba8c0 Update ingress controller image, ingress cookie annotations
This updates the ingress controller image to v0.23.0, which was
required to add support for configuring cookie max age and expires
for ingresses via annotations on the ingress.

This also removes the --enable-dynamic-configuration flag, as the
flag is no longer valid in 0.23.0 due to the functionality being
a default behavior of the nginx ingress controller in recent
releases

Change-Id: I4917797c43ec973ed0bb311fc305b01f10abd4e5
2019-03-07 20:39:03 +00:00
Steve Wilkerson 4c0fd492ee Update logging format and config for apache reverse proxies
This updates the logging format and configuration for the apache
reverse proxies used for elasticsearch, kibana, nagios and
prometheus to enable logging of the remote clients used to access
these services

Change-Id: Id07e4294ea18203fbb890b78424a232c2d59cb82
2019-02-25 09:21:41 -06:00
Steve Wilkerson 6a78fa2eae Kibana: Include kernel and journal indexes in register job
This updates the Kibana chart to include the kernel and journal
indexes as part of the default indexes that get registered with
the register-indexes job

Change-Id: Icd8678debb3dd9620548c6a7c5f02dbb1da048ba
2019-01-08 13:11:18 -06:00
Steve Wilkerson 30d2cf00d4 Remove unused pod-etc-apache volumes
This removes unused pod-etc-apache volumes from the charts that
use an apache sidecar container as a reverse proxy.

Change-Id: Ibafff3b53f9d3c20f5aed30d40ee6470cb515a8a
2019-01-04 10:31:35 -06:00
Chris Wedgwood 0c4e37391f 'NOP' cleanup for more consistent white-space use in charts
Where we have the style '{{ ...' we should use the style '... }}'.

Change-Id: Ic3e779e4681370d396f95d3804ca27db5b9d3642
2019-01-03 22:45:49 +00:00