This begins to split the fluent-logging chart into two separate
charts, one for fluentbit and one for fluentd. This is to help
isolate each chart and its dependencies better, and to treat each
service as its own entity.
This also moves the job for creating Elasticsearch templates to
the Elasticsearch chart, as the elasticsearch chart should have
ownership of creating the templates for its indices.
This also performs some general cleanup of values keys that are
not currently used
Change-Id: I827277d5faa62b8b59c5960330703d23c297ca47
Signed-off-by: Steve Wilkerson <sw5822@att.com>
This updates the ClusterRole object for fluentd by removing a
duplicate rules: key and also adds 'get' to the list of verbs for
the "" apiGroups (as it's required for the kubernetes metadata
plugin)
Change-Id: Ia901d9fe9a0784038f0f882297c64afcce58ca7e
This removes the utilities for generating the fluentd, fluentbit,
and parser configuration files from yaml and moves to instead
consume the configuration files as strings from the values.yaml.
This allows for easier creation and maintenance of configuration
files for fluentd and fluentbit, as the utilities became unwieldy
with complex configuration files.
This necessitated the removal of the core test executed by the
charts helm tests, but this would be required as we move to split
the charts regardless
Change-Id: Ied4a76bbdf58b54a6d702db04a7120b64f54dcac
Signed-off-by: Steve Wilkerson <sw5822@att.com>
We now have a process for OSH-images image building,
using Zuul, so we should point the images by default to those
images, instead of pointing to stale images.
Without this, the osh-images build process is completely not
in use (and completely opaque to deployers), and updating the
osh-images process or patching its code has no impact on OSH.
This should fix it.
Change-Id: Ic00bd98c151669dc2485cd88e0e8c2ab05445959
This ps exposes the anti-affinity weight value, including
default, that will be consumed by the updated htk function.
Change-Id: Id8eb303674764ef8b0664f62040723aaf77e0a54
This adds a basic egress policy to the charts run by the
network-policy check. A change was recently merged requiring
the eggress tag to be in the chart but did not add it, this
addresses that
Change-Id: I60669c9351db7854cba8c69723eb783a966d2a56
This updates the fluentd-exporter to use the bitnami image for the
chart instead of a personal image
Change-Id: I162dca4556646eb781c380acea307d2feb156d18
This PS adds emptydirs backing the /tmp directory in pods, which
is required in most cases for full operation when using a read only
filesystem backing the container.
Additionally some yaml indent issues are resolved.
Change-Id: I8b7f1614da059783254aa6efc09facf23fca3cad
Signed-off-by: Pete Birley <pete@port.direct>
This adds ingress network policy for the fluent-logging, kibana
and Elasticsearch charts. This leverages the helm-toolkit template
that was used in openstack-helm for the openstack services
Change-Id: I2a89b62f1002851346e9a25de40113078e9c518f
securityContext with allowPrivilegeEscalation: false is implemented at
container level and leveraged the helm-toolkit snippet
Change-Id: Iddb18c87993fd3dc005c55f5678829c2a19718db
This adds the release-annotation to the pod spec for the charts in
openstack-helm-infra. This also adds missing configmap annotations
to charts in openstack-helm-infra
Change-Id: Ie23f0c16a7a21d3929e98928db2bbcef69ae6490
The change that modified the output configurations for fluentd
accidentally removed the type_name from the default elasticsearch
output, which prevents the output from using the fluent template
that's defined in the chart. This replaces the type_name for that
output
Change-Id: I2098ca8c243d55f0446ea623a80b5b40e3acff8c
This updates the fluentbit configuration for tail inputs to remove
the values for utilizing mysqlite databases to track its location
in each file it's configured to tail. This is intended to reduce
the pressure fluentbit exerts on the host through writing to
/var/log/foo.db. To help mitigate large amounts of traffic
sent from fluentbit to fluentd upon a pod restart, this also
adds a throttle filter to fluentbit.
As a result, Fluentbit no longer needs a writable mount to its
hostPath on /var/log on the host. Thus, this change includes
updating the Fluentbit daemonset's mount on /var/log to be
readOnly
Change-Id: If4381f4ff47e887f3ea10beded4f6172edaf08ba
This removes an unused section of configuration for fluentd, as
well as cleans up the values for filtering fluentd logs
Change-Id: I0c58d3ac236af7723c64c3b9fcba877736b1f606
This adds a liveness probe to the fluentd chart. This probe will
simply perform a tcpSocket check on the same port the readiness
probe executes the check on.
Change-Id: I768b23d36d50d6f6938f5588bea71e97aeb624b9
This updates the fluentd configuration to use 8 threads for the
Elasticsearch output configuration by default. This uses the
correct buffer output settings for the fluent-elasticsearch
plugin
This also updates the buffer output settings to the defaults used
for fluentd
Change-Id: I976cddaa973e850dabe4de495cd3bf1a4acdd4e7
This adds the security context snippet to the fluentd and
fluentd exporter templates. This changes the users for these two
pods from root to the nobody user instead
This also adds the container security context to explicitly set
allowPrivilegeEscalation to false
Change-Id: Ibf1da152f4aa78d425bbd00f514c2787d8ad9c5f
This adds an input to Fluentbit for capturing all qemu instance
logs in /var/log/libvirt/qemu/, and adds an Elasticsearch output
for those entries
Change-Id: I0802023f9861a5944e7989fd5469133c325349e7
This modifies the libvirt chart to write logs directly to the
host by default. This also modifies the fluentbit and fluentd
charts to capture libvirt logs from the host and index them into
Elasticsearch
Change-Id: I0bbc49d2c0d4cf4895f797e48f309f308ffd021f
This PS implements the helm toolkit function to generate the
Egress in kubernetes network policy manifest based on overrideable values.
It also enbale the K8s network policy at Osh-infra gate.
Change-Id: Icbe2a18c98dba795d15398dcdcac64228f6a7b4c
This adds the Decode_Field_As configuration key to the docker
parser for fluentbit. This is required to escape utf-8 encoded
characters appropriately in the log field
Change-Id: Ie2600cfe22045e3ab651fddf61ed2f676ab8a1d5
Adding auth tags for the logs to support special filter
for openstack and application security logs
Change-Id: Ifbd2395e4268d8d8fc4a2a3ac4d351db3d3e0845
This removes the tolerations key from the labels entries. As the
boolean check is on the pod.tolerations.enabled key instead, the
labels.foo.tolerations key is no longer used and should be removed
Change-Id: I00536dabadf9bd354219058d8efd054c60952bbd
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent
This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints
Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
This removes the fluentbit sidecars from the ceph-mon and ceph-osd
charts. Instead, we mount /var/log/ceph as a hostpath, and use the
fluentbit daemonset to target the mounted log files instead
This also updates the fluentd configuration to better handle the
correct configuration type for flush_interval (time vs int), as
well as updates the fluentd elasticsearch output values to help
address the gate failures resulting from the Elasticsearch bulk
endpoints failing
Change-Id: If3f2ff6371f267ed72379de25ff463079ba4cddc
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.
Additionally, implementation is done for some infrastructure charts.
Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
This changes the image used for various jobs and helm tests in the
osh-infra charts. This replaces the kolla heat image with the loci
based heat image used for jobs and helm tests in openstack-helm in
order to drive consistency
Change-Id: Ie9deedadb7507282fe62723ec4641dd508040364
This updates the helm tests for the fluent-logging chart to make
them more robust in being able to check for indexes defined in the
chart. This is done by calculating the combined flush interval
for both fluentbit and fluentd, and sleeping for at least one
flush cycle to ensure all functional indexes have received logged
events.
Then, the test determines what indexes should exist by checking
all Elasticsearch output configuration entries, determining
whether to use the default logstash-* index or the logstash_prefix
configuration value if it exists. For each of these indexes, the
test checks whether the indexes have successful hits (ie: there
have been successful entries into these indexes)
Change-Id: I36ed7b707491e92da6ac4b422936a1d65c92e0ac
This updates the logging interval values for the Elasticsearch
outputs to integers (20) vs the previous string value (20s)
Change-Id: I681bdaf807ba0136fef3b6dc1c7ddaa689ae77a3
This updates the helm test pod templates in the charts with helm
tests defined. This change includes the addition of:
- Generate test pod cluster roles and role bindings
- Generate service accounts for test pods
- Add node selectors to the test pods
- Add service accounts to the test pods
- Addition of entrypoint container to the test pods
- Indentation fix for rabbitmq test pod template
Change-Id: I9a0dd8a1a87bfe5eaf1362e92b37bc004f9c2cdb
This adds inputs for kernel logs on the host, as well as dockerd
and kubelet logs via the systemd plugin. This also adds a filter
for adding the hostname to the kernel log events, for renaming the
fields for systemd logs as kibana can not visualize fields that
begin with an underscore, and adds elasticsearch indexes for both
kernel and systemd logs
Change-Id: I026470dd45a971047f1e5bd1cd49bd0889589d12
This updates the configuration for fluentd, providing a mechanism
for basic determination of the log level of a logged event via
entries from /var/log/containers. This log level is prepended to
the tag for that event, and also added as a new `level` key in
the resulting event. These two improvements allow for querying
specific log level events via the tag.
This also adds similar functionality to any events captured via
the oslo log fluentd handler/formatter. This allows for
elasticsearch queries akin to `error.openstack.keystone`, which
can be used by nagios or another alerting mechanism to raise
alerts when a particular level event has been captured.
Change-Id: I016ddcfcf7408de7b6511ddf7009e1e6a5f3a1d9
Steve Wilkerson