We indicated to the OpenStack TC that this service would be going away
after the Yoga cycle if no one stepped up to start maintaining it. That
help didn't arrive in the form of OpenDev assistance (there is effort
to use OpenSearch external to OpenDev) and Yoga has released. This means
we are now clear to retire and shutdown this service.
This change attempts to remove our configuration management for these
services so that we can shutdown the servers afterwards. It was a good
run. Sad to see it go but it wasn't sustainable anymore.
Note a follow-up will clean up elastic-recheck which runs on the status
server.
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/837619
Change-Id: I5f7f73affe7b97c74680d182e68eb4bfebbe23e1
Not updating the gerrit git links thing, because that needs to
be a wider patch that updates the link syntax too.
Change-Id: I98013ba79e707540879e0cf2849a35c52f3371e8
Attempt to fix the following error:
puppet-user[13761]: (/Stage[main]/Openstack_project::Logstash_worker/Log_processor::Worker[D]/File[/etc/logprocessor/jenkins-log-worker-D.yaml])
Could not evaluate: Could not retrieve information from environment production source(s) file:/etc/logprocessor/worker.yaml
Change-Id: I7c8eef36cad7b62f3f70ef7069ec7e14a1ea8bd2
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Otherwise, we raise a puppet error about /etc/logstash not existing.
Change-Id: I073034582bf7a28e14eb504d02e6d91dd1c0ae61
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The logstash workers don't have a subunit user so don't use it to set
file ownership. Instead just set file ownership to root as the file just
needs to be readable by the logstash worker processes.
Change-Id: I14a4fbf879d64b883bceb1bb227bb8b392f654ba
This commit enables the mqtt notifications in the gearman worker. Most
of the setup was already there from our previous failed attempts to
leverage the logstash plugin directly. This just pivots that to use the
gearman worker instead.
Change-Id: I6becec12604c61fe50d3e6b9c7ed9f9e9be311ae
Depends-On: I0a17444cce18dd4b63f1f924e393483f6d8fe8eb
Depends-On: I43be3562780c61591ebede61f3a8929e8217f199
In order to run the script directly in the test, the script needs to
have the executable bit set. Currently, the test is failing with a
"Permission denied" error when attempting to run the script.
We also need to run install_modules.sh in the logstash-filters test, so
move that invocation to the shared prep-apply.sh script.
The vcsrepo puppet define doesn't understand revisions of the form
"refs/zuul/foo", so instead parse the HEAD sha and pass that to vcsrepo.
Finally, we need to ensure that $elasticsearch_nodes is populated with
an array, otherwise the function @elasticsearch_nodes.map in the
output.conf template will fail.
Change-Id: Ic1ab6d04232abb3cf347f0d27afbf9172f62b0e4
This commit will enable the mqtt output plugin on logstash workers. So
now we'll be emitting logstash events to the firehose for anyone to
listen to.
Change-Id: I570a461ee13f5dfa5494554df1cc321fc6cdbf6c
Depends-On: I7255f6c256ab3e3ca33caf69b71bf4ffab02c1bb
Add a script for integration testing with
openstack-infra/logstash-filters and refactor the apply-test script to
leverage much of the prep work done there.
Also add some needed parameters to install arbitrary versions of the
logstash-filters repo.
Change-Id: Ifcc7fb13c4fb21986e33327487c739966e4bb107
This uses new logstash conf.d features to properly link in the
configuration from the logstash-filters repo. This should make
configuring logstash far more flexible and reconsumable.
Change-Id: Ia304eb7e73c82ca5ce85967fbf442bb4bc5f8b7a
Depends-On: Icbca7a6ba0c5a94a273ef158f707311b588483fd
Now that we are running logstash 2.0 the logstash daemon can talk to
elasticsearch directly and load balance across the cluster. This means
we don't need a local elasticsearch daemon to do that for us. The big
savings here is in memory so stop installing and running elasticsearch
completely on the workers.
Note this will not uninstall an existing ES install you will need to
clean that up if a preexisting install is present.
Change-Id: I9b622674a74a26e7c3024e684e05291f43aec021
The actual upgrade is handled by ansible but this will enforce the
correct version for any new host builds. This should be merged after the
ansible managed upgrade is complete.
Change-Id: Ieda9cdffe7486d24f17012de5fc094d18a45ec78
Remove elasticsearch01 as it has been turned off.
Allow all elasticsearch running nodes to talk to each other over ports
9200 to 9400. This includes the logstash-worker nodes running non data
elasticsearch clients. This appears to be necessary to keep logs from
filling up with gigs of NoRouteToHost exceptions on the worker nodes.
Change-Id: I3e3365a717d22d881654206692d17dd05ed32c98
Set up small non master non data elasticsearch daemons on logstash
workers to act as local load balancers for the elasticsearch http
protocol.
Change-Id: Ie3729f851ebef3331a6b69f718e57d663209bfc2
The logstash elasticsearch output seems to degrade over time and slow
down. Restarting the logstash daemon temporarily corrects this problem.
Switch to the elasticsearch HTTP output to see if that corrects the
problem as well.
Note the logstash watchdog is disabled by this change as logstash
daemons using elasticsearch HTTP output will not join the elasticsearch
cluster which will force the watchdog to always trip. To avoid this
issue disable the watchdog.
Change-Id: I77044b26fa10fb1fc3690a0464d79d55bed2fe00
The logstash-indexer services sometimes fall out of the elasticsearch
cluster when it is under heavy load and unable to respond to pings.
Logstash doesn't do anything to reconnect :( so restart the service if a
node detects that it has fallen out of the cluster.
Upstream bug submitted at https://logstash.jira.com/browse/LOGSTASH-1951
Change-Id: I2e7767c5fe20cff279366fec2ddadd7710dbb4a9
The default java heap size is too small for our logstash indexers.
Double it to 2g. Do this by adding an /etc/default/logstash-indexer file
that the upstart configs source if it is present. This required some
tweaks to the upstart configuration to load the defaults properly.
Co-Authored-By: K Jonathan Harker <k.jonathan.harker@hp.com>
Change-Id: I63447f59f3fa6d466a7d275476121fe8339479dc
Now that the new elasticsearch cluster members are joined, we need
to move the primary discover node from elasticsearch.openstack.org
to one of the new servers before we can remove it from service.
Change-Id: I79fe4e2154def6b0404b6620601f4a02b63fee8a
Initial load testing suggests we should be able to handle at least
4 processes per host.
Also re-enable crm114.
Change-Id: Ia0158ad5f7f524c4fa0a80d479e2b74d28f0d1a6
Separate the jenkins log client and worker bits into a new module
called log_processor with ::client and ::worker classes.
Instantiate two workers on each logstash worker node.
Change-Id: I7cfec410983c25633e6b555f22a85e9435884cfb
Try to identify the probability that each log line indicates an error.
Pass that information on to logstash.
Change-Id: I0b298c2e8c00d8fdf1c907215a7bbf27086bc80c
* modules/openstack_project/manifests/logstash_worker.pp: Use logrotate
to rotate Jenkins log worker logs. These log files were not being
rotated and get fairly large making debugging failures difficult. Rotate
the logs to simplify debugging with logs and prevent unbounded log
growth.
Change-Id: Ic7e983c1b88ea45430c732c0dadb0ba3f08c6cc6
We need to expand our elasticsearch install base. Update puppet to make
this possible.
Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* modules/openstack_project/manifests/logstash.pp: Concat is not
available in our version of puppetlabs stdlib. Use flatten instead
which is available. Remove dependency on non existant logstash::indexer
class. Fix requires orders.
* modules/openstack_project/manifests/logstash_worker.pp: Fix requires
orders.
* modules/openstack_project/files/logstash/jenkins-log-client.init
* modules/openstack_project/files/logstash/jenkins-log-worker.init:
Set pidfile argument when calling scripts.
* modules/openstack_project/files/logstash/log-gearman-worker.py:
Use python2 compatible gzip.GzipFile instead of gzip.decompress. Send
work exception instead of work fail when an exception happens. Log these
exceptions locally as well.
Change-Id: Idf0a873215acb72187e058a0306a21ccd928d464
Reviewed-on: https://review.openstack.org/32804
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
This change reorgs the logstash log pushing so that there is a central
gearman server that listens to Jenkins ZMQ events which are then
converted to per log file gearman jobs which are processed by gearman
workers. The central gearman server will live on logstash.o.o and the
existing logstash-worker hosts will be converted to gearman log pusher
workers.
This commit includes relavent documentation changes.
Change-Id: I45f7185c2479c54b090d223408dff268e1e8d7db
Reviewed-on: https://review.openstack.org/32455
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Logstash performs filtering in a single thread so it does not scale up
very well. Work around this by scaling Logstash out to multiple indexer
hosts.
Current plan is to have a small (2GB) kibana web front end host that
does nothing but talk to elasticsearch, three 4GB logstash indexers that
will run a single log-pusher.py + logstash indexer with some partition
of the logfiles assigned to each indexer, and finally the existing large
elasticsearch node.
Eventually properly load balancing log processing across the worker
nodes would be great, but the current partition method should work well
enough with little additional effort.
Change-Id: Ifc6396560934314ffd6a7c47eb2acff9e9c2a7af
Reviewed-on: https://review.openstack.org/30573
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Clark Boylan