Commit Graph

39 Commits

Author SHA1 Message Date
Clark Boylan 4279e20293 Remove configuration management for ELK stack
We indicated to the OpenStack TC that this service would be going away
after the Yoga cycle if no one stepped up to start maintaining it. That
help didn't arrive in the form of OpenDev assistance (there is effort
to use OpenSearch external to OpenDev) and Yoga has released. This means
we are now clear to retire and shutdown this service.

This change attempts to remove our configuration management for these
services so that we can shutdown the servers afterwards. It was a good
run. Sad to see it go but it wasn't sustainable anymore.

Note a follow-up will clean up elastic-recheck which runs on the status
server.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/837619
Change-Id: I5f7f73affe7b97c74680d182e68eb4bfebbe23e1
2022-04-18 10:04:06 -07:00
Clark Boylan 6f986a4fcc Remove elasticsearch01
This host hasn't existed in years. Make that clearer in system-config by
removing it.

Change-Id: Ie11151c00ab32ea249be0f539952660cefc0fa2d
2020-06-17 14:51:58 -07:00
Ian Wienand 2acfc176b0 Remove graphite.openstack.org
The server has been removed, remove it from inventory.

While we're here, s/graphite.openstack.org/graphite.opendev.org/'
... it's a CNAME redirect but we might as well clean up.

Change-Id: I36c951c85316cd65dde748b1e50ffa2e058c9a88
2019-05-08 05:55:33 +10:00
Clark Boylan 328aa4b593 Run standalone geard for log processing
We don't have a jenkins 0mq event stream with zuulv3 instead jobs
directly submit their log processing jobs to gearman. Convert to running
a standalone geard to take advantage of this.

Depends-On: Iedcb5b29875494b8e18fa125adb08ec2e34d0064
Change-Id: Ie3f814e6d3278d87f2a20a72e40b6b92217684fc
2017-12-22 10:19:53 -08:00
George Peristerakis 6000c580ab Moved the server class out from the logstash and logstash_worker class
Change-Id: I96df327f278714fb393dca887b8db1e01ca7504d
Story: 2000172
Spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/server_base_template_refactor.html
2015-05-28 13:59:48 -04:00
Mikhail S Medvedev 6c5199d5a9 Split out mysql_proxy module and rename it
The spec:
http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html

For this particular module, we will take this opportunity to rename the
module to "simpleproxy" to reflect the fact that eventually it will have
no dependency on mysql.

Depends-On: I42597f8bc5ded5b4cdf89b4eae780ae4bcf9f61e
Change-Id: I3cbd5983423ff3a4f59b79af3378c91eff323a49
2015-01-29 19:52:25 +00:00
Matthew Treinish a86df2042c Open port 3306 on logstash.o.o
When mysql-proxy was being used we had to open port 4040 to allow
connections to the proxy. However, since that approach was abandoned
and simpleproxy is being used instead, it's just forwarding port 3306.
This commit fixes it by switch 4040 to 3306 in the firewall rules.

Change-Id: Ie6d04e20996ba4c7f567526a2428ce5015b8b135
2014-12-12 16:43:37 -05:00
Matthew Treinish d69df1515f Switch mysql-proxy to simpleproxy
Mysql-proxy turns out too unreliable and unstable for use in
production. The packaged version on Ubuntu suffers from a critical DOS
by using telnet. This patch switches from mysql-proxy to simpleproxy,
which is just a tcp proxy to forward incoming port 3306 connections to
the subunit2sql db.

Change-Id: Iffea64aea46cc34969bbaa970e5d91bd0cc05232
2014-12-12 15:40:54 -05:00
Matthew Treinish 364e5ca681 Add mysql-proxy to enable read-only access to a db
This commit adds a mysql_proxy module which will setup a read-only
proxy to a mysql db. This also configures a proxy to the subunit2sql
db to run on logstash.o.o to provide read only access to the data in
the database.

Change-Id: I478baca354354347fe50074a8e3b9f66ca890d55
2014-12-05 19:27:23 +00:00
Matthew Treinish 5941f835ac Switch subunit2sql to use decomposed uri
This commit switches the subunit2sql_uri which get's passed into the
config file template to use decomposed values. The advantage of this
is that it enables using subsets of the information elsewhere.

Change-Id: I30b5117c12897fa3a1cb1e2579b4eec59ba57b2b
2014-12-05 14:27:00 -05:00
Matthew Treinish a0600d594c Add subunit2sql server configuration
This commit adds the policy around keeping the schema for the
subunit2sql db up to date. It will run the db migrations on updates
to the version of subunit2sql installed.

Change-Id: Ice6c2607bb627b789bf265b38566b7b279ac0f9a
2014-10-29 15:39:16 -04:00
Jeremy Stanley d90a9bc3fb Move primary elasticsearch discover node
Now that the new elasticsearch cluster members are joined, we need
to move the primary discover node from elasticsearch.openstack.org
to one of the new servers before we can remove it from service.

Change-Id: I79fe4e2154def6b0404b6620601f4a02b63fee8a
2014-02-26 22:00:08 +00:00
James E. Blair 032ab0d470 Enable statsd for jenkins-log-client
Change-Id: I4395ce2151fc666cae1d9ea76b43bc3663e98ac8
2014-02-21 10:54:24 -08:00
James E. Blair 2bb28339cb Move elastic-recheck bot to status.o.o
Also, normalize some parameters to the status manifest to make it
more readable (since they follow the same pattern).  And make the
ssh config for elastic-recheck bot match reviewday.

Change-Id: I2417f121e7b3685aab9540504cdd4c6db1754e67
2013-12-31 19:35:47 +00:00
James E. Blair 920fc0c42b Refactor logstash_worker into log_processor module
Separate the jenkins log client and worker bits into a new module
called log_processor with ::client and ::worker classes.

Instantiate two workers on each logstash worker node.

Change-Id: I7cfec410983c25633e6b555f22a85e9435884cfb
2013-12-18 00:40:49 +00:00
Sean Dague b3af619900 move to elastic-recheck web console in share
this provides the changes needed to handle the new elastic-recheck
web console installed from the elastic-recheck package in the
share tree.

factor out bot code to elastic_recheck::bot

this change moves elastic_recheck bot starting code out into
a separate module, which lets us include all the elastic_recheck
base module into the static site config.

Change-Id: If53523754494a746c152c5d1384db5426b096fc1
2013-12-06 15:52:40 -05:00
Clark Boylan e0427cfbb7 Deploy elastic-recheck on logstash.openstack.org.
This elastic recheck thing that classifies failures based on logstash
queries is awesome. We should run it on the OpenStack developer
infrastructure. Deploy it on logstash.o.o to keep related services
together.

Change-Id: I16ae7a32c2c48474454df17931779d0f60c3ed85
2013-09-30 18:56:23 +00:00
Clark Boylan 8cf1cc37ac Rotate Jenkisn log client logs.
* modules/openstack_project/manifests/logstash.pp: Rotate jenkins log
client logs with logrotate. Keep one week of logs and rotate daily.

Change-Id: I9b6a320afda76cc7285c959196a24853a5a9561e
2013-09-03 10:16:27 -07:00
Clark Boylan 443539b7fa Better elasticsearch cluster settings.
Use mutliple discover nodes to determine elasticsearch cluster
membership. Put a timeout on recovery starting instead of the default
to recovery immediately. Describe cluster topology in elasticsearch yaml
config so that it can make smarter decisions. Round robin kibana
requests across each discover node.

Change-Id: I08ef9dd158ddf6a6ce01dfb2050626f543d45b10
Reviewed-on: https://review.openstack.org/34106
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-01 17:49:09 +00:00
Clark Boylan 33367c88de Add support for elasticsearch cluster.
We need to expand our elasticsearch install base. Update puppet to make
this possible.

Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-01 17:48:16 +00:00
Clark Boylan f51d974205 Reverse proxy subset of elasticsearch API.
It has been requested that we provide programmatic access to the
elasticsearch API in addition to the Kibana web UI. Do this by reverse
proxying http://logstash.openstack.org/elasticearch/$API_REQUEST to
http://elasticsearch.openstack.org:9200/$API_REQUEST.

The only values for $API_REQUEST that will be passed through are
_aliases, _status, and _search.

Change-Id: Ib41f6d91e2e59d493218074a67155af450ec8c93
Reviewed-on: https://review.openstack.org/33316
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-18 19:01:18 +00:00
Clark Boylan abe222fa9e Fix logstash gearman puppet.
* modules/openstack_project/manifests/logstash.pp: Concat is not
available in our version of puppetlabs stdlib. Use flatten instead
which is available. Remove dependency on non existant logstash::indexer
class. Fix requires orders.

* modules/openstack_project/manifests/logstash_worker.pp: Fix requires
orders.

* modules/openstack_project/files/logstash/jenkins-log-client.init
* modules/openstack_project/files/logstash/jenkins-log-worker.init:
Set pidfile argument when calling scripts.

* modules/openstack_project/files/logstash/log-gearman-worker.py:
Use python2 compatible gzip.GzipFile instead of gzip.decompress. Send
work exception instead of work fail when an exception happens. Log these
exceptions locally as well.

Change-Id: Idf0a873215acb72187e058a0306a21ccd928d464
Reviewed-on: https://review.openstack.org/32804
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-17 21:08:48 +00:00
Clark Boylan 4aba312807 Use gearman to distribute logstash log pushing.
This change reorgs the logstash log pushing so that there is a central
gearman server that listens to Jenkins ZMQ events which are then
converted to per log file gearman jobs which are processed by gearman
workers. The central gearman server will live on logstash.o.o and the
existing logstash-worker hosts will be converted to gearman log pusher
workers.

This commit includes relavent documentation changes.

Change-Id: I45f7185c2479c54b090d223408dff268e1e8d7db
Reviewed-on: https://review.openstack.org/32455
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-12 21:45:42 +00:00
Clark Boylan 5bbe7824e0 Scale out logstash indexing to multiple hosts.
Logstash performs filtering in a single thread so it does not scale up
very well. Work around this by scaling Logstash out to multiple indexer
hosts.

Current plan is to have a small (2GB) kibana web front end host that
does nothing but talk to elasticsearch, three 4GB logstash indexers that
will run a single log-pusher.py + logstash indexer with some partition
of the logfiles assigned to each indexer, and finally the existing large
elasticsearch node.

Eventually properly load balancing log processing across the worker
nodes would be great, but the current partition method should work well
enough with little additional effort.

Change-Id: Ifc6396560934314ffd6a7c47eb2acff9e9c2a7af
Reviewed-on: https://review.openstack.org/30573
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-29 00:33:36 +00:00
Clark Boylan 5dd697f3c4 Fix elasticsearch connectivity on logstash.o.o.
The new elasticsearch server needs to be able to connect back to
logstash over the elasticsearch ports. Also make kibana talk to the new
elasticsearch server.

Change-Id: I88af40c7885b263fcef5ff16210f073cd0a696a5
Reviewed-on: https://review.openstack.org/30354
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-23 23:46:30 +00:00
Clark Boylan e43e208a99 Switch to dedicated elasticsearch node.
Switch to a large dedicated elasticsearch node as sharing resources
between logstash, kibana, jenkins-log-pusher, and elasticsearch results
in a constrained environment.

Change-Id: I39e6210f2c577429be2cb38aca09111a0f56f9be
Reviewed-on: https://review.openstack.org/30344
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-23 21:37:01 +00:00
Clark Boylan 1fed9e39e4 Restart log pusher when its config is updated.
* modules/openstack_project/manifests/logstash.pp:
Subscribe the jenkins-log-pusher service to the log pusher yaml config
file so that updates to the config force the pusher service to restart
and reload its config.

Change-Id: I4a675907133075bf8fa716b8c57c011489842858
Reviewed-on: https://review.openstack.org/29997
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-22 22:33:53 +00:00
Clark Boylan 868ad1ff0f Further elasticsearch config tweaks.
* modules/logstash/files/elasticsearch.yml: Set the number of
elasticsearch shards to 1 and the number of replicas to 0 as we are
not running a cluster, just a single node.

* modules/openstack_project/manifests/logstash.pp: Update elasticsearch
indice optimization cron job's max_num_segments value from 1 to 2.

These options are suggested at
https://github.com/logstash/logstash/wiki/Elasticsearch-Storage-Optimization

Change-Id: I747ab8952f0102ba582e213407909aaeebcf2778
Reviewed-on: https://review.openstack.org/28616
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-10 16:45:30 +00:00
Clark Boylan f07756b5bd Make log-pusher.py properly configurable.
* modules/openstack_project/files/logstash/log-pusher.py: Make the log
pusher properly configurable with a yaml configuration. As part of this
change support multiple zmq publisher inputs, multiple file retrievers,
job name filtering, and event tagging (with the filename).

* modules/openstack_project/files/logstash/jenkins-log-pusher.yaml:
Initial config for the log pusher.

* modules/openstack_project/manifests/logstash.pp: Put new log pusher:
config in place.

* modules/openstack_proejct/files/logstash/jenkins-log-pusher.init: Run
the log pusher service with the new config file.

Change-Id: I4c8405b1edfa16bbcc8f998627c6240bef23f302
Reviewed-on: https://review.openstack.org/28113
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-06 19:32:07 +00:00
Clark Boylan aae5ffc65e Make the Jenkins log pusher a service.
* modules/openstack_project/manifests/logstash.pp: Run the Jenkins log
pusher script as a service. This is the first step in making Logstash
use the TCP inputs instead of pipe inputs.

* modules/openstack_project/files/logstash/jenkins-log-pusher.init: Add
a simple init script for the Jenkins log pusher.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Switch
to TCP input instead of pipe input as the new Jenkins log pusher service
will push log events over TCP.

Change-Id: Id80c710abd5facd71d18afb2b250b2d7d92dec2d
Reviewed-on: https://review.openstack.org/28074
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-06 19:32:05 +00:00
K Jonathan Harker 901e706aea Use kibana as the web frontend to logstash.
Add a kibana module that installs kibana and flat configuration files,
and also disables the previous boring bland web frontend.

Change-Id: I772c66f07d2bc3a88128e101074be9d4162e3f8a
Reviewed-on: https://review.openstack.org/27089
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-25 20:45:19 +00:00
Clark Boylan 7086b8f459 Run single node logstash without redis.
logstash.o.o is running into memory contention. The three logstash
processes (agent, indexer, web), redis, and elasticsearch are all memory
hogs. Remove redis which allows us to consolidate the work of the agent
and indexer into a single process.

If we decide to add remote logstash agents we will want to add redis to
the mix again. This is probably best done on a second host dedicated to
redis.

Change-Id: I5ba96b4a27d91cb73c9380ed47d0495b9005effa
Reviewed-on: https://review.openstack.org/27452
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-25 17:44:22 +00:00
Clark Boylan 2c92932ba0 Use curl -sS to silence non error output.
The logstash daily curl commands output non error messages on normal
runs. Use the -sS options to curl to silence all non error message
output.

Change-Id: I7df3abdd74f6e058cf0208cefb7639202e4f9d0e
Reviewed-on: https://review.openstack.org/27316
Reviewed-by: Anita Kuno <anita.kuno@enovance.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-23 17:22:27 +00:00
Clark Boylan ffd659db39 Escape use of '%' in cron entries.
The '%' has special meaning in cron. Escape the use of this character in
cron entries to allow these cron entries to run properly.

Change-Id: I049a8a2a4454f99aedb7fe3017db231b8a13e569
Reviewed-on: https://review.openstack.org/27294
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-22 20:53:36 +00:00
Clark Boylan f6ae11e3b5 Manage elasticsearch logstash indices.
* modules/openstack_project/manifests/logstash.pp: Add two cron jobs.
One to delete indices that are a week old and a second to optimize the
previous days' index.

Change-Id: I14767b3a590c04fa60cdc5ec350e6f49c155050b
Reviewed-on: https://review.openstack.org/26642
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-04-11 21:05:17 +00:00
Clark Boylan 1876e51ef1 Install python3-zmq on logstash.o.o.
* modules/openstack_project/manifests/logstash.pp: The log-pusher.py
script which feeds jenkins logs into logstash depends on pyzmq for
python3. Install python3-zmq through apt to satisfy this dependency.

Change-Id: Ibd7cec25b865ce93fd8f5ff0c482af2982c68ce8
Reviewed-on: https://review.openstack.org/26406
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-04-09 17:20:33 +00:00
Clark Boylan f316096bea Fix typo in log-pusher.py source path.
* openstack_project/manifests/logstash.pp: Change source path from
puppet://modules/... to puppet:///modules/... so that puppet can find
the script.

Change-Id: I9c41f0b158d9ace9e839e34bcba37698b3a8cd18
Reviewed-on: https://review.openstack.org/26402
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-04-08 21:22:08 +00:00
Clark Boylan 9dfa44abca Put jenkins console logs into logstash.
Add a script that listens to jenkins ZMQ events and uses that
information to grab log files and pipe them into logstash. Configure
logstash to use this script and allow this script to connect to jenkins
via tcp port 8888.

Change-Id: Iced7d3a3147c116170a238b4fe07b46b7fb2a2a3
Reviewed-on: https://review.openstack.org/26035
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-08 20:29:35 +00:00
Clark Boylan 457a9d8764 Add skeleton logstash module.
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.

Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.

Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-01-29 20:44:09 +00:00