Merge "Remove configuration management for ELK stack"

This commit is contained in:
Zuul 2022-04-22 16:04:22 +00:00 committed by Gerrit Code Review
commit 7b09f7baab
21 changed files with 0 additions and 839 deletions

View File

@ -1,293 +0,0 @@
:title: Logstash
.. _logstash:
Logstash
########
Logstash is a high-performance indexing and search engine for logs.
At a Glance
===========
:Hosts:
* http://logstash.openstack.org
* logstash-worker\*.openstack.org
* elasticsearch\*.openstack.org
:Puppet:
* https://opendev.org/opendev/puppet-logstash
* :git_file:`modules/openstack_project/manifests/logstash.pp`
* :git_file:`modules/openstack_project/manifests/logstash_worker.pp`
* :git_file:`modules/openstack_project/manifests/elasticsearch.pp`
:Configuration:
* :git_file:`modules/openstack_project/files/logstash`
* :git_file:`modules/openstack_project/templates/logstash`
* `submit-logstash-jobs defaults`_
:Projects:
* http://logstash.net/
* http://kibana.org/
* http://www.elasticsearch.org/
* http://crm114.sourceforge.net/
:Bugs:
* https://storyboard.openstack.org/#!/project/748
* https://logstash.jira.com/secure/Dashboard.jspa
* https://github.com/rashidkpc/Kibana/issues
* https://github.com/elasticsearch/elasticsearch/issues
Overview
========
Logs from Zuul test runs are sent to logstash where they are
indexed and stored. Logstash facilitates reviewing logs from multiple
sources in a single test run, searching for errors or particular
events within a test run, as well as searching for log event trends
across test runs.
System Architecture
===================
There are four major layers in our Logstash setup.
1. Submit Logstash Jobs.
The `logs post-playbook`_ in the Zuul ``base`` job submit logs defined
in the `submit-logstash-jobs defaults`_ to a Logstash Indexer.
2. Logstash Indexer.
Reads these log events from the log pusher, filters them to remove
unwanted lines, collapses multiline events together, and parses
useful information out of the events before shipping them to
ElasticSearch for storage and indexing.
3. ElasticSearch.
Provides log storage, indexing, and search.
4. Kibana.
A Logstash oriented web client for ElasticSearch. You can perform
queries on your Logstash logs in ElasticSearch through Kibana using
the Lucene query language.
Each layer scales horizontally. As the number of logs grows we can add
more log pushers, more Logstash indexers, and more ElasticSearch nodes.
Currently we have multiple Logstash worker nodes that pair a log pusher
with a Logstash indexer. We did this as each Logstash process can only
dedicate a single thread to filtering log events which turns into a
bottleneck very quickly. This looks something like:
::
zuul post-logs playbook
|
|
gearman-client
/ | \
/ | \
gearman gearman gearman
worker1 worker2 worker3
| | |
logstash logstash logstash
indexer1 indexer2 indexer3
\ | /
\ | /
elasticsearch
cluster
|
|
kibana
Log Pusher
----------
This is an ansible module in the `submit-log-processor-jobs role`_. It
submits Gearman jobs to push log files into logstash.
Log pushing looks like this:
* Zuul runs post-playbook on job completion.
* Using info in the Gearman job log files are retrieved.
* Log files are processed then shipped to Logstash.
Using Gearman allows us to scale the number of log pushers
horizontally. It is as simple as adding another process that talks to
the Gearman server.
If you are interested in technical details the source of these scripts
can be found at
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-client.py
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
Logstash
--------
Logstash does the heavy lifting of squashing all of our log lines into
events with a common format. It reads the JSON log events from the log
pusher connected to it, deletes events we don't want, parses log lines
to set the timestamp, message, and other fields for the event, then
ships these processed events off to ElasticSearch where they are stored
and made queryable.
At a high level Logstash takes:
::
{
"fields" {
"build_name": "gate-foo",
"build_numer": "10",
"event_message": "2013-05-31T17:31:39.113 DEBUG Something happened",
},
}
And turns that into:
::
{
"fields" {
"build_name": "gate-foo",
"build_numer": "10",
"loglevel": "DEBUG"
},
"@message": "Something happened",
"@timestamp": "2013-05-31T17:31:39.113Z",
}
It flattens each log line into something that looks very much like
all of the other events regardless of the source log line format. This
makes querying your logs for lines from a specific build that failed
between two timestamps with specific message content very easy. You
don't need to write complicated greps instead you query against a
schema.
The config file that tells Logstash how to do this flattening can be
found at
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
This works via the tags that are associated with a given message.
The tags in
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
are used to tell logstash how to parse a given file's messages, based
on the file's message format.
When adding a new file to be indexed to
https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
at least one tag from the openstack-filters.conf file should be associated
with the new file. One can expect to see '{%logmessage%}' instead of
actual message data if indexing is not working properly.
In the event a new file's format is not covered, a patch for
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
should be submitted with an appropriate parsing pattern.
ElasticSearch
-------------
ElasticSearch is basically a REST API layer for Lucene. It provides
the storage and search engine for Logstash. It scales horizontally and
loves it when you give it more memory. Currently we run a multi-node
cluster on large VMs to give ElasticSearch both memory and disk space.
Per index (Logstash creates one index per day) we have N+1 replica
redundancy to distribute disk utilization and provide high availability.
Each replica is broken into multiple shards providing increased indexing
and search throughput as each shard is essentially a valid mini index.
To check on the cluster health, run this command on any es.* node::
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
Kibana
------
Kibana is a ruby app sitting behind Apache that provides a nice web UI
for querying Logstash events stored in ElasticSearch. Our install can
be reached at http://logstash.openstack.org. See
:ref:`query-logstash` for more info on using Kibana to perform
queries.
simpleproxy
-----------
Simpleproxy is a simple tcp proxy which allows forwarding tcp connections from
one host to another. We use it to forward mysql traffic from a publicly
accessible host to the trove instance running the subunit2sql MySQL DB. This
allows for public access to the data on the database through the host
logstash.openstack.org.
.. _query-logstash:
Querying Logstash
=================
Hop on over to http://logstash.openstack.org and by default you get the
last 15 minutes of everything Logstash knows about in chunks of 100.
We run a lot of tests but it is possible no logs have come in over the
last 15 minutes, change the dropdown in the top left from ``Last 15m``
to ``Last 60m`` to get a better window on the logs. At this point you
should see a list of logs, if you click on a log event it will expand
and show you all of the fields associated with that event and their
values (note Chromium and Kibana seem to have trouble with this at times
and some fields end up without values, use Firefox if this happens).
You can search based on all of these fields and if you click the
magnifying glass next to a field in the expanded event view it will add
that field and value to your search. This is a good way of refining
searches without a lot of typing.
The above is good info for poking around in the Logstash logs, but
one of your changes has a failing test and you want to know why. We
can jumpstart the refining process with a simple query.
``@fields.build_change:"$FAILING_CHANGE" AND @fields.build_patchset:"$FAILING_PATCHSET" AND @fields.build_name:"$FAILING_BUILD_NAME" AND @fields.build_number:"$FAILING_BUILD_NUMBER"``
This will show you all logs available from the patchset and build pair
that failed. Chances are that this is still a significant number of
logs and you will want to do more filtering. You can add more filters
to the query using ``AND`` and ``OR`` and parentheses can be used to
group sections of the query. Potential additions to the above query
might be
* ``AND @fields.filename:"logs/syslog.txt"`` to get syslog events.
* ``AND @fields.filename:"logs/screen-n-api.txt"`` to get Nova API events.
* ``AND @fields.loglevel:"ERROR"`` to get ERROR level events.
* ``AND @message"error"`` to get events with error in their message.
and so on.
General query tips:
* Don't search ``All time``. ElasticSearch is bad at trying to find all
the things it ever knew about. Give it a window of time to look
through. You can use the presets in the dropdown to select a window or
use the ``foo`` to ``bar`` boxes above the frequency graph.
* Only the @message field can have fuzzy searches performed on it. Other
fields require specific information.
* This system is growing fast and may not always keep up with the load.
Be patient. If expected logs do not show up immediately after the
Zuul job completes wait a few minutes.
crm114
=======
In an effort to assist with automated failure detection, the infra team
has started leveraging crm114 to classify and analyze the messages stored
by logstash.
The tool utilizes a statistical approach for classifying data, and is
frequently used as an email spam detector. For logstash data, the idea
is to flag those log entries that are not in passing runs and only in
failing ones, which should be useful in pinpointing what caused the
failures.
In the OpenStack logstash system, crm114 attaches an error_pr attribute
to all indexed entries. Values from -1000.00 to -10.00 should be considered
sufficient to get all potential errors as identified by the program.
Used in a kibana query, it would be structured like this:
::
error_pr:["-1000.0" TO "-10.0"]
This is still an early effort and additional tuning and refinement should
be expected. Should the crm114 settings need to be tuned or expanded,
a patch may be submitted for this file, which controls the process:
https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/classify-log.crm
.. _logs post-playbook: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml
.. _submit-logstash-jobs defaults: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
.. _submit-log-processor-jobs role: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-log-processor-jobs

View File

@ -16,7 +16,6 @@ Major Systems
grafyaml grafyaml
keycloak keycloak
zuul zuul
logstash
elastic-recheck elastic-recheck
devstack-gate devstack-gate
nodepool nodepool

View File

@ -14,12 +14,6 @@ cacti_hosts:
- bridge.openstack.org - bridge.openstack.org
- cacti.openstack.org - cacti.openstack.org
- eavesdrop01.opendev.org - eavesdrop01.opendev.org
- elasticsearch02.openstack.org
- elasticsearch03.openstack.org
- elasticsearch04.openstack.org
- elasticsearch05.openstack.org
- elasticsearch06.openstack.org
- elasticsearch07.openstack.org
- ethercalc02.openstack.org - ethercalc02.openstack.org
- etherpad01.opendev.org - etherpad01.opendev.org
- gitea-lb01.opendev.org - gitea-lb01.opendev.org
@ -39,27 +33,6 @@ cacti_hosts:
- kdc04.openstack.org - kdc04.openstack.org
- keycloak01.opendev.org - keycloak01.opendev.org
- lists.openstack.org - lists.openstack.org
- logstash-worker01.openstack.org
- logstash-worker02.openstack.org
- logstash-worker03.openstack.org
- logstash-worker04.openstack.org
- logstash-worker05.openstack.org
- logstash-worker06.openstack.org
- logstash-worker07.openstack.org
- logstash-worker08.openstack.org
- logstash-worker09.openstack.org
- logstash-worker10.openstack.org
- logstash-worker11.openstack.org
- logstash-worker12.openstack.org
- logstash-worker13.openstack.org
- logstash-worker14.openstack.org
- logstash-worker15.openstack.org
- logstash-worker16.openstack.org
- logstash-worker17.openstack.org
- logstash-worker18.openstack.org
- logstash-worker19.openstack.org
- logstash-worker20.openstack.org
- logstash.openstack.org
- nb01.opendev.org - nb01.opendev.org
- nb02.opendev.org - nb02.opendev.org
- nb03.opendev.org - nb03.opendev.org

View File

@ -91,48 +91,6 @@ all:
region_name: DFW region_name: DFW
public_v4: 104.239.144.232 public_v4: 104.239.144.232
public_v6: 2001:4800:7818:104:be76:4eff:fe04:46c8 public_v6: 2001:4800:7818:104:be76:4eff:fe04:46c8
elasticsearch02.openstack.org:
ansible_host: 104.130.159.19
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 104.130.159.19
public_v6: 2001:4800:7818:102:be76:4eff:fe05:40b
elasticsearch03.openstack.org:
ansible_host: 104.130.246.141
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 104.130.246.141
public_v6: 2001:4800:7819:103:be76:4eff:fe05:f26
elasticsearch04.openstack.org:
ansible_host: 104.130.246.138
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 104.130.246.138
public_v6: 2001:4800:7819:103:be76:4eff:fe02:3042
elasticsearch05.openstack.org:
ansible_host: 104.130.159.15
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 104.130.159.15
public_v6: 2001:4800:7818:102:be76:4eff:fe04:55b5
elasticsearch06.openstack.org:
ansible_host: 104.130.246.111
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 104.130.246.111
public_v6: 2001:4800:7819:103:be76:4eff:fe04:b9d7
elasticsearch07.openstack.org:
ansible_host: 104.130.246.44
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 104.130.246.44
public_v6: 2001:4800:7819:103:be76:4eff:fe04:35bb
ethercalc02.openstack.org: ethercalc02.openstack.org:
ansible_host: 162.242.144.125 ansible_host: 162.242.144.125
location: location:
@ -279,153 +237,6 @@ all:
region_name: DFW region_name: DFW
public_v4: 50.56.173.222 public_v4: 50.56.173.222
public_v6: 2001:4800:780e:510:3bc3:d7f6:ff04:b736 public_v6: 2001:4800:780e:510:3bc3:d7f6:ff04:b736
logstash-worker01.openstack.org:
ansible_host: 172.99.116.137
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 172.99.116.137
public_v6: 2001:4800:7821:105:be76:4eff:fe04:b9e7
logstash-worker02.openstack.org:
ansible_host: 23.253.253.66
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.253.66
public_v6: 2001:4800:7817:104:be76:4eff:fe04:aee1
logstash-worker03.openstack.org:
ansible_host: 23.253.242.246
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.242.246
public_v6: 2001:4800:7817:104:be76:4eff:fe04:81e6
logstash-worker04.openstack.org:
ansible_host: 166.78.47.27
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 166.78.47.27
public_v6: 2001:4800:7817:101:be76:4eff:fe04:46ec
logstash-worker05.openstack.org:
ansible_host: 23.253.100.127
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.100.127
public_v6: 2001:4800:7817:101:be76:4eff:fe04:3b27
logstash-worker06.openstack.org:
ansible_host: 23.253.100.168
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.100.168
public_v6: 2001:4800:7817:101:be76:4eff:fe04:5e21
logstash-worker07.openstack.org:
ansible_host: 172.99.116.187
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 172.99.116.187
public_v6: 2001:4800:7821:105:be76:4eff:fe04:bca2
logstash-worker08.openstack.org:
ansible_host: 166.78.47.61
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 166.78.47.61
public_v6: 2001:4800:7817:101:be76:4eff:fe04:432c
logstash-worker09.openstack.org:
ansible_host: 23.253.119.179
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.119.179
public_v6: 2001:4800:7817:103:be76:4eff:fe04:38d1
logstash-worker10.openstack.org:
ansible_host: 172.99.116.111
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 172.99.116.111
public_v6: 2001:4800:7821:105:be76:4eff:fe04:3a04
logstash-worker11.openstack.org:
ansible_host: 166.78.47.86
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 166.78.47.86
public_v6: 2001:4800:7817:101:be76:4eff:fe05:3e1
logstash-worker12.openstack.org:
ansible_host: 166.78.174.249
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 166.78.174.249
public_v6: 2001:4800:7815:105:be76:4eff:fe01:61e4
logstash-worker13.openstack.org:
ansible_host: 23.253.230.28
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.230.28
public_v6: 2001:4800:7817:103:be76:4eff:fe04:6ff1
logstash-worker14.openstack.org:
ansible_host: 172.99.116.170
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 172.99.116.170
public_v6: 2001:4800:7821:105:be76:4eff:fe04:4a4a
logstash-worker15.openstack.org:
ansible_host: 23.253.242.14
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.242.14
public_v6: 2001:4800:7817:104:be76:4eff:fe04:829b
logstash-worker16.openstack.org:
ansible_host: 23.253.230.58
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.230.58
public_v6: 2001:4800:7817:103:be76:4eff:fe03:5f45
logstash-worker17.openstack.org:
ansible_host: 172.99.117.120
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 172.99.117.120
public_v6: 2001:4800:7821:105:be76:4eff:fe04:6173
logstash-worker18.openstack.org:
ansible_host: 172.99.116.203
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 172.99.116.203
public_v6: 2001:4800:7821:105:be76:4eff:fe05:435
logstash-worker19.openstack.org:
ansible_host: 166.78.47.173
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 166.78.47.173
public_v6: 2001:4800:7817:101:be76:4eff:fe03:a5ec
logstash-worker20.openstack.org:
ansible_host: 23.253.76.170
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 23.253.76.170
public_v6: 2001:4800:7815:105:be76:4eff:fe04:59a6
logstash01.openstack.org:
ansible_host: 104.239.141.90
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 104.239.141.90
public_v6: 2001:4800:7819:104:be76:4eff:fe04:935b
meetpad01.opendev.org: meetpad01.opendev.org:
ansible_host: 104.239.240.194 ansible_host: 104.239.240.194
location: location:

View File

@ -1,4 +0,0 @@
iptables_extra_allowed_groups:
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'elasticsearch'}
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash'}
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash-worker'}

View File

@ -8,7 +8,6 @@ iptables_extra_allowed_hosts:
iptables_extra_allowed_groups: iptables_extra_allowed_groups:
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'} - {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'} - {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
- {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'} - {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'}
- {'protocol': 'udp', 'port': '8125', 'group': 'zuul'} - {'protocol': 'udp', 'port': '8125', 'group': 'zuul'}

View File

@ -8,7 +8,6 @@ iptables_extra_allowed_hosts:
iptables_extra_allowed_groups: iptables_extra_allowed_groups:
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'} - {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'} - {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
- {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'} - {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'}
- {'protocol': 'udp', 'port': '8125', 'group': 'zuul'} - {'protocol': 'udp', 'port': '8125', 'group': 'zuul'}

View File

@ -1,6 +0,0 @@
iptables_extra_public_tcp_ports:
- 80
- 3306
iptables_extra_allowed_groups:
- {'protocol': 'tcp', 'port': '4730', 'group': 'logstash-worker'}
- {'protocol': 'tcp', 'port': '4730', 'group': 'zuul-executor'}

View File

@ -52,7 +52,6 @@ groups:
- adns*.opendev.org - adns*.opendev.org
- ns*.opendev.org - ns*.opendev.org
eavesdrop: eavesdrop[0-9]*.opendev.org eavesdrop: eavesdrop[0-9]*.opendev.org
elasticsearch: elasticsearch[0-9]*.open*.org
ethercalc: ethercalc*.open*.org ethercalc: ethercalc*.open*.org
etherpad: etherpad[0-9]*.open*.org etherpad: etherpad[0-9]*.open*.org
gitea: gitea:
@ -105,10 +104,6 @@ groups:
- storyboard[0-9]*.opendev.org - storyboard[0-9]*.opendev.org
- translate[0-9]*.open*.org - translate[0-9]*.open*.org
- zuul[0-9]*.opendev.org - zuul[0-9]*.opendev.org
logstash:
- logstash[0-9]*.open*.org
logstash-worker:
- logstash-worker[0-9]*.open*.org
mailman: mailman:
- lists*.katacontainers.io - lists*.katacontainers.io
- lists*.open*.org - lists*.open*.org
@ -131,11 +126,8 @@ groups:
- paste[0-9]*.opendev.org - paste[0-9]*.opendev.org
puppet: puppet:
- cacti[0-9]*.open*.org - cacti[0-9]*.open*.org
- elasticsearch[0-9]*.open*.org
- ethercalc[0-9]*.open*.org - ethercalc[0-9]*.open*.org
- health[0-9]*.openstack.org - health[0-9]*.openstack.org
- logstash-worker[0-9]*.open*.org
- logstash[0-9]*.open*.org
- status*.open*.org - status*.open*.org
- storyboard-dev[0-9]*.opendev.org - storyboard-dev[0-9]*.opendev.org
- storyboard[0-9]*.opendev.org - storyboard[0-9]*.opendev.org
@ -143,11 +135,8 @@ groups:
- translate[0-9]*.open*.org - translate[0-9]*.open*.org
puppet4: puppet4:
- cacti[0-9]*.open*.org - cacti[0-9]*.open*.org
- elasticsearch[0-9]*.open*.org
- ethercalc[0-9]*.open*.org - ethercalc[0-9]*.open*.org
- health[0-9]*.openstack.org - health[0-9]*.openstack.org
- logstash-worker[0-9]*.open*.org
- logstash[0-9]*.open*.org
- status*.open*.org - status*.open*.org
- storyboard[0-9]*.opendev.org - storyboard[0-9]*.opendev.org
- storyboard-dev[0-9]*.opendev.org - storyboard-dev[0-9]*.opendev.org

View File

@ -20,66 +20,6 @@ node /^ethercalc\d+\.open.*\.org$/ {
} }
} }
# Node-OS: xenial
node /^logstash\d*\.open.*\.org$/ {
class { 'openstack_project::server': }
class { 'openstack_project::logstash':
discover_nodes => [
'elasticsearch03.openstack.org:9200',
'elasticsearch04.openstack.org:9200',
'elasticsearch05.openstack.org:9200',
'elasticsearch06.openstack.org:9200',
'elasticsearch07.openstack.org:9200',
'elasticsearch02.openstack.org:9200',
],
subunit2sql_db_host => hiera('subunit2sql_db_host', ''),
subunit2sql_db_pass => hiera('subunit2sql_db_password', ''),
}
}
# Node-OS: xenial
node /^logstash-worker\d+\.open.*\.org$/ {
$group = 'logstash-worker'
$elasticsearch_nodes = [
'elasticsearch02.openstack.org',
'elasticsearch03.openstack.org',
'elasticsearch04.openstack.org',
'elasticsearch05.openstack.org',
'elasticsearch06.openstack.org',
'elasticsearch07.openstack.org',
]
class { 'openstack_project::server': }
class { 'openstack_project::logstash_worker':
discover_node => 'elasticsearch03.openstack.org',
enable_mqtt => false,
mqtt_password => hiera('mqtt_service_user_password'),
mqtt_ca_cert_contents => hiera('mosquitto_tls_ca_file'),
}
}
# Node-OS: xenial
node /^elasticsearch\d+\.open.*\.org$/ {
$group = "elasticsearch"
$elasticsearch_nodes = [
'elasticsearch02.openstack.org',
'elasticsearch03.openstack.org',
'elasticsearch04.openstack.org',
'elasticsearch05.openstack.org',
'elasticsearch06.openstack.org',
'elasticsearch07.openstack.org',
]
class { 'openstack_project::server': }
class { 'openstack_project::elasticsearch_node':
discover_nodes => $elasticsearch_nodes,
}
}
# A machine to run Storyboard # A machine to run Storyboard
# Node-OS: xenial # Node-OS: xenial
node /^storyboard\d+\.opendev\.org$/ { node /^storyboard\d+\.opendev\.org$/ {

View File

@ -58,15 +58,12 @@ SOURCE_MODULES["https://github.com/voxpupuli/puppet-nodejs"]="v2.3.0"
# Please keep sorted # Please keep sorted
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-bup"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-bup"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elastic_recheck"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elastic_recheck"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elasticsearch"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ethercalc"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ethercalc"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-httpd"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-httpd"]="origin/master"
# Storyboard and translate use the jeepyb module # Storyboard and translate use the jeepyb module
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-jeepyb"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-jeepyb"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-kibana"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-kibana"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-log_processor"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logrotate"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logrotate"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logstash"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-mysql_backup"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-mysql_backup"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-pip"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-pip"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-project_config"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-project_config"]="origin/master"
@ -76,7 +73,6 @@ INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-reviewday"]="origin/mast
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-simpleproxy"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-simpleproxy"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ssh"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ssh"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-storyboard"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-storyboard"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-subunit2sql"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-tmpreaper"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-tmpreaper"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ulimit"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ulimit"]="origin/master"
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-user"]="origin/master" INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-user"]="origin/master"

View File

@ -1,3 +0,0 @@
# Increase the max heap size to twice the default.
# Default is 25% of memory or 1g whichever is less.
JAVA_ARGS='-Xmx2g'

View File

@ -1,46 +0,0 @@
# Copyright 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# Elasticsearch server glue class.
#
class openstack_project::elasticsearch_node (
$discover_nodes = ['localhost'],
$heap_size = '30g',
) {
class { 'logstash::elasticsearch': }
class { '::elasticsearch':
es_template_config => {
'index.store.compress.stored' => true,
'index.store.compress.tv' => true,
'indices.memory.index_buffer_size' => '33%',
'indices.breaker.fielddata.limit' => '70%',
'bootstrap.mlockall' => true,
'gateway.recover_after_nodes' => '5',
'gateway.recover_after_time' => '5m',
'gateway.expected_nodes' => '6',
'discovery.zen.minimum_master_nodes' => '4',
'discovery.zen.ping.multicast.enabled' => false,
'discovery.zen.ping.unicast.hosts' => $discover_nodes,
'http.cors.enabled' => true,
'http.cors.allow-origin' => "'*'", # lint:ignore:double_quoted_strings
},
heap_size => $heap_size,
version => '1.7.6',
}
class { 'logstash::curator':
keep_for_days => '7',
}
}

View File

@ -1,47 +0,0 @@
# Copyright 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# Logstash web frontend glue class.
#
class openstack_project::logstash (
$discover_nodes = ['elasticsearch02.openstack.org:9200'],
$statsd_host = 'graphite.opendev.org',
$subunit2sql_db_host,
$subunit2sql_db_pass,
) {
class { 'logstash::web':
frontend => 'kibana',
discover_nodes => $discover_nodes,
proxy_elasticsearch => true,
}
class { 'log_processor': }
class { 'log_processor::geard':
statsd_host => $statsd_host,
}
include 'subunit2sql'
class { 'subunit2sql::server':
db_host => $subunit2sql_db_host,
db_pass => $subunit2sql_db_pass,
}
include 'simpleproxy'
class { 'simpleproxy::server':
db_host => $subunit2sql_db_host,
}
}

View File

@ -1,100 +0,0 @@
# Copyright 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
# Logstash indexer worker glue class.
#
class openstack_project::logstash_worker (
$discover_node = 'elasticsearch02.openstack.org',
$filter_rev = 'master',
$filter_source = 'https://opendev.org/openstack/logstash-filters',
$enable_mqtt = true,
$mqtt_hostname = 'firehose.openstack.org',
$mqtt_port = 8883,
$mqtt_topic = "logstash/${::hostname}",
$mqtt_username = 'infra',
$mqtt_password = undef,
$mqtt_ca_cert_contents = undef,
) {
file { '/etc/logprocessor/worker.yaml':
ensure => present,
owner => 'root',
group => 'root',
mode => '0644',
content => template('openstack_project/logstash/jenkins-log-worker.yaml.erb'),
require => Class['::log_processor'],
}
file { '/etc/default/logstash-indexer':
ensure => present,
owner => 'root',
group => 'root',
mode => '0644',
source => 'puppet:///modules/openstack_project/logstash/logstash-indexer.default',
}
vcsrepo { '/opt/logstash-filters':
ensure => latest,
provider => git,
revision => $filter_rev,
source => $filter_source,
}
include ::logstash
logstash::filter { 'openstack-logstash-filters':
level => '50',
target => '/opt/logstash-filters/filters/openstack-filters.conf',
require => [
Class['::logstash'],
Vcsrepo['/opt/logstash-filters'],
],
notify => Service['logstash'],
}
file { '/etc/logstash/mqtt-root-CA.pem.crt':
ensure => present,
content => $mqtt_ca_cert_contents,
replace => true,
owner => 'root',
group => 'root',
mode => '0555',
require => Class['::logstash'],
}
validate_array($elasticsearch_nodes) # needed by output.conf.erb
class { '::logstash::indexer':
input_template => 'openstack_project/logstash/input.conf.erb',
output_template => 'openstack_project/logstash/output.conf.erb',
require => Logstash::Filter['openstack-logstash-filters'],
}
include ::log_processor
log_processor::worker { 'A':
config_file => '/etc/logprocessor/worker.yaml',
require => File['/etc/logprocessor/worker.yaml'],
}
log_processor::worker { 'B':
config_file => '/etc/logprocessor/worker.yaml',
require => File['/etc/logprocessor/worker.yaml'],
}
log_processor::worker { 'C':
config_file => '/etc/logprocessor/worker.yaml',
require => File['/etc/logprocessor/worker.yaml'],
}
log_processor::worker { 'D':
config_file => '/etc/logprocessor/worker.yaml',
require => File['/etc/logprocessor/worker.yaml'],
}
}

View File

@ -1,8 +0,0 @@
input {
tcp {
host => "localhost"
port => 9999
codec => json_lines {}
type => "jenkins"
}
}

View File

@ -1,13 +0,0 @@
gearman-host: logstash.openstack.org
gearman-port: 4730
output-host: localhost
output-port: 9999
output-mode: tcp
crm114-script: /usr/local/bin/classify-log.crm
crm114-data: /var/lib/crm114
mqtt-host: <%= @mqtt_hostname %>
mqtt-port: <%= @mqtt_port %>
mqtt-topic: gearman-logstash/<%= @hostname %>
mqtt-user: <%= @mqtt_username %>
mqtt-pass: <%= @mqtt_password %>
mqtt-ca-certs: /etc/logstash/mqtt-root-CA.pem.crt

View File

@ -1,9 +0,0 @@
output {
elasticsearch {
hosts => <%= @elasticsearch_nodes.map { |node| node + ":9200" }.inspect %>
manage_template => false
flush_size => 1024
timeout => 300
}
}

View File

@ -1,3 +0,0 @@
output {
redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
}

View File

@ -22,11 +22,6 @@ results:
- letsencrypt - letsencrypt
- mailman - mailman
logstash-worker02.openstack.org:
- logstash-worker
- puppet
- puppet4
mirror02.regionone.linaro-us.opendev.org: mirror02.regionone.linaro-us.opendev.org:
- afs-client - afs-client
- kerberos-client - kerberos-client

View File

@ -11,14 +11,11 @@
- opendev/ansible-role-puppet - opendev/ansible-role-puppet
- opendev/puppet-bup - opendev/puppet-bup
- opendev/puppet-elastic_recheck - opendev/puppet-elastic_recheck
- opendev/puppet-elasticsearch
- opendev/puppet-ethercalc - opendev/puppet-ethercalc
- opendev/puppet-httpd - opendev/puppet-httpd
- opendev/puppet-jeepyb - opendev/puppet-jeepyb
- opendev/puppet-kibana - opendev/puppet-kibana
- opendev/puppet-log_processor
- opendev/puppet-logrotate - opendev/puppet-logrotate
- opendev/puppet-logstash
- opendev/puppet-mysql_backup - opendev/puppet-mysql_backup
- opendev/puppet-openstack_infra_spec_helper - opendev/puppet-openstack_infra_spec_helper
- opendev/puppet-pip - opendev/puppet-pip
@ -28,7 +25,6 @@
- opendev/puppet-simpleproxy - opendev/puppet-simpleproxy
- opendev/puppet-ssh - opendev/puppet-ssh
- opendev/puppet-storyboard - opendev/puppet-storyboard
- opendev/puppet-subunit2sql
- opendev/puppet-tmpreaper - opendev/puppet-tmpreaper
- opendev/puppet-ulimit - opendev/puppet-ulimit
- opendev/puppet-user - opendev/puppet-user
@ -87,19 +83,15 @@
- opendev/puppet-project_config - opendev/puppet-project_config
- opendev/puppet-ethercalc - opendev/puppet-ethercalc
- opendev/puppet-httpd - opendev/puppet-httpd
- opendev/puppet-subunit2sql
- opendev/puppet-reviewday - opendev/puppet-reviewday
- opendev/puppet-kibana - opendev/puppet-kibana
- opendev/puppet-redis - opendev/puppet-redis
- opendev/puppet-zanata - opendev/puppet-zanata
- opendev/puppet-logstash
- opendev/puppet-tmpreaper - opendev/puppet-tmpreaper
- opendev/puppet-elastic_recheck - opendev/puppet-elastic_recheck
- opendev/puppet-ulimit - opendev/puppet-ulimit
- opendev/puppet-logrotate - opendev/puppet-logrotate
- opendev/puppet-elasticsearch
- opendev/puppet-storyboard - opendev/puppet-storyboard
- opendev/puppet-log_processor
- opendev/puppet-simpleproxy - opendev/puppet-simpleproxy
- opendev/puppet-bup - opendev/puppet-bup
- opendev/puppet-ssh - opendev/puppet-ssh