Commit Graph

45 Commits

Author SHA1 Message Date
OpenDev Sysadmins cedde0d8a5 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:43:00 +00:00
Clark Boylan 41a4d92703 Handle journald timestamps with microseconds
Our existing timestamp parsing rules could handle syslog format
timestamps with seconds but not microseconds. Journald output is giving
us microseconds so be sure to parse those too.

Change-Id: I44165b4744495ce825bc3b3b2cd908df004c5325
2017-05-03 11:33:17 -07:00
Clark Boylan 9e1dd3b7e1 Handle the new journald based logs
These logs have slightly more information. Also they have syslog
timestamps not iso8601 timestamps.

Change-Id: I7eaa35b3259d17f7e1f5f69e5acf41bddec685f7
2017-05-03 10:43:21 -07:00
Jonathan Harker d33c95310d Move the input and output sections to example files
In preparation for consuming the filter section by system-config,
move the input and output to example files to not consume them as well.

Change-Id: I5e0853783db8c793dde387a5f625a526f6c3b4aa
2016-04-25 12:27:49 -07:00
Jonathan Harker 73fc04fce1 Add .gitreview file
Change-Id: I9730e4982a52ced1aa0a36963e25cd2023a541b5
2016-04-25 12:26:04 -07:00
Jonathan Harker 145bed2b9e Sync filter updates with system-config templates
Pull in some changes that have merged in the system-config template
in preparation for converting system-config to use this project.

Change-Id: I84241379a88ea2a5c1bba6273ba063ae8615de99
2016-04-25 12:25:41 -07:00
James E. Blair 9729fb3840 Split grok filters into their own repo 2015-08-21 14:52:28 -07:00
Matt Riedemann 8556df1a4b Handle 'ERROR' for multiline oslofmt
oslo.log commit 27f7fe52fefcb81c5fe01736841d7a20149caf02 changed the
"logging_exception_prefix" config option to use the ERROR rather than
TRACE level which breaks multi-line queries in logstash.o.o, which is
kind of a big deal for elastic-recheck to not have super generic
queries.

This makes the log level an OR for TRACE or ERROR for versions of
oslo.log used in stable branches and ERROR for master.

This should behave OK otherwise because of the %{NUMBER} in the pattern
where NUMBER corresponds to the pid in logging_exception_prefix:

  %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s

Closes-Bug: #1474545

Change-Id: Iddd7f51b6a57deb6d56fc9f702992e7b14cf5cde
2015-07-14 14:03:46 -07:00
Jenkins 9ea0ba8e6c Merge "Move keystone logs over to oslofrmt" 2014-09-07 20:43:24 +00:00
pcrews c65bfd6d61 Altering indexer config to properly match on libvirtd files
The tags associated with libvirtd.txt in logstash are libvirtd.txt
and libvirt, the index config looked for libvirtd as the current
setup does not appear to match - we only have {%logmessage%} for message
content now. Setting search tag to 'libvirt'.

Change-Id: I27ae3b034581571285f42dab674ed85ff820cb6f
Partial-bug: 1350121
2014-08-01 16:13:49 -07:00
Joe Gordon febff756c8 Move keystone logs over to oslofrmt
After the successful implementation of
https://blueprints.launchpad.net/keystone/+spec/unified-logging-in-keystone
keystone uses oslo logging instead of its own. Update logstash settings
to reflect this.

We currently don't have any keystone logs in logstash, this should fix
that.

Change-Id: I7acfa35bfd112a15d2ec3aa7338fb28b37a827cb
2014-08-01 15:28:09 +10:00
Clark Boylan fbca077b50 Use local ES balancers on logstash workers
Set up small non master non data elasticsearch daemons on logstash
workers to act as local load balancers for the elasticsearch http
protocol.

Change-Id: Ie3729f851ebef3331a6b69f718e57d663209bfc2
2014-07-07 17:01:57 -07:00
Clark Boylan e3423df338 Convert logstash ES output to HTTP.
The logstash elasticsearch output seems to degrade over time and slow
down. Restarting the logstash daemon temporarily corrects this problem.
Switch to the elasticsearch HTTP output to see if that corrects the
problem as well.

Note the logstash watchdog is disabled by this change as logstash
daemons using elasticsearch HTTP output will not join the elasticsearch
cluster which will force the watchdog to always trip. To avoid this
issue disable the watchdog.

Change-Id: I77044b26fa10fb1fc3690a0464d79d55bed2fe00
2014-07-07 13:14:01 -07:00
Sean Dague 9cfe7b7133 add horizon_error to the indexed logs
this adds horizon_error to the indexed log files, which is very
useful in determining how horizon fails, as we get very specific
stack traces.

Change-Id: Ifb323e327dbc2931100a4552d029d91209c4bbba
2014-03-17 12:37:39 -07:00
Clark Boylan 7c9473ec26 Better logstash field data.
We are currently using a lot of wildcard searches in elasticsearch which
are slow. Provide better field data so that we can replace those
wildcard searches with filters. In particular add a short uuid field and
make the filename tag field the basename of the filepath so that grenade
and non grenade files all end up with the same tags.

Change-Id: If558017fceae96bcf197e611ab5cac1cfe7ae9bf
2014-03-13 14:42:58 -07:00
Clark Boylan e1c228c3b0 Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9
Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9. Logstash 1.3.3 comes
with many performance improvements and bug fixes and needs ElasticSearch
0.90.9 to use the ElasticSearch output so upgrade ES as well. These
upgrades come with config changes for logstash as well to use new
features and disable others that are not wanted.

Change-Id: Ia180c5edfa755214a801f2cad06f091125814b0e
2014-02-11 17:07:34 -08:00
Clark Boylan 594da47694 Remove deprecated logstash max_inflight_requests
The logstash ElasticSearch output max_inflight_requests option has been
deprecated and should not be used. Remove it.

Change-Id: I3644523f8605f17be7d9ed6f20f2020290f0fb28
2014-02-11 15:28:46 -08:00
Clark Boylan 1fd943286c Parse timestamps in libvirt logs properly.
The libvirt logs use a slightly different timestamp than the other log
files (iso8601 including the timezone offset) which makes logstash
unhappy when it receives these timestamps. Parse them properly to keep
logstash logs under control and hopefully happy.

Change-Id: I18b7f263d083d9babbdf1404298cb7def0b8501a
2014-02-11 11:31:19 -08:00
Joe Gordon e78d152c2a Add logstash rules to parse libvirtd.txt
Change-Id: I09185dae08d46821f804e3ad43205205aab9345c
2014-01-31 15:27:54 -08:00
Clark Boylan 4698b088a2 Upgrade Logstash to 1.2.1.
* modules/logstash/manifests/init.pp: Download and install Logstash
1.2.1.

* modules/openstack_project/files/logstash/log-gearman-client.py:
Logstash 1.2.1 comes with a new schema. Update the job data sent to log
push workers to better accomodate the new schema.

* modules/openstack_project/files/logstash/log-gearman-worker.py: Push
Logstash 1.2.1 schema compliant JSON to the Logstash TCP input.

* modules/openstack_project/templates/logstash/indexer.conf.erb:
Logstash 1.2.1 comes with a new schema and many input and filter
changes. Use the newly supported features like conditionals to keep the
config up to date.

* modules/kibana/templates/config.rb.erb: Change the default field for
kibana to 'message'. It was @message which is deprecated in the new
logstash schema.

Change-Id: Id19fc05bcce8d42c5c0cf33df3da7e95f5794107
2013-10-21 10:17:48 -07:00
Clark Boylan d2d138c09c Don't index logs with DEBUG log level.
ElasticSearch has a hard time performing queries on large amounts of
data. It must load the fields it is searching on into memory and that
can cause the node to use all of the memory allocated to it which in
turn causes a lot of garbage collection essentially taking the node
offline. Filter out DEBUG log messages in the devstack logs to reduce
the amount of data that must be loaded into memory when performing
searches.

Change-Id: Icfe9c8c17ccef4f9379c774eef791f43463dcf6d
2013-08-06 11:52:26 -07:00
Clark Boylan d320162dc4 Add support for elasticsearch cluster.
We need to expand our elasticsearch install base. Update puppet to make
this possible.

Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-01 17:48:16 +00:00
Clark Boylan 2f2169bc56 Handle mutliline oslofmt traceback in logstash.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Add a
mutliline filter that matches oslo format TRACE level log lines. This
will collect all lines related to the traceback in a single event.

Change-Id: I201e407b6bd6fbb5b2db94fa6757f184709f2eaa
Reviewed-on: https://review.openstack.org/32962
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-18 19:03:57 +00:00
Clark Boylan aa925dd95e Add swift logs to Logstash.
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the swift log files to the list of files to be processed by
logstash-worker1. This does not include the swift proxy log file as it
will require extra parsing.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for apache combined log format files.

Change-Id: I7545ace8f7601bdca453e0d2ac1b2233823878ce
Reviewed-on: https://review.openstack.org/31103
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-30 20:44:48 +00:00
Clark Boylan 06ae62b256 Add syslog logs to Logstash.
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the syslog log file to the list of files to be processed by
logstash-worker1.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for syslog format files.

Change-Id: I0f8f58ab484949eb0506842bdb98385767a50333
Reviewed-on: https://review.openstack.org/31097
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-05-30 20:44:42 +00:00
Clark Boylan 700a9df757 Add keystone logs to Logstash.
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the keystone log file to the list of files to be processed by
logstash-worker1.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for keystone format files.

Change-Id: I5a72fc17ed1f37b816581faabe44f26f8cc36db2
Reviewed-on: https://review.openstack.org/31096
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-05-30 20:44:10 +00:00
Clark Boylan a388e4cc04 Handle multiline Logstash events per file.
* modules/openstack_project/files/logstash/log-pusher.py: Add a filename
field to logstash events that can be used to associate multiline events
in files to their appropriate parents in the same file.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
stream_identities to the mutliline filters that use the source host and
file name to determine relationships between mutliline events.

Change-Id: Ia325c0e1257131ab1b721c4df8f70f6bea1d0b99
Reviewed-on: https://review.openstack.org/30953
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-05-30 20:44:08 +00:00
Clark Boylan fa476974c2 Set ElasticSearch output node name to hostname.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Set the
node name parameter in the ElasticSearch output template to be the
hostname of the host running logstash.

Change-Id: I8ade373403ca59d6d5768df175405167c9c8b4be
Reviewed-on: https://review.openstack.org/30925
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-30 17:17:24 +00:00
Clark Boylan a5f8f1d12a Increase Logstash ES output max_inflight_requests.
The default value for the Logstash ElasticSearch max_inflight_requests
parameters is 50. Bump that up to 512, to have a bigger firehose to
ElasticSearch. ElasticSearch is the piece of the Logstash puzzle that
doesn't seem to have trouble with bigger firehoses.

Change-Id: I3eeaa036c68b00a8daa0896a4256b30043f04aaa
Reviewed-on: https://review.openstack.org/30769
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-29 00:33:30 +00:00
Clark Boylan 00c31589c0 Fix elasticsearch connectivity on logstash.o.o.
The new elasticsearch server needs to be able to connect back to
logstash over the elasticsearch ports. Also make kibana talk to the new
elasticsearch server.

Change-Id: I88af40c7885b263fcef5ff16210f073cd0a696a5
Reviewed-on: https://review.openstack.org/30354
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-23 23:46:30 +00:00
Clark Boylan 8fb3f9d288 Add cinder and quantum screen logs to logstash.
* modules/openstack_project/files/logstash/jenkins-log-pusher.yaml:
Add the cinder and quantum screen log files to the log pusher. Tag them
with screen and oslofmt (screen because they are screen logs and oslofmt
because they use the oslo log message format).

* modules/openstack_project/templates/logstash/indexer.conf.erb:
Add a grep filter to remove the screen log header lines. Add a multiline
filter to handle oslo log format multi line events. Add a grok filter to
parse the oslo format logs. Handle timestamps without millisecond
precision. Remove event_message field if that message was properly
parsed.

Change-Id: Icd18e252a512416e0cce5ee0e27942b072a25e09
Reviewed-on: https://review.openstack.org/29985
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-23 22:10:42 +00:00
Clark Boylan 017aba78ea Switch to dedicated elasticsearch node.
Switch to a large dedicated elasticsearch node as sharing resources
between logstash, kibana, jenkins-log-pusher, and elasticsearch results
in a constrained environment.

Change-Id: I39e6210f2c577429be2cb38aca09111a0f56f9be
Reviewed-on: https://review.openstack.org/30344
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-23 21:37:01 +00:00
Clark Boylan 99635df2e9 Make logstash GREEDYDATA accept newlines.
* modules/openstack_project/templates/logstash/indexer.conf.erb: The
mutliline filter can add newlines to the @message field of events. Make
the grok filter consume the newlines.

Change-Id: I8353a482baf83ec1e728e4852c59c37b5e078e8a
Reviewed-on: https://review.openstack.org/29984
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-22 22:33:58 +00:00
Clark Boylan 929f152d49 Use log-pusher injected tags during logstash grok
* modules/openstack_project/templates/logstash/indexer.conf.erb: The
log-pusher.py script tags events with the filename of the log generating
the event. Use these values instead of different types to differentiate
Jenkins console logs from nova logs and so on.

Note that filters must match all of the values in the tags array (they
are ANDed together). This may mean that as the logstash filter rulesets
grow we will need a set of initial mutates to convert
'logs/screen-n-api.txt' and 'logs/screen-n-cpu.txt' tags to 'nova' to
allow for common grok and parsing filters.

Change-Id: I2769bc05a2e9cc7e8dbc46849e052146b9fee75e
Reviewed-on: https://review.openstack.org/28119
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-06 19:32:10 +00:00
Clark Boylan aa411bef1a Make the Jenkins log pusher a service.
* modules/openstack_project/manifests/logstash.pp: Run the Jenkins log
pusher script as a service. This is the first step in making Logstash
use the TCP inputs instead of pipe inputs.

* modules/openstack_project/files/logstash/jenkins-log-pusher.init: Add
a simple init script for the Jenkins log pusher.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Switch
to TCP input instead of pipe input as the new Jenkins log pusher service
will push log events over TCP.

Change-Id: Id80c710abd5facd71d18afb2b250b2d7d92dec2d
Reviewed-on: https://review.openstack.org/28074
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-06 19:32:05 +00:00
Clark Boylan 8a2b2af255 Turn on log-pusher.py debug logging.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Enable
log-pusher.py debug logging to help determine why logstash is restarting
the log-pusher and to see why the logs stop flowing at times.

Change-Id: I250a6400c2fd29201865e9f8c5761dc2f4fbf5c3
Reviewed-on: https://review.openstack.org/27871
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-01 00:28:05 +00:00
Clark Boylan 3a3e0a1f22 Run single node logstash without redis.
logstash.o.o is running into memory contention. The three logstash
processes (agent, indexer, web), redis, and elasticsearch are all memory
hogs. Remove redis which allows us to consolidate the work of the agent
and indexer into a single process.

If we decide to add remote logstash agents we will want to add redis to
the mix again. This is probably best done on a second host dedicated to
redis.

Change-Id: I5ba96b4a27d91cb73c9380ed47d0495b9005effa
Reviewed-on: https://review.openstack.org/27452
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-25 17:44:22 +00:00
Clark Boylan 17ac82d873 Better multiline handling for Jenkins console logs
* modules/openstack_project/templates/logstash/agent.conf.erb: Treat any
line not beginning with a timestamp in Jenkins console logs as belonging
to the previous log message. This keeps timestamps sane for messages
that do not have timestamps.

Change-Id: I5959e766c28efebbd6c44567262beb63ad3a0a8e
Reviewed-on: https://review.openstack.org/26777
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-04-11 21:07:58 +00:00
Clark Boylan 07261689b6 Remove redundant log fields.
* modules/openstack_project/templates/logstash/agent.conf.erb: Remove
logdate and logmessage fields when we are done with them. This reduces
the size of each log event.

Change-Id: I0b102d437397858d07cfc38498bc6b431199d3a4
Reviewed-on: https://review.openstack.org/26531
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-10 22:03:00 +00:00
Clark Boylan 34a1e989c7 Handle mulitline console log lines.
* modules/openstack_project/templates/logstash/agent.conf.erb: Add
multiline filters to properly capture multiline log lines that do not
include their own timestamp.

Change-Id: I127e327cf6bdde580dcdfbe185f77fae3bd8d2c5
Reviewed-on: https://review.openstack.org/26530
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-10 22:02:57 +00:00
Clark Boylan 24d3f11326 Clean up logstash Jenkins console logs.
* modules/openstack_project/files/logstash/log-pusher.py: Change
@message to event_messsage. Using @message here appears to confuse
logstash and results in duplicate events.

* modules/openstack_project/templates/logstash/agent.conf.erb: Change
@message to event_messsage. Using @message here appears to confuse
logstash and results in duplicate events. Also drop the opening and
closing pre tags that end up in the html log files.

Change-Id: I3cf679dc644392cfb8ef9002cf149d8b449ad4f5
Reviewed-on: https://review.openstack.org/26518
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-10 22:02:54 +00:00
Clark Boylan 5770730f1e Parse Jenkins' log timestamps with logstash.
* modules/openstack_project/templates/logstash/agent.conf.erb: Use the
grok, date, and mutate filters to transform Jenkins' logs from
"2013-04-08 22:46:43.659 | Started by user OpenStack Hudson" to

  @timestamp: 2013-04-08 22:46:43.659
  @message: Started by user OpenStack Hudson

Change-Id: I6aac49b4717f25838c13a2563360d03e40bb19e3
Reviewed-on: https://review.openstack.org/26446
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-09 17:16:19 +00:00
Clark Boylan d9d2af5eaf Set a message_format for logstash log-pusher input
* modules/openstack_project/templates/logstash/agent.conf.erb: Set
message_format to "%{@message}".

Change-Id: Ia8e753ff8bc9a2e0b16a6b43a7cbaf74c6a59eb2
Reviewed-on: https://review.openstack.org/26412
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: K Jonathan Harker <jesusaurus@mail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-09 16:43:24 +00:00
Clark Boylan 40d012787e Put jenkins console logs into logstash.
Add a script that listens to jenkins ZMQ events and uses that
information to grab log files and pipe them into logstash. Configure
logstash to use this script and allow this script to connect to jenkins
via tcp port 8888.

Change-Id: Iced7d3a3147c116170a238b4fe07b46b7fb2a2a3
Reviewed-on: https://review.openstack.org/26035
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-08 20:29:35 +00:00
Clark Boylan 25d167565a Add skeleton logstash module.
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.

Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.

Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-01-29 20:44:09 +00:00