Commit Graph

49 Commits

Author SHA1 Message Date
Clark Boylan d82984ddf2 Retire this project
We've shutdown the logstash service and don't need to manage it with
puppet any more.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/839235
Change-Id: I2b5d3ac705d246f705b4ef91f12a4895e6854fc1
2022-04-25 09:50:21 -07:00
Jenkins 515357a301 Merge "Remove the mqtt options from logstash::indexer" 2017-08-02 18:20:16 +00:00
Colleen Murphy 399a2ca633 Parameterize JRE package
Add a params class so that puppet can distinguish between JRE packages
on different Ubuntu releases.

Change-Id: Ia1d22379d915d347c549c8cb8785920d97470f06
2017-06-14 14:38:27 +02:00
Colleen Murphy 567ae88a96 Fix curator cron command string
The puppet syntax check complains about the '\%' in the cron command
string, but the % still does need to be escaped for the cron to work
properly. We double escape it here in the puppet string which will
result in a properly single-escaped command string in the crontab.

Change-Id: I3d97463a611c43ded12ddb788e9da5e13466e0c5
2017-06-14 00:32:56 +02:00
Matthew Treinish f183b53d30
Remove the mqtt options from logstash::indexer
The logstash::indexer no longer sets up the mqtt output plugin for
logstash. So we don't need to keep the options around to enable it and
write the create the cert file. This commit removes those pieces.

Change-Id: I6ebdb5dc8fea8309609d837a758eeaf1dc5ee760
Depends-On: I6becec12604c61fe50d3e6b9c7ed9f9e9be311ae
2017-05-03 15:58:57 -04:00
Matthew Treinish 2f689ee899
Remove logstash mqtt plugin install
The logstash mqtt plugin caused issues in production and we're not going
to use it. We'll be emitting mqtt notifications directly from the
gearman worker instead. So this commit removes the plugin install so we
can set enable_mqtt to true to install the cert.

Change-Id: I0a17444cce18dd4b63f1f924e393483f6d8fe8eb
2017-04-27 15:57:33 -04:00
Clark Boylan 0e3d046cd6 Update to latest 2.x release of logstash
Hopes are this fixes the bug where logstash stops processing events and
ends up stalled out requiring forceful stoppage. Has been running
successfully on logstash-worker01 for almost a day now.

Change-Id: I62f311c33d8f31f26baee887881d6b672b0f2916
2017-01-03 08:45:14 -08:00
Matthew Treinish ce51abae9c
Switch onlyif to unless
We want unless not onlyif so we actually run the install command

Change-Id: I7ae6b05e38e2234b51f6263134a77d305fa46648
2016-09-21 08:18:25 -04:00
Matthew Treinish 616156f9a1
Add mqtt output plugin to logstash indexers
This commit adds support for configuring the mqtt logstash output
plugin. This will enable us to push events from our logstash workers
to firehose.o.o. The plugin can be found here:

https://github.com/kompa3/logstash-output-mqtt (althought it's README
is from a template and you have to look at the inline docs for how
to use it)

and the gem is published on rubygems.org here:

https://rubygems.org/gems/logstash-output-mqtt/

Change-Id: I7255f6c256ab3e3ca33caf69b71bf4ffab02c1bb
2016-09-20 05:57:34 -04:00
Clark Boylan f671fcf2c9 Pin curator to 3.5.1
elasticsearch-curator 4.0 introduced completely new everything. It will
take some work to support it and all of the new features so pin to 3.5.1
for now. This will keep index deletes working while we add the support.

Change-Id: Iea4e8794d2b953cfdd59ff7907420dffb3de4dff
2016-07-08 15:23:52 -07:00
Philip Marc Schwartz 38571dfae5 Update pip provider to use openstack_pip provider
Change-Id: Ib7a8740b7e7f83b1e93e54dc0e05b0d5e8ed61a9
Signed-off-by: Philip Marc Schwartz <philip@progmad.com>
2016-06-16 10:20:00 -04:00
Clark Boylan eb1fad4e80 Cleanup apache configuration
There is a puppet kibana module now that configures apache for us so we
don't need to do that here. Remove the redundant config.

Change-Id: I9e25c637b645f28b167bf248ee213608a1dfe615
Depends-On: Ic05f26d61f2ac4b5adffee98782d1a02931d5731
2016-05-26 10:41:47 -07:00
K Jonathan Harker c55836e555 Remove old config when switching to split configs
When we switch from a single config to split configs, make sure the old
config is removed. Leaving it in place prevents logstash from starting,
since we have multiple inputs attempting to listen on the same port.

Change-Id: I12e9da295e892ec70bcd9717fbdc0526b0822b86
2016-05-12 14:46:34 -07:00
Clark Boylan 049a6e2b87 Properly support logstash conf.d
Add in proper support for conf.d configs. Create input at 00-input,
output at 99-output, and filters at user specifiable levels in between
using a new defined type for filters.

Co-Authored-By: Jonathan Harker <code@gentlydownthe.net>
Change-Id: Icbca7a6ba0c5a94a273ef158f707311b588483fd
2016-04-29 14:15:29 -07:00
Jonathan Harker b91deb2b0c Fix beaker tests
Now that we install logstash 2.0, and its package manages service files
and user creation, update our beaker tests to reflect these changes.

Also, the logstash deb depends on the logrotate package, so make sure that's
installed.

Depends-On: Id50f98459b677460c28772c406c86482dcd0e9e4
Change-Id: I4c062c82d8d0aae5dd27036021a9464d788244d1
2016-04-29 12:41:02 -07:00
Andrey Nikitin 04f3167e9a Order of the classes parameters is refactored
Order and intendation of those parameters are changed
to follow Puppet Style Guide recommendation [0].
Moreover, it will allow to an user to find much faster
a variable in a list of variables.

[0]. https://docs.puppetlabs.com/guides/style_guide.html

Change-Id: I4457f7c6dd3cfca24084a252f360c45d1aa2b2d7
2016-03-21 13:13:04 +03:00
Jenkins ee59b4dae7 Merge "Install logstash 2.0" 2016-03-10 22:23:42 +00:00
Clark Boylan f326d9af28 Install logstash 2.0
This makes a whole host of changes. First we install logstash from the
deb packages that are now available. This package sets up the logstash
user, group, logrotate for log files, a conf.d dir, and an init script
for a service for us. This means all of that stuff can be removed from
puppet and be provided by the package instead.

In addition to cleanup related to the package we remove things that no
longer make sense with newer logstash. There is no more logstash web
service for example use kibana instead. Also the distinction between
agent and indexer is no longer strong and logstash has a separate
shipper tool which should be used in place of the agent (though backward
compat for the agent class is preserved by setting it up as an indexer
using an agent config).

Depends-On: I2354fbe9d3ab25134c52bfe58f562dfdf9ff6786
Change-Id: If3fa423a80a03d09993e58f6d68a3a212b457468
2016-03-10 09:24:45 -08:00
K Jonathan Harker 03844fb02d Escape % in cron command
The % character needs to be escaped in cron commands as it has a special
meaning.

Change-Id: Iab028133276c8186458c3f80098c9ef2ae58e9a9
2015-12-09 08:47:52 -08:00
K Jonathan Harker 4a8f77c2fb Look for curator in both /usr/bin and /usr/local/bin
Change-Id: I7adacd984aa8175ad991e0b0e42765ddcb4b8306
2015-11-17 09:43:49 -08:00
Jenkins 9779295da6 Merge "Upgrade from kibana2 to kibana3" 2015-10-15 21:01:13 +00:00
K Jonathan Harker 5f9717b803 Upgrade from kibana2 to kibana3
Change-Id: I176148c3bdaf603a1703d215ac2086101ecabe21
Depends-On: I13f9dff0bbd6498a36dc75b026c9042a9bb05e8f
2015-10-14 14:02:19 -07:00
jesusaurus 3c904c866e Create a class to install and run curator
The curator project manages elasticsearch indices, with knowledge of the
naming format used by logstash and the ability to delete indicies
matching a given pattern.

Needed-By: Iba3c0726b789dcbcf58f2da071e6fa93d4cd67ae
Change-Id: Ic49c8649a0ad849a8d72900425e589e40b82a37f
2015-09-28 22:56:27 -07:00
Colleen Murphy 466d44326b Add Gemfile and puppet 4 checks
In anticipation of puppet 4, start trying to deal with puppet 4 things
that can be helpfully predicted by puppet lint plugins. Also fix lint
errors caught by the puppet-lint-absolute_classname-check gem and arrow
alignment errors not caught before.

Change-Id: Icb76cc5019c378524344640e9670229e142a9627
2015-08-04 13:18:30 -07:00
Paul Belanger 60d1e2b295 Migrate to puppet-httpd module
puppet-httpd is the openstack-infra version of puppetlabs-apache
(0.0.4) release.

This patchset will remove the puppetlabs-apache namespace from -infra
allowing for possible future patchsets to use newer puppetlabs-apache
modules.

Change-Id: Idff8228de0a5c309b02d95feb6e6f25aafaa5f2a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2015-07-16 15:38:11 -04:00
Clark Boylan 79d4e6bbf6 Logrotate the indexer.log logfile.
Over time this logfile can grow quite large filling disk. Rotate it
daily and keep one week worth of logs to keep a reasonable amount of
logs.

Change-Id: Ic5f3acf290d3febc8af83773eb120df774937f1b
2015-05-14 10:31:51 -07:00
Yolanda Robla 9d3ba41410 Add missing vhost_name
Add a missing parameter into apache inclusion,
that was causing vhost_name to don't be passed properly
and generating the vhost incorrectly.

Change-Id: Ia39c005fffb82b5f1d75ac704cf24a1a2a978d6d
2015-04-23 08:20:25 +01:00
Spencer Krum fd00522597 Enhance and refactor head-of-file docs
The strange looking syntax the head-of-file docs are in is in
compliance with the new puppet-strings system for inline documentation

This enables neato stuff like 'puppet string <modulename>' to get docs
on the command line, and generating html, which we could then ship to
somewhere.

Change-Id: Icb0be2c733e4d3ee6beff32780c171b9f750b06f
2015-02-11 21:53:56 -08:00
Isaias Piña 0a11722253 Watchdog class will not be installed in logstash_worker class anymore
Change-Id: I68b48fd26eae73775d2e5200acd29dc1c375047e
2014-09-04 20:49:02 -05:00
Clark Boylan f754afd753 Use local ES balancers on logstash workers
Set up small non master non data elasticsearch daemons on logstash
workers to act as local load balancers for the elasticsearch http
protocol.

Change-Id: Ie3729f851ebef3331a6b69f718e57d663209bfc2
2014-07-07 17:01:57 -07:00
Clark Boylan a6248632db Convert logstash ES output to HTTP.
The logstash elasticsearch output seems to degrade over time and slow
down. Restarting the logstash daemon temporarily corrects this problem.
Switch to the elasticsearch HTTP output to see if that corrects the
problem as well.

Note the logstash watchdog is disabled by this change as logstash
daemons using elasticsearch HTTP output will not join the elasticsearch
cluster which will force the watchdog to always trip. To avoid this
issue disable the watchdog.

Change-Id: I77044b26fa10fb1fc3690a0464d79d55bed2fe00
2014-07-07 13:14:01 -07:00
Clark Boylan 2dfd6ee26a Install curl when logstash watchdog is used.
The logstash watchdog depends on curl and assumed that curl would just
be available. Not all cloud images are created equal and come with curl.
Be explicit about the need for curl and install it with puppet in the
watchdog manifest.

Change-Id: If90cae9a64b9f48f04b5643b643d2c83650c7d86
2014-06-25 10:24:30 -07:00
Clark Boylan 2e035245c9 Add a simple watchdog for logstash-indexer service
The logstash-indexer services sometimes fall out of the elasticsearch
cluster when it is under heavy load and unable to respond to pings.
Logstash doesn't do anything to reconnect :( so restart the service if a
node detects that it has fallen out of the cluster.

Upstream bug submitted at https://logstash.jira.com/browse/LOGSTASH-1951

Change-Id: I2e7767c5fe20cff279366fec2ddadd7710dbb4a9
2014-02-28 13:53:53 -08:00
Clark Boylan 71b5997918 Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9
Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9. Logstash 1.3.3 comes
with many performance improvements and bug fixes and needs ElasticSearch
0.90.9 to use the ElasticSearch output so upgrade ES as well. These
upgrades come with config changes for logstash as well to use new
features and disable others that are not wanted.

Change-Id: Ia180c5edfa755214a801f2cad06f091125814b0e
2014-02-11 17:07:34 -08:00
Clark Boylan 4f5c1c282c Upgrade Logstash to 1.2.1.
* modules/logstash/manifests/init.pp: Download and install Logstash
1.2.1.

* modules/openstack_project/files/logstash/log-gearman-client.py:
Logstash 1.2.1 comes with a new schema. Update the job data sent to log
push workers to better accomodate the new schema.

* modules/openstack_project/files/logstash/log-gearman-worker.py: Push
Logstash 1.2.1 schema compliant JSON to the Logstash TCP input.

* modules/openstack_project/templates/logstash/indexer.conf.erb:
Logstash 1.2.1 comes with a new schema and many input and filter
changes. Use the newly supported features like conditionals to keep the
config up to date.

* modules/kibana/templates/config.rb.erb: Change the default field for
kibana to 'message'. It was @message which is deprecated in the new
logstash schema.

Change-Id: Id19fc05bcce8d42c5c0cf33df3da7e95f5794107
2013-10-21 10:17:48 -07:00
Ryan Lane fdf5c0810e Separate logstash/elasticsearch and verify hash
To allow for reuse of elasticsearch I'm splitting it away from
logstash. Also, I'm doing a hash check of the elasticsearch wget
for better security.

Change-Id: Iff42d538cd941abd50b000879ea4a237ea48d40e
2013-10-15 11:57:51 -07:00
Clark Boylan 3a0f518d52 Tune ElasticSearch settings.
* modules/logstash/templates/elasticsearch.yml.erb:
Give 33% of memory to indexing instead of 40%.

* modules/logstash/files/es-logstash-template.json:
Flush the translog every 50k operations instead of the default 5k
(we can do 5k ops every second or two so the flushing was happening too
often). Limit the total number of shards per index per node to 3. We
have 5 nodes and 10 shards per index, 5*3 > 10 so we should be fine.
Set the field cache to soft type so that its entries can be garbage
collected in memory constricted situations.

* modules/logstash/manifests/elasticsearch.pp:
Apply es-logstash-template.json to the elasticsearch servers.

Change-Id: I2337fc41998fd00e090b0acfd29f007dfb6ec8df
2013-07-19 16:11:49 -07:00
Clark Boylan 7805734dcb Add two more elasticsearch nodes.
* manifests/site.pp: List the two new elasticsearch nodes in the
appropriate lists.

* modules/logstash/manifests/elasticsearch.pp: Do not restart
elasticsearch when config files change. Service restarts are costly and
should be manually performed when necessary. Otherwise puppet should
simply update the config files.

* modules/logstash/templates/elasticsearch.yml.erb: Update elasticsearch
config with new cluster topology. Increase memory available for
indexing.

* modules/openstack_project/manifests/cacti.pp: Add new nodes to cacti
monitoring list.

Adding two more elasticsearch nodes to relieve memory pressure (more
nodes means fewer indexes per nodes which requires less memory to
manage). And two more nodes gives us more disk to retain older indexes
in. These new nodes should allow us to retain at least 3 weeks of
indexed logs.

Change-Id: I3a5a02311e939c8147e401110c7b96d085eb8274
Reviewed-on: https://review.openstack.org/36305
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-09 18:40:21 +00:00
Clark Boylan 3ef10e90ed Better elasticsearch cluster settings.
Use mutliple discover nodes to determine elasticsearch cluster
membership. Put a timeout on recovery starting instead of the default
to recovery immediately. Describe cluster topology in elasticsearch yaml
config so that it can make smarter decisions. Round robin kibana
requests across each discover node.

Change-Id: I08ef9dd158ddf6a6ce01dfb2050626f543d45b10
Reviewed-on: https://review.openstack.org/34106
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-01 17:49:09 +00:00
Clark Boylan 6c33ba3b70 Add support for elasticsearch cluster.
We need to expand our elasticsearch install base. Update puppet to make
this possible.

Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-01 17:48:16 +00:00
Clark Boylan 60934a5cdd Reverse proxy subset of elasticsearch API.
It has been requested that we provide programmatic access to the
elasticsearch API in addition to the Kibana web UI. Do this by reverse
proxying http://logstash.openstack.org/elasticearch/$API_REQUEST to
http://elasticsearch.openstack.org:9200/$API_REQUEST.

The only values for $API_REQUEST that will be passed through are
_aliases, _status, and _search.

Change-Id: Ib41f6d91e2e59d493218074a67155af450ec8c93
Reviewed-on: https://review.openstack.org/33316
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-18 19:01:18 +00:00
Clark Boylan efabfa0e97 Install java7 runtime with logstash root manifest.
Logstash depends on java but the runtime wasn't being installed by
the Logstash manifest. Fix this.

Change-Id: I63f2c9831a9c9eae63f4eebd2493800dd606f8fb
Reviewed-on: https://review.openstack.org/30919
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-29 18:20:14 +00:00
Clark Boylan a45c34a36d Fix elasticsearch connectivity on logstash.o.o.
The new elasticsearch server needs to be able to connect back to
logstash over the elasticsearch ports. Also make kibana talk to the new
elasticsearch server.

Change-Id: I88af40c7885b263fcef5ff16210f073cd0a696a5
Reviewed-on: https://review.openstack.org/30354
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-23 23:46:30 +00:00
Clark Boylan e662dc7514 Upgrade logstash and elasticsearch. Retune ES.
Upgrade logstash to 1.1.12. Upgrade elasticsearch to 0.20.5 (the version
corresponding to logstash 1.1.12). Go back to default elasticsearch
shard and replica counts.

Change-Id: I8e5497442e918f78677335ec29d03542c93734e5
Reviewed-on: https://review.openstack.org/29447
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-17 19:15:53 +00:00
Clark Boylan ab1cec255c Fix elasticsearch mapping path and content.
* modules/logstash/manifests/elasticsearch.pp: The file name for the
default elasticsearch mapping is default-mapping.json not
default_mapping.json. Fix this path.

* modules/logstash/files/elasticsearch.mapping.json: Add a missing
comma.

Change-Id: I36bbe6e125157b97f3333f383a8949b2a6b3ef5c
Reviewed-on: https://review.openstack.org/28484
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-08 18:39:34 +00:00
K Jonathan Harker 2594d16c6b Add a default_mapping to elasticsearch.
* Don't store or index `_all`.
* Compress `_source` (which is stored but not indexed by default).
* Create a dynamic template for the string type.
* Only analyze the index of `@message` strings.

This is a very rudimentary config, and should probably be expanded
on by someone who has a deeper understanding of elasticsearch.

Change-Id: I14052c48c7bb6f05e259615a5203dc1ad87a08af
Reviewed-on: https://review.openstack.org/28216
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-05-04 12:36:12 +00:00
Clark Boylan 4b1d2cf531 Give elasticsearch more heap memory.
* modules/logstash/files/elasticseach.default: Elasticsearch does not
handle running out of memory very gracefully. Recommended allocation is
half of the hosts memory. Give elasticsearch 4GB of the 8GB host.

* modules/logstash/manifests/elasticseach.pp: Copy new elasticsearch
defaults file into place on the host.

Change-Id: I9ed14148c901cd3fd4ec38333a722821adb77ff5
Reviewed-on: https://review.openstack.org/27870
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-01 00:28:02 +00:00
K Jonathan Harker 6d5bc62b64 Use kibana as the web frontend to logstash.
Add a kibana module that installs kibana and flat configuration files,
and also disables the previous boring bland web frontend.

Change-Id: I772c66f07d2bc3a88128e101074be9d4162e3f8a
Reviewed-on: https://review.openstack.org/27089
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-25 20:45:19 +00:00
Clark Boylan 4ca0838652 Add skeleton logstash module.
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.

Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.

Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-01-29 20:44:09 +00:00