Add a params class so that puppet can distinguish between JRE packages
on different Ubuntu releases.
Change-Id: Ia1d22379d915d347c549c8cb8785920d97470f06
The puppet syntax check complains about the '\%' in the cron command
string, but the % still does need to be escaped for the cron to work
properly. We double escape it here in the puppet string which will
result in a properly single-escaped command string in the crontab.
Change-Id: I3d97463a611c43ded12ddb788e9da5e13466e0c5
The logstash::indexer no longer sets up the mqtt output plugin for
logstash. So we don't need to keep the options around to enable it and
write the create the cert file. This commit removes those pieces.
Change-Id: I6ebdb5dc8fea8309609d837a758eeaf1dc5ee760
Depends-On: I6becec12604c61fe50d3e6b9c7ed9f9e9be311ae
The logstash mqtt plugin caused issues in production and we're not going
to use it. We'll be emitting mqtt notifications directly from the
gearman worker instead. So this commit removes the plugin install so we
can set enable_mqtt to true to install the cert.
Change-Id: I0a17444cce18dd4b63f1f924e393483f6d8fe8eb
Hopes are this fixes the bug where logstash stops processing events and
ends up stalled out requiring forceful stoppage. Has been running
successfully on logstash-worker01 for almost a day now.
Change-Id: I62f311c33d8f31f26baee887881d6b672b0f2916
This commit adds support for configuring the mqtt logstash output
plugin. This will enable us to push events from our logstash workers
to firehose.o.o. The plugin can be found here:
https://github.com/kompa3/logstash-output-mqtt (althought it's README
is from a template and you have to look at the inline docs for how
to use it)
and the gem is published on rubygems.org here:
https://rubygems.org/gems/logstash-output-mqtt/
Change-Id: I7255f6c256ab3e3ca33caf69b71bf4ffab02c1bb
elasticsearch-curator 4.0 introduced completely new everything. It will
take some work to support it and all of the new features so pin to 3.5.1
for now. This will keep index deletes working while we add the support.
Change-Id: Iea4e8794d2b953cfdd59ff7907420dffb3de4dff
There is a puppet kibana module now that configures apache for us so we
don't need to do that here. Remove the redundant config.
Change-Id: I9e25c637b645f28b167bf248ee213608a1dfe615
Depends-On: Ic05f26d61f2ac4b5adffee98782d1a02931d5731
When we switch from a single config to split configs, make sure the old
config is removed. Leaving it in place prevents logstash from starting,
since we have multiple inputs attempting to listen on the same port.
Change-Id: I12e9da295e892ec70bcd9717fbdc0526b0822b86
Add in proper support for conf.d configs. Create input at 00-input,
output at 99-output, and filters at user specifiable levels in between
using a new defined type for filters.
Co-Authored-By: Jonathan Harker <code@gentlydownthe.net>
Change-Id: Icbca7a6ba0c5a94a273ef158f707311b588483fd
Now that we install logstash 2.0, and its package manages service files
and user creation, update our beaker tests to reflect these changes.
Also, the logstash deb depends on the logrotate package, so make sure that's
installed.
Depends-On: Id50f98459b677460c28772c406c86482dcd0e9e4
Change-Id: I4c062c82d8d0aae5dd27036021a9464d788244d1
Order and intendation of those parameters are changed
to follow Puppet Style Guide recommendation [0].
Moreover, it will allow to an user to find much faster
a variable in a list of variables.
[0]. https://docs.puppetlabs.com/guides/style_guide.html
Change-Id: I4457f7c6dd3cfca24084a252f360c45d1aa2b2d7
This makes a whole host of changes. First we install logstash from the
deb packages that are now available. This package sets up the logstash
user, group, logrotate for log files, a conf.d dir, and an init script
for a service for us. This means all of that stuff can be removed from
puppet and be provided by the package instead.
In addition to cleanup related to the package we remove things that no
longer make sense with newer logstash. There is no more logstash web
service for example use kibana instead. Also the distinction between
agent and indexer is no longer strong and logstash has a separate
shipper tool which should be used in place of the agent (though backward
compat for the agent class is preserved by setting it up as an indexer
using an agent config).
Depends-On: I2354fbe9d3ab25134c52bfe58f562dfdf9ff6786
Change-Id: If3fa423a80a03d09993e58f6d68a3a212b457468
The curator project manages elasticsearch indices, with knowledge of the
naming format used by logstash and the ability to delete indicies
matching a given pattern.
Needed-By: Iba3c0726b789dcbcf58f2da071e6fa93d4cd67ae
Change-Id: Ic49c8649a0ad849a8d72900425e589e40b82a37f
In anticipation of puppet 4, start trying to deal with puppet 4 things
that can be helpfully predicted by puppet lint plugins. Also fix lint
errors caught by the puppet-lint-absolute_classname-check gem and arrow
alignment errors not caught before.
Change-Id: Icb76cc5019c378524344640e9670229e142a9627
puppet-httpd is the openstack-infra version of puppetlabs-apache
(0.0.4) release.
This patchset will remove the puppetlabs-apache namespace from -infra
allowing for possible future patchsets to use newer puppetlabs-apache
modules.
Change-Id: Idff8228de0a5c309b02d95feb6e6f25aafaa5f2a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Over time this logfile can grow quite large filling disk. Rotate it
daily and keep one week worth of logs to keep a reasonable amount of
logs.
Change-Id: Ic5f3acf290d3febc8af83773eb120df774937f1b
Add a missing parameter into apache inclusion,
that was causing vhost_name to don't be passed properly
and generating the vhost incorrectly.
Change-Id: Ia39c005fffb82b5f1d75ac704cf24a1a2a978d6d
The strange looking syntax the head-of-file docs are in is in
compliance with the new puppet-strings system for inline documentation
This enables neato stuff like 'puppet string <modulename>' to get docs
on the command line, and generating html, which we could then ship to
somewhere.
Change-Id: Icb0be2c733e4d3ee6beff32780c171b9f750b06f
Set up small non master non data elasticsearch daemons on logstash
workers to act as local load balancers for the elasticsearch http
protocol.
Change-Id: Ie3729f851ebef3331a6b69f718e57d663209bfc2
The logstash elasticsearch output seems to degrade over time and slow
down. Restarting the logstash daemon temporarily corrects this problem.
Switch to the elasticsearch HTTP output to see if that corrects the
problem as well.
Note the logstash watchdog is disabled by this change as logstash
daemons using elasticsearch HTTP output will not join the elasticsearch
cluster which will force the watchdog to always trip. To avoid this
issue disable the watchdog.
Change-Id: I77044b26fa10fb1fc3690a0464d79d55bed2fe00
The logstash watchdog depends on curl and assumed that curl would just
be available. Not all cloud images are created equal and come with curl.
Be explicit about the need for curl and install it with puppet in the
watchdog manifest.
Change-Id: If90cae9a64b9f48f04b5643b643d2c83650c7d86
The logstash-indexer services sometimes fall out of the elasticsearch
cluster when it is under heavy load and unable to respond to pings.
Logstash doesn't do anything to reconnect :( so restart the service if a
node detects that it has fallen out of the cluster.
Upstream bug submitted at https://logstash.jira.com/browse/LOGSTASH-1951
Change-Id: I2e7767c5fe20cff279366fec2ddadd7710dbb4a9
Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9. Logstash 1.3.3 comes
with many performance improvements and bug fixes and needs ElasticSearch
0.90.9 to use the ElasticSearch output so upgrade ES as well. These
upgrades come with config changes for logstash as well to use new
features and disable others that are not wanted.
Change-Id: Ia180c5edfa755214a801f2cad06f091125814b0e
* modules/logstash/manifests/init.pp: Download and install Logstash
1.2.1.
* modules/openstack_project/files/logstash/log-gearman-client.py:
Logstash 1.2.1 comes with a new schema. Update the job data sent to log
push workers to better accomodate the new schema.
* modules/openstack_project/files/logstash/log-gearman-worker.py: Push
Logstash 1.2.1 schema compliant JSON to the Logstash TCP input.
* modules/openstack_project/templates/logstash/indexer.conf.erb:
Logstash 1.2.1 comes with a new schema and many input and filter
changes. Use the newly supported features like conditionals to keep the
config up to date.
* modules/kibana/templates/config.rb.erb: Change the default field for
kibana to 'message'. It was @message which is deprecated in the new
logstash schema.
Change-Id: Id19fc05bcce8d42c5c0cf33df3da7e95f5794107
To allow for reuse of elasticsearch I'm splitting it away from
logstash. Also, I'm doing a hash check of the elasticsearch wget
for better security.
Change-Id: Iff42d538cd941abd50b000879ea4a237ea48d40e
* modules/logstash/templates/elasticsearch.yml.erb:
Give 33% of memory to indexing instead of 40%.
* modules/logstash/files/es-logstash-template.json:
Flush the translog every 50k operations instead of the default 5k
(we can do 5k ops every second or two so the flushing was happening too
often). Limit the total number of shards per index per node to 3. We
have 5 nodes and 10 shards per index, 5*3 > 10 so we should be fine.
Set the field cache to soft type so that its entries can be garbage
collected in memory constricted situations.
* modules/logstash/manifests/elasticsearch.pp:
Apply es-logstash-template.json to the elasticsearch servers.
Change-Id: I2337fc41998fd00e090b0acfd29f007dfb6ec8df
* manifests/site.pp: List the two new elasticsearch nodes in the
appropriate lists.
* modules/logstash/manifests/elasticsearch.pp: Do not restart
elasticsearch when config files change. Service restarts are costly and
should be manually performed when necessary. Otherwise puppet should
simply update the config files.
* modules/logstash/templates/elasticsearch.yml.erb: Update elasticsearch
config with new cluster topology. Increase memory available for
indexing.
* modules/openstack_project/manifests/cacti.pp: Add new nodes to cacti
monitoring list.
Adding two more elasticsearch nodes to relieve memory pressure (more
nodes means fewer indexes per nodes which requires less memory to
manage). And two more nodes gives us more disk to retain older indexes
in. These new nodes should allow us to retain at least 3 weeks of
indexed logs.
Change-Id: I3a5a02311e939c8147e401110c7b96d085eb8274
Reviewed-on: https://review.openstack.org/36305
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Use mutliple discover nodes to determine elasticsearch cluster
membership. Put a timeout on recovery starting instead of the default
to recovery immediately. Describe cluster topology in elasticsearch yaml
config so that it can make smarter decisions. Round robin kibana
requests across each discover node.
Change-Id: I08ef9dd158ddf6a6ce01dfb2050626f543d45b10
Reviewed-on: https://review.openstack.org/34106
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
We need to expand our elasticsearch install base. Update puppet to make
this possible.
Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
It has been requested that we provide programmatic access to the
elasticsearch API in addition to the Kibana web UI. Do this by reverse
proxying http://logstash.openstack.org/elasticearch/$API_REQUEST to
http://elasticsearch.openstack.org:9200/$API_REQUEST.
The only values for $API_REQUEST that will be passed through are
_aliases, _status, and _search.
Change-Id: Ib41f6d91e2e59d493218074a67155af450ec8c93
Reviewed-on: https://review.openstack.org/33316
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Logstash depends on java but the runtime wasn't being installed by
the Logstash manifest. Fix this.
Change-Id: I63f2c9831a9c9eae63f4eebd2493800dd606f8fb
Reviewed-on: https://review.openstack.org/30919
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
The new elasticsearch server needs to be able to connect back to
logstash over the elasticsearch ports. Also make kibana talk to the new
elasticsearch server.
Change-Id: I88af40c7885b263fcef5ff16210f073cd0a696a5
Reviewed-on: https://review.openstack.org/30354
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Upgrade logstash to 1.1.12. Upgrade elasticsearch to 0.20.5 (the version
corresponding to logstash 1.1.12). Go back to default elasticsearch
shard and replica counts.
Change-Id: I8e5497442e918f78677335ec29d03542c93734e5
Reviewed-on: https://review.openstack.org/29447
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* modules/logstash/manifests/elasticsearch.pp: The file name for the
default elasticsearch mapping is default-mapping.json not
default_mapping.json. Fix this path.
* modules/logstash/files/elasticsearch.mapping.json: Add a missing
comma.
Change-Id: I36bbe6e125157b97f3333f383a8949b2a6b3ef5c
Reviewed-on: https://review.openstack.org/28484
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* Don't store or index `_all`.
* Compress `_source` (which is stored but not indexed by default).
* Create a dynamic template for the string type.
* Only analyze the index of `@message` strings.
This is a very rudimentary config, and should probably be expanded
on by someone who has a deeper understanding of elasticsearch.
Change-Id: I14052c48c7bb6f05e259615a5203dc1ad87a08af
Reviewed-on: https://review.openstack.org/28216
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
* modules/logstash/files/elasticseach.default: Elasticsearch does not
handle running out of memory very gracefully. Recommended allocation is
half of the hosts memory. Give elasticsearch 4GB of the 8GB host.
* modules/logstash/manifests/elasticseach.pp: Copy new elasticsearch
defaults file into place on the host.
Change-Id: I9ed14148c901cd3fd4ec38333a722821adb77ff5
Reviewed-on: https://review.openstack.org/27870
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Add a kibana module that installs kibana and flat configuration files,
and also disables the previous boring bland web frontend.
Change-Id: I772c66f07d2bc3a88128e101074be9d4162e3f8a
Reviewed-on: https://review.openstack.org/27089
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.
Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.
Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins