There shouldn't be a need to do this, as the contents are generated
directly from elasticsearch.
Change-Id: I99ec3cfcc146477ee011f6dd926191c65495e90f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Since the beaker jobs are being run on xenial, we need a special nodeset
for it, otherwise beaker gives an error:
beaker-hostgenerator was not able to use this value as input.
Exiting with an Error.
We also want to install puppet from the Ubuntu repos rather than from
puppetlabs, since puppetlabs doesn't support puppet 3 for Xenial. For
centos we can keep the install process the same.
Additionally, this patch updates params.pp to distinguish between JRE
packages based on the Ubuntu release so that these module will work on
Xenial. We still keep support for trusty.
Finally, remove spec assertions that are not useful, including
verifications that packages are installed and config files existing. It
is enough to assume that if the puppet ran without errors that it was
able to also successfully find and install packages and lay down config
files. We remove these tests instead of updating the package check for
the new JRE package.
Change-Id: Ifd2244ae9dd212b2475f9cd6adb994bc058a4769
Depends-On: I053d437ceb7895fe2b1e2c46bf9cf19f73bdb20c
It is possible when we move volumes from one server to another that
our UID permissions are not currect. Have puppet ensure they are setup
properly.
Change-Id: I8f15b60507d1d52235d2db49753df144bea046fe
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Elasticsearch now hosts at elastic.co. The old name still functions but
update this before upstream can break us.
Change-Id: Ia39471881f2946eaab6e37dea8a19ed5d36d0662
Order and intendation of those parameters are changed
to follow Puppet Style Guide recommendation [0].
Moreover, it will allow to an user to find much faster
a variable in a list of variables.
[0]. https://docs.puppetlabs.com/guides/style_guide.html
Change-Id: Ie516f4a5aafc724d8f290c6d12f5ee2f6fe99a94
Keep elasticsearch from fillings its disk with logs by deleting logs
older than two weeks. Note we do not use logrotate because elasticsearch
expects to be doing the log rotation itself, it just doesn't delete
older logs. Instead we setup a cron with find to delete the older logs.
Change-Id: Ia2611cbf6e4aa1945ec8b3914d7ba042d2a7b0c1
Set up small non master non data elasticsearch daemons on logstash
workers to act as local load balancers for the elasticsearch http
protocol.
Change-Id: Ie3729f851ebef3331a6b69f718e57d663209bfc2
The resource has _deb in it, which was left out.
Also, the java package name resource was replaced in only
one of two locations.
Change-Id: Ia47beeeff80f2cba1e7dfff0538156bd6464b9a1
java7-runtime-headless is an empty package that can never be
installed, so it attempts to install every time. Replace it with
openjdk-7-jre-headless, which is the real package.
Also, add a refreshonly on the exec for checking the diff, because
it runs every time as well.
Change-Id: I5195d27ae90cd7f9f3e20d92aca43d7b8d0e504a
We rely on curl to query the elasticsearch api for cluster state,
deleting indexes and so on. Ensure it is installed on the elasticsearch
nodes.
Change-Id: I843c6be65e5750856cb0b68f68eb7fac47e04e29
The package maintainer scripts start elasticsearch when it gets
installed, but it seems to return control before the pidfile is
created which allows puppet to race for it and ultimately launch a
second daemon resulting in an OOM condition. It's configured to
start at boot anyway, so the puppet ensure running isn't buying us
much.
Change-Id: I1b374f90c198b9cb73d412fe63b439623f434d9a
* modules/elasticsearch/manifests/init.pp: Brown bag fix... the
package installation used to make this directory, but now we want to
put files in it prior to package installation.
Change-Id: Id3069999d59f965226bcae8f21ef7a93e4d44c72
* modules/elasticsearch/manifests/init.pp: The config files need to
be in place before package installation starts the service, and they
don't depend on the package itself anyway.
Change-Id: Iabc57e3091533eb5168c670f5c29d0d2dac9a1a2
wiki.o.o is much smaller than our typical elasticsearch nodes. For it to
use elasticsearch it needs to allocate a much smaller heap. Make the
heap size configurable (keep the 16g default) and set the heap size to
1g for wiki.o.o.
Change-Id: I0c5fb4865eb07c0fa5bbe4cf698a9f0e9ea9f2fc
To allow for reuse of elasticsearch I'm splitting it away from
logstash. Also, I'm doing a hash check of the elasticsearch wget
for better security.
Change-Id: Iff42d538cd941abd50b000879ea4a237ea48d40e
* modules/logstash/templates/elasticsearch.yml.erb:
Give 33% of memory to indexing instead of 40%.
* modules/logstash/files/es-logstash-template.json:
Flush the translog every 50k operations instead of the default 5k
(we can do 5k ops every second or two so the flushing was happening too
often). Limit the total number of shards per index per node to 3. We
have 5 nodes and 10 shards per index, 5*3 > 10 so we should be fine.
Set the field cache to soft type so that its entries can be garbage
collected in memory constricted situations.
* modules/logstash/manifests/elasticsearch.pp:
Apply es-logstash-template.json to the elasticsearch servers.
Change-Id: I2337fc41998fd00e090b0acfd29f007dfb6ec8df
* manifests/site.pp: List the two new elasticsearch nodes in the
appropriate lists.
* modules/logstash/manifests/elasticsearch.pp: Do not restart
elasticsearch when config files change. Service restarts are costly and
should be manually performed when necessary. Otherwise puppet should
simply update the config files.
* modules/logstash/templates/elasticsearch.yml.erb: Update elasticsearch
config with new cluster topology. Increase memory available for
indexing.
* modules/openstack_project/manifests/cacti.pp: Add new nodes to cacti
monitoring list.
Adding two more elasticsearch nodes to relieve memory pressure (more
nodes means fewer indexes per nodes which requires less memory to
manage). And two more nodes gives us more disk to retain older indexes
in. These new nodes should allow us to retain at least 3 weeks of
indexed logs.
Change-Id: I3a5a02311e939c8147e401110c7b96d085eb8274
Reviewed-on: https://review.openstack.org/36305
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Use mutliple discover nodes to determine elasticsearch cluster
membership. Put a timeout on recovery starting instead of the default
to recovery immediately. Describe cluster topology in elasticsearch yaml
config so that it can make smarter decisions. Round robin kibana
requests across each discover node.
Change-Id: I08ef9dd158ddf6a6ce01dfb2050626f543d45b10
Reviewed-on: https://review.openstack.org/34106
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
We need to expand our elasticsearch install base. Update puppet to make
this possible.
Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Upgrade logstash to 1.1.12. Upgrade elasticsearch to 0.20.5 (the version
corresponding to logstash 1.1.12). Go back to default elasticsearch
shard and replica counts.
Change-Id: I8e5497442e918f78677335ec29d03542c93734e5
Reviewed-on: https://review.openstack.org/29447
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* modules/logstash/manifests/elasticsearch.pp: The file name for the
default elasticsearch mapping is default-mapping.json not
default_mapping.json. Fix this path.
* modules/logstash/files/elasticsearch.mapping.json: Add a missing
comma.
Change-Id: I36bbe6e125157b97f3333f383a8949b2a6b3ef5c
Reviewed-on: https://review.openstack.org/28484
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* Don't store or index `_all`.
* Compress `_source` (which is stored but not indexed by default).
* Create a dynamic template for the string type.
* Only analyze the index of `@message` strings.
This is a very rudimentary config, and should probably be expanded
on by someone who has a deeper understanding of elasticsearch.
Change-Id: I14052c48c7bb6f05e259615a5203dc1ad87a08af
Reviewed-on: https://review.openstack.org/28216
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
* modules/logstash/files/elasticseach.default: Elasticsearch does not
handle running out of memory very gracefully. Recommended allocation is
half of the hosts memory. Give elasticsearch 4GB of the 8GB host.
* modules/logstash/manifests/elasticseach.pp: Copy new elasticsearch
defaults file into place on the host.
Change-Id: I9ed14148c901cd3fd4ec38333a722821adb77ff5
Reviewed-on: https://review.openstack.org/27870
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.
Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.
Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Clark Boylan