Commit Graph

20 Commits

Author SHA1 Message Date
Clark Boylan 4279e20293 Remove configuration management for ELK stack
We indicated to the OpenStack TC that this service would be going away
after the Yoga cycle if no one stepped up to start maintaining it. That
help didn't arrive in the form of OpenDev assistance (there is effort
to use OpenSearch external to OpenDev) and Yoga has released. This means
we are now clear to retire and shutdown this service.

This change attempts to remove our configuration management for these
services so that we can shutdown the servers afterwards. It was a good
run. Sad to see it go but it wasn't sustainable anymore.

Note a follow-up will clean up elastic-recheck which runs on the status
server.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/837619
Change-Id: I5f7f73affe7b97c74680d182e68eb4bfebbe23e1
2022-04-18 10:04:06 -07:00
Clark Boylan bd752a0bfe Keep only 7 days of records in ElasticSearch
We have been running out of disk recently with some indexes requiring
more than 400GB of space per index replica. Actual disk space
requirements are double that as we run with a replica. On top of that
the idea is that 5 of 6 elasticsearch nodes have enough space for all
our data so that we are resilient to losing a node.

Napkin math:

  400 * 10 * 2 = ~8TB of disk
  400 * 7 * 2 = ~5.6TB of disk

Each of the six ES nodes has 1TB of disk allocated to ES so 5.6TB should
get us just under the limit. Then for handling a node outage weekends
tend to not have as many records so our actual usage should be a little
lower.

Change-Id: Ie677bd47a9886870bc83876d2407742133299861
2020-02-06 13:50:56 -08:00
Clark Boylan 1b6d4e30d6 Revert "Reduce elasticsearch retention to 7 days"
This reverts commit 42e159434d.

Change-Id: Iad1f9dff3e986a7f50fdb735f60366d5f501f5c6
2017-11-28 09:10:58 -08:00
Clark Boylan 42e159434d Reduce elasticsearch retention to 7 days
We were keeping 10 days of logs in elasticsearch but now that we've
caught up on the job backlog that has filled the disks on our
elasticsearch cluster. Address this by reducing log retention by 30%.

Change-Id: I8b1e126e08eb4745d01534a63c558bf1494ac6bf
2017-11-22 08:31:02 -08:00
Clark Boylan a57925c9ed Upgrade elasticsearch to 1.7.6
This can be done as a rolling upgrade as we are currently running 1.7.5
which shares a majory version with 1.7.6.

Change-Id: Icd759967dc8a6a09d4db0a78d2dfef72042c171e
2017-01-03 14:46:54 -08:00
Clark Boylan 6fcc8d3f26 Upgrade elasticsearch to 1.7.5
There have been a couple bug fix release between latest 1.7.x and 1.7.3
which we are running. Pull in this fixes in hopes that it makes shard
allocation more reliable.

Change-Id: I48f46663c73cb178ca3cf95a166b3df2ea121459
2016-05-04 12:06:58 -07:00
Jenkins af13b1960e Merge "Switch from a cron to curator" 2015-11-16 21:30:50 +00:00
K Jonathan Harker 1d1b206620 Switch from a cron to curator
Instead of running curl in a cron, switch to the logstash::curator class
that runs a slightly more intelligent script with logging in /var/log.

Change-Id: Iba3c0726b789dcbcf58f2da071e6fa93d4cd67ae
Depends-On: Ic49c8649a0ad849a8d72900425e589e40b82a37f
2015-11-16 12:05:19 -08:00
Clark Boylan 285e90e16d Set indices.breaker.fielddata.limit to 70% on ES
We had to bump indices.breaker.fielddata.limit to 70% against the
running cluster in order for the elastic-recheck queries to run without
erroring due to not enough memory for the @timestamp field. Add this to
the config so that it persists across cluster installs.

Change-Id: Ia2f9c2ffff166bf3cc8f32c90b230249b3392406
Depends-On: I46c0cb5157aae40a0029ff1b425ecc663d171768
2015-11-13 12:08:14 -08:00
Clark Boylan 723a574ef8 Allow CORS against elasticsearch API
Since we do fine grained access control to the API via a proxy that
limits what requests can be made from anywhere go ahead and allow CORS
requests from anywhere as they won't be able to write data anyways.

This is useful so that you can test a new version of $tool hosted
locally against the actual cluster or for admins to run admin tools
hosted locally against the cluster.

Change-Id: I774d0ad0b246315794ab387acc39c41c7cfac3cd
Depends-On: I0aa8d5167c770c1024b7596da582d6cc089b1b47
2015-11-12 11:24:38 -08:00
Clark Boylan 11b297faf2 Update ES to 1.7.3
The actual upgrade is handled by ansible but this will enforce the
correct version for any new host builds. This should be merged after the
ansible managed upgrade is complete.

Change-Id: Ieda9cdffe7486d24f17012de5fc094d18a45ec78
2015-11-09 09:45:39 -08:00
Jenkins 2966b2e221 Merge "Move out server class from elasticsearch_node.pp" 2015-10-24 23:46:09 +00:00
Mikhail S Medvedev 053b4c657b Move out server class from elasticsearch_node.pp
Change-Id: Ic49e815e7346280de31891eb11faccb7cad3adad
Story: 2000172
Spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/server_base_template_refactor.htm
2015-07-09 15:36:44 -05:00
K Jonathan Harker 78cde77666 Remove unused cron
Change-Id: Ife90f2210f3cdc1621e55251957a986a6d198941
2015-06-29 11:45:19 -07:00
Clark Boylan 411844b50a Shift elasticsearch retention to 10 days.
We were keeping 14 days of elasticsearch indexes. Unfortunately we are
running out of available disk space on our elasticsearch nodes and do
not have quota to add more volumes to increase available disk. Shift to
10 days of retention while we sort out how to get more disk.

Change-Id: I011d4f16f9a74e2eb7665ef66d2193e5fda476b7
2014-06-16 13:18:11 -07:00
Jeremy Stanley cde1a4abee Create new 30g heap elasticsearch workers
Temporarily leave the old node names in place with an override to
16g heap size, but pass the openstack_project default of 30g into
the elasticsearch module as a new variable. Also don't actually
configure the new servers to install and run elasticsearch yet,
since there are manual steps needed to prep the additional block
devices on them.

Change-Id: I9fca2115c8996ae2f1a6c605bcdf16a8b6342a24
2014-02-25 00:04:37 +00:00
Clark Boylan 5a8351635d Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9
Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9. Logstash 1.3.3 comes
with many performance improvements and bug fixes and needs ElasticSearch
0.90.9 to use the ElasticSearch output so upgrade ES as well. These
upgrades come with config changes for logstash as well to use new
features and disable others that are not wanted.

Change-Id: Ia180c5edfa755214a801f2cad06f091125814b0e
2014-02-11 17:07:34 -08:00
Ryan Lane 712349aae7 Configure elasticsearch via a hash
Change-Id: I70067e371d930dcecab6e85399465b4c3bf67ab2
2013-10-21 18:48:29 -07:00
Clark Boylan 0423b59c6c Upgrade elasticsearch to version 0.90.3.
* modules/openstack_project/manifests/elasticsearch.pp: Upgrade
elasticsearch to 0.90.3 to support logstash 1.2.1. This version is
required to use the elasticsearch output in logstash 1.2.1.

* modules/elasticsearch/files/elasticsearch.mapping.json: Add setting
for new "message" field in logstash 1.2.1's new schema.

Change-Id: Ice1a8613ea11aeb954c600752a2de745dd56ef4b
2013-10-21 10:06:28 -07:00
Ryan Lane 2f4c686edc Rename openstack_project::elasticsearch
To avoid namespace issues, rename openstack_project::elasticsearch
to openstack_project::elasticsearch_node.

Change-Id: Icd215cd0c13f06d279a477d0885d33665ce125ae
2013-10-18 16:22:13 -07:00