two policies are added to handle short and long term indices.
life of the indices can be configured using the 'elasticsearch.life'
field at browbeat-config.yml file
shortterm will have: 125 days
longterm will have 2 years
the policy and the policy based templates can be created using the
'es-template' install playbook.
Change-Id: I0f4a4a9acc03092fd582ae4ff50f688850def953
Elasticsearch5 changed some of the template structure. We also need to
update the number of fields avaialble.
Change-Id: I7de9fe8e41cb8663467249b1bcad603a214d1624
Right now Rsyslog->ES will log like:
logstash-YYYY.MM.DD.HH
This creates a ton of indexes making search much harder. This commit
removes the .HH so we have:
logstash-YYYY.MM.DD
This also patches the template to work with ES5
Change-Id: Iaef70f51601da504b769165e8593603adc484225
This implements rsyslog -> elasticsearch logging as well
as rsyslog forwarder -> rsyslog aggregator -> elasticsearch logging
using the common logging template as a base and adding
in dynamic detection of containerized services and log path
detection.
Services can be moved into and out of containers and add
or remove log files and the log detector script will create a template
that reflects these changes dynamically.
Logging inherits cloud name and elasticsearch info from the existing
group_vars variables, so this should be no additional work to setup
beyond setting logging_backend: rsyslog and either running the install
playbook or the rsyslog-logging playbook.
Finally additional variables can be passed into the deployment with
-e or just being in the ansible namespace, this way things like a
unique build ID can be templated into the logs automatically. I've
added support for browbeat_uuid, dlrn_hash, and rhos_puddle others
should be trivial to add.
There are also additional tunables to configure if logging instaces
should be standalone (viable for small clouds) or rely on a server
side aggregator service (more efficient for large deployments).
Disk backed mode is another tunable that will create a variable
disk load that may be undesierable in some deployments, but if
collecting every last log is important it can be turned on creating
a one or two layer queueing structure in case of Elasticsearch downtime
or overload depending on if the aggregation server is in use.
If you want to see examples from both containerized and non
container clouds check out elk.browbeatproject.org's logstash
index.
Change-Id: I3e6652223a08ab8a716a40b7a0e21b7fcea6c000
This patch enhances the already existing template in order to not
tokenize some fields that are used in visualizations.
+ Verified
Change-Id: Idd5a791a999de5d8dee0ec50694e38d0fcb47469
This work will help Elastic/Kibana from processing our UUID's into
multiple strings. Before this work, Kibana would display parts of our
UUID since they contained '-'.
If the user installs ELK from Browbeat they will have these templates
installed for them. If the user wants to install these templates into a
existing Elasticsearch they can with the es-template.yml
Change-Id: I04165c50dfac9cbd1f05b4bcd2ac78fbdb4da0a2
Masco