History

jkilpatr bb44cd830c Rsyslog -> Elasticsearch logging This implements rsyslog -> elasticsearch logging as well as rsyslog forwarder -> rsyslog aggregator -> elasticsearch logging using the common logging template as a base and adding in dynamic detection of containerized services and log path detection. Services can be moved into and out of containers and add or remove log files and the log detector script will create a template that reflects these changes dynamically. Logging inherits cloud name and elasticsearch info from the existing group_vars variables, so this should be no additional work to setup beyond setting logging_backend: rsyslog and either running the install playbook or the rsyslog-logging playbook. Finally additional variables can be passed into the deployment with -e or just being in the ansible namespace, this way things like a unique build ID can be templated into the logs automatically. I've added support for browbeat_uuid, dlrn_hash, and rhos_puddle others should be trivial to add. There are also additional tunables to configure if logging instaces should be standalone (viable for small clouds) or rely on a server side aggregator service (more efficient for large deployments). Disk backed mode is another tunable that will create a variable disk load that may be undesierable in some deployments, but if collecting every last log is important it can be turned on creating a one or two layer queueing structure in case of Elasticsearch downtime or overload depending on if the aggregation server is in use. If you want to see examples from both containerized and non container clouds check out elk.browbeatproject.org's logstash index. Change-Id: I3e6652223a08ab8a716a40b7a0e21b7fcea6c000	2017-10-16 12:08:26 +00:00
..
templates	Rsyslog -> Elasticsearch logging	2017-10-16 12:08:26 +00:00
README.rst	Add template for Elastic	2016-10-07 16:01:12 -04:00

jkilpatr bb44cd830c Rsyslog -> Elasticsearch logging

This implements rsyslog -> elasticsearch logging as well
as rsyslog forwarder -> rsyslog aggregator -> elasticsearch logging
using the common logging template as a base and adding
in dynamic detection of containerized services and log path
detection.

Services can be moved into and out of containers and add
or remove log files and the log detector script will create a template
that reflects these changes dynamically.

Logging inherits cloud name and elasticsearch info from the existing
group_vars variables, so this should be no additional work to setup
beyond setting logging_backend: rsyslog and either running the install
playbook or the rsyslog-logging playbook.

Finally additional variables can be passed into the deployment with
-e or just being in the ansible namespace, this way things like a
unique build ID can be templated into the logs automatically. I've
added support for browbeat_uuid, dlrn_hash, and rhos_puddle others
should be trivial to add.

There are also additional tunables to configure if logging instaces
should be standalone (viable for small clouds) or rely on a server
side aggregator service (more efficient for large deployments).
Disk backed mode is another tunable that will create a variable
disk load that may be undesierable in some deployments, but if
collecting every last log is important it can be turned on creating
a one or two layer queueing structure in case of Elasticsearch downtime
or overload depending on if the aggregation server is in use.

If you want to see examples from both containerized and non
container clouds check out elk.browbeatproject.org's logstash
index.

Change-Id: I3e6652223a08ab8a716a40b7a0e21b7fcea6c000

2017-10-16 12:08:26 +00:00

templates

Rsyslog -> Elasticsearch logging

2017-10-16 12:08:26 +00:00

README.rst

Add template for Elastic

2016-10-07 16:01:12 -04:00

README.rst

ElasticSearch Configuration for Browbeat

+ templates/

Will contain Elasticsearch templates to account for things like the Browbeat UUID. These Templates will be installed if you run through our Elasticsearch installer. If you already have a Elasticsearch Host, you can install them by running the following:

$ cd ../ansible
$ vi install/group_vars/all.yml
# Update your es_ip
$ ansible-playbook -i hosts install/es-template.yml