openstack
/
os-loganalyze
Code Issues Proposed changes
RETIRED, log analyzer tools for logs.openstack.org
117 Commits 1 Branch 1 Tag 6.2 MiB
Go to file
Clark Boylan a6b2565e93 Only provide html if client supports it 
We can specify via config that a file should be served as html but we
should still only do so if the client supports html. Otherwise fallback
on text. This allows us to have a default for log files that is rendered
html in a browser but text when wgetted or viewed in a text editor.

This corrects an issue where we are attempting to index a bunch of html
into elasticsearch because we always get html for the log files even
though the logstash http client doesn't support html.

Change-Id: I35d78498034a9303bd940f57ac34f423673448db
 		2015-08-19 09:09:28 -07:00
apache update apache sample file to have the kind of stanza we want 2013-10-12 07:04:02 -04:00
devstack Set DocumentRoot for devstack plugin 2015-07-07 21:10:46 +10:00
doc/source cleaning up index.rst file 2014-06-03 07:30:52 +02:00
etc/os_loganalyze Add support for setting filters and views per file 2015-07-15 18:41:39 +10:00
os_loganalyze Only provide html if client supports it 2015-08-19 09:09:28 -07:00
.coveragerc initial cookiecutter import 2013-10-11 10:03:03 -04:00
.gitignore Add ignored Editors to .gitignore 2014-11-12 07:51:59 -08:00
.gitreview fix pep8 in test 2013-10-12 06:17:54 -04:00
.mailmap initial cookiecutter import 2013-10-11 10:03:03 -04:00
.testr.conf initial cookiecutter import 2013-10-11 10:03:03 -04:00
CONTRIBUTING.rst Workflow documentation is now in infra-manual 2014-12-05 12:03:56 -08:00
LICENSE initial cookiecutter import 2013-10-11 10:03:03 -04:00
MANIFEST.in initial cookiecutter import 2013-10-11 10:03:03 -04:00
README.rst Add support for non-text files 2015-04-13 12:17:10 +10:00
babel.cfg initial cookiecutter import 2013-10-11 10:03:03 -04:00
openstack-common.conf initial cookiecutter import 2013-10-11 10:03:03 -04:00
requirements.txt Add support for setting filters and views per file 2015-07-15 18:41:39 +10:00
setup.cfg Provide a script that starts a simple WSGI server 2013-12-17 14:15:33 +00:00
setup.py fix pep8 errors from cookiecutter 2013-10-11 10:04:09 -04:00
test-requirements.txt Bump hacking to 0.9.x series 2014-07-21 10:24:21 +02:00
tox.ini Bump hacking to 0.9.x series 2014-07-21 10:24:21 +02:00

README.rst

os_loganalyze

OpenStack tools for gate log analysis

os_loganalyze is designed as a lightweight wsgi filter for openstack logs, making it easier to interact with them on OpenStack's logs.openstack.org repository. This includes colorizing the logs based on log level severity, having bookmarkable links to timestamps in the logs for easy reference, and being able to filter by log level.

This is implemented as a low level wsgi application which returns a generator so that it can act like a pipeline. Some of our logs are 35 MB uncompressed, so if we used a more advanced framework that required we load the entire data stream into memory, the user response would be very poor. As a pipeline and generator the delay added by this script to the user grabbing the logs is largely not noticeable (< 1s).

  • Free software: Apache 2.0 license

Features

  • Supports text/html or text/plain dynamically based on content negotiation
  • html highlighting based on severity
  • filtering based on severity using the level=XXXX parameter (works in either text/html or text/plain responses
  • linking and highlighting of lines based on timestamp
  • control of max number of lines that will be returned using the limit=XXXX parameter
  • Support for specifying which backend source will be used. As we support filesystems and swift containers simultaneously, you can specify source=swift to only use the swift container backend.
  • Provides a script named htmlify_server.py that serves htmlified logs over HTTP. To view devstack logs: set SCREEN_LOGDIR=$DEST/logs/screen and LOG_COLOR=false in localrc before running stack.sh, run htmlify_server.py, and point your browser at http://devstack-ip:8000/

Todo

Next steps, roughly in order

  • provide links to logstash for request streams (link well know request ids to logstash queries for them)

Hacking

If you are working on making changes one of the easiest ways to do this is to run the server stack locally to see how your changes look on same data included for the tests.

This can be done with tox -e run, which will use the script designed for devstack locally pointed at the sample data. A url where you can browse the resultant content will be provided on the command line.