Commit Graph

158 Commits

Author SHA1 Message Date
Clark Boylan 88434e0ecb Retire os-loganalyze
This project has been unmaintained for a while. CI jobs don't currently
pass. Zuulv3's dashboard has largely replaced the need for this service
as well as it renders the logs for us instead.

If there is a need for this project elsewhere it can be resurrected and
maintained by others, but for now lets retire it and move on.

Depends-On: https://review.opendev.org/753398
Change-Id: I476b14c7cd9c49270f67897206ec7ad90643703b
2020-09-22 11:39:55 -07:00
Zuul d80e6a55e0 Merge "Add support for Senlin" 2019-05-05 16:25:41 +00:00
Erik Olof Gunnar Andersson 5ec0b46df6 Add support for Senlin
Depends-On: https://review.opendev.org/#/c/615688/
Change-Id: Ief8522fedd711ee3f922c4f58c639f5e6dbb5083
2019-04-21 03:57:16 +00:00
OpenDev Sysadmins 71c5673b3b OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:32:57 +00:00
Sorin Sbarnea 4deb74f593 Move zuul job definition inside the respository
Change-Id: I0a949406ced0bf7f50643010e6d5189887d967f1
2018-11-07 18:56:59 +00:00
Sorin Sbarnea bdc06873ca Adds libmagic on plaforms missing it
Installs libmagic on platforms where is not installed by defaut,
to allow development work on them.

Change-Id: Ia9f7319c375c2dce7dd86c271c731b44a70ec036
2018-11-07 18:28:41 +00:00
Sorin Sbarnea 9a10e3a826 Assures that wrapping on PRE occurs on any kind of characters
Fixes problem where some lines, like pip requirements were not wrapped
because they didn contain any white-space.

This prevents the need to use horizontal browser scrolling, even
on huge screens.

Change-Id: Ia8e32195fc8d9ca5e171506f03099ebc03859bd1
2018-10-25 18:18:43 +01:00
James E. Blair 21c2a8234a Fix HTMLify CLI
The facility to HTMLify a file from the command line has been
broken since the severity filter was added.  This gets it working
again.

Change-Id: I3cb0894014bfbf0e376b8bb60c8da738d93aaffb
2018-07-31 16:15:18 +00:00
Tristan Cacqueray f614806d1f Add missing </head> tag
Change-Id: I094d60ed9ba19c6d5b51821acee6a0ccbaf6d3d7
2018-07-08 23:30:00 +00:00
Clark Boylan 0f40dfd634 Better supports sev checking
Previously we attempted to check supports sev by running re.search
against the full log path of the log file. This meannt that the re that
roughly said: r'/neutron-' could match job logs like this:

  http://logs.openstack.org/17/553617/16/check/neutron-tempest-plugin-dvr-multinode-scenario/42fb158/job-output.txt.gz

Unfortunately that meant that all neutron logs had to support sev and if
they didn't (which job output doesn't) then we'd serve no logs to the
user if they set a log level.

We address this by switching to matching on the basename of the path and
by switching to re.match so that we only match from the beginning of the
re.

Change-Id: Ia428dc7339f06a15c58001e99ca677cb6059186b
2018-04-17 15:33:38 -07:00
Matt Riedemann 33ee1b89b8 Handle screen-placement-api
Since Pike the placement-api logs show up under
screen-placement-api since placement-api is run
using uwsgi rather than Apache.

Change-Id: Ie5428066e6c185de2bd25e2e84823393c6b87d02
2017-12-13 10:20:58 -05:00
Zuul 5231f7c8cd Merge "Support severity filtering for dragonflow logs" 2017-12-05 20:07:06 +00:00
Shachar Snapiri 6f82fd8ec9 Support severity filtering for dragonflow logs
Adding the logs of the dragonflow project
(https://github.com/openstack/dragonflow) to the list of logs that
support filtering by severity, to allow better log analyzing for the
dragonflow project.

Change-Id: Id71c25e1987c589cdcd2fc226cfc4755c17cf7dd
2017-11-16 08:53:32 +02:00
John L. Villalovos 961f35ec3b Add clickable CRITICAL link
Previous commit 9dff1f2af9 added support
for showing loglevel CRITICAL. And now when clicking a loglevel like
ERROR you will see both ERROR and CRITICAL.

But the previous patch forget to add a clickable link for just
CRITICAL messages. This patch adds that.

Change-Id: I472db26dc9f21e0205dbfd0e21c75037d71a07b8
2017-11-15 22:13:19 -08:00
John L. Villalovos 9dff1f2af9 Support the log level "CRITICAL"
"CRITICAL" is a valid Python log level [1]

We should support it or otherwise if we select to display the "ERROR"
log level we won't see the "CRITICAL" log level messages.

[1] https://docs.python.org/2.7/library/logging.html#logging-levels

Change-Id: Icdf8792dfa605c17f6186b53032ed103b3bfdd79
2017-10-04 16:51:59 -07:00
Adam Harwell a0a4cadabd Add Octavia logs to filter regex
Add 'o-' to SUPPORTS_SEV regex to enable Octavia logs
support in filter.

Change-Id: Ie29830320fd6c4524d151b291d836aa0c6be71cc
2017-08-02 11:11:29 -07:00
Clark Boylan 0ed87b88fe Use last sev if current sev is NONE
There are examples of logs from cinder volume at least where we get
lines without severity belonging to the last line with severity.
Properly associate these lines with the last not NONE severity.

Change-Id: I154ac99fe02de7f035c36e18603496f954aac961
2017-05-31 10:13:09 -07:00
Sean Dague f7aea1b29c fix missing space in systemd logs
The regex ended up removing a space between date and host in systemd
log processing, this puts it back.

Change-Id: If0516e1187422cc76f2b3eaac82280504f94cec0
2017-04-28 07:45:19 -04:00
Sean Dague bbef755ed8 Fix systemd line parsing
There was a bug in systemd parsing with a missing space so that it was
never matching. That's fixed.

In addition, the systemd and syslog line parsing are close enough to
each other, that in our previous patterns they would match each
other's lines. The important distinction is that systemd includes
[pid] after the host which syslog doesn't.

We also have to realize that status might not be in a systemd line,
and that's ok, but still match it, given that otherwise we run the
risk of falling into the syslog match.

Change-Id: Id91c167118894c86b9b0d875a1bb36bdbe239ebd
2017-04-12 06:29:20 -04:00
Sean Dague 35819eb8d6 Add support for systemd logs
Change-Id: I3dcf42704e55401b7274af5d87f012b84d76fb73
2017-04-10 16:48:25 -04:00
Sean Dague 5b7ad194dd Remove swift support
The swift support was never moved into production for infra because of
an unsolved issue around index files, so the logs in swift approach
was ended. This takes out all the swift supporting code to drop the
complexity of the code base. As this is very intermittently maintained
tool, less code makes any future work easier to do.

Change-Id: Iabc4cffc35633fd808556ebb68a8bdd5a5d1fde0
2017-04-05 12:53:05 -04:00
Ihar Hrachyshka 9916cb3a3d Capture logs for lib/neutron services
Services supported by the new lib/neutron devstack library use neutron-
prefix not q-. Support both for the time being. Eventually we will get
rid of q- prefix.

The patch also adds neutron-l3 sample file to test suite, to make sure
we don't regress in the future.

Change-Id: I48ca66421d39a8fdd8700adf691823e20a03ab30
2017-02-24 15:46:28 +00:00
Ihar Hrachyshka 1048d8b2a4 Made test_wsgi somewhat smarter
Now it can distinguish between random strings and actual log messages
(with timestamps and all). The patch also fixes some cases where levels
were incorrectly classified depending on the order of messages parsed.
The fix triggers some renumbering in calculated dicts with number of
matching messages per level.

The test is still a bit dumb because it does not apply elaborate
classification filtering as the actual middleware does, but that's
enough for now.

The patch is needed for the next patch that will add neutron-l3 log
file.

Change-Id: I0232c416a4dbcafb6741a40eeb75a81037cbf0c8
2017-02-24 15:46:21 +00:00
Ihar Hrachyshka 43f9353878 Bump hacking to the latest version
Change-Id: I821f19fbcc8921f584cd1d2aea44cf370afece63
2017-02-24 15:39:27 +00:00
Sean Dague af0d9f04e8 add placement-api to list of understood logs
Change-Id: Ie140c62eddebf563b47dee4d0bac4798452f4000
2016-09-06 08:43:12 -04:00
Ian Wienand 85e117da7d Test high-precision timestamps in console log
Test that we detect microseconds correctly in console logs.

Change-Id: I922283891c557e489e4d75f17e78e8c31a5ca2ca
2016-06-21 10:58:45 +10:00
Jenkins 2fd29fe305 Merge "Whitelist ironic- prefix in filter.py" 2016-05-16 13:13:28 +00:00
Dmitry Tantsur c824312cdd Whitelist ironic- prefix in filter.py
While ironic itself uses ir- prefix, ironic-inspector used ironic-.
This change will allow it to have better-looking logs in gate.

Change-Id: Id692f64c7fcf2e7c70c264e9a4e6f0d38d48a311
2015-12-09 13:26:21 +01:00
Ryan Hsu c4a1cf8f46 Whitelist .log.gz files from the passthrough
Currently, files with extension .log.gz are not exempt from the
passthrough filter and will be downloaded by the browser instead
of formatted for viewing in the browser. Since some log files in
stable releases use .log for the extension, it's useful to
whitelist these files when served up for log analyzation.

Change-Id: I3d391167bbc3e3b8a4fffd368a0a16f944f548ca
2015-11-30 21:33:23 +00:00
K Jonathan Harker 5828d818ed Add support for rfc7233 range headers.
This is primarily needed by our log_processor gearman workers which currently
attempt to retry failed downloads with partial GETs.

Change-Id: Ic49579d253ea773651e4b9641223c2d2ae961c94
2015-11-11 16:05:06 -08:00
Jenkins ec9a338ddc Merge "Change ignore-errors to ignore_errors" 2015-10-11 00:18:27 +00:00
Clark Boylan 8a0619c666 Add mtime to generated indexes
It is valuable when looking for the most recent test run to know the mod
time of log files. Add this to the generated index files to provide that
information to our users.

Change-Id: I03669afd73d1f0ce14adce17a3136ce26377b403
2015-10-08 13:19:16 -07:00
Clark Boylan aab3f8b234 Report object sizes from swift when writing index
As with the reporting of file size for on disk files report the file
size for objects in swift.

Change-Id: Ife8e1a9c7b9b793aafc1e85648c24adeac3fca2a
2015-10-08 11:23:31 -07:00
Clark Boylan a9b91f6a05 Print file sizes in indexes for on disk files
File sizes are an important piece of information in log file indexes as
they help readers pick out inappropriately sized files which may
indicate the bug can be found there. Add file sizes to the index so that
we don't lose this info when compared to mod_autoindex.

Change-Id: I248a4fc9bd5ff3fa536e30bcc981a2d5f640307c
2015-10-08 11:21:51 -07:00
Clark Boylan a6fa3bfd2d Dedup index entries
When we generate indexes we query disk and swift, if any files have
overlapping paths between them we were generating duplicate index
entries for the same paths. This happened because we used lists to store
the entries which allow for duplicates.

Fix this by using sets until we need to sort (sets are unsorted so we
have to have a list at that point). This will remove any duplicates and
make the index pages more correct.

Change-Id: I6dfa3b30819d6633c3e483d3a386bdce3e26b572
2015-10-07 11:07:09 -07:00
Joshua Hesketh 082d0b942a Skip whitespace for html view
In some cases (such as the new coverage report) there is leading
whitespace before the <html> tag. Skip these lines when trying to
detect if we should escape any content.

This makes os-loganlayze lossy whereby it'll ignore leading whitespace
in html files.

Change-Id: Ie7b183dfa78f58f0b39339ba41850393b248b872
2015-09-30 23:28:09 +10:00
Robert Collins e185cd0dd3 Update requirements
The key thing I wanted to update was the stale pbr req, but I figured
I'd just run update-requirements and take what it made :).

Change-Id: I65e2556a19d81defb4ee6ceb31a0bec6635efe55
2015-09-30 14:36:16 +13:00
Monty Taylor b9a5056847 Change ignore-errors to ignore_errors
Needed for coverage 4.0

Change-Id: I56b03a195f6282fa5c43122301cd1c2d79b29320
2015-09-21 14:23:34 +00:00
Joshua Hesketh e1079394e3 Add support for listing folder indexes
If a file isn't found, instead list an index with everything within
that folder from both disk and swift.

Change-Id: I60ef6c625904de6af098df1906598ef107f74796
2015-09-02 23:25:33 +10:00
Jenkins f6bd80673b Merge "Fix comment on way indexes are loaded" 2015-08-21 17:26:25 +00:00
Clark Boylan a6b2565e93 Only provide html if client supports it
We can specify via config that a file should be served as html but we
should still only do so if the client supports html. Otherwise fallback
on text. This allows us to have a default for log files that is rendered
html in a browser but text when wgetted or viewed in a text editor.

This corrects an issue where we are attempting to index a bunch of html
into elasticsearch because we always get html for the log files even
though the logstash http client doesn't support html.

Change-Id: I35d78498034a9303bd940f57ac34f423673448db
2015-08-19 09:09:28 -07:00
Joshua Hesketh 3e59f98a01 Fix comment on way indexes are loaded
Change-Id: If7b5abe616b6132d9eea7c6918af383ef62589b3
2015-08-04 13:10:48 +10:00
Joshua Hesketh 6d7e65df3d Surpress 404 errors from swift
Because we try swift and then fall back to disk or alternative
path names we get a lot of 404's when looking for a log. These
are currently filling up apache's error.log from our catch-all
exception handler.

We still want to catch all exceptions from attempting swift (as
these will range from 403's through to the swiftclient not existing)
but we want to ignore errors from 404's as they are handled
separately.

Change-Id: I7c03685dfde12a3c09f9a2a9caca811281779875
2015-07-18 13:07:45 +10:00
Jenkins 03f726cb73 Merge "Add support for setting filters and views per file" 2015-07-18 02:44:03 +00:00
Jenkins fb4add1c28 Merge "Allow the config to set the filter and view" 2015-07-18 02:40:30 +00:00
Jenkins 9350510329 Merge "Reapply tidy up generators into contained objects" 2015-07-18 02:29:39 +00:00
Jenkins 9e40b05024 Merge "Add Manila logs to filter regex" 2015-07-15 10:37:52 +00:00
Joshua Hesketh 2dac23f208 Add support for setting filters and views per file
Match filters and views to files based off their filename. If no
match for a file is found the default set in wsgi.conf will be used
or else a best (legacy) guess is made.

Change-Id: Idac2849b52471acb06eb43b64edb1762a74c6fe1
2015-07-15 18:41:39 +10:00
Joshua Hesketh d4ac63b711 Allow the config to set the filter and view
Currently os-loganalyze will detect what filter and view to use based
off the file type, name and other headers. Instead, allow the config
to define exactly what view to use. If none is set then the legacy
detection will still be applied.

Next change will allow the config to define filters and views based
off file match conditions.

Change-Id: I8955577c100b13ce20609426025a68fbbd052423
2015-07-15 18:41:39 +10:00
Joshua Hesketh 0a4c50260d Reapply tidy up generators into contained objects
Neaten up the generators by returning an iterable object that
has the headers as attributes.

Change-Id: I8a07c7c862e5a72cdd347a80da9b789f437e4d2d
2015-07-15 18:41:37 +10:00