This test adds testing for var log messages format. In this commit we
also enhance the functional.sh script to:
1) make sed usage more compatible with other systems (--expression -> -e)
2) Add color to output PASSED / FAILED
3) Generate template files for non existing outputs
4) Generate failed outputs for manual verification
Change-Id: I1146b96fe9c91473c0127d0c47f890db1ddfb7b5
The funcional testing will include, per case, input files
and output files, plus a simple bash script which makes
sure the output stays as it should, and otherwise shows
a diff and fails the case.
Change-Id: I4c5f1143188f49c6a1448ca8918ab8b30710b865
Adds py37 to the env list and switches pep8 python to
python3, including necessary changes to make pylint
and pep8 checks happy in python3:
* Unpins pylint, as 1.4.5 explodes (exception) when trying
to check our code.
* Unpins flake8-import-order from an old version.
* Stops making use of the builtin cmp function, and replaces
it with the recommended (a > b) - (a < b).
* Disables one py3 specific warning, since it's still necessary
for python2 (useless-object-inheritance)
* Removes a non existing argument in netprobe parser (version).
* Inserts an additional blank line between __future__ imports
and stdlib imports.
Change-Id: If4a45108fe93080f9967e2ac26d681b8dd5c7fc2
The purpose of this change is to help the licensecheck tool
identifying the license of the sourcecode.
Change-Id: I3cc325e825fefe3c124329d8ab4b10821c9b1676
One of the reasons that pbr exists is to have versions be driven by
signed git tags instead of by content in the source repository. This has
the benefit of being both easier and preventing people from forgetting
to push a git tag when they cut a release.
Additionally, this code is doesn't work for sdist jobs in Zuul upstream
because it requires the project to be importable to run setup.py, but
sdist/wheel generation does not otherwise need to install dependencies.
Remove the override of version in setup.py and replace the hard-coded
version string in oslogmerger.py with the appropriate pbr code.
Change-Id: Ifdbb649b02d7e2fde6e8682e138c81817543fb08
Now the code is python2 and python3 compatible, and they both
generate the same output regardless of subtle differences.
Change-Id: Ic6b908387becf4d0de5d9da72ce6915d3f7c3b1c
This is a simplification of the previous code which used global state.
It was prompted by a bug: iterating over a log file containing 3
lines, we were returning lines 1 and 3, and omitting line 2.
Change-Id: Ic7388f96a6201f30e16662ff4a5f3c081ac2c4bd
TSLogParser was searching for any instance of [<number>] in a log
line, which was incorrectly matching on, eg:
Sep 15 18:09:46 clgrabguye21.localdomain su[160015]: ...
With this change we assert that the timestamp occurs at the beginning
of the line.
Change-Id: I04da2f1fa661bccfd3b046dfc22e5a10f6cb37f8
Add a new option --os-logs/-ol to explicitly specify OpenStack format
logs. Logs whose format is not given explicitly are auto-detected.
Change-Id: Ie97127f24f34ab386138db28b626abf4c259ac24
All log entries are now the lighter weight LogEntry class, which is
also responsible for its own output. Parsing is simplified and
separated into independent parser classes.
Change-Id: I264cf20933e8af007556efd7a36639f854460f49
heapq.merge does an extremely efficient merge sort. Being a merge sort
it has constant memory overhead, and is very fast. It also starts
producing output immediately. It obsoletes both previous sorting
methods, so we deprecate the --min-memory option.
Change-Id: I8384b7214ba54dffa61d1c2195f3b4c238ba253a
You never know what's in the input string, so catch all possible
failures without crashing the tool.
Related-Bug: #1672921
Change-Id: I3a518be2e6087d0a096fa3c36aa40dcad8aecb98
Without that patch, the tool chokes on lines with null-bytes in the
middle, which as it turned out sometimes happens in neutron logs.
Related-Bug: #1672921
Change-Id: I2ec2c637dc03008f4d964f88a788af0d69abe282
This is an standalone .py file that can be uploaded to remote
servers and used to log network packets over network interfaces.
It will discover new interfaces automatically, in the root or
inside network namespaces.
Interfaces and namespaces are filtered via regular expressions.
usage: netprobe [-h] [-v] [--netns-re NETNS_REGEX]
[--netdev-re NETDEV_REGEX]
[--tcpdump-filter TCPDUMP_FILTER]
[--check-interval CHECK_INTERVAL]
This tool will track system network devices as they appear in a host,
and start tcpdump processes for each of them, while the output of all
the tcpdumps goes in a single openstack-like log.
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
--netns-re NETNS_REGEX, -n NETNS_REGEX
--netdev-re NETDEV_REGEX, -d NETDEV_REGEX
--tcpdump-filter TCPDUMP_FILTER, -t TCPDUMP_FILTER
--check-interval CHECK_INTERVAL, -i CHECK_INTERVAL
The interval between interface checks
This patch adds support for timestamped log files with the format
"[ 0.003036]".
Since timestamp many times will not take epoc time as the source of the
timestamp but the time the system started, the initial datetime will be
calculated by substracting from the file modified datetime the last
timestamp in the file.
Option added is --timestamp-logs or -tl, and it also supports ALIAS and
globs, and is also affected by base directory and postfix options.
This patch adds support for log files that use the same datetime format
as /var/log/messages, which is in the for of Oct 15 14:11:19.
Year will from last modification to the file will be used to complete
missing year.
In the process of refactoring the read of the logs the OpenStack log
reading has been simplified to not require the presence of PID or LEVEL,
thus providing more compatibility with custom log formats (as long as
default datetime format is being used).
With this refactoring it is now easier to add new datetime formats to
the script.
Option added is --msg-logs or -ml and it also supports ALIAS and globs,
and is also affected by base directory and postfix options.
TODO: If year of file creation and file last modification are different
we should start with the cration year and then change to the next year
once the months go back.
When using an absolute path an auto alias generation level 3 we would
get IndexError: pop from empty list.
This patch fixes this issue by removing the empty string generated from
splitting the path by "/" and adding a lead "/" to the first directory.
During auto alias generation with levels 2 and 3, when we are removing
unnecessary directories we were not taking into consideration existing
files in parent directory, so we could end up with duplicated aliases.
For example if we had cinder/vol.log and vol.log we would end up with 2
[vol] aliases.
This patch fixes an error with auto alias generation levels 2 and 3 were
we would get a KeyError exception on certain cases.
For example with `os-log-merger -a2 2/cinder/vol.log 2/vol.log` we would
be getting KeyError: 'cinder'.
The reason for this was that we were overwriting subtrees so when we
added 2/vol.log we would be destroying the subtree generated when we
added 2/cinder/vol.log.
When one of the log files is empty and we use `-m` option we will get no
output, unlike normal operation that would output contents of all other
files correctly.
This patch fixes this by removing from the files to read empty ones like
we do once we start processing the files.
Current code would modify the datetime format on the output file adding
000 to the microseconds count, so an input of "2016-03-07 12:15:27.805"
would become "2016-03-07 12:15:27.805000".
This patch changes the way we output the datetime, and instead of using
the datetime object used for sorting we will output the original
datetime string.
This will probably be useful later when we support custom log formats or
other log sources.
This patch adds an option to reduce program's memory footprint during
runtime and updates README file accordingly.
Instead of loading all contents into memory and then sorting the
entries, it will go entry by entry saving the oldest one.
Memory reduced footprint operation increases time to merge logs by
approximately 25%.
Option is `-m` or `--min-memory`.
This patch adds optional alias generation from filenames.
There are 4 generation levels:
- Level 0: means disabled and will return the full file.
- Level 1: use filename without base or postfix
- Level 2: like 1, change filenames, reduce directories.
- Level 3: like 2, plus reduce directory names
If an alias is given auto alias generation will be skipped for that
file.
Miguel Angel Ajo