Commit Graph

86 Commits

Author SHA1 Message Date
OpenDev Sysadmins 13f7d4c542 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:51:18 +00:00
Miguel Angel Ajo 6b87af03cd Add functional test for var log messages and enhance functional script
This test adds testing for var log messages format. In this commit we
also enhance the functional.sh script to:

1) make sed usage more compatible with other systems (--expression -> -e)
2) Add color to output PASSED / FAILED
3) Generate template files for non existing outputs
4) Generate failed outputs for manual verification

Change-Id: I1146b96fe9c91473c0127d0c47f890db1ddfb7b5
2018-10-15 09:50:13 +00:00
Miguel Angel Ajo 222bb2a984 Create and enable funcional testing
The funcional testing will include, per case, input files
and output files, plus a simple bash script which makes
sure the output stays as it should, and otherwise shows
a diff and fails the case.

Change-Id: I4c5f1143188f49c6a1448ca8918ab8b30710b865
2018-10-15 08:10:16 +00:00
Miguel Angel Ajo 384b6b5a1e Fix setup.cfg categories
"Topic:: Utilities" is missing a space. We are also adding python 3.6.

Change-Id: Icfef38cb5cf75684766be6be563069e71e751bad
2018-10-04 18:45:30 +02:00
Miguel Angel Ajo efdf6487d8 Switches to openstack-python templates for jobs adding py3.6
Instead of manually re-creating the gate/check elements, we
can make use of the openstack-python* templates, which
in addition will start publishing sources after merge.

Change-Id: If23427f68e0ab7627c3a79e73690cd43d783b8ac
2018-10-04 12:30:30 +00:00
Miguel Angel Ajo 20645e4f9a Making sure we're ready for python3
Adds py37 to the env list and switches pep8 python to
python3, including necessary changes to make pylint
and pep8 checks happy in python3:

* Unpins pylint, as 1.4.5 explodes (exception) when trying
  to check our code.

* Unpins flake8-import-order from an old version.

* Stops making use of the builtin cmp function, and replaces
  it with the recommended (a > b) - (a < b).

* Disables one py3 specific warning, since it's still necessary
  for python2 (useless-object-inheritance)

* Removes a non existing argument in netprobe parser (version).

* Inserts an additional blank line between __future__ imports
  and stdlib imports.

Change-Id: If4a45108fe93080f9967e2ac26d681b8dd5c7fc2
2018-10-03 15:10:59 +02:00
Zuul 352a5ab8f2 Merge "Add Apache license headers to the .py files" 2018-09-27 10:26:32 +00:00
Zuul 546fee53dc Merge "Stop setting version manually" 2018-09-27 09:09:41 +00:00
Zuul 4ab53a4e7a Merge "Parse message logs containining timestamps with microseconds" 2018-09-27 09:09:41 +00:00
Miguel Angel Ajo e518ac9bc7 Add Apache license headers to the .py files
The purpose of this change is to help the licensecheck tool
identifying the license of the sourcecode.

Change-Id: I3cc325e825fefe3c124329d8ab4b10821c9b1676
2018-09-27 10:15:53 +02:00
Monty Taylor 44d81d873e
Stop setting version manually
One of the reasons that pbr exists is to have versions be driven by
signed git tags instead of by content in the source repository. This has
the benefit of being both easier and preventing people from forgetting
to push a git tag when they cut a release.

Additionally, this code is doesn't work for sdist jobs in Zuul upstream
because it requires the project to be importable to run setup.py, but
sdist/wheel generation does not otherwise need to install dependencies.

Remove the override of version in setup.py and replace the hard-coded
version string in oslogmerger.py with the appropriate pbr code.

Change-Id: Ifdbb649b02d7e2fde6e8682e138c81817543fb08
2018-07-05 17:01:26 -05:00
XiaojueGuan 3ab0acd63a Trivial: Update pypi url to new url
Change-Id: I4d2afc18167ed3b7945330fb4db62535e72f426c
2018-05-01 15:45:04 +00:00
Matthew Booth 9dd538d2e8 Parse message logs containining timestamps with microseconds
Change-Id: I9ffb52dccf3c18dac5cf5dfe8c957a95301d72be
2018-04-20 11:33:51 +01:00
Miguel Angel Ajo e08b086914 Enable py27 and py35 unit testing
This also includes a couple of basic unit tests as proof
that this is working.

Change-Id: Iff8241b4bdfd3c97319198873527911e8ed318e1
2018-03-01 17:14:45 +00:00
Miguel Angel Ajo 5193d0010e Setup tox.ini for unit and pep8 testing
This patch includes fixes to all the pep8 warnings.

Change-Id: I4f87e1428258824509d2d83879312a1f0dff0a8a
2018-02-28 23:19:52 +00:00
Miguel Angel Ajo 809bb6add3 Python 3.x compatibility
Now the code is python2 and python3 compatible, and they both
generate the same output regardless of subtle differences.

Change-Id: Ic6b908387becf4d0de5d9da72ce6915d3f7c3b1c
2018-02-28 12:18:37 +00:00
Zuul c095b83d35 Merge "Make iter(LogFile) return an independent generator" 2018-02-11 08:04:58 +00:00
Zuul 358f2eb694 Merge "Add --default-tz" 2018-02-11 08:01:47 +00:00
Zuul 749b15bfc5 Merge "Fix incorrect auto-detection as timestamp log" 2018-02-11 08:00:45 +00:00
Zuul 06a41984d0 Merge "Delete some dead code" 2018-02-11 08:00:09 +00:00
Zuul 3a6b12679d Merge "Add auto-detect parser for raw syslog" 2018-02-11 07:59:13 +00:00
Zuul 18891d4142 Merge "Add auto-detect parser for libvirt domain logs" 2018-02-11 07:57:51 +00:00
Zuul 7733dc1c6d Merge "Allow zero positional arguments" 2018-02-11 07:54:01 +00:00
Zuul 18985c7485 Merge "Implement format auto-detection" 2018-02-11 07:53:26 +00:00
Zuul e2a4066918 Merge "Refactor log parsing into separate parser classes" 2018-02-11 07:53:25 +00:00
Matthew Booth c9e5f14026 Make iter(LogFile) return an independent generator
This is a simplification of the previous code which used global state.
It was prompted by a bug: iterating over a log file containing 3
lines, we were returning lines 1 and 3, and omitting line 2.

Change-Id: Ic7388f96a6201f30e16662ff4a5f3c081ac2c4bd
2017-09-29 18:27:13 +01:00
Matthew Booth 4583d5c340 Add --default-tz
Change-Id: I4b827b257879eae396852e7464236fb86ff0f57d
2017-09-29 18:27:12 +01:00
Matthew Booth e73a2a35b2 Fix incorrect auto-detection as timestamp log
TSLogParser was searching for any instance of [<number>] in a log
line, which was incorrectly matching on, eg:

  Sep 15 18:09:46 clgrabguye21.localdomain su[160015]: ...

With this change we assert that the timestamp occurs at the beginning
of the line.

Change-Id: I04da2f1fa661bccfd3b046dfc22e5a10f6cb37f8
2017-09-29 18:26:51 +01:00
Matthew Booth 1c6d22503d Delete some dead code
Change-Id: I92c05a7520541b937badbbd87308daf12a76baf5
2017-09-29 18:26:51 +01:00
Matthew Booth 1a31f79327 Add auto-detect parser for raw syslog
Change-Id: Iba7e80d97cffc809defdfdd87f88a8b698f28019
2017-09-29 18:26:51 +01:00
Matthew Booth 1be8b23dfc Add auto-detect parser for libvirt domain logs
Change-Id: I7d98327bebf20b01e41525b56ef59c68cbb8bbd3
2017-09-29 18:26:49 +01:00
Matthew Booth 11e139a706 Allow zero positional arguments
Allow all log files to have their formats given explicitly, with no
auto-detection.

Change-Id: I0a0a614642bc543f03d394b5e8822a445e7bde5b
2017-09-21 11:30:48 +01:00
Matthew Booth 06de0ba054 Implement format auto-detection
Add a new option --os-logs/-ol to explicitly specify OpenStack format
logs. Logs whose format is not given explicitly are auto-detected.

Change-Id: Ie97127f24f34ab386138db28b626abf4c259ac24
2017-09-21 11:30:48 +01:00
Matthew Booth 27599915cb Refactor log parsing into separate parser classes
All log entries are now the lighter weight LogEntry class, which is
also responsible for its own output. Parsing is simplified and
separated into independent parser classes.

Change-Id: I264cf20933e8af007556efd7a36639f854460f49
2017-09-20 16:53:05 +01:00
Matthew Booth dedd78ec27 README.rst: fix typo
Change-Id: I4f59a5a70b004c173c2c4718e153f0f721098c4e
2017-05-25 10:18:10 +01:00
Matthew Booth f2be96fe6a Always merge sort input files with heapq.merge
heapq.merge does an extremely efficient merge sort. Being a merge sort
it has constant memory overhead, and is very fast. It also starts
producing output immediately. It obsoletes both previous sorting
methods, so we deprecate the --min-memory option.

Change-Id: I8384b7214ba54dffa61d1c2195f3b4c238ba253a
2017-05-25 10:18:10 +01:00
Miguel Angel Ajo 5b9c82746b Bump version to 1.1.0 for release
Change-Id: I0461a7bf6ef78ff2afeb2c38ea4478b4b3cda46c
2017-03-15 11:18:33 +01:00
Jenkins 27b28c12f2 Merge "Guard against all exceptions when processing a line" 2017-03-15 09:27:22 +00:00
Ihar Hrachyshka 881b6fd359 Guard against all exceptions when processing a line
You never know what's in the input string, so catch all possible
failures without crashing the tool.

Related-Bug: #1672921
Change-Id: I3a518be2e6087d0a096fa3c36aa40dcad8aecb98
2017-03-15 04:37:58 +00:00
Ihar Hrachyshka 733b98a8ec Guard against null-strings in lines
Without that patch, the tool chokes on lines with null-bytes in the
middle, which as it turned out sometimes happens in neutron logs.

Related-Bug: #1672921
Change-Id: I2ec2c637dc03008f4d964f88a788af0d69abe282
2017-03-15 04:37:46 +00:00
Miguel Angel Ajo 77c2819793 Add .gitreview file
Change-Id: If00b1ac133d632b736435a9f9ea0cd5167e6a781
2016-09-20 19:56:10 +02:00
Miguel Ángel Ajo 158f64ae4c Merge pull request #28 from danalsan/master
Add spec file for Fedora distros
2016-07-22 12:54:36 +02:00
root 6a70bd6b49 clean empty bullet at the end of the changelog section 2016-07-22 08:37:05 +02:00
dani 139a6f6d51 Add spec file for Fedora distros 2016-07-21 15:48:33 +02:00
Miguel Angel Ajo f39e434d99 Fix variable size timestamp for OS LogEntry 2016-04-26 14:46:59 -05:00
Miguel Angel Ajo afb9ec3b44 Add Gorka as author too 2016-04-26 11:06:19 -05:00
Miguel Angel Ajo 6086787765 Remove README.rst from datafiles to avoid warnings in pip
This will also fix an issue with pip install on OSX.
2016-04-26 10:59:43 -05:00
Miguel Angel Ajo e901627f16 Updated netprobe, and added netprobe.log to auto alias 2016-04-26 10:50:19 -05:00
Miguel Angel Ajo 7eb4f383db Added a new probe: netprobe
This is an standalone .py file that can be uploaded to remote
servers and used to log network packets over network interfaces.

It will discover new interfaces automatically, in the root or
inside network namespaces.

Interfaces and namespaces are filtered via regular expressions.

usage: netprobe [-h] [-v] [--netns-re NETNS_REGEX]
                [--netdev-re NETDEV_REGEX]
                [--tcpdump-filter TCPDUMP_FILTER]
                [--check-interval CHECK_INTERVAL]

This tool will track system network devices as they appear in a host,
and start tcpdump processes for each of them, while the output of all
the tcpdumps goes in a single openstack-like log.

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  --netns-re NETNS_REGEX, -n NETNS_REGEX
  --netdev-re NETDEV_REGEX, -d NETDEV_REGEX
  --tcpdump-filter TCPDUMP_FILTER, -t TCPDUMP_FILTER
  --check-interval CHECK_INTERVAL, -i CHECK_INTERVAL
                        The interval between interface checks
2016-04-23 18:53:37 +02:00
Miguel Ángel Ajo 561b4310f7 Merge pull request #25 from Akrog/issue7/other_log_formats
Add support for other log formats
2016-03-29 12:21:01 +02:00