Commit Graph

2364 Commits

Author SHA1 Message Date
frenzyfriday 01b6b8299b Fixing: TypeError: '<' not supported between instances of 'list' and 'str'
The above exception [1] occurs for example [2] when elasticsearch returns data
with more than one zuul_executor as a list.

This is what l#58 is able to sort
[(12.5, '5'), (12.5, '4'), (12.5, '3'), (25.0, '6'), (18.75, '2'), (6.25, '13'), (12.5, '1')]

This  is when it throws exception
[(8.13953488372093, 'ze06.opendev.org'),
(12.790697674418604, 'ze10.opendev.org'),
(5.813953488372093, 'ze05.opendev.org'),
(8.13953488372093, 'ze01.opendev.org'),
(16.27906976744186, 'ze04.opendev.org'),
(4.651162790697675, 'ze03.opendev.org'),
(3.488372093023256, 'ze02.opendev.org'),
(4.651162790697675, 'ze08.opendev.org'),
(12.790697674418604, 'ze09.opendev.org'),
(20.930232558139537, 'ze12.opendev.org'),
(1.1627906976744187, 'ze11.opendev.org'),
(1.1627906976744187, ['ze12.opendev.org', 'ze11.opendev.org'])]

[1] https://0050cb9fd8118437e3e0-3c2a18acb5109e625907972e3aa6a592.ssl.cf5.rackcdn.com/790065/7/check/openstack-tox-py38/4968a73/tox/test_results/1449136.yaml.log
[2] https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/787569/6/output/elastic-recheck/1449136.yaml

Change-Id: Ie559d5764d9f68420119a7f9608389f0745a9c02
2021-05-19 22:10:24 +02:00
melanie witt 668269bbcd Add query for bug 1882521
Jobs are failing with the following error in n-cpu:

  Guest refused to detach volume <uuid>:
  nova.exception.DeviceDetachFailed: Device detach failed for vdb:
  Unable to detach the device from the live config.

At this time, this query has:

20 hits in the last 7 days, check and gate, all failures

Related-Bug: #1882521

Change-Id: Ib92b679f2d1dbd8131f58c8bb85fc2a3f65dbfb5
2020-12-18 20:25:46 +00:00
Zuul b556165402 Merge "Add query for bug 1903979" 2020-12-03 12:03:43 +00:00
Stephen Finucane de3ee8d0f9 gitignore: Ignore .eggs directory
Created by simply installing the package in a local venv.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ie27eb9fca09b2b204e5bc95076fc5ada8128944f
2020-12-02 10:52:37 +00:00
Stephen Finucane 3d4d39ca77 Add query for bug 1906428
Change-Id: Ie67945572e1cd30144cb8b4bba339e66a8b33639
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Related-Bug: #1906428
2020-12-02 10:48:18 +00:00
Balazs Gibizer 2615a3292c Add query for bug 1903979
Change-Id: I357ffbafc78cb9c22013481d659dbedc931b6668
Related-Bug: #1903979
2020-11-12 12:13:58 +01:00
Sorin Sbarnea 5104f277b8 Make file writing atomic
This should render the need to use wrappers obsolete as all
file writing operations are now atomic, assuring that we either
write the entire file or fail.

That is important as we do not want to end-up serving partial files
with the web-server.

Change-Id: I696e2474b557e6b5fea707a198f32cea721cc150
2020-11-09 10:02:18 +00:00
melanie witt 43e7ee9699 Add query for bug 1901739
Related-Bug: #1901739

Change-Id: I48e5e05256ef6f6366521795e12c1eb888373162
2020-10-29 12:33:17 +00:00
Zuul 798c970240 Merge "Replace testr with pytest" 2020-09-18 12:13:40 +00:00
Zuul 262ec2c995 Merge "Enable configuration via environment variables" 2020-09-18 07:14:43 +00:00
Sorin Sbarnea 0cbc0cd524 Replace testr with pytest
Using a single test run orchestrator makes it easier to maintain
the project, especially as pytest is actively maintained.

Change-Id: I5c843984bc0a1b9264e744373e6a3fd9d43e99cd
2020-09-17 15:49:32 +01:00
Sorin Sbarnea e6b1354d91 Enforce use of safe yaml loader
Sorts runtime warning related to use of unsafe loading on yaml files.

Change-Id: I11f1332f34fe341b13100d8ae4c263cfbd5b90e0
2020-09-17 15:18:36 +01:00
Sorin Sbarnea 3d1411f3a1 Enable configuration via environment variables
Refactors configuration loading in order to simplify it and to
allow overriding defaults using environment variables.

This behavior is similar to other tools like pip or ansible, which
can load any configurable option from env.

This step ease migration towards containerized use, where we do not
want to keep any secrets inside containers and we may want to
avoid volume mounting, especially when testing.

Change-Id: I0d3a9f19b0ba8d1604d0ca63db01296a3219fb47
2020-09-17 10:59:38 +01:00
Zuul dc942c06f1 Merge "Made parse_jenkins_failure a non static" 2020-09-17 07:21:36 +00:00
Zuul 0b3f6478ef Merge "Create elastic-recheck container image" 2020-09-15 16:14:44 +00:00
Sorin Sbarnea 3901d2fd93 Made parse_jenkins_failure a non static
Replaces static implementation that received password and a member
function that can make use of the config object.

Change-Id: If9617b6db73eb49c5193f098d45e357a267529dd
2020-09-15 10:04:15 +01:00
Sorin Sbarnea 88211997df Create elastic-recheck container image
- part of replacing puppet deployment with ansible and docker-compose
- current change tests container building

Change-Id: Id70f156e63751ebd14908cc5da969e964f63645b
Story: TRIPLEOCI-177
2020-09-14 18:14:39 +01:00
Sorin Sbarnea 9d37c88c8f pylint: 4 more
Change-Id: I4cc928d8212a5192927a994b4248f32fe05ca723
2020-09-11 11:19:35 +01:00
Sorin Sbarnea 360c57118c pylint: 6 more
Change-Id: Ic16db7972fe6f9da86592d56f4983572d7c68989
2020-09-10 15:28:52 +01:00
Sorin Sbarnea c41b9c6fa0 pylint: fixed logging-not-lazy
Change-Id: Ic25366a9afdfc67ab2beddbe2b8d02544c51e480
2020-09-10 14:56:52 +01:00
Sorin Sbarnea ed5296999e pylint: fixed imports
Fixed pylint violations around imports. Implements
standard import ordering (isort).

Change-Id: Ib89108925487e49109d18ae315cd4892b8b48837
2020-09-10 13:46:38 +01:00
Zuul bbd3a2f2d5 Merge "pylint fixes" 2020-09-10 12:14:05 +00:00
Sorin Sbarnea 78a8098354 pylint fixes
Resolves several code style violations.

Change-Id: Id03dad8f8ce141eb1e630a77d0c9ae497de9f2ed
2020-09-10 12:42:12 +01:00
Sorin Sbarnea c6f07d7f93 Use pytest for queries
Switches queries testing to use of pytest which provides the following:
- test generator for each query (parametrize)
- ability to test a single query test
- generate html report with test results, making easier to investigate
  failures.
- parallel executions
- minor bugfix which prevented running queries from running with py38
  as the  config parser requires only strings (None being invalid).

Change-Id: I982c694a5160a9ecfd117d177d30b911cfe53425
2020-09-10 10:24:37 +00:00
Sorin Sbarnea df62575028 Refreshed bindep
Removes all historical leftovers from bindep.txt file.

Change-Id: I9df8eb8aa321549acc90f58eff9a37774822b36e
2020-09-10 09:26:20 +01:00
Zuul 50cd9d6aa3 Merge "Enable pylint" 2020-09-08 13:02:49 +00:00
Zuul ee0c3ef6dc Merge "Avoid pip breakage due to keyring" 2020-09-08 13:01:46 +00:00
Sorin Sbarnea 01a290f2c0 Enable pylint
- enable pylint execution with current violations temporary disabled
- avoid installing dependencies for linting (heavy)
- pin linters to avoid suprises

Change-Id: Ica47efcec7a4da3b0bf949612cf9f1a49016c488
2020-09-08 12:12:07 +00:00
Sorin Sbarnea baea37e12c Avoid pip breakage due to keyring
Apparently under linux pip can become fully blocked due to keyring
presence. That is a known open bug, so we apply the workaround until
it is fixed.

Change-Id: I30a5ec1b04b57a5604cb3caa3bc56ea2476e89ba
2020-09-08 11:54:23 +01:00
Sorin Sbarnea 97ca1c24c3 Drop py27 and add py38 jobs
- Dropping py27 as is out of support
- Enable py38 testing, already default python on several distros.
- removes six as a dependency as is no longer needed for pure py3

Change-Id: I1e825073abc6cd55aa2fdc363358f2701152c57b
2020-09-08 10:21:02 +01:00
Sorin Sbarnea 8f709c1d67 Resolve unsafe yaml.load use
Fxed deprecation warning about unsafe call

Change-Id: I474454f438d6345dea76daf788be14c93fee6fb6
2020-08-19 09:52:17 +01:00
Sorin Sbarnea 673a054b17 Replace pep8 jobs with linters
Change-Id: I9a59157bf85325a105837828a099915ae79ad003
2020-05-28 10:19:47 +01:00
Sorin Sbarnea f68a8719af Bumped flake8
- Upgraded hacking(flake8)
- Added more modern tox linters environment (pep8 alias)
- Temporary added skips for broken newer rules
- Fixed few basic rule violations
- Moved flake8 config to setup.cfg (tox.ini is not recommended)

Change-Id: I75b3ce5d2ce965a9dc5bdfaa49b2aacd8f0195ad
2020-05-23 08:54:14 +01:00
Clark Boylan 6fc19ee5b9 Simplify 1708704's query
The previous query for 1708704 was causing e-r to OOM and resulting in a
lack of graph data. This is beacuse tripleo's logstash.txt file is not
getting parsed properly and ends up with the entire file as the single
event. Then e-r downloads all those file copies and fills its memory and
breaks. We can work around this by only looking in job-output.txt files
for this bug. Then when tripleo's fix has flushed the bad events out (10
days after fix merges) we can revert this change.

Change-Id: Id619f90ffe84b3d4de334ea4b17026b9b3239d33
2020-01-29 12:16:04 -08:00
Zuul 67e6bf30e9 Merge "Don't pretty print json files" 2020-01-29 18:32:09 +00:00
Clark Boylan 94ab7eb16b Don't pretty print json files
The json file outputs of e-r are loaded by web browsers in order to
render our graphs. These json files are actually quite large and part of
the reason why is we pretty print them with 4 space indents and they
have large nesting. Stop pretty printing (humans can pass the files
through a filter if necessary) in order to reduce the size of these
files and make browsers happier (less time spent downloading).

Change-Id: I19dedc2994169932eb0e90b6cdea3856637f5ef0
2020-01-29 10:05:38 -08:00
Clark Boylan e83ce5d729 Fix flot xaxis labels
After the flot update the xaxis labels aren't meaningful to humans (in
fact I'm not quite sure what they were showing us). We can explicitly
state the input type as milliseconds and the label render format. Doing
this gives us labels that are meaningful to humans.

Change-Id: I7912a536f3de2756404f8c7e7f31d8bd5890ab22
2020-01-29 09:58:09 -08:00
Clark Boylan 265217b0cc Update flot js file inclusion to match new flot
This was necessary after upgrading the status server which pulled in
newer flot.

Change-Id: I28c844d216eefdf6d186b04ce47889fcd044d23a
2020-01-28 16:43:32 -08:00
Matt Riedemann c7bb8a099b Add query for WARN: failed: route add -net bug 1857151
308 hits in 7 days, check and gate, all failures, multiple providers.

Change-Id: I205bce7f7ebace0fa7c41f0917cde8463baff20f
2019-12-20 12:00:51 -05:00
Zuul 464ee63b90 Merge "Add query for test_create_servers_with_vpmem bug 1856902" 2019-12-18 22:56:29 +00:00
Matt Riedemann 151a056216 Add query for test_create_servers_with_vpmem bug 1856902
18 hits in 7 days, check and gate, all failures.

Change-Id: Ia99f3c6aec92b83769c050b42914a3166a252b34
2019-12-18 17:08:58 -05:00
Matt Riedemann 62e42f4322 Handle ElasticHttpError in graph generation
Getting elasticsearch data for bug 1708704 is failing
in the check queue with:

  pyelasticsearch.exceptions.ElasticHttpError: \
  (500, 'ArrayIndexOutOfBoundsException[null]')

This might have to do with the size of the resulting
messages from the hits on the tripleo and kolla jobs,
I'm not sure.

What's clear though is the graph generation is blowing
up in the check queue on that bug but not the gate queue,
maybe due to a smaller result set, so this adds some
error handling in the graph generation for when a specific
bug query fails so it does not halt the entire build of the
graph.

Change-Id: Ibe18c9cccc421a6549a18148f1a2ce3c1e4339d4
2019-12-18 15:46:53 -05:00
Matt Riedemann 9d788302e6 Add query for network_sanity_check ping check fail bug 1856760
45 hits in the last 7 days, with a spike since Dec 16, mostly
fortnebula nodes, all failures, check and gate.

Change-Id: Ic856d11e183075244de556b86d4ecdc7bcc78abd
2019-12-17 16:20:37 -05:00
Matt Riedemann 54d728e2f1 Update query for bug 1820007
This also shows up in multi-node jobs when attaching a volume
so update the query to include volume attach failures.

Change-Id: Ie77e3998b2ff4a508fa3b6078acb3ceec15d7e37
2019-12-13 09:22:13 -05:00
Matt Riedemann 0b5eac82ee Add query for NetworkAmbiguous fail bug 1844568
211 hits in 7 days, check and gate, all failures.

The message shows up in the n-api logs but filtering
on n-api logs isn't sufficient to get 100% failure rate
in logstash because there are tempest tests like
test_create_list_show_delete_interfaces_by_network_port
which handle the error and work around it so even though
the message shows up not all jobs fail because of it so
we use the tempest failure log in the console to fingerprint
this bug.

Change-Id: I7b4a3f4a483c5166e9aee1507f12bb31069a8fe0
2019-12-05 09:39:16 -05:00
Matt Riedemann 149b5017b3 Update query for bug 1853453
This brings the hit count up from 14 to 74. Now we get hits
in the gate queue as well.

Change-Id: Ieb50f0c796cfccbbd6b3da0a534f6b7a7ae3800e
2019-11-21 09:19:59 -05:00
Matt Riedemann 39c87d1bb1 Add query for test_shelve_volume_backed_instance bug 1853453
This is split off from bug 1848078 since this test specifically
seems to hit this issue a lot in multinode jobs where the instance
is shelved from one node and unshelved on another.

14 hits in 7 days, all failures.

Change-Id: I9bd41e356abf72ff08415693b7b2b11a035a542d
2019-11-21 08:01:39 -05:00
Matt Riedemann d3ae71f373 Add query for nova install on py27 bug 1853166
908 hits in the last 24 hours, check and gate, all failures.

Change-Id: I62f0ad207eee41ce4738b0117bd6718248502d61
2019-11-19 10:59:52 -05:00
Matt Riedemann 242082d815 Add query for inap package mirror bug 1852617
80 hits in 7 days (really the last 2), check and gate,
all failures on all branches.

Change-Id: I1d6f827903f4fd4ccf7c0398575ae2ed26a4b539
2019-11-14 12:43:11 -05:00
Eric Fried e54ffc076e Add query for python-openstackclient bug 1851391
8 hits in 7 days, check and gate, affects osc-functional-devstack and
osc-functional-devstack-tips.

Related-Bug: #1851391
Story: 2006811
Task: 37369

Change-Id: I9d8c12c9ef502ed41de8905feaaf1e20b6945395
2019-11-05 17:47:49 +00:00