As CentOS 8 is not compatible with old ara, we installing new one isnted
Also providing py3 as default python for CentOS 8
Change-Id: I264e6f8bfa507668dada7b629af52a8a13001a45
As for U we don't support debian stretch, there is no need
in having stretch CI jobs. Buster ones should be used instead.
This also fixes ara report
Needed-by: https://review.opendev.org/706285
Change-Id: I554e512968718492ce6056b29b0612e450f0856b
It's good to check for tools before launching them. The common practise
is used in openstack-ansible/scripts/log-collect.sh
Change-Id: I2b2866bb0ba36eb658da71fe0670d39b911d90c1
--safe-links was used, which breaks openstack service collection on
the metal /etc paths, because they are symlinked into the venv like:
/etc/neutron -> ../openstack/venvs/neutron-19.1.0.dev12/etc/neutron
Instead we can use --copy-links which will resolve the symlink and
copy the data.
Note: This is a fix copied from I40905d79efa1bef9ae6a7678515fd07fde1bdd00
in the integrated repo. Since the scripts used are virtually identical
the fix must be applied here too.
Change-Id: I2cb193322cacd135214f5c3c5d0736842c458531
When collecting logs we do not care about file deltas so we can use
the --whole-file option to not perform any kind of diff between the
source and the destination. The --inplace option can also improve
performance by copying files directly on the destination instead of
creating intermediate ones first.
Change-Id: I044cd2d56b9b907e5948be1d3bd11ce4a876b1a2
zypper performs some network operations when collecting the list of
installed packages which slow down the overall process. We already have
the per host list of repositories so we can do some maths to figure out
where each package is coming from.
As we see below, disabling the repositories saves us ~9s.
~$ time zypper --quiet pa -i
real 0m10.607s
user 0m9.024s
sys 0m0.193s
~$ time zypper --disable-repositories pa -i
real 0m1.043s
user 0m0.833s
sys 0m0.136s
On a deployment with multiple containers, the 9s difference causes a lot
of overhead and jobs are timing out from time to time. As such, lets
just disable the repositories when collecting the list of the installed
packages.
Change-Id: I3e48dad0e213f6e24a2ceb4033b1a53873b6e298
I have no idea why this is only showing up after a merge,
but it is doing so on stable/rocky patches in the linters
(bashate) test. This patch should resolve it.
Change-Id: Ib5e34b16c4bacc459883d2be8d503d9f1b22ea0d
Compressing the journals breaks them making it impossible to review the
data later. This change adds anything with the suffix ".journal" to the
list of files the "find_files" function will return.
Change-Id: If521f1186797a74816b7a2d1eb8b0e501d6719e1
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change adds several services to the collection list which are part
of our deliverables yet we've no logging for. This change also
implements journal log collection when a given service is found which
should help folks better debug issues with various services, especially
as we transition to all journal logging.
Change-Id: Ie6f06eb5b0488005263aed397fab7dedc5a287a0
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The lxc_cmd variable is being set when we collect log information for
containers but it is not unset when we do the same thing on the host. As
a result of which, the host repo and pkg logs were wrong since they were
collected from the last container instead of the host.
Change-Id: If2d0a84b46538e44f383967108ca9f67e91f5054
This patch ensures that the ARA reports are run on successful jobs
but the results are put into a tarball. This should allow quicker
access to ARA reports without consuming lots of inodes on OpenStack's
CI infrastructure.
Change-Id: Ibb24c74d989f1a24f5f9a610b3996e359ceaeb58
Signed-off-by: Major Hayden <major@mhtx.net>
The LXC containers are now using networkd to configure the network
interfaces so we should collect the appropriate files. Moreover, we
also need to collect the resolv.conf file so we know how the DNS
resolution is configured.
Change-Id: I797fb2ca013c8cc9f2c7a9871187f2a67be9276f
Depending on the VM distro the working directory can either be rsync'd
to the VM or mounted. When the directory is mounted, various artifacts
are left behind from the test execution which is not desired especially
when the directory is shared between multiple VMs for parallel testing.
As such, it's best to simply copy the directory to the VM using 'rsync'
so each VM can get its own copy. This also restores the missing new line
at the end of the test-log-collect.sh script which got removed in
I4475e4dfe13acb4c88444f1f7aa88272bf87f6ee in order to make the linters
happy.
Change-Id: I1aab7d3f0d26f35dea84ae088fe78cfd2de79366
Sometimes gzip log collection will fail because certain linked
log file are missing. This change notice those error and let the
test pass
Change-Id: I0720460dba3ec959429b577017713e1d5a209337
It's helpful to know how the glocal pip.conf configuration file is
constructed so add it to the list of files we collect at the end of
each job.
Change-Id: I70e2b9b20c3a92c436f37c086476c41b97ab3a55
gzip during log collection sometimes fails because there are too
many levels of links in log directory. This change use gzip --force
flag to make it work in this scenario.
Change-Id: Ifa121fe56a6bdc7376b5a79800eae0f4d3f6b4f0
We record installed packages and enabled repositories for the host but
we also need to collect such information for the containers since this
is where the infrastructure and OpenStack services are running.
Change-Id: I009cbfbc87a0b8b6cf3013b3fae41343876810fd
The ethtool command can fail when run against some devices in some
configurations. This change simply makes the test collections pass
no matter the interface configuration.
Change-Id: I4475e4dfe13acb4c88444f1f7aa88272bf87f6ee
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Regular users can't check whether /proc/$CONTAINER_PID/root exist
since it's normally only accessible by root so run the test command
using 'sudo'
Change-Id: I95997b697f85a223d22b1d0a5ad2b6237193fb89
We export many variables throughout the CI jobs so it's helpful to
collect them all as part of the job results.
Change-Id: Ic12bb7efa6e841108d56a9368917157c4e104c77
This change updates our log collection script so that it is once again
grabbing logs from services and containers but this time far more
conservatively.
Depends-On: I3c1b755e4bed6a209c127258b79d3ef557caf19e
Change-Id: Ifae67fa3bb3dd790f806bc73de5068f0998cb4e6
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The script supports collecting information for packages and
repositories for openSUSE and CentOS but not for Ubuntu so add the
appropriate commands for it. This also fixes some indentation issues.
Change-Id: I9a2eca110cdf6d1186c867b5bb632ba48580f070
We'd like to see our tests results show up in OpenStack Health
regardless of whether the test passed or failed. This patch
makes sure that it will always try to generate them.
It also implements a routine to try and find where ARA is
installed, because it's not always in a tox venv called
'functional' which was the previously hard-coded location.
Change-Id: I3d967567af9d9c9583f34c89789a38e60717781e
When zuul executes jobs it adds all the post plays together
and executes them in the inherited sequence. This is causing
some confusion in our cross-repo testing where the logs are
trying to be collected twice as the cross-repo test is based
on the standard functional test.
Instead of defining two sets of plays we can consolidate them
into a single set, eliminating the confusion and repetition.
At the same time we also do not try to collect logs if the
script to prepare them was not able to execute, and we fix
the conditional for the ARA report generation so that it
works properly with the value being given it by zuul which
is a boolean word. To make sure it still works outside of
Zuul we leave the conditional which uses the standard success
return code of 0.
Change-Id: Id5027415f33a7a41650332a983cb87fab7ba2a4a
This reverts commit af3a08a5b1
but adds the tests repository as a required repo to make sure that
it works properly across other repositories. This will be tested
through another review using depends-on so that it does not
break the role tests again.
Change-Id: If9bded9d60e2fe839c0ec8c49cc5102b5a541474
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This reverts commit fd4db8ea90.
The playbook used here needs to be in every repo calling that job
(which is all our branches and all our repos).
I suggest we revert it until alll the other zuul v3 changes are settled.
We can then take a more organized approach.
Change-Id: I8ba967af69311fc0684635462d27cdc03728ca81
To ensure that logs are collected regardless of whether
a job succeeds, fails or times out, we move the log
collection to the 'post' stage of the job definition.
As the job playbook only gets run in nodepool, we remove
the detection of nodepool in the log collection script
and also remove the trap which executes it so that it
is only ever run when executed in zuul.
Change-Id: I5ab31cc141804cbcb6d8721032a79d0c724fc0a2
In order to reduce the quantity of unnecessary log content
being kept in OpenStack-Infra we only generate the ARA report
when the test result is a failure. The ARA sqlite database is
still available for self generation if desired for successful
tests.
Change-Id: I3ab9752650911bba0b8be5e73b65019e97bfef42
Jonathan Rosser