Update master grenade jobs to upgrade testing
- grenade job to test 2025.1(N-1) to master(N) upgrade
- grenade skip level job to test 2024.2(N-2) to master(N) upgrade
Also, heat job is failing 100%, so making it non voting to merge this change.
Depends-On: https://review.opendev.org/c/openstack/devstack/+/945237
Change-Id: I5e0110a61a5a5635530813cb76fb0c86f3ef661c
Service name "neutron-api" can be used instead of the "q-svc" to enable
Neutron API server in the Devstack based deployment. It should be
checked by the Grenade to and treated the same way as "q-svc" service.
Related-Bug: #2096912
Change-Id: I9e0b3522c87e8f4f89cce0c60da11796525c2f25
Its implementation was moved to lib/libraries long ago[1] and now it
is just a thin wrapper to keep compatibility.
[1] 3ed99c0b27122ff00e2d236086ab16b0cc1887c1
Change-Id: If9d3e4b355a5fec8c71b2e4a146d8009cc8daf15
pre-commit has been introduced to number of projects like oslo to run
lint checks such as hacking. The pre-commit config file does not affect
functionality, so grenade job is not needed when only the file is
updated.
Change-Id: I8e4eb7a023ab67d9c96deb228784ba88e3aa9048
nova-status was not there before Ocata due to that we
had condition to check if nova-status exist or not. As
Ocata and older releases are EOL, we should not have that
command exist check. The check can give false result when
nova-status does not exist due to any reason and in that
case greande will not fail.
Change-Id: I465d4159f04775890bfb0da20cd14c5e1de19cc4
Due to bug#2080600, grenade job failing on
user creation because domain-id is None. OSC
has fixed it but that is not yet released and
u-c, meanwhile let's fix it in grenade itself.
- https://review.opendev.org/q/I1204ca611a74d134c879467d6c2b73f16e043213
Related-Bug: #2080600
Change-Id: Icd295af041d244f1772a33b10b0e9e06a610dd21
grenade job is inherited from the multinode job which is
defined in zuul-job. Devstack lib/tempest script
needs the project os-test-images repo to be present, so
we need to add it in required project in grenade job.
Same way we did for devstack base job
- 03bc214525/.zuul.yaml (L513)
It worked in the grenade job on 2024.1 (master till now) because the old node was stable/2024.1
which did not require the os-test-images but on the current master the old node is stable/2024.2
where the grenade job needs this repo to be present.
Change-Id: I020e05403cb2c75eff3aa0fba0f975d0688cf92d
Seen in a nova-grenade-multinode job failure today:
In grenade/projects/70_cinder/resources.sh:create, a server is booted
from a volume and attaches one non-bootable non-encrypted volume
followed by one non-bootable encrypted volume. After the volumes attach
successfully, there is an attempt to SSH to the server and it fails due
to a timeout if SSH does not connect within 30 seconds.
After a failure, the server console log is dumped and in it the last
message is "Starting dropbear sshd: OK" which makes it seem likely that
sshd was not up and running yet and that's why SSH failed.
We could increase the timeout here to give the server a bit more time
to get sshd running before giving up.
Closes-Bug: #2066072
Change-Id: I823d1b984fee9f66b11dc58889cc40ab5555dd62
Current master branch (2024.2) is non SLURP release and
grenade-skip-level job is not supposed to run on non SLURP
gate.If any project want to test upgrade from stable/2023.2
to 2024.2 then they can use grenade-skip-level-always job.
Keeping its job definition create more issue and confusion when
zuul looks for the job selection.
Change-Id: I705996a9eefd40b975080cdc51b68c4467b94437
we have notice failures in the grenade jobs
corralated with OOM events. grenade-base
previously used the default value of 1G
for configure_swap_size: but now its updated
to configure_swap_size: 8192
most devstack jobs have already been changed
to either 4096 or 8192. all jobs that inherit
form devstack-tempest use 4096 where as more
complex jobs such as those that have ceph use
8192. as grenade is a non trivial job this patch
uses 8192 MB
Change-Id: Ibf6dfe84f15019d0360e2c2042856aeecb22209d
Currently, because we don't run the n-novnc (console proxy) service on
the subnode, devstack won't enable VNC for guests hosted on the
subnode.
We see occasional failures in the gate when VNC tempest tests attempt
to obtain a VNC console for a guest that happened to get scheduled to
the subnode with the following error:
Details: {'code': 400, 'message': 'Unavailable console type novnc.'}
This sets NOVA_VNC_ENABLED = true for the subnode only, with the
thinking that the enabled n-novnc service is sufficient for the
controller node.
Jobs that derive from the "devstack" job [1] are already setting
NOVA_VNC_ENABLED: true for both the controller node and subnode.
[1] 3ffcc89d5d/.zuul.yaml (L535)
Change-Id: I1acbdb605e0e0d9e5b4604fcd21af850b06823b9
This is occasionally failing with a timeout. It has been for months,
but it seems like the frequency has increased lately.
Change-Id: Ib1270e4f5bada8680a5d19133a888a8ade8f73c3
Closes-Bug: #2039707
grenade is branched for stable/2023.2, so updating
the current master default setting for:
- Update grenade testing from stable/2023.2 to master.
- Update grenade-skip-level and grenade-skip-level-always
jobs to upgrade from stable/2023.1 -> master
Depends-On: https://review.opendev.org/c/openstack/grenade/+/897381
Change-Id: I1dbea0caad676d6be7a4a58495a76a351e3c41e5
Python regexes are deprecated in Zuul, so update the use of regexes
in branch matchers with re2-compatible syntax.
The commit also removes matches for branches that are EOL - all
branches until stable/rocky (including).
Change-Id: I43c696ef2029cbf351f902b293df05f296dbb2a2
We don't want to go through the hassle of changing existing deployments
from global pip installs to using a virtual environment, so disable this
feature in grenade jobs for now until we can use it on the old side,
too.
Needed-for: https://review.opendev.org/558930
Change-Id: I5e5a67cceade05a4a32ddd1b86802f9496d1bb79
In troubleshooting grenade failures in ironic,
we've observed a sporatic failure causing the
ironic-grenade job to fail upwards of 20-40%
of the time where it seems cirros, due to the
boot sequencing and interfaces, is not
online for networking until after sixty seconds
have passed. In one case 83 seconds before the
networking was fully online.
In consulting with neutron contributors, a random
job check seems to reveal that even with pure VM
workloads, it takes on average 35 seconds into the
ping check. As such, it seems reasonable to make
the setting configurable so ironic-grenade can
execute with an increased timeout more appropriate
to the job settings and test environment.
Adds a INSTANCE_WAIT variable, which defaults to
the prior setting which was static, 60 seconds.
Change-Id: Ie5acf1ad7f8dca39db07f7e61035a8916439004d
We fixed the tempest venv constraints setting for tempest test run
on base branch but forgot to do the same for target branch test run.
It cause issue when base and target branch have different Tempest
and constraints to use (for example stable/wallaby use old tmepest
and stable/wallaby constraints but stable/xena use tempest master
and master constraints), in such cases we need to set proper constraints
defined in devstack and then run tempest.
We do not see this issue as none of the job run tempest on target except
heat (https://review.opendev.org/c/openstack/heat/+/872055/5).
I reproduced the issue in grenade stable/xena by enaling the tempest run
on target:
- https://review.opendev.org/c/openstack/grenade/+/878247/1
- https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184
Closes-Bug: 2003993
Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f
grenade is branched for stable/2023.1, so updating
the current master default setting for:
- Update grenade testing from stable/2023.1 to master.
- Update grenade-skip-level job not to run for current non SLURP master
release
octavia-grenade job is broken due to diskimage-builder so to unblock
grenade gate, stop running it and we can run it once DIB fix is merged
and released
- https://review.opendev.org/c/openstack/diskimage-builder/+/878089
Change-Id: I400c7f386e5b900774039223c9199508229494a2
For projects that want to opt-in to testing N-2 upgrades on all
releases (for early warning of issues, best practices, etc), this
pulls out a grenade-skip-level-always job that can track N-2 regardless
of whether master or N-2 is a SLURP release. The regular
grenade-skip-level job can inherit from that, with the branch
restriction and most recent SLURP as the from_branch.
Note that this makes the specific job a subset of the generic for
easier application of config to both. Right now they use different
nodeset rules, but in the next release they will be the same, so it
seems clearer to make it a hierarchy.
Change-Id: I39d1cbc4a2633c9908ba0e25205fb2f6a4aa729e
Nova has deprecated the server-side keypair generation process, which
is used by cinder's resource create phase in grenade. Even though it
still works for old microversions, it generates RSA keys which will
not be accepted (by default) in newer distros. This makes us generate
the keypair locally and upgrade the public key instead of generating
it server-side.
Change-Id: I0275a44f2ec4977b085b8129fd06e80a0ed6e68d
The jobs fails intermittently on ping check
with current timeout of 30 seconds.
The testvm can take time to boot and have network
ready on slow systems and using qemu hypervisor
and lead to ping timeout.
With [1] included noticed it taking more than 40 seconds
to boot testvm, so bumping timeout to 60 seconds will
help in such cases.
[1] https://review.opendev.org/c/openstack/grenade/+/874417
Related-Bug: #1463631
Related-Bug: #2007357
Change-Id: Ibdd030e126d508e6ff24cde180c611ada7f24cb3
Grenade set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
which are used to use the constraints during Tempest virtual env
installation. Those env var are set to non-master constraint when
we need to use non-master constraints but when we need to use the
master constraints we do not set/reset them point to master constraints.
This create the issue when grenade script install the tempest second time
on new devstack where we need to use master constraints but env var
TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE set previously in tempest
installation in old devstack point to the stable constraints.
One case where it started failing when we tried to pin the stable/wallaby with
Tempest 29.0.0
- https://review.opendev.org/c/openstack/devstack/+/871782
and stable/xena grenade job failed (stable/xena use master tempest and supposed
to use master constraints)
- https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641
We should set/reset those constraint env var to master constraints if configuration
tell to use the master constraints.
Closes-Bug: #2003993
Change-Id: I8c623008292f3b2dd2cd6d596feea552194acd89
As reported in the related bug, the rootwrap filters are not updated
between versions. This patch calls the devstack method that setups
the Neutron rootwrap configuration and filters.
Closes-Bug: #1999235
Change-Id: Iaebf1b33ccf3bfd64191f9a898408bcfe11dd557
In [1] we finally got rid of the unfinished lib/neutron module and kept
only lib/neutron-legacy. It's renamed to lib/neutron now and it's the
only neutron related module in Devstack.
So this patch removes leftovers related to the old lib/neutron-legacy.
[1] https://review.opendev.org/c/openstack/devstack/+/865014
Change-Id: I2a856b15eda992f0e78ee8eff65f39646e24c936
When devstack was switching to ML2/OVN as default backend,
in the base grenade job it was explicitly configured to use old default
ML2/OVS backend. It was done like that to avoid problems with upgrade
and change backend in the same job.
Now, as ML2/OVN is default backend in Devstack since at least
couple of cycles it's fine to switch grenade jobs to use default
backend.
Depends-On: https://review.opendev.org/c/openstack/ironic/+/866993
Depends-On: https://review.opendev.org/c/openstack/devstack/+/867065
Change-Id: Iede9fe71d81fc86a92122800d5a16f45442dd79e
This new Devstack's variable is introduced by patch [1] but as it's not
in the stable branches of Devstack yet, it can't be used in Grenade jobs
with default setting.
So lets set it to the emtpy string to make it working as it was before
[1].
[1] https://review.opendev.org/c/openstack/devstack/+/849145
Change-Id: I2de65d4764f23ae85a086d4d09ed80c81777434b
Grenade jobs stop services, check fip connectivity
for a nova server and then upgrade to next release.
But since ovn data plane and db services are stopped along
with other services, fip connectivity fails as a result.
We shouldn't stop these services along with other
neutron services. This patch sets "SKIP_STOP_OVN" to True
to skip stop of ovn services.
This fixes ovn grenade jobs.
Depends-On: https://review.opendev.org/c/openstack/devstack/+/839362
Change-Id: I2bd3f7e5f0af9a6532db7a1cdb4bc45a963042ca
grenade-skip-level is added and supposed to be run from Yoga
release. We have added this job in integrated template which
is in branchless tempest side and applicable for stable branches
also.
We can control to run this job on Yoga and SLURP release onwards but
that leave if anyone running it directly without template So controlling
the branches in the job definition will work for everyone.
Change-Id: Ie560b25f83c3f102f408077f5023f07facf6095d
Tripleo-ci jobs are broken after latest release of setuptools 61.0
because of breaking changes which are not backwork compatible,
details in related bug and [1].
Users that don't set ``packages``, ``py_modules``, or `configuration`
are still likely to observe the auto-discovery behavior, which may
halt the build if the project contains multiple directories and/or
multiple Python files directly under the project root.
To disable auto discovery, one can do below in setup.py
~~~
setuptools.setup(..,packages=[],..)
~~~
or
~~~
setuptools.setup(..,py_modules=[],..)
~~~
[1] https://github.com/pypa/setuptools/issues/3197
Releated-Bug: #1966382
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: I8ba2f6eccb9f279406321915793709e50e8279d4
