We fixed the tempest venv constraints setting for tempest test run
on base branch but forgot to do the same for target branch test run.
It cause issue when base and target branch have different Tempest
and constraints to use (for example stable/wallaby use old tmepest
and stable/wallaby constraints but stable/xena use tempest master
and master constraints), in such cases we need to set proper constraints
defined in devstack and then run tempest.
We do not see this issue as none of the job run tempest on target except
heat (https://review.opendev.org/c/openstack/heat/+/872055/5).
I reproduced the issue in grenade stable/xena by enaling the tempest run
on target:
- https://review.opendev.org/c/openstack/grenade/+/878247/1
- https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184
Closes-Bug: 2003993
Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f
In [1] we finally got rid of the unfinished lib/neutron module and kept
only lib/neutron-legacy. It's renamed to lib/neutron now and it's the
only neutron related module in Devstack.
So this patch removes leftovers related to the old lib/neutron-legacy.
[1] https://review.opendev.org/c/openstack/devstack/+/865014
Change-Id: I2a856b15eda992f0e78ee8eff65f39646e24c936
This adds some conditional logic around when mysql
is used based on the configured database type in
the devstack rc file. TODOs are left in place for
supporting postgresql which can be fleshed out when
we have a grenade-postgresql CI job to verify the
changes.
Change-Id: I30e7ee0c8de1add3c51824e30ca7ad103bb4a876
The neutron experimental multinode job is experiencing some network
issues, which causes the ssh connections to take 5 minutes to
fail. This means we hit the job timeout and don't capture our dump
information properly. Force a smaller connect timeout on all ssh
connections to avoid this.
Change-Id: I8bc2c46b8fa91f35c285ac0dcf1d669c38156f5d
As a follow-on to Iba7739eada5711d9c269cb4127fa712e9f961695 in
devstack, where we rename the XTRACE storage calls to not conflict, do
the same in grenade
Also some minor turning-down of tracing when importing stackrc; it
floods the logs with irrelevant stuff. And now we're tracing
properly, cleanup "echo_summary" output in the logs.
Change-Id: Ibb67a6454e4465a8d0493461e32dbae96c6a03e7
Ceilometer is moving to using in-tree devstack and grenade plugins. This
removes ceilometer from core grenade to make room for the plugin.
Change-Id: I09f47a8c24ac59e5db840bcbc7970bfb5d259591
This makes our ssh-into-test-instances commands completely stateless. That
means they accept any key automatically (which we already did) and not
write them out to the known_hosts file of the user running grenade. Without
this, multiple runs of grenade (or existing host keys for the IPs we use)
will conflict and cause us to fail when validating things over ssh because
of the key mismatches.
Change-Id: I0be425764cd640ada4f00300167df77fd870214e
This will make it easier to understand the failures by not having to
time corolate these files, but know which are the good ones.
Depends-On: I4e40eff6d0b1ef194e43b151a83206fbd50deb66
Change-Id: I7080b1d7d3f71d858ad6063c923376e49339b411
The upgrade scripts have a *ton* of historical cruft in them. This is
a bulk clean up patch that does the following things:
- ensures we source grenaderc before functions, we need this for
devstack dir definitions.
- moves xtraces until after sources to trim down irrelevant output
- removes as many variables and library sources as possible to ensure
that we include the minimum possible code in these scripts (for
sanity and readability).
This skips the neutron upgrade script, much like in devstack, the
neutron code has so many extra variables it's not clear what's really
needed.
Change-Id: Idac9190fd33478a479f7c7dd5cb1be894bb03444
We need to turn off errexit in the EXIT trap otherwise we break out if
it early. Rookie mistake, and prevents the rest of the debug from
happening.
Change-Id: Ida7e277dd1693a6c594b2f721ddf15c9a169f07b
When including functions file turn off xtrace before we include other
files, to keep the output a bit cleaner.
Change-Id: I4becdba925ae6472a80b2ab279705d34af7a4c44
sourcing a file under xtrace causes a huge amount of gorp in the
console, and it's rarely useful. Create a source_quiet to source
things with tracing turned off.
Change-Id: Ic0569d78314b85a8a7dab028e5895db8f37bc30e
This provides global variable comments for various function files, as
well as license info and the #!/bin/bash lines so that gerrit will
syntax highlight correctly.
Change-Id: Ice55aff1e39084a55e1a3da6d6418a103df4d04f
This adds nova modular service checks, including the use of the
verify_noapi phase to ensure the server doesn't go away when
nova-compute is down (which is a substantial enhancement of our
existing validation).
There is slight regression in checking multiple servers, however that
did not seem to be a bug that we actually ran into.
add our own local copy of ping_check to grenade. The version in
devstack gets diverted down a neutron path even though we actually
want to use it in a public context. We need to change that code to
work on public networks regardless of backend. Until then, we'll keep
our own copy.
Change-Id: I985ca38ef670b03802f9ad67ce19cf94cfc802ae
let projects declare that they will have a database to be saved, and
we'll do it for them, in the mysql case. Eventually other saving
issues should be left to per project save-state.sh scripts.
services which aren't always enabled in grenade jobs have their
settings files guarded with conditionals. Eventually we'll do
conditional loading of these settings files, however for now this lets
the rest of the infrastructure around running actions based on
PLUGIN_DIRS and UPGRADE_SERVICES be *non* conditional, which
simplifies them.
Change-Id: I06cad95ca319e4403772e5fb394c0f8965e0bf9d
This moves the exit trap into grenade/functions, which means all
grenade scripts will get the exit trap registered on them. This also
adds stack tracing into the scripts so that we can determine where we
were when we failed.
Change-Id: I4100628180498ff8723a1af6c0a4c10b4d8244c8
This moves all the project specific upgrade code into project specific
directories. This forshadows how we might break out this code into
external plugins. It happens all at once to keep a working grenade
environment on both sides of this change.
Change-Id: If47395c796bdb0ae2b8fe65bfe4b4556a6487c7a
This refactor's the grenade code to remove the copy/pasted
functions-common from devstack and instead use the functions from
TARGET_DEVSTACK. In order to accomplish this we need to refactor when
the devstack trees are pulled, so that we always pull both the base
and target devstack trees *extremely* early.
We also need to include a small number of functions (mostly related to
git) in grenade to be able to bootstrap these trees. This is done in
the inc/bootstrap tree.
There are some slightly unrelated function moves into inc/upgrade to
consolidate all the upgrade functions which were scattered through the
codebase for clarity. inc/* will be the contract library interfaces
out of grenade (mirroring devstack).
To do this grenaderc has to be split between variables which can be
set without devstack functions, and variables we compute later. These
later variables will be moved into project specific directories based
on the plugin plan.
Change-Id: I1a2d1a1fed7858c4f3b51a19be2d12cc2d260a24
Moves checks of DO_NOT_UPGRADE_SERVICES to a function, and
uses it to also setup for partial Ironic upgrades, where we
upgrade the entire cloud except for ironic-api and ironic-condcutor.
This also ends up adding ironic to services that are acted upon
in check-sanity and stop-base.
Change-Id: I392b41d956bc1b0c0eb8383f0eaad9b0c5461760
Syncing devstack/functions-common -> grenade/functions is confusing,
and as a result grenade/functions has been updated with grenade-specific
stuff. This moves existing functions to 'functions-common' and isolates
grenade specific things to 'functions'. 'functions' sources 'functions-common'
so this should not require any other changes and will make it easier to keep
code synced with devstack.
Change-Id: I3b6324fc3067833075fc5487684d31593627b779
This adds the ability for grenade to be used to run a 'sideways'
upgrade within the Juno release to migrate nodes from Nova Baremetal
to Ironic. The process is as follows:
* Deploy base Nova with VIRT_DRIVER=fake, stop base.
* Deploy target with VIRT_DRIVER=ironic
* Create Nova baremetal nodes and interfaces in db that map to
VMs
* Run Ironic migration scripts to migrate Nova baremetal nodes and
interfaces to Ironic
* Update baremetal flavor to reference Ironic deployment ramdisk
and kernel
* Run smoke tests against target
Change-Id: Icca6ed4d304c5177a8e41db14c85354d73f10edf
There was duplication between base and target dumps that is resolved
by use of arguments.
By extracting to function it becomes easier to make changes to how
database dumps are performed. The current setup is a bit hairy,
aggregating in a function makes it easier to improve later.
Change-Id: Ic5d4f13eaa7a4d842c9af0073de17e1cffad0a88
This adds logic to upgrade_project() to call project upgrade
scripts from the within-$release directory if performing a
sideways upgrade within the same release. A symlink named
within-master is added and linked to the current development
release to allow sideways migration testing of trunk. When Juno
tags are cut and K opens, a from-kilo directoy should be created
and this symlink updated.
Change-Id: I770075442d05f38515e5dadf24f2c642d14d7845
style checks can now be run with 'tox'. This does not yet integrate
this to the gate, but it does clean up all the existing code to
work with the currently released bash8 pip.
Done as an easy example to create a pattern for other shell programs.
Change-Id: I5ecf28cfa2551f0cf3502263a379242c8803252c
stop-base already stop services, so don't need to do it again in
upgrade-nova. This is needed So we can keep the old nova-compute up for
rolling upgrade testing. Instead of killing the entire screen process
kill all the services inside of it.
stop-base is the preferred place to manage the stop logic, because it
has access to the base devstack's stop-* commands.
Change-Id: Ifa2b6305f1f6bbe41407ca12a0f3cbc99b09f301
systems may have a ton of apt repos on them, many of which aren't
really needed. We'd rather take the updates we can, and not fail
on secondary repositories, especially as secondary repository fails
will stop us dead in our tracks.
Closes-Bug: #1259911
Change-Id: I08b19cd4d48fba3b4884fa4bc011cde451136594
Any new configurations for Icehouse should go in one of the upgrade files in
from-havana directory.
Also moves {BASE,TARGET}_RELEASE vars forward to Havana->Icehouse pair.
Change-Id: Iac520a8de4c3b9f14e2289cfb9b6a50c4ab0ce32
This allow grenade to move and test from a brand new release to master without
any change.
This way it allows to enable d-g grenade testing new-release->master where for
a just released Openstack means that we don't need any new configuration yet.
Change-Id: I3e685deb61b95116420eae9cacb6b38dc405ffe9
From now on all releases configurations will be available, which will make
easier to test more than two versions upgrade (meaning test upgrades like
g->h->i).
Notes:
* from-folsom files will not be necessary since we don't test
folsom->grizzly.
* from-havana files will be added in the next patch(with the change-id follow
change-id Iac520a8de4c3b9f14e2289cfb9b6a50c4ab0ce32).
Change-Id: I3312732b8c344ce8844a82282a032e383e07e07c
Add necessary policies to support Cinder's volume migration, using a
new policy_add function which can be re-used.
Fixes: bug 1214569
Change-Id: I9baad5f933e14c5d3647786478992462d190f53e
to make grenade work like havana master we need to bring in infra
and oslo repositories from upstream git and apply global
requirements to all the repositories.
Update functions file from devstack.
Attempt to uninstall oslo.config first, to deal with weird pip
issues where 1.1.1 is still installed, but 1.2.x from git is in
the easy repo files, and nova-manage explodes.
Change-Id: I310d40a0adc9355eea91e79598bf867ad6818b79
We have some complex logic in here to try to do the right things with
the requirements before doing the install of the package which still
winds up being wrong in some cases. Since having written this code,
we've learned that the logic we're trying to achieve is actually what
pip install -e does. So just use that. We have to follow up with a chown
of the resulting egg-info directory, because the sudo command will cause
it to be written by root, which prevents subsequent commands from
operating without privilege in the directory.
Change-Id: I7454f0053c36b15051b616363a5fc04c0045b57c
After we supposedly shut down Folsom, actually check that it's all
the way down.
After we start grizzly, before we do anything, make sure that the
services look up, and the final vm that should have survived, did.
Change-Id: I13159c84444c3b3e9744acac7110b5cbd5ff9233