We fixed the tempest venv constraints setting for tempest test run
on base branch but forgot to do the same for target branch test run.
It cause issue when base and target branch have different Tempest
and constraints to use (for example stable/wallaby use old tmepest
and stable/wallaby constraints but stable/xena use tempest master
and master constraints), in such cases we need to set proper constraints
defined in devstack and then run tempest.
We do not see this issue as none of the job run tempest on target except
heat (https://review.opendev.org/c/openstack/heat/+/872055/5).
I reproduced the issue in grenade stable/xena by enaling the tempest run
on target:
- https://review.opendev.org/c/openstack/grenade/+/878247/1
- https://zuul.opendev.org/t/openstack/build/1b503d359717459c9c77010608068e27/log/controller/logs/grenade.sh_log.txt#17184
Closes-Bug: 2003993
Change-Id: I05fcdb5045a33997bd1a4c340a16458d88399f5f
Grenade set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
which are used to use the constraints during Tempest virtual env
installation. Those env var are set to non-master constraint when
we need to use non-master constraints but when we need to use the
master constraints we do not set/reset them point to master constraints.
This create the issue when grenade script install the tempest second time
on new devstack where we need to use master constraints but env var
TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE set previously in tempest
installation in old devstack point to the stable constraints.
One case where it started failing when we tried to pin the stable/wallaby with
Tempest 29.0.0
- https://review.opendev.org/c/openstack/devstack/+/871782
and stable/xena grenade job failed (stable/xena use master tempest and supposed
to use master constraints)
- https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641
We should set/reset those constraint env var to master constraints if configuration
tell to use the master constraints.
Closes-Bug: #2003993
Change-Id: I8c623008292f3b2dd2cd6d596feea552194acd89
The initial source is too early when GRENADE_USE_EXTERNAL_DEVSTACK
is False (as, e.g., in DSVM [aka devstack-gate, aka legacy] jobs).
The issue was introduced in
I66b5d128c3dbe51289410760ea1cbf7f61c55c2f
fixing the Tempest upper constraints to be properly pinned.
It did not affect Zuul v3 (aka new) jobs.
Also, let's only source the one variable that we need rather than
the whole file.
See https://review.opendev.org/c/openstack/devstack/+/785547
for an example failure.
Closes-Bug: #1923343
Change-Id: If5f14654ab9aee2a140bbfb869b50d63cb289fdf
We use Tempest master for testing the supported stable
branches, so using master upper constraints works fine for
such master or stable branch testing, but when we need to
use old Tempest in the below cases then master upper constraints
do not work for Tempest venv:
- Testing Extended Maintenance branch
- Testing py2.7 jobs until stable/train with in-tree tempest plugins
grenade run smoke test on old devstack and if old devstack does not
use the master constraints for Tempest then we need to take care of
the same while running the tests too otherwise, it will end up recreating
the Tempest venv and flush the already installed tempest plugins
and their deps.
This commit set the compatible upper constraint
according to TEMPEST_VENV_UPPER_CONSTRAINTS variable, which is
what used in devstack while creating the Tempest venv and install
tempest plugins in that venv.
Closes-Bug: 1922597
Change-Id: I66b5d128c3dbe51289410760ea1cbf7f61c55c2f
Setup a native Zuul v3 grenade-base job that defines base folders and
base devstack settings. The grenade play checks out repositories
in two locations, old and new, and sets up devstack config
in old and new as well.
Define a grenade job that sets up devstack services.
This job runs:
- devstack from grenade_from_branch (without tempest)
- grenade from grenade_to_branch
- projects in old from grenade_from_branch
and then tries to run grenade and then tempest.
The configure-grenade-branches role sets the base/target
branch variables (grenade_{from,to}_branch) and must be
updated when cutting a new branch.
Also, define the native versions of the grenade-postgresql
and grenade-multinode jobs, replacing non-native jobs
(like neutron-grenade and neutron-grenade-multinode) when possible.
Even though Python 3 is now the default, define grenade-py3
for compatibility reasons.
Finally, define a basic grenade-forward job which should
be used for forward upgrade testing. Forward testing
requires the user to set the destination branch,
and it is relevant for stable branches only, so disable it
from master.
At least for legacy grenade jobs, Zuul seems to be taking care
of Depends-On on the stable branch.
It is worth noting that tls-proxy is set to False as
it happens with the current legacy jobs. It does not
work by just flipping it to true.
Co-Authored-By: Luigi Toscano <ltoscano@redhat.com>
Depends-On: https://review.opendev.org/637523
Depends-On: https://review.opendev.org/649275
Change-Id: Iefe8d1d7d13bb56cbc9e80fb009d19218f8b1a64
When GRENADE_USE_EXTERNAL_DEVSTACK is set to True,
the initial steps performed by grenade are skipped, namely:
- grabbing and configuring the base and the target devstacks;
- running devstacks on the base target.
This change is required to allow a native Zuul v3 job to use
the existing workflow to setup and run devstack.
Change-Id: I232db8de05141849c81851dd29440959cb0d8533
Setting WSGI_MODE was removed in change
Ib504ab21dfc5e32eb3f73f57d636981963e20520, but left the comment
about setting it, which blows my mind, man, so let's remove it.
Change-Id: I50e5be2a661bcec73a78246684c334c23ea01aa8
Tempest is in the process of switching the tempest run command to use
stestr internally instead of testrepository. This means that results
will be stored in .stestr instead of .testrepository now. This commit
updates the base job tempest run post-processing to check which
directory exists before attempting to copy the results dir.
Change-Id: Ic1fa3a98b6bcd151c489b078028687892655a19b
In the pike release we migrated everything to use uwsgi. There is no
need to force everything back to eventlet. This was necessary when pike
was master because in ocata we used eventlet. We don't want to switch
deployment mechanisms in an upgrade. But now that pike is base we don't
need to have any of these since everything is consistent on both sides
of the upgrade.
Depends-On: I066f5f87ff22d7da2e3814f8c2de75f2af625d2b
Change-Id: Ib504ab21dfc5e32eb3f73f57d636981963e20520
* Stop nova-conductor explicitly if it is still running
(also filed a devstack specific change in
I9ffd6d09df6f390a842b8a374097f144564d2db4)
* Run keystone, cinder, nova etc under mod_wsgi as we run
into problems with uwsgi (need to fix those but not
right now during the release process)
* Make sure we use singleconductor as grenade we don't yet
support multiple rabbitmq(s) in the multinode scenario
* Hack to pass through some of the variables from above
to the additional node before it runs stack.sh as
the defaults won't work (superconductor etc)
Depends-On: I075eb5a88113acfa36519e2c6e2aab87836be065
Depends-On: I9ffd6d09df6f390a842b8a374097f144564d2db4
Change-Id: If4c82ca12fe7b8b1ca7cfd8181d24dbd8dad3baa
This ensure that for Pike we will not use the new uwsgi
functionality for the cinder-api service, ensuring that
we continue to use the default infrastructure over the
release transition. This can be removed when both cycles
are using uwsgi.
Change-Id: I14b68f36e7fcc5341bbdbcf165274d9d50f7dd04
This ensure that for Pike we will not use the new uwsgi
functionality for the nova-api anod nova-metadata services, ensuring
that we continue to use the default infrastructure over the
release transition. This can be removed when both cycles are using
uwsgi.
Change-Id: I7b22692ea9135547d93b08724f1f9ef060f13007
With changes to the way that openrc is working, especially with
keystone not just being a simple host / port combo, we need to copy
over out stackenv to the new side.
Change-Id: Ie59c6f824ae0801b2373c50eb2047f05a008790e
This is needed because we're changing up some things in devstack, and
this ensures the pike defaults don't flip the deployment mode mid way
through an ocata upgrade.
Change-Id: I5aa1e3017365df120fa8c8ea676c63ddf99f21b4
Add DSTOOLS_VERSION setting here. Eventually devstack-gate will set
this, but for now we can move forward without it.
Change-Id: I9bf5ac24dc87404e054068d7ed643e0bf0fac311
This attempts to provide local.conf support by making it possible to
have local.conf files during initial processing and concattenating
them to the end of the final local.conf.
This is not fool proof, but given the limited ways that we currently
use configs in devstack-gate, it's good enough to start the
transition.
Depends-On: Ia19d64ba240adc642284e4b942f6ce12310f862b
Change-Id: I3347c0886b1c2c8c56c902f485f45fb126446451
This flag is not used in any project gates, but is used for testing
grenade locally. This patch documents that fact, so situations like
https://review.openstack.org/#/c/342275/ don't occur again.
Change-Id: I2ae9b2f2c9d611aedd008df1ee35588442db041b
On Tempest side, some scenario tests are smoke tests, but all scenario
tests don't have to be smoke. Grenade just runs smoke tests.
So current comment seems misleading, this patch removes the word.
Change-Id: Iaec4106c1a89d34930b0962865e96fb622715264
This makes the resource verify commands take a second argument that
indicates if we are pre-upgrade or post-upgrade. This may be needed
in order to validate things that will not be true before, but must
be true after an upgrade.
Change-Id: I6383c5f641bb8eb7fee6f1fbd41bca3a281d9636
For projects that need to setup custom networking we need an
early_create phase that has only gotten the base neutron network setup
which they can then work with and make changes as appropriate.
Create this new hook point, and move the neutron create to this early
phase.
Change-Id: Ifb1903f8d3cb11bc4db33f8acdea342e6373f864
As a follow-on to Iba7739eada5711d9c269cb4127fa712e9f961695 in
devstack, where we rename the XTRACE storage calls to not conflict, do
the same in grenade
Also some minor turning-down of tracing when importing stackrc; it
floods the logs with irrelevant stuff. And now we're tracing
properly, cleanup "echo_summary" output in the logs.
Change-Id: Ibb67a6454e4465a8d0493461e32dbae96c6a03e7
init_grenade_db will clean the db file. If we are running grenade -t
we expect the db have resources saved at base run. so move it down to
run only for base installation.
Change-Id: I39fcee54db08c163352d314d68e014191a43d07c
Closes-bug: #1488008
After the main stack.sh is called, provide the ability for a
post-stack script to be called. This can assist in supporting
multinode testing.
Change-Id: I42c2d8ae88a5abe871f286eb9f59b320376056f4
Currently rsync output is very verbose. So drop the -v and -P flags.
(search grenade.sh for 'sending incremental file list')
-v, --verbose increase verbosity
-P same as --partial --progress
--progress show progress during transfer
--partial keep partially transferred files
Change-Id: I0492329d7c39279ccc6877a87eb3853ae423a99c
Implement an external plugin mechanism for grenade in the spirit of
devstack plugins. This assumes that external plugins will live in the
devstack/upgrade tree in their respective repository.
External plugins are enabled by adding
enable_grenade_plugin <name> <giturl> [branch]
to ``pluginrc`` at the top of the grenade directory. (``localrc``
could not be used due to timing issues on parsing)
a new ``devstack_localrc <base|target> ...`` is also added to grenade
that allows settings files to add content to base and target devstack
localrcs. This is quite critical for out of tree support, because for
upgrade testing there will often be a need to add services to the
localrc.
Partially tested on the existing Heat plugin implemented in
I0847004a8ac023a113cabaee46896abc823a01da, though that doesn't yet
fully pass due to conflicts in it's upgrade process.
Include documentation for out of tree best practices.
Change-Id: I02a777077d40408766204b05ed284fe98efbce8e
Instead of recreating the tox and testrepository environments from
scratch, copy the contents of the base runs into the target
runs. Tempest tox creation takes about 2 minutes, so this should drop
2 minutes from all the grenade runs.
Change-Id: I4dfc33183ab69c4984a0d5bd234a4b767ee68228
javelin is obsoleted for direct use in grenade by the resource plugin
scripts, so remove the coupling and reference to it.
Change-Id: Ieaf63f0162f8d87a96058e941fb5b1f3e750dd2b
This provides the infrastructure for modular resource checking, as
well as an example implementation for keystone.
The infrastructure includes:
- the resource script dispatcher, including the ability to disable
using it all together via config.
- resource_save / resource_get scripts for our grenadedb registry
(wrapper on ini files, but abstracted so we could do something
different in the future)
- keystone resource script.
Change-Id: I8666a96a817cfc333e50eed3b9aeffa5ac4aeec6
This is a smorgasbord cleanup patch which does the following:
- moves all the inline function definitions in grenade.sh out to
either bootstrap or upgrade libraries.
- gets rid of grenaderc.settings, this is handled by modular shutdown
/ upgrade service checking.
- consolidates all settings into grenaderc
- removes all variables that were extraneous, and came from very very
early history of grenade.
- Documents the current flow and assumptions in grenade.sh as much as
possible.
Change-Id: I3d2fa30240247690e3f45dcb0be93672808914f6
There were a remarkable number of scripts that we ran during the
target setup phase, that all boiled down to very small
activities.
- upgrade-devstack - copied 1 file
- upgrade-infra - cloned gr
- upgrade-oslo - actually *did nothing* (now that we install
libraries from pypi)
Consolidating them all into a unified prep-target makes understanding
what's happening on the target setup simpler.
Change-Id: I0a1a41714fd9749857ccfc0825414f331f44ec2a
Move the logic of check-sanity, which looks for services that should
be started and logs that should exist, into the per service upgrade
scripts.
Change-Id: I3e5a90a3d2c47b8224a7e7ce3a45a19837872252
This breaks the content out of stop-base into a set of modular
shutdown.sh scripts so that every service can control it's own
shutdown. Services are shut down in reverse order under the theory
that will prevent services from calling services they depend on and
blocking their shutdown. Keystone had a habit of not going down if it
was first in the list.
We also need to manage TOP_DIR because many devstack scripts source
other devstack scripts by TOP_DIR. In the base case we make it
TARGET_DEVSTACK_DIR, however when we run shutdown we reference the
BASE_DEVSTACK_DIR instead.
This also moves all the stop-base service checks into the modular
shutdown scripts.
Depends-On: I9138a8370be06e6c9d2dacbec1b0ffd4928e603d
... as apparently stop_neutron does do anything useful in juno, and
the neutron project was relying on global pkills to make their
services actually shut down.
Change-Id: I02599b95bde61ade34220251e896f6d4c4122c22
let projects declare that they will have a database to be saved, and
we'll do it for them, in the mysql case. Eventually other saving
issues should be left to per project save-state.sh scripts.
services which aren't always enabled in grenade jobs have their
settings files guarded with conditionals. Eventually we'll do
conditional loading of these settings files, however for now this lets
the rest of the infrastructure around running actions based on
PLUGIN_DIRS and UPGRADE_SERVICES be *non* conditional, which
simplifies them.
Change-Id: I06cad95ca319e4403772e5fb394c0f8965e0bf9d
This moves the exit trap into grenade/functions, which means all
grenade scripts will get the exit trap registered on them. This also
adds stack tracing into the scripts so that we can determine where we
were when we failed.
Change-Id: I4100628180498ff8723a1af6c0a4c10b4d8244c8
This moves all the project specific upgrade code into project specific
directories. This forshadows how we might break out this code into
external plugins. It happens all at once to keep a working grenade
environment on both sides of this change.
Change-Id: If47395c796bdb0ae2b8fe65bfe4b4556a6487c7a
A bunch of the trueorfalse usage was wrong, and thus could not be
overridden. Also, if tempest is attempted to upgrade when the smoke
tests haven't run, it doesn't actually work. So adjust that
conditional appropriately.
Change-Id: I847a8c2ca7cd044cf02e18321eba00b6cc55ebed
This refactor's the grenade code to remove the copy/pasted
functions-common from devstack and instead use the functions from
TARGET_DEVSTACK. In order to accomplish this we need to refactor when
the devstack trees are pulled, so that we always pull both the base
and target devstack trees *extremely* early.
We also need to include a small number of functions (mostly related to
git) in grenade to be able to bootstrap these trees. This is done in
the inc/bootstrap tree.
There are some slightly unrelated function moves into inc/upgrade to
consolidate all the upgrade functions which were scattered through the
codebase for clarity. inc/* will be the contract library interfaces
out of grenade (mirroring devstack).
To do this grenaderc has to be split between variables which can be
set without devstack functions, and variables we compute later. These
later variables will be moved into project specific directories based
on the plugin plan.
Change-Id: I1a2d1a1fed7858c4f3b51a19be2d12cc2d260a24
Introduce variable that is a list of projects to upgrade, this becomes
a setting that can be overridden by out of tree projects.
Change-Id: I84be46fcdc544c61e09bc5e1238f54f53a63d893
Ghanshyam Mann