When GRENADE_USE_EXTERNAL_DEVSTACK is set to True,
the initial steps performed by grenade are skipped, namely:
- grabbing and configuring the base and the target devstacks;
- running devstacks on the base target.
This change is required to allow a native Zuul v3 job to use
the existing workflow to setup and run devstack.
Change-Id: I232db8de05141849c81851dd29440959cb0d8533
When grenade.sh is invoked by devstack-gate, PYTHON3_VERSION
is not set so grenadrc defaults it to 3.5, which breaks grenade
on nodes where python 3.5 does not exist, such as ubuntu bionic.
One solution is to set PYTHON3_VERSION in devstack-gate [1].
Alternatively, this change uses a bit of code borrowed from
stackrc in devstack to determine the version of python
within grenade if it was not set by the caller.
Closes-bug: #1812208
[1] https://review.openstack.org/#/c/607379/
Change-Id: I0349de2026c49279ba7f262d5e86d37018d66326
With the switch to uwsgi (for most services) just doing a ps and
grepping for the process name isn't as simple as it was when each
binary command was unique. When using uwsgi the command is 'uwsgi'
which isn't unique between processes. Luckily we've also switched
everything to run under systemd which gives us an easy alternative
to check if the service is running. This commit pivots the checking
to leverage systemd to check the status of the process instead of
using ps. If the unit file can't be found it will fallback to using
ps. Moving forward we'll delete that path once everything is updated.
Change-Id: I2bc2674c1c29f8970983a12d75ccb3d05fdb9103
In change I075eb5a88113acfa36519e2c6e2aab87836be065 to workaround the
potentially missing dbs in a cells environment it turned off exiting on
error. This is a bad idea since it will mask real failures. This commmit
fixes the function to check if the db exists before trying to do a
mysqldump and report whether it's missing or not.
Change-Id: I54aac82e412a298cd1d2066a0fa51131a6da2af0
Nova now has multiple databases, let's add them to the list
nova-api nova_cell0 nova_cell1
Since it's hard to detect which set of databases are present,
we should disable bailing out when trying to save these.
Change-Id: I075eb5a88113acfa36519e2c6e2aab87836be065
In some situations there are additional sections in the devstack
localrc file and they may be at the end of the file. This caused
errors as the desired changes would not be placed in the
'[[local|localrc]]' section of the file.
Using 'dsconf setlc_raw' solves that issue.
Change-Id: If4bc197d685e20fadbc5ffbd2151cae9d8129135
When the fetch_devstacks function runs as part of grenade.sh
it tries to install devstack tools. As part of the devstack tools
installation, python3 gets installed through a function called
install_python3 which uses a function called is_ubuntu to
determine if the system is ubuntu. However this is_ubuntu
function is not available at that moment, so the script fails.
This commit sources functions-common from the devstack directory
which includes is_ubuntu so the grenade.sh script does not break.
Change-Id: I92b9f731e99df0840be296bfa8ce6a70e02a1f91
Closes-bug: 1659081
A user can set ERROR_ON_CLONE to yes (true) to fail a run if a git
clone fails. This patch will document that behavior in grenaderc
and set the flag to its default value of no (unless a user overrides).
Change-Id: I1b1f80b00b4a4838ef10241306d9190520a93701
Add DSTOOLS_VERSION setting here. Eventually devstack-gate will set
this, but for now we can move forward without it.
Change-Id: I9bf5ac24dc87404e054068d7ed643e0bf0fac311
This attempts to provide local.conf support by making it possible to
have local.conf files during initial processing and concattenating
them to the end of the final local.conf.
This is not fool proof, but given the limited ways that we currently
use configs in devstack-gate, it's good enough to start the
transition.
Depends-On: Ia19d64ba240adc642284e4b942f6ce12310f862b
Change-Id: I3347c0886b1c2c8c56c902f485f45fb126446451
Some of the phases, for example 'verify', have an additional
component, for example 'pre-upgrade'. Show that detail in the summary
line generated.
Change-Id: Icfc346bfd0aa432a3c01f70bfc98c7e0b6790c88
mariadb does not use DATABASE_USER = 'root' by default.
New versions of MySQL flavor MariaDB do not allow remote access for
'root' user by default.
This patch enables non-root users by loading DATABASE_USER variable
from stackrc (which is loaded by MYSQL_USER in localrc). It keeps 'root'
user as default for MySQL.
Change-Id: I15f5f7ca1fe91d97b3f5bf268982b133513e5e3c
This makes the resource verify commands take a second argument that
indicates if we are pre-upgrade or post-upgrade. This may be needed
in order to validate things that will not be true before, but must
be true after an upgrade.
Change-Id: I6383c5f641bb8eb7fee6f1fbd41bca3a281d9636
Net delete will fail if there are still instances on this network,
destroy the resources reversely.
Change-Id: Ib0976352ae81ac94d8b31ebaed93758ca4210ec5
Closes-bug: #1488757
Partial upgrade support was a very hacky hybrid state of no stopping
some services so not everything is upgraded to new code levels. Now
that we have a multinode upgrade model in the gate we don't need this
any more, and we should burn this with fire so that more folks don't
go down this path.
Change-Id: I2b1b440aa3cccfb2b72e65e41809c359438024be
I noticed this in I45894911c0ee9cd6c9b142d6192350269873d82f where the
"old" database isn't dumped. This seems to be because "save-state"
calls inc/upgrade:save_mysql_dbs, but it doesn't have access to
DATABASES_TO_SAVE because it's not exported. An additional trick is
that save-state includes inc/upgrade, so be sure not to overwrite the
variable as blank.
Depends-On: I45894911c0ee9cd6c9b142d6192350269873d82f
Change-Id: I49626dcc277497eaf66d4d998ba8b2655293d28f
As a follow-on to Iba7739eada5711d9c269cb4127fa712e9f961695 in
devstack, where we rename the XTRACE storage calls to not conflict, do
the same in grenade
Also some minor turning-down of tracing when importing stackrc; it
floods the logs with irrelevant stuff. And now we're tracing
properly, cleanup "echo_summary" output in the logs.
Change-Id: Ibb67a6454e4465a8d0493461e32dbae96c6a03e7
Even though you've told grenade to UPGRADE a project, we still go and
check to see if it's enabled in devstack before we believe you. For
components that are fully implemented in the grenade and devstack
trees, this is fine. However for plugins this is weird, because
stackrc doesn't provide enough information to figure this out.
We should not do this second check, just trust your grenade config is
correct. This code was probably vestigial to getting the refactor
working correctly before the kilo release.
Change-Id: I01d00f95d602b9b159b3b64ff46a7cd3e23c3983
This allows us to set SERVICE_TIMEOUT per job to handle services which
might take longer to stop / start. We'll default to 30 as a reasonably
safe value.
Change-Id: I306a9ddfb957a947915eecc2d3121a15909f54d3
If there are whitespaces in the process names (such as a complete args
list), the current code does not work as it breaks on spaces. This fixes
it.
Change-Id: I4a2a982d4153a31971a5f1a1b504b96ce29236f9
Closes-Bug: #1484528
grenade/save-state need DATABASES_TO_SAVE to save the base release databases.
The DATABASES_TO_SAVE is updated by load_settings.
Change-Id: I023248e6416fa0e8163883226bc74a7db6f7919e
Closes-bug: #1487287
In devstack openrc -> stackrc -> functions
I think it's fair to say nobody is really thinking about idempotency
when they write these rc files; it's assumed they're sourced once
per-run. I3ca4e576aa3fb8992c08ca44900a8c53dd4b4163 is an example of
this.
Reduce some of the duplicate imports here by just importing the
top-level (either openrc if that's wanted, or stackrc)
Change-Id: I6fbae12f950a03afab39f341132746d3db9f788c
We still have an intermittent connectivity problem in grenade, doing
dumps during normal runs at points of success might help us figure out
what went horribly wrong.
Change-Id: I835071d04d3fa463a9c710c80850ae35f40f7aa5
Make the echo_summary code also print to the normal logs which makes
for easier understanding of what's going on.
Change-Id: I2fb7d7eba9912ca90988fc05c9353a918007f1cf
Implement an external plugin mechanism for grenade in the spirit of
devstack plugins. This assumes that external plugins will live in the
devstack/upgrade tree in their respective repository.
External plugins are enabled by adding
enable_grenade_plugin <name> <giturl> [branch]
to ``pluginrc`` at the top of the grenade directory. (``localrc``
could not be used due to timing issues on parsing)
a new ``devstack_localrc <base|target> ...`` is also added to grenade
that allows settings files to add content to base and target devstack
localrcs. This is quite critical for out of tree support, because for
upgrade testing there will often be a need to add services to the
localrc.
Partially tested on the existing Heat plugin implemented in
I0847004a8ac023a113cabaee46896abc823a01da, though that doesn't yet
fully pass due to conflicts in it's upgrade process.
Include documentation for out of tree best practices.
Change-Id: I02a777077d40408766204b05ed284fe98efbce8e
The upgrade scripts have a *ton* of historical cruft in them. This is
a bulk clean up patch that does the following things:
- ensures we source grenaderc before functions, we need this for
devstack dir definitions.
- moves xtraces until after sources to trim down irrelevant output
- removes as many variables and library sources as possible to ensure
that we include the minimum possible code in these scripts (for
sanity and readability).
This skips the neutron upgrade script, much like in devstack, the
neutron code has so many extra variables it's not clear what's really
needed.
Change-Id: Idac9190fd33478a479f7c7dd5cb1be894bb03444
When things fail to shut down look for open sockets to all services
not just keystone, this seems to be a problem with more projects.
Change-Id: I003115e72a29e4c7f201ef2017795ace6a885ead
This adds additional debug information to try to catch artifacts
for keystone-all failing to shut down. In future we'll generalize
it into worlddump, but right now it's good debug to keep around.
Change-Id: Id62fb2ddeae2001313e98e567f5c227a8f2fadc6
The grenade db is really just an ini file, but there is no reason why
it should persist in a second run of grenade, and it actually causes
issues if it does. Blank it out in the init phase to ensure we're
starting with a blank store.
Change-Id: I22a2b53739153011f5c250739278719259ad7860
This provides global variable comments for various function files, as
well as license info and the #!/bin/bash lines so that gerrit will
syntax highlight correctly.
Change-Id: Ice55aff1e39084a55e1a3da6d6418a103df4d04f
javelin is obsoleted for direct use in grenade by the resource plugin
scripts, so remove the coupling and reference to it.
Change-Id: Ieaf63f0162f8d87a96058e941fb5b1f3e750dd2b
This provides the infrastructure for modular resource checking, as
well as an example implementation for keystone.
The infrastructure includes:
- the resource script dispatcher, including the ability to disable
using it all together via config.
- resource_save / resource_get scripts for our grenadedb registry
(wrapper on ini files, but abstracted so we could do something
different in the future)
- keystone resource script.
Change-Id: I8666a96a817cfc333e50eed3b9aeffa5ac4aeec6
This is a smorgasbord cleanup patch which does the following:
- moves all the inline function definitions in grenade.sh out to
either bootstrap or upgrade libraries.
- gets rid of grenaderc.settings, this is handled by modular shutdown
/ upgrade service checking.
- consolidates all settings into grenaderc
- removes all variables that were extraneous, and came from very very
early history of grenade.
- Documents the current flow and assumptions in grenade.sh as much as
possible.
Change-Id: I3d2fa30240247690e3f45dcb0be93672808914f6
Move the logic of check-sanity, which looks for services that should
be started and logs that should exist, into the per service upgrade
scripts.
Change-Id: I3e5a90a3d2c47b8224a7e7ce3a45a19837872252
This breaks the content out of stop-base into a set of modular
shutdown.sh scripts so that every service can control it's own
shutdown. Services are shut down in reverse order under the theory
that will prevent services from calling services they depend on and
blocking their shutdown. Keystone had a habit of not going down if it
was first in the list.
We also need to manage TOP_DIR because many devstack scripts source
other devstack scripts by TOP_DIR. In the base case we make it
TARGET_DEVSTACK_DIR, however when we run shutdown we reference the
BASE_DEVSTACK_DIR instead.
This also moves all the stop-base service checks into the modular
shutdown scripts.
Depends-On: I9138a8370be06e6c9d2dacbec1b0ffd4928e603d
... as apparently stop_neutron does do anything useful in juno, and
the neutron project was relying on global pkills to make their
services actually shut down.
Change-Id: I02599b95bde61ade34220251e896f6d4c4122c22
let projects declare that they will have a database to be saved, and
we'll do it for them, in the mysql case. Eventually other saving
issues should be left to per project save-state.sh scripts.
services which aren't always enabled in grenade jobs have their
settings files guarded with conditionals. Eventually we'll do
conditional loading of these settings files, however for now this lets
the rest of the infrastructure around running actions based on
PLUGIN_DIRS and UPGRADE_SERVICES be *non* conditional, which
simplifies them.
Change-Id: I06cad95ca319e4403772e5fb394c0f8965e0bf9d
This moves all the project specific upgrade code into project specific
directories. This forshadows how we might break out this code into
external plugins. It happens all at once to keep a working grenade
environment on both sides of this change.
Change-Id: If47395c796bdb0ae2b8fe65bfe4b4556a6487c7a
This refactor's the grenade code to remove the copy/pasted
functions-common from devstack and instead use the functions from
TARGET_DEVSTACK. In order to accomplish this we need to refactor when
the devstack trees are pulled, so that we always pull both the base
and target devstack trees *extremely* early.
We also need to include a small number of functions (mostly related to
git) in grenade to be able to bootstrap these trees. This is done in
the inc/bootstrap tree.
There are some slightly unrelated function moves into inc/upgrade to
consolidate all the upgrade functions which were scattered through the
codebase for clarity. inc/* will be the contract library interfaces
out of grenade (mirroring devstack).
To do this grenaderc has to be split between variables which can be
set without devstack functions, and variables we compute later. These
later variables will be moved into project specific directories based
on the plugin plan.
Change-Id: I1a2d1a1fed7858c4f3b51a19be2d12cc2d260a24
Luigi Toscano