Commit Graph

1182 Commits

Author SHA1 Message Date
Jonathan Rosser e910144dd2 Bump ansible version to 2.15.9
Depends-On: https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/910477
Change-Id: Iea39f6859cd85cf4ff2e9686ff33a905f7ea33c7
2024-03-14 08:56:29 +00:00
Jonathan Rosser ccdb5932b0 Determine if upgrade source branch is stable/ or unmaintained/
Branch renaming to unmaintanted/ breaks the gate-check-commit
script when it sets the source branch for an upgrade job.

This patch determines the source branch prefix dynamically in
an attempt to make upgrade jobs still work from branches marked
as unmaintained under wider OpenStack policy.

Change-Id: I9662156fe0e9463e54ccc5c6ede0624a85887ebe
2024-03-07 14:11:28 +00:00
Zuul 52c04d4624 Merge "[doc] Update dist upgrade guide for 2023.1 / Ubuntu Jammy" 2024-02-12 15:26:13 +00:00
Andrew Bonney ded73432b8 [doc] Update dist upgrade guide for 2023.1 / Ubuntu Jammy
Change-Id: Ic708385ce2a75e42e9a70fa53dbd6030ba318a7c
2024-02-12 08:51:04 +00:00
Jonathan Rosser 10cb93ca2b Extra PIP_OPTS in bootstrap_ansible script must be space separated
Currently PIP_OPTS concatenate with no spaces and result in an
invalid set of command line options for pip.

Change-Id: I07b88856d8c1c30843eed1f991d475b431d0486f
2024-01-24 09:05:07 +00:00
Dmitriy Rabotyagov 5b57f10eec Add support for extra Python packages inside Ansible runtime
At the moment there is no well defined way to supply list of extra Python
requirements for Ansible venv. However, some collections for their work might
require presence of extra libraries inside the execution environment.

While PIP_OPTS might be used right for that, it's usage is not transparent
or documented.

In order to handle such need, `user-ansible-venv-requirements.txt ` is being implemented
that reside inside OSA_CONFIG_DIR and contains list of required packages
for installation when running bootstrap-ansible.sh

Change-Id: Ic99f7eff200e2e672dcc3edd875b155af84232b7
2024-01-11 14:35:09 +00:00
Zuul 3414202400 Merge "Switch back to track master" 2023-12-30 01:10:16 +00:00
Zuul 5f9173cb75 Merge "Refactor log collection from lxc containers" 2023-12-14 12:24:55 +00:00
Dmitriy Rabotyagov 6d4ca83f2c Switch back to track master
Once we've branched it's time to switch back to tracking master and continue 2024.1 development

Change-Id: I15a2e706ec97c617475b2a2f13022147adf94ee1
2023-12-14 09:47:31 +00:00
Jonathan Rosser 475621bb26 Allow installing collections from repos containing more than one
A git repo may contain one collection, several collections at
the top level or be organised into namespace/collection named
directories. See [1].

The '#' symbol is used to identify a particular collection from
a repository to install. This patch ensures that the paths
searched in CI for zuul source directories do not include parameters
beyond the '#' symbol, and that the contents of the collection
requirements file is also correct for installing any required
collections using the 'dir' type.

[1] https://docs.ansible.com/ansible/latest/dev_guide/developing_collections_distributing.html#specifying-the-collection-location-within-the-git-repository

Change-Id: I081212819c61b46c2ad143025e7287cd3ce5390a
2023-12-05 14:04:28 +00:00
Jonathan Rosser 9a9c8c734f Refactor log collection from lxc containers
The previous code retrieved logs from the lxc containers by copying
them from the /proc/<pid>/root/... directory which worked except
when the files being collected were symlinks, which would then
potentially point to the wrong, or a missing file on the host itself.

This patch changes the log collection to collect from inside the
running containers using tar, giving a consistent view of the
source files. Any symlinks are replaced with the content of files
they reference in the collected logs.

A side effect of using the more complex lxc-attach/tar approach
was corruption of stdout due to the previous simple method of
parallelising the log collection by running each collection function
in the background. This patch switches to use gnu parallel
as the mechanism to run multiple log collections simultaneously
and keep the console output clean.

Change-Id: Ief03c33b76eac6e256477cfaa16c8a59acd7d702
2023-11-13 17:50:18 +01:00
Zuul 2e93f05e09 Merge "Do not try to install packages on each log_instance_info run" 2023-11-13 16:10:20 +00:00
Dmitriy Rabotyagov 6bd9c19cf7 Do not try to install packages on each log_instance_info run
Function log_instance_info is being called from gate-check-commit script
after each major step. With that we update cache and try to install
packages that are already installed.

Change-Id: Iaacadb3785b834bc9288d498a3a3c7028bd64c5b
2023-10-25 18:23:02 +00:00
Jonathan Rosser 6efe795998 Bump ansible version to 2.15.5
Change-Id: I883c5d254d0209a7ed8d354a8ba3ad9a42bcee85
2023-10-25 13:13:41 +01:00
Dmitriy Rabotyagov 49b268634a Do not fail add-compute.sh script if exit code is 4
Right now exit code 4 stands for hosts being unreachable [1], that might
take place in envoronments at the moment of adding new computes

So instead of failing the script and asking for user to re-run commands
manually we silently proceed when anything fails with unreachable hosts.

[1] 74f6e6a134/lib/ansible/executor/task_queue_manager.py (L128C5-L128C26)

Change-Id: Ib422f181256a67cbde4b485c1f1e9079d0400faf
2023-10-18 12:28:31 +02:00
Jonathan Rosser 5910a88c95 Always use on-disk openstack service git repos in CI jobs
This is not dependant on if we are running a regular or an upgrade
job. Move the code to a more logical location.

Change-Id: Ibdb29a679504ce30bcdc7591b8e68c054dbea580
2023-10-10 06:59:38 +00:00
Zuul 68643fb110 Merge "Allow deployment on debian bookworm hosts" 2023-10-04 19:13:55 +00:00
Dmitriy Rabotyagov 4130be36d0 Bump ansible-core to 2.15.3 and ansible-lint
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/895688
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/895682
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/895681
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/895689

Change-Id: I38f91c716cfdb82eaec20ab3c7de6fcda363a11d
2023-09-19 10:11:44 +00:00
Jonathan Rosser b00b4db3e2 Allow deployment on debian bookworm hosts
Change-Id: I1a2d8e6ecbc12168971b3983308ffaf537b87d19
2023-09-11 17:23:22 +01:00
Jonathan Rosser 7ba58e15fd Add default name for user collections file
The get-ansible-collection-requirements playbook cannot be used
standalone without passing an extra var for the name of the
user collections file.

This patch allows the playbook to be run standalone and the
default name will be used automatically, still allowing an override
to be used as currently happens in scripts/bootstrap-ansible.sh

Sonetimes it is desireable to have a simple way to update the
installed collections without running other parts of
bootstrap-ansible, or look up flags to disable other side-effects.

Change-Id: I27131ecaa3bc0ef459d2c353be9531608dd9d9ab
2023-08-30 18:06:48 +01:00
Zuul 3d19e6fb71 Merge "Allow to update AIO config prior to an upgrade" 2023-07-22 14:24:26 +00:00
Dmitriy Rabotyagov 6204256ff4 Allow to skip roles/collections bootstrap separately
At the moment we control need of installing both roles and
collections by `SKIP_OSA_ROLE_CLONE` environment variable.

This patch introduces SKIP_OSA_COLLECTION_CLONE variable in addition
to that, to enable operators more finely manage what needs
bootstrapping.

Change-Id: I07374bb52bebe27c94c229f1e759016cd34b286d
2023-07-12 14:58:57 +00:00
Damian Dabrowski 230004a38a Allow to update AIO config prior to an upgrade
Upgraded OSA version may contain some updates to the default config
files used in CI.
For some scenarios it may be required to apply these changes prior to
the upgrade.

I tried to re-execute the whole bootstrap-aio.sh script but then I got
some issues with OVN agents after an upgrade. They were down until I
restarted neutron-server. So it is safer to only run
'prepare-aio-config' tag.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/885187
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/885188
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/885189
Change-Id: I1be9c669e9f4c996d587ef52a6d384d89b75a3f6
2023-07-06 18:44:13 +02:00
Dmitriy Rabotyagov d458b1f46a Return PIP_OPTS for load_nodepool_pip_opts
At the moment for some scenarios, like linters, where ansible bootstrap
is skipped, PIP_OPTS are undefined and linters fail due to that.

With patch we define PIP_OPTS to an empty value if
we're not in CI.

Alternatively we can patch scenarios independently
not to rely on existance of PIP_OPTS or export default in
gate-check-commit instead.

Change-Id: If451b4bc12bb0b0bfe3fe20494e99a07b59ab798
2023-07-06 18:36:24 +02:00
Zuul 42b7c85528 Merge "Ensure management_address is used instead of ansible_host" 2023-06-19 16:16:25 +00:00
Zuul ad5a3c1339 Merge "Start 2023.2 (Bobcat) development" 2023-06-07 20:36:00 +00:00
Dmitriy Rabotyagov b97c4860eb Start 2023.2 (Bobcat) development
Depends-On: https://review.opendev.org/c/openstack/tempest/+/884584
Change-Id: I124942ace678765b01bcb642798bdca6d14f3141
2023-06-06 17:08:01 +00:00
Damian Dabrowski 0eb9ccab3a Allow to pass BOOTSTRAP_EXTRA_PARAMS to bootstrap-aio.yml
This patch allows to pass custom parameters to the bootstrap-aio.yml
playbook by the `BOOTSTRAP_EXTRA_PARAMS` variable.
With `BOOTSTRAP_OPTS` it was only possible to set extra variables.
As an exmaple, `BOOTSTRAP_EXTRA_PARAMS` can be used to define custom
set of tags that should be executed.

Change-Id: Ie53b51f4e4d5ccbd0e7ff638a9d5bce236a561bd
2023-06-04 16:43:32 +02:00
Dmitriy Rabotyagov 9fca8555df Move repo_packages to group_vars
At the moment it's not possible to apply different versions of
services to the different groups due to playbook vars having
prescedence over group_vars. However, it can be quite important
to  such use cases, for example for phased rollouts of newer versions.

This will also reduce amount of unnecessary variables that are included
for each host, since only required git details will be loaded.

Closes-Bug: #2007296
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/881954
Change-Id: Icaa3a958926d9f9aa6cb649bd9f3da9449dd7490
2023-05-18 08:49:51 +00:00
Zuul c1d63ccd9e Merge "Add Yoga upgrade jobs" 2023-05-02 13:35:58 +00:00
Zuul 9c46f11de7 Merge "Fix dstat run in gates" 2023-04-27 21:33:21 +00:00
Dmitriy Rabotyagov 9f56dc611a Add Yoga upgrade jobs
This patch aims to implement testing of SLURP upgrades. While
2023.1 is the first SLURP release, upgrade from Yoga to 2023.1 is
considered as unofficial SLURP upgrade.

With that we're adding changes to gate-check-commit to be able to
provide source release from which upgrade will be perfromed.

While running upgrade script user is required to provide source
release as input to script for SLURP releases.

Also regular upgrades are now switched to Zed as they supposed to be.

Change-Id: I64e55b2c685782d23bb84e7e9f7c60708c276cc3
2023-04-25 07:34:43 +00:00
Damian Dąbrowski 4dca2883f7 Revert "Skip haproxy with setup-infrastructure for upgrades"
This reverts commit befd8424e2.

Reason for revert:

This change is not needed anymore for A release, especially with
separated-haproxy-service-config feature in place.

It creates a problem now, because haproxy-install.yml is responsible for
creating "base" haproxy service which is required for horizon to work.
If horizon does not work during setup-openstack.yml, tempest tests
may fail(along with the whole upgrade job in our CI)

That's why we need to run haproxy-install.yml, traditionally during
setup-infrastructure.

Change-Id: I5531e56610a8a63f601867d236831ce7d5e4d583
Needed-By: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879517
2023-04-18 20:20:37 +00:00
Dmitriy Rabotyagov aef89acf21 Install zuul-provided collections as directories
At the moment we don't really use collection version that is
provided by depends-on. This results in testing different content
that should be. Presumably, this happens because ansible-galaxy
does install from default branch HEAD, while zuul does check-out
to refs when using depends-on. To ensure that git repo state is not
the issue, we switch type to be directory for collections that are
contained in Zuul.

Change-Id: I7a0e2f7702a9f8630ed916523b9f0e6c2bbc202b
2023-04-11 17:04:55 +00:00
Zuul c8b22b37be Merge "Add documentation on refreshing hosts file" 2023-04-04 19:09:42 +00:00
Zuul 7c0ea4f0d6 Merge "Drop `echo` from add-compute.sh script" 2023-04-04 14:49:00 +00:00
Dmitriy Rabotyagov 03319fc690 Add documentation on refreshing hosts file
Since Yoga release we've wrongly have dropped hosts file refresh task
This task is still required, as despite /etc/hosts now are fully provisioned
on new computes, it still does not make old ones to recognize new
ones.

[1] 8cd65feae2

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/877824
Change-Id: I494caa4a4104129781159ea924f3aeae94dad9a8
2023-04-04 09:28:37 +00:00
Dmitriy Rabotyagov cae3e206e0 Fix dstat run in gates
With previous commit [1] a condition was introduced on when to run
dstat. Though, ZUUL_PROJECT env variable used is defined
only in post jobs, while dstat should be started in pre-jobs and then
collected with logs. So we adjust condition to start up dstat in gates
at the first place.

[1] 0369605a1c

Change-Id: I7fd4a846360847496b1db7b505b35b9eff57ebbe
2023-04-04 09:22:20 +00:00
Dmitriy Rabotyagov 333108a5e3 Drop `echo` from add-compute.sh script
Due to this `echo` that has been added during script testing,
reall commands are printed to stdout instead of being executed.

Change-Id: I2abe15526b630cd7c68c3b79c148525e16fa2563
2023-03-17 16:44:34 +01:00
Jonathan Rosser d408b96898 Bump ansible to latest release of 2.13
Update to 2.13.8

Change-Id: I2a5a5a7d315a0e0d172c38348983d58fb0724c33
2023-03-13 12:58:46 +00:00
Zuul 34761f07e8 Merge "Disable fact variables" 2023-03-07 18:16:28 +00:00
Zuul 400578ed40 Merge "Remove support for calico ml2 driver." 2023-03-07 18:03:54 +00:00
Zuul 17a37653e6 Merge "Do not use openstack.osa.linear strategy plugin" 2023-03-01 21:20:27 +00:00
Jonathan Rosser 9ca06ea5e6 Disable fact variables
This patch disables automatic generation of ansible_<var>
variables which are duplicate of ansible_facts['var'].

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/860424
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/860480
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/866119
Change-Id: I7f39aa2c628761d5f56eea7eb8d05e715efd00e9
2023-02-23 11:14:16 +00:00
Jonathan Rosser dc1f76c823 Remove support for calico ml2 driver.
The deployment of etcd as a service on the control plane is no
longer needed as calico was the only user of this.

The etcd-server role remains in our requirements as it is used
internally as part of the Zun playbook.

Change-Id: I2a158fd2b85ec0e637071ed4ef7c123a6583ecc0
2023-02-23 12:13:55 +01:00
Damian Dabrowski 8a05eda535 Do not use openstack.osa.linear strategy plugin
Custom linear plugin was added long time ago.
Nowadays it causes issues with loop conditionals.
It's not really needed these times. Everything works fine without it.
I also didn't notice any performance degradation after disabling it on
my AIO.

Closes-Bug: #2007849
Change-Id: I566004cfeaa4d562265f22adc044cff3458430d1
2023-02-22 21:39:12 +00:00
Zuul 878206ee6b Merge "Allow git servers for openstack services and tempest to be overridden" 2023-02-21 21:42:43 +00:00
Damian Dabrowski 68cb6f950a Install curl by defining binary that is provided
Currently, curl conflicts with curl-minimal package on Rocky:
package curl-minimal-7.76.1-19.el9_1.1.x86_64 conflicts with curl
provided by curl-7.76.1-19.el9_1.1.x86_64

To avoid complex logic between curl/curl-minimal we just install
package by providing a path to the binary it should provide.

Based on the distro this will result in instalation of either
curl or curl-minimal.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/872973
Change-Id: Ia1b657ae6a21d9c3be499e7ecf6b36c11c9f08d9
2023-02-08 08:39:57 +00:00
Jonathan Rosser df4758ab1b Allow git servers for openstack services and tempest to be overridden
This will allow the services to be cloned from github or a local
mirror by setting a small number of variables rather than overriding
every git repo URL individually.

Change-Id: I750d897e9e3c8ca161c0740c73cdc4c6e42b6440
2023-02-01 14:57:14 +00:00
Dmitriy Rabotyagov 6a0646470a Ensure management_address is used instead of ansible_host
ansible_host in deployments is designed to represent SSH address.
In cases, when address that used for SSH is different from management
network, this might lead to undpredictable results on bare metal hosts.

Thus we're ensuring that management (container) address is used
for services to listen on and to interact through.

Related-Bug: #2002645
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/870113
Change-Id: I8a5e817d024eef5453fa072d8fee5aeca9bed67b
2023-01-23 16:07:48 +01:00