Commit Graph

397 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov 27078e06e8 Cleanup upgrade tasks
There were bunch of tasks for upgrading from lsyncd to the shared fs
mount, which can be safely cleaned-up now.

Change-Id: Ia0f5cbced196467007eafc61d3152ebea7559b84
2023-11-07 19:13:32 +00:00
Dmitriy Rabotyagov 6a2affb1a5 Ensure mounts are present only when they are expected to exist
At the moment there is a check, which ensures for race condition to
not happen between asking for a mount and mounting the point.
However, it tries to check for the mount when there is no mount
defined for the directory.

We add extra check to wait for the mount only when mounts are defined.

Change-Id: I900a55a6f4edce3d3fe419821c47cf56d641192f
2023-11-07 19:13:28 +00:00
Dmitriy Rabotyagov 2f8dc163a2 Fix example playbook linters
Change-Id: I81d63abe8e6c3010ddcffddac08956a6e88e480a
2023-11-07 20:13:05 +01:00
Dmitriy Rabotyagov 8ae6540d0b Fix linters and metadata
With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.

With that we also update metdata to reflect current state.

Change-Id: I74cefdfa885fa26dd7199fd0798527f511bf329d
2023-07-12 16:07:09 +02:00
Damian Dabrowski 2d0e465fd3 Add TLS support to repo_server backends
By overriding the variable `repo_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the repo_server backend.

The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: I5c5d3dd5689ac122781303ad21dacc8a1fa746eb
2023-04-28 11:27:09 +02:00
Damian Dabrowski 3d3f610245 Turn off absolute_redirect for nginx
Nginx adds trailing slashes to the URLs ending with directories.
So by default, when accessing http://172.29.236.101:8181/pools, nginx
will return 301 redirect to http://172.29.236.101:8181/pools/.
It's an absolute redirect which causes a problem when haproxy frontend
listens on HTTPS but its backends listen on HTTP.
In this case, when accessing https://172.29.236.101:8181/pools, nginx
will return 301 redirect to http://172.29.236.101:8181/pools/ (http)
that won't work.

This patch changes behavior by disabling absolute_redirects, so when
accessing https://172.29.236.101:8181/pools, nginx will return a
redirect to relative location '/pools/' without changing protocol.

Change-Id: I9e55508996d9b24437870f2f23dca5db7827fee1
2023-03-14 23:35:19 +01:00
Dmitriy Rabotyagov b9d9ccfc8a Update tox.ini to work with 4.0
With tox release of 4.0, some parameters were deprecated and are ignored now
which causes tox failures. One of the most spread issues we have is using
`whitelist_externals` isntead of `allowlist_externals`


Change-Id: Ib6f505093477a8b19ca8ae31ab7759904369a8c0
2022-12-27 17:53:34 +01:00
OpenStack Release Bot 87e601f951 Update master for stable/zed
Add file to the reno documentation build to show release notes for
stable/zed.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/zed.

Sem-Ver: feature
Change-Id: I89224e297ba377656774b3b62a176226558b6bc7
2022-12-13 13:23:31 +00:00
Kevin Carter f3fe518781 Convert include to include_tasks
Include is deprecated, additionally include_tasks is now faster.

Signed-off-by: Kevin Carter <kevin@cloudnull.com>
Change-Id: I20f92b972b794513cb774983f3aa6044769987d7
2022-09-03 21:14:29 -05:00
Jonathan Rosser a9ecec103d Restart nginx after removing old repo content
The version of nginx on centos-8 appears to keep file handles open,
possibly the old /var/www/repo directory persistently. Once the old
content is removed and the new shared filesystem mount is created at
the web root, ensure that nginx is restarted to close any file handles
which are now stale.

Change-Id: I941359b1b42aa4a874230a32b438dcefddfb2acb
2022-05-30 16:13:27 +01:00
Dmitriy Rabotyagov b0fe7fd14d Switch sphinx language to en
With sphinx release of 5.0.0, they changed default for language variable
to 'en' from None. With that current None valuable is not valid and should
not be used.


Change-Id: Ieb0129ffb8964eba380a5d829e4033f19c892451
2022-05-30 16:01:48 +02:00
Zuul 746c156311 Merge "Use distro packages for nginx on centos." 2022-05-30 09:17:45 +00:00
Zuul eafa818812 Merge "Use the same vars file for all versions of centos" 2022-05-29 20:36:50 +00:00
Jonathan Rosser e31bee556c Use distro packages for nginx on centos.
There is no longer any need to get these from EPEL or the
nginx repos.

Change-Id: I24a031b5e14359f08a231dfc3429468561d48126
2022-05-29 17:33:00 +00:00
Zuul ce74fc77b5 Merge "Avoid a race condition between mounting and using repo filesystem" 2022-05-27 19:24:27 +00:00
Zuul c9d33a7054 Merge "Clean up legacy lsycnd, rsync and ssh key config" 2022-05-27 17:32:45 +00:00
Zuul 8382f16001 Merge "Remove all code for lsync, rsync and ssh" 2022-05-27 13:57:54 +00:00
Jonathan Rosser 3b9a0f49a3 Avoid a race condition between mounting and using repo filesystem
Wait until the path is confirmed to be a mountpoint.

Change-Id: If220d073147d8f424cfe4f0d1ab494144b406860
2022-05-26 17:58:16 +00:00
Zuul 4e5947dc1e Merge "Add infra upgrade jobs" 2022-05-25 18:39:11 +00:00
Jonathan Rosser 961ceb258d Add infra upgrade jobs
Repo server has complexity with glusterfs and migration path from
lsyncd so we should cover this with an upgrade job.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/843267/
Change-Id: I11eb3f3208cbe3858b642dd86145a806983c692b
2022-05-25 15:03:20 +00:00
Dmitriy Rabotyagov 703563c59c Run content restore only on first host
Currently we're trying to restore content for each host independently

This is not needed as we already storing content on shared FS,
that's been mounted.

We don't implement variable since that's only for migration purposes.

Change-Id: I0795fa1936aaeb9b9005a35685d85b6a4a619835
2022-05-25 13:58:02 +02:00
Dmitriy Rabotyagov a5df0d1a9b Have a symlink to u_c versioned file
With current behavior of u-c file that is stored on repo server and
migration to shared FS instead of lsyncd, we don't have any reliable
path to check for.

Also we have issue, that with updated u-c SHA users catch "unrelated"
error in python_venv_build role.

As a solution, we can create a symlink to the u-c file, that will have
persistant name and be updated with repo_server role run.

That would give us both file to verify and it can be used as u-c URI.

Change-Id: Ie7bdb9137ed69be465f014fa811b750dbec4e428
Related-Bug: #1943978
2022-05-20 06:56:39 +00:00
Jonathan Rosser 309c235a8a Use the same vars file for all versions of centos
Change-Id: I6e61e5e226c22d84d0a427359979dbb4338e890d
2022-05-19 16:41:55 +00:00
Jonathan Rosser 45d07fcf0c Clean up legacy lsycnd, rsync and ssh key config
These are no longer needed when a shared filesystem is used to
syncronise the repo server contents

Change-Id: I3109bd891d4c6b3522f5f741d9970093b1e882c8
2022-05-19 16:41:32 +00:00
Jonathan Rosser 03b55edaae Remove all code for lsync, rsync and ssh
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/842571
Change-Id: I4f32c03179a1d8814548a92fc714a5fd9dd3f433
2022-05-19 16:33:18 +00:00
Jonathan Rosser 1f39aa0239 Add upgrade path from lsyncd to shared filesystem.
This patch checks if the repo server content directory is a mountpoint,
and if it is not, creates an archive of the repo server contents
before mounting the shared filesystem. After the mount completes, the
archive of repo server contents is copied to the shared filesystem.

This runs on every repo server in the deployment to include corner
cases of repo servers running different OS versions or CPU architecures
and will gather all of the content onto a single shared mount.

Change-Id: I976a5ea5f6b6ebd65c22e89657763fef87cf4b23
2022-05-16 15:29:59 +00:00
Jonathan Rosser c966363bd4 Add facility to store repo contents on a remote mount
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/837706
Change-Id: I9008680a5f41287599d67f4ce70605b60bccabf3
2022-04-20 06:55:06 +00:00
Zuul 5aefc76d47 Merge "Use ssh_keypairs role to generate keys for repo sync" 2022-04-05 21:50:29 +00:00
Jonathan Rosser 8bdf307151 Use ssh_keypairs role to generate keys for repo sync
This uses ssh signed certificates so there is no longer the need
to distribute the repo_server public key from each repo_server to all
other repo_servers.

The legacy scripts and authorized key files are removed as a
migration step.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/836377
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/825292
Change-Id: I27770f3a781bdf62d2a37659e087b12db2fb459e
2022-04-04 17:09:53 +00:00
Dmitriy Rabotyagov 4f3bec1680 Use /run/nginx.pid
In modern systems /var/run folder is just a symlink to
/run with potential to be removed in the future.

Currently, nginx might fail on Debian jobs because of that [1]

[1] https://zuul.opendev.org/t/openstack/build/54263ee3816c484d94505a629558218c/log/logs/openstack/aio1_repo_container-da3488c9/nginx.service.journal-12-02-06.log.txt

Change-Id: I19321a61cce978e781c3581fcc136609b5be8092
2022-04-04 17:46:14 +02:00
OpenStack Proposal Bot baeea3b398 Updated from OpenStack Ansible Tests
Change-Id: I8b0204c86093c37a80eee0d0493c1016ca10db63
2022-03-29 17:19:51 +00:00
Zuul 4f1f7b0387 Merge "Ensure insist=true is always set for lsyncd" 2022-02-10 15:31:13 +00:00
Jonathan Rosser ee0a6d5b37 Ensure insist=true is always set for lsyncd
If insist is not set to true then lsyncd will exit if it cannot
perform an initial rsync to the target hosts.

Due to the order in which the repo servers are configured, lsyncd
may be installed and started on the first host in the repo_servers
group before the ssh keys and other necessary configuration have
been placed on the remaining hosts. This leads to a failure to
start lsyncd.

This patch moves the setting of insist into the lua config file
for all operating systems, and removes the need to template a
defaults file on debian derivatives.

Change-Id: I26bb0e21d797c2bfbe67e03003da01c355c27561
2022-02-10 09:47:24 +00:00
Jonathan Rosser a91f79a43b Refactor use of include_vars
Use a first_found lookup instead of a with_first_found loop so that
the 'paths' parameter can be used.

This ensures that only vars from the role are included, and not vars
from a parent calling role. This can happen when a parent role has
a higher priority vars file available for inclusion than the role
it calls.

Change-Id: I14afe3040d8ecd8702edca19877ab50b6e57007f
2022-01-12 08:03:04 +00:00
OpenStack Proposal Bot 0ff7578897 Updated from OpenStack Ansible Tests
Change-Id: I3b1f72ae1739ca6b10713ad6bf2ff6d5e36f64f2
2021-12-17 16:49:38 +00:00
Dmitriy Rabotyagov 28b2f59e35 Cleanup CentOS 7 vars file
Change-Id: I983bdc86f61768b1e1cfb713a220c75ddf8ec5ec
2021-08-20 17:52:30 +03:00
Dmitriy Rabotyagov 3b402db6b1 Replace linters test with integarted one
We've created integrated linters check job a while back and it's successfully
working for several releases. At the moment we experience difficulties
with future maintenance of the linters check from the openstack-ansible-tests
repo. So instead of fixing current one, we replace it with modern version of
the test.


Change-Id: I73ff28f76aed64cf527cd4c58d25954b5e864f20
2021-05-21 16:07:06 +03:00
OpenStack Proposal Bot 7f3caeb047 Updated from OpenStack Ansible Tests
Change-Id: I2d9f195d9374e14ee39ad103a9f280a0046584f0
2021-03-22 08:51:49 +00:00
Zuul 8f105a7c72 Merge "Allow constraints files to be hosted on the repo server" 2021-03-17 09:41:48 +00:00
Jonathan Rosser dfe91c5ac0 Remove references to unsupported operating systems
All references to Gentoo, SUSE, Debian stretch and Centos-7  are removed.
Conditional tasks, ternary operators and variables are simplified where possible
OS specific variables files are generalised where possible

Change-Id: Ibe914d0d547b930b8167bfa475cc838df8d7ae25
2021-03-16 08:51:12 +00:00
OpenStack Proposal Bot 1a8c0e75aa Updated from OpenStack Ansible Tests
Change-Id: Ifd11fb59ef62021f03dcdc0335b80ed436e7b5b6
2021-03-12 22:22:38 +00:00
Jonathan Rosser aab7090e4d Use ansible_facts[] instead of fact variables
See https://github.com/ansible/ansible/issues/73654

Change-Id: I3e48000a4685d4df46cd60113ce4c0c02b63dc0c
2021-02-23 09:24:07 +00:00
Jonathan Rosser 4400c9203f Allow constraints files to be hosted on the repo server
Change-Id: I2a28a180e0bb947da2b091ec0671a48ef857e8e5
2021-02-22 23:56:14 +00:00
dmitriy 46c4b8e104 [reno] Stop publishing release notes
Since we copy all release notes to the integrated repo there is not need
in publishing release notes for each repository. We should only verify their
validity and linting.


Change-Id: Icd69125b861d292bfdbfca264e3867e4acf9b469
2021-01-22 18:28:44 +02:00
Marc Gariepy 64c683aa13 Fix order for removing nginx file.
the default config needs to be removed after the pkg is installed.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/766030
Change-Id: Ibb9ccd8b85f673103f3ce863a8eb0641ebb056fb
2020-12-10 07:44:42 -05:00
Dmitriy Rabotyagov 0e38f6e575 Use infra jobs for testing
Infra jobs are more appropriate for testing infra resources, such
as repo server.

Depends-On: https://review.opendev.org/660477
Change-Id: Ie696c1b6b6732d0a59f03351d3135554afc14ce4
2020-11-02 18:59:18 +02:00
Zuul 6860cc39cc Merge "Allow remote detection of repo sync status" 2020-10-31 14:57:14 +00:00
Zuul 720f37fbda Merge "Updated the git source" 2020-10-29 00:07:29 +00:00
Georgina 7132acbd3b Allow remote detection of repo sync status
If a repo container and its data are deleted and recreated then it is
not currently possible for a loadbalancer healthcheck to differentiate
between an empty repo server and a correctly synchronised one.

This patch creates a file 'repo-sync-complete' as part of the process
of synchronising repo contents from master repo servers to slaves. The
presence of this file on the slave can then be used as the loadbalancer
healthcheck to ensure that repo contents are only served once sync has
completed.

In addition, this patch ensures that synchronisation occurs from the
master to a reprovisioned slave by triggering a master repo server lsyncd
restart handler during the initial setup of the slave repo server.
Currently, a freshly provisioned repo server will remain empty
for an indeterminate amount of time, this patch forces a complete re-sync
to occur.

Change-Id: I6913341674dbde5524c2270e824bda4544211eca
2020-10-27 13:37:21 +02:00
OpenStack Proposal Bot 7f58e1cc61 Updated from OpenStack Ansible Tests
Change-Id: Ia2206bbb2a5a4634b327bac8e779d12d3b80e5d2
2020-09-24 16:59:21 +00:00