Commit Graph

50 Commits

Author SHA1 Message Date
Damian Dabrowski 2d0e465fd3 Add TLS support to repo_server backends
By overriding the variable `repo_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the repo_server backend.

The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: I5c5d3dd5689ac122781303ad21dacc8a1fa746eb
2023-04-28 11:27:09 +02:00
Damian Dabrowski 3d3f610245 Turn off absolute_redirect for nginx
Nginx adds trailing slashes to the URLs ending with directories.
So by default, when accessing http://172.29.236.101:8181/pools, nginx
will return 301 redirect to http://172.29.236.101:8181/pools/.
It's an absolute redirect which causes a problem when haproxy frontend
listens on HTTPS but its backends listen on HTTP.
In this case, when accessing https://172.29.236.101:8181/pools, nginx
will return 301 redirect to http://172.29.236.101:8181/pools/ (http)
that won't work.

This patch changes behavior by disabling absolute_redirects, so when
accessing https://172.29.236.101:8181/pools, nginx will return a
redirect to relative location '/pools/' without changing protocol.

Change-Id: I9e55508996d9b24437870f2f23dca5db7827fee1
2023-03-14 23:35:19 +01:00
Jonathan Rosser 03b55edaae Remove all code for lsync, rsync and ssh
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/842571
Change-Id: I4f32c03179a1d8814548a92fc714a5fd9dd3f433
2022-05-19 16:33:18 +00:00
Jonathan Rosser ee0a6d5b37 Ensure insist=true is always set for lsyncd
If insist is not set to true then lsyncd will exit if it cannot
perform an initial rsync to the target hosts.

Due to the order in which the repo servers are configured, lsyncd
may be installed and started on the first host in the repo_servers
group before the ssh keys and other necessary configuration have
been placed on the remaining hosts. This leads to a failure to
start lsyncd.

This patch moves the setting of insist into the lua config file
for all operating systems, and removes the need to template a
defaults file on debian derivatives.

Change-Id: I26bb0e21d797c2bfbe67e03003da01c355c27561
2022-02-10 09:47:24 +00:00
Jonathan Rosser aab7090e4d Use ansible_facts[] instead of fact variables
See https://github.com/ansible/ansible/issues/73654

Change-Id: I3e48000a4685d4df46cd60113ce4c0c02b63dc0c
2021-02-23 09:24:07 +00:00
Georgina 7132acbd3b Allow remote detection of repo sync status
If a repo container and its data are deleted and recreated then it is
not currently possible for a loadbalancer healthcheck to differentiate
between an empty repo server and a correctly synchronised one.

This patch creates a file 'repo-sync-complete' as part of the process
of synchronising repo contents from master repo servers to slaves. The
presence of this file on the slave can then be used as the loadbalancer
healthcheck to ensure that repo contents are only served once sync has
completed.

In addition, this patch ensures that synchronisation occurs from the
master to a reprovisioned slave by triggering a master repo server lsyncd
restart handler during the initial setup of the slave repo server.
Currently, a freshly provisioned repo server will remain empty
for an indeterminate amount of time, this patch forces a complete re-sync
to occur.

Change-Id: I6913341674dbde5524c2270e824bda4544211eca
2020-10-27 13:37:21 +02:00
Dmitriy Rabotyagov 1ac51ddea7 Bind services to mgmt network addresses
These addresses are given defaults of 0.0.0.0 in the role defaults
but in a deployment we know which address each service should bind to.

The variable repo_server_bind_address should hold the mgmt network IP
address for either containerised or metal deployments and drives the
bind addresses where necessary.

Co-Authored-By: Jonathan Rosser <jonathan.rosser@rd.bbc.co.uk>
Change-Id: Iff95282b91a94d22fc8f6cdbadefacb53cae5b79
2020-06-04 06:55:14 +00:00
Matthew Thode 220a481071
add missing quote to lsyncd.lua template
Change-Id: I0c56df1d184553a6128c9645811e7ed6950e893b
2020-05-29 09:42:20 -05:00
Dmitriy Rabotyagov 748d86411b Remove git deamon functionality
Caching git repositories has been deprecated in Queens, so it's high time
we removed this functionality.
This shouldn't influece OSA deployments in any way.

Change-Id: I35829aa35489f06dbb3b65f522f0a08318eccbfa
2020-05-02 16:53:16 +01:00
Zuul a4a5d87ce9 Merge "lsyncd.lua.j2: update coding style" 2020-04-06 17:55:11 +00:00
Erik Berg 860224c64e Exclude repo_build_global_links_path from sync
These files are only used by the repo_build during wheel requirements
and should only be needed and used on the repo_container they're on.

When you're transitioning from one distribution release to another,
e.g. xenial -> bionic, syncing the links directory between these
repo_containers can break the wheel building in weird ways.

Depends-on: I3bd6d3d987e32ee11c5f1fcb5c1b4b0fc797e7f9
Change-Id: Iaa2e52b26ba89802e06665ebe43fdf18e515abd7
2020-03-12 21:14:19 +01:00
Erik Berg 547b79fae4 lsyncd.lua.j2: update coding style
Updating this file to better match the coding style currently
used in the file it was based on;
https://github.com/axkibe/lsyncd/blob/master/default-rsync.lua

Change-Id: I75281da097784b55748722b0b16957fb823b9fdd
2020-03-12 10:05:02 +01:00
Erik Berg 9cb92db21b lsyncd: use relative paths in exclude statement
It seems only relative paths should be used in the exclude
statement. This has probably gone unnoticed since there is a second
sync for these files.

Change-Id: Ife4c1ab05e135930da8706d96af5b23648cee800
2020-02-03 10:46:52 +01:00
Mohammed Naser 44547c7b7b pypiserver: drop pypi server
It is no longer needed because of how we are using python_venv_build
at the moment, so let's remove it.

Depends-On: https://review.openstack.org/648477
Change-Id: I56531388fb49a8c3d098fd762392299742b0e120
2019-03-29 10:02:36 +00:00
Jesse Pretorius 6663637374 Remove apt-cacher-ng
The repo container's package cache causes quite a bit of confusion
given that it's a 'hidden' feature which catches deployers off-guard
when they already have their own cache configured. This is really
the kind of service which people should manage outside of OSA. It
also makes no sense if the deployer is using their own local mirror
which is a fairly common practise. Adding to that, it seems that it
is broken in bionic, causing massive delays in package installs.
Finally, it also adds to quite a bit of complexity due to the fact
that it's in a container - so in the playbooks prior to the container's
existence we have to detect whether it's there and add/remove the config
accordingly.

Let's just remove it and let deployers managing their own caching
infrastructure if they want it.

Change-Id: I829b9cfa16fbd1f9f4d33b5943f1e46623e1b157
2018-10-08 14:48:32 +01:00
Jesse Pretorius dab934bdb9 Remove the upstream pypi reverse proxy
Trying to reverse proxy upstream pypi has not turned out to
be very stable, or very useful. We've had many, many reports
of stability issues and the additional complexity for offline
and proxy usage is just not worth it.

Given we already have a mechanism in place to handle using
upstream pypi if the repo server is not there yet, disabling
this should just result in that mechanism kicking in and all
will be well again.

Once the repo is built, the reverse proxy to pypiserver will
then be exclusively used and the upstream pypi proxy is not
necessary anyway.

Depends-On: https://review.openstack.org/584393
Change-Id: Ie407c6a346de6b46c8f4d30caea8664a7f6bd341
2018-07-20 14:25:19 +00:00
Mohammed Naser d125a8a58f Remove dependency on EPEL
The EPEL repositories currently conflict with the RDO packages,
this commit drops it from openstack_hosts so new installs do not
get it.

Change-Id: Iecc021a294befb64f54d73cc926faeea21cba372
2018-06-07 14:00:09 -04:00
Jonathan Rosser bcb29bd5c5 Support devpi as the upstream pip server
devpi serves packages from url starting +f/. The devpi source code
suggets that +e may also be used but is not seen when using devpi
as a caching proxy.

Change-Id: Ib391d17e5038a355a558aa3f041ed58ede7dad4a
2018-05-11 13:34:14 +01:00
Jesse Pretorius 09a058f3f7 Use correct protocol for pypi reverse proxy
When reverse proxying pypi the current implementation uses
https to communicate upstream. This works just fine if the
upstream pypi server is serving data via https, but causes
the handshake to fail if the upstream pypi server is serving
via http instead.

This patch implements a check to validate the upstream pypi
server port set and adapts the reverse proxy configuration
appropriately.

Change-Id: I1a986fef5bf1e069212bc432c3a775be15df11ef
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2018-03-11 09:54:07 +00:00
Jesse Pretorius a340308444 Allow the upstream pypi mirror to be changed
Currently the upstream pypi mirror is hard set to
pypi, but sometimes it is preferred to use a different
mirror. This allows the upstream mirror to be changed.

Change-Id: Icd93c0c801bfee1b4fdc8154d078067722c0640a
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2017-12-04 11:11:10 +00:00
Jean-Philippe Evrard 2b5f1f5cc0 Fix swift port conflict
To avoid any port conflict, whether on LB or on bare metal nodes,
we should make sure each service runs on its own port.

The 8080 port is already used by swift, and opendaylight.
We keep 8080 for swift, move opendaylight to 8180, and the
pypiserver to 8280 to avoid overlaps when everything runs on metal.

Closes-Bug: 1735764
Change-Id: I69dd043efe5d2e50e83014bdbd6a848bfcc2aa39
2017-12-02 17:10:37 +00:00
Jesse Pretorius 6320c00217 Implement pypiserver and pypi proxy cache
This patch implements nginx as a reverse proxy for python
packages. The initial query will be to a local deployment
of pypiserver in order to serve any locally built packages,
but if the package is not available locally it will retry
the query against pypi and cache the response.

Depends-On: Id20a43fed833d53ca0f147f517deafba6587352d
Change-Id: Ic4fd64f4dc82121a65088f3d7f4ae53f373df608
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2017-11-24 11:58:18 +00:00
Jimmy McCrory 94255aa931 Reload nginx during syncs
Instead of stopping nginx during a sync of repository files, temporarily
disable the openstack-slushee site. This friendlier behavior will allow
for the repo server to share a web server with other sites and services.

Change-Id: I95cbd104d948f51cbff6a8396d1ba115ad99d7c6
2017-11-21 08:06:11 +00:00
Major Hayden f20c70fbba
Escape periods in DontCache regex
This patch ensures that periods are escaped in the DontCache
regex so that they avoid matching things they shouldn't.

Change-Id: If27857cf7f5c22cb466ce3b94c34f272ff7df94d
2017-07-21 08:01:47 -05:00
Major Hayden dc80ec4412
Add comment for acng DontCache line
This patch adds a comment that explains the DontCache regular
expressions in the apt-cacher-ng configuration file.

Change-Id: Idbce7a5446f10482ab01fb4d45a62561b79c1520
2017-07-20 07:34:45 -05:00
Major Hayden 639594217d
Don't cache MariaDB repodata
yum occasionally gets 503's when retrieving MariaDB's repodata bz2
files through apt-cacher-ng. This patch causes apt-cacher-ng to
stop caching the repodata files and allow them to pass through.

Change-Id: I76d70c2ccbc1d6dc17f7de5bef21aeeab6efc4bd
2017-07-19 10:14:00 -05:00
Jenkins 79353be533 Merge "Merge mirror list for centos." 2017-07-17 20:36:20 +00:00
Marc Gariepy b9892084f1 Adding DontCache for mirrorlist.centos.org
Dont cache mirrorlist.centos.org to prevent hitting:
- Could not retrieve mirrorlist * -14: HTTP Error 503 - Service Unavailable

Change-Id: I3fd41f192f8178f0b73250fc60a9309c15a0a1bb
2017-07-17 14:44:11 -04:00
Marc Gariepy e65301a755 Merge mirror list for centos.
Merge mirror list for centos

Change-Id: I0c6916ccf7a0dacaaae6753ef06d567fa1cc0206
2017-07-17 18:39:13 +00:00
Major Hayden c43efe58fc
Add LSYNCD_OPTIONS for CentOS
The lsyncd daemon on CentOS can't find the location of its
configuration file because the LSYNCD_OPTIONS variable is
missing in the defaults file.

Closes-bug: 1689965
Change-Id: I1dc24570f0d724cfe7b3338e04ad8cf50ed8b558
2017-05-10 19:20:05 -04:00
Ravi Gummadi 33e7ad0400 Cap the number of worker threads
Users can configure the number of worker threads however when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs.

Change-Id: Idda6e476e9e9b5842c4cc03e9853ec31d123abc5
2017-02-23 05:45:07 -05:00
Jenkins aae3f2ba69 Merge "Fix apt-cacher-ng file owners during rsync" 2016-12-21 12:38:48 +00:00
Kyle L. Henderson 1169edc47b Fix apt-cacher-ng file owners during rsync
The lsyncd service runs as the 'nginx' user such that files sync'd
from the master node to the backups will have 'nginx' as the owner.
However, the apt-cacher-ng service needs to be the owner to function
properly. This fix consolidates the pre and post sync tasks into
a script that can be called by lsyncd. The script can then change
the file owners as needed before and after the rsync.  The owners
need to be 'nginx' before the rsync so that lsyncd can update
files and 'apt-cacher-ng' after the sync so the cacher service works.

Additionally, setup lsyncd to sync each service's directory separately
rather than being rsync'd all together. This avoids lsyncd bouncing
services when their respective files are not being sync'd.

Change-Id: Ifaba17b89035398917f2b3257574e18eb9027c08
Closes-bug: #1649339
2016-12-19 18:48:53 -06:00
Andy McCrae 997047b558 Remove Trusty Support from repo_server role
Change-Id: Ib5e24fcc7509a312ca8ee6c5811c3f194f16d662
Implements: blueprint trusty-removal
2016-12-15 15:32:32 +00:00
Kyle L. Henderson 55eb0f8830 Wait for lsyncd rsync to finish successfully
Add logic to the lsyncd rsync configuration to wait for a
successful rsync event before issuing the postcmd. The existing
code issued the postcmd immediately after spawning the rsync,
which caused a race condition.

Change-Id: I412b5ed7762d825c345a2e2afa6f6088d69ba6b4
Closes-bug: #1649760
2016-12-13 21:55:35 -06:00
ArchiFleKs b000586d14 Add proxy configuration for apt-cacher-ng
apt-cacher-ng does not use environment variable and proxy needs to
be set in apt-cacher-ng configuration file.

Change-Id: I0f9b7e3dc0a24a13a290d286100f9da672ca4d37
2016-12-06 19:31:39 +00:00
Marc Gariepy ab30c01607 Add VfilePatternEx to apt-cacher-ng for centos.
This will enable it on 16.04 and CentOS, version of apt-cacher-ng on 14.04 is a bit too old.

Change-Id: Ie5d3efb6908fd664efb4effc3cbce04183f9c3c0
2016-11-14 12:13:10 -05:00
Ravi Gummadi 31c8937271 Fix errors due to repo_service_user_name setting
Changing the paths of nginx logs to not rely on the
 user name of repo service. This is because nginx related
 configurations, playbooks assume nginx in the directory names.

Change-Id: I458293c687c1857e7d8451a200173bec554d9559
Closes-Bug: #1633739
2016-10-23 05:08:56 -05:00
Jesse Pretorius (odyssey4me) e80911be67 Revert "add VfilePatternEx to apt-cacher-ng for centos."
This reverts commit 95ac11e15c.

Change-Id: I56d51b1bcd0a327b11df15b08b5a160db4974011
Closes-Bug: #1633936
2016-10-21 12:07:29 +00:00
Marc Gariepy 95ac11e15c add VfilePatternEx to apt-cacher-ng for centos.
Centos mirror fail with 403 when using mirrorlist.centos.org. Adding
the pattern to match the url.

https://www.pitt-pladdy.com/blog/_20150720-132951_0100_Home_Lab_Project_apt-cacher-ng_with_CentOS/

Change-Id: I64bc7c2bd85a8b4c5ca03564c1671967f5b197a6
2016-09-28 14:00:40 +00:00
Kevin Carter a888f8fd07 Convert role testing to use Ansible 2.1.1
Change-Id: I40cecfc65daeee8e0c45a8cf47f27289097d9e93
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-09-14 15:53:35 +00:00
Jesse Pretorius 6cad87f853 Disable slave git servers while syncing
Now that the git repository is served from a different process
it is possible for there to be a race condition where the slaves
do not yet have the updated copy of the data before they get a
request for it.

This patch ensures that the git service is shut down on the slave
nodes during the synchronisation of repo data.

Change-Id: I83650253924354d0e63621474ba69f8cf254bde8
2016-09-08 15:38:14 +01:00
Jimmy McCrory c87a8c1d4c Remove fastcgi and related configuration
In I62321a7b62dabca469eb072ddbf4e8f250ce0fb3, git daemon was added to
support hosting git repos from the repo server over the git protocol.
When the integrated build transitions to using it, fastcgi and all
related configuration can be removed.

Depends-On: I09bc504490d4b5114895f7f646fc8254748a7f41
Change-Id: I7ec8277d3883d1f8891de6ae2b0881fe026a34c8
2016-07-26 16:15:49 -07:00
Kevin Carter 02e58dfda8 Implemented package caching on the repo server
This change implements package caching on the repo server.
To take advantage of this a deploy will need to do nothing more
than setup an apt-proxy configuration file. This will speed up
package delivery while also providing ha capabilities within the
environment.

Change-Id: I78b2fba6a1f294751bd7098513060015cb41300c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-07-22 10:17:54 +00:00
Jimmy McCrory f59bafd778 Updated role for multi-distro support
Separate files have been created for vars and tasks related to a
specific package manager.

The 'repo_apt_packages' variable has been deprecated and renamed to
the more generalized 'repo_server_packages' to better describe its
purpose and to simplify reuse of existing install tasks between multiple
distros.

git daemon is configured to host git repositories from the repo servers
using the git protocol.

Currently, openstack-ansible uses git over http to access repositories
on servers created by this role.
fcgiwrap and its configuration within nginx should be removed in a
follow-up patch after openstack-ansible has been updated to use the git
protocol.

Change-Id: I62321a7b62dabca469eb072ddbf4e8f250ce0fb3
2016-05-13 10:48:57 +00:00
Hugh Saunders bbc038c87b Add insist option to lsyncd
Without insist, lsyncd will quit if any of its slaves are unavailable at
startup. This enforces a strict restart ordering - the master repo
server must come up last.

In order to remove this restriction the -insist option is added to
lsyncd. This causes lsyncd to retry the initial connection to each slave
so that even if the master comes up first, connections will eventually
be established.

Change-Id: I12e3dca147b5cb25ed982d5aceb9d521728c4e53
Closes-Bug: #1572433
2016-04-20 09:13:01 +01:00
Jenkins 26e68251ea Merge "Trim apt package list" 2016-03-17 16:04:37 +00:00
Jimmy McCrory b0836d3b22 Trim apt package list
Limit repo_apt_packages to a more minimally viable list of requirements
for deploying Nginx servers and syncing files between them.

Change-Id: I677c78473b7f0442f8c334cd59b8c676973f4535
Depends-On: I03d5c061ec506a9dc142ff55a50fb3ecb18c238f
Closes-Bug: #1550418
2016-03-16 07:40:44 +00:00
Hugh Saunders b457f3bda6 Disable slave repo servers while syncing
Currently there is a race between the repo servers syncing and the first
role that attempts to install a pip package. This change ensures that
only the primary repo server is accessible until the slaves are synced.

This is achieved by adding a hook into lsyncd that allows a command to
be run before and after each sync. This command is an ssh command to
connect to the relevant secondary container and stop/start nginx. As the
nginx user is unprivileged, a sudoers file is added to allow nginx to be
stopped and started.

Notes on adding the hook into lsyncd:
 * There is an existing script in lsyncd/examples for postcmd. This
   works at a higher level by adding an event onto the stack for executing a
   command once the sync has finished. I experimented with that but
   events dont get fired for the initial recursive sync, only on
   subsequent changes. As it is the initial sync that causes the problem
   that this patch is addressing, I had to look at a lower level.

 * The lsync lua C lib has an exec function, but it is hidden from
   config scripts except through the spawn(...) function. However spawn
   requires an event so can't be used for the initial sync.

 * I ended up going outside the lsync framework and using lua's own
   os.execute() function for pre/post cmds.

While this looks like a big patch, its actually a relatively small
change to the default rsync script. See
https://github.com/hughsaunders/lsyncd/compare/master...hughsaunders:rsync_prepost
for a comparison.

Bug: #1543146
Change-Id: I045a4a6bf722d6f1e01d21fbbec733872acb87a5
2016-03-16 07:19:20 +00:00
Kevin Carter 725222afee
first commit
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2015-12-09 09:25:37 -06:00