With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.
With that we also update metdata to reflect current state.
Change-Id: I74cefdfa885fa26dd7199fd0798527f511bf329d
By overriding the variable `repo_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the repo_server backend.
The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: I5c5d3dd5689ac122781303ad21dacc8a1fa746eb
These addresses are given defaults of 0.0.0.0 in the role defaults
but in a deployment we know which address each service should bind to.
The variable repo_server_bind_address should hold the mgmt network IP
address for either containerised or metal deployments and drives the
bind addresses where necessary.
Co-Authored-By: Jonathan Rosser <jonathan.rosser@rd.bbc.co.uk>
Change-Id: Iff95282b91a94d22fc8f6cdbadefacb53cae5b79
The apt-cache was removed a long time ago and these tasks should
already have been removed in the Train release. Remove them now.
There is no change for existing deployments as these tasks are no
longer required as the apt-cache component of the repo server is already
retired.
Change-Id: I42785ee153955eb6e72bef01c26791ae8eec2087
Caching git repositories has been deprecated in Queens, so it's high time
we removed this functionality.
This shouldn't influece OSA deployments in any way.
Change-Id: I35829aa35489f06dbb3b65f522f0a08318eccbfa
These files are only used by the repo_build during wheel requirements
and should only be needed and used on the repo_container they're on.
When you're transitioning from one distribution release to another,
e.g. xenial -> bionic, syncing the links directory between these
repo_containers can break the wheel building in weird ways.
Depends-on: I3bd6d3d987e32ee11c5f1fcb5c1b4b0fc797e7f9
Change-Id: Iaa2e52b26ba89802e06665ebe43fdf18e515abd7
For Centos, we use by default public repo for nginx. You can change this
behaviour with these role-wide variables:
- repo_centos_nginx_mirror
- repo_centos_nginx_key
Or with these osa-wide variables:
- centos_nginx_mirror
- centos_nginx_key
Change-Id: I8dcb3c97e9593877a4a420bd32b50ae29d9d311c
When using a custom repo with repo_centos_epel_mirror, you maybe need to
change the gpg key url because offline env
You can use this variable: repo_centos_epel_key
Change-Id: I9bb305d866f3d65653b95a25a5b9f5ecde5af0b0
It is no longer needed because of how we are using python_venv_build
at the moment, so let's remove it.
Depends-On: https://review.openstack.org/648477
Change-Id: I56531388fb49a8c3d098fd762392299742b0e120
The repo container's package cache causes quite a bit of confusion
given that it's a 'hidden' feature which catches deployers off-guard
when they already have their own cache configured. This is really
the kind of service which people should manage outside of OSA. It
also makes no sense if the deployer is using their own local mirror
which is a fairly common practise. Adding to that, it seems that it
is broken in bionic, causing massive delays in package installs.
Finally, it also adds to quite a bit of complexity due to the fact
that it's in a container - so in the playbooks prior to the container's
existence we have to detect whether it's there and add/remove the config
accordingly.
Let's just remove it and let deployers managing their own caching
infrastructure if they want it.
Change-Id: I829b9cfa16fbd1f9f4d33b5943f1e46623e1b157
Given that the openstack_hosts role installs pip and virtualenv,
we do not need this extra meta-dependency and extra task/var.
Change-Id: Iac9f72586f6b26bd31d59a4fa5055687ff77f78b
Trying to reverse proxy upstream pypi has not turned out to
be very stable, or very useful. We've had many, many reports
of stability issues and the additional complexity for offline
and proxy usage is just not worth it.
Given we already have a mechanism in place to handle using
upstream pypi if the repo server is not there yet, disabling
this should just result in that mechanism kicking in and all
will be well again.
Once the repo is built, the reverse proxy to pypiserver will
then be exclusively used and the upstream pypi proxy is not
necessary anyway.
Depends-On: https://review.openstack.org/584393
Change-Id: Ie407c6a346de6b46c8f4d30caea8664a7f6bd341
The folder from which pypiserver will serve the wheels that are
pre-built in the repo-build process was incorrectly set, resulting
in pypiserver never, ever serving the wheels but instead always
forwarding on to the nginx reverse proxy to fetch things from
pypi.
Now that the root folder is correctly set, two things will happen:
* Wheels will be served from pypi, negating the need to use the
pip_links, so [1] should pass once this merges.
* Once the repo build is complete, the packages available to
install will be restricted properly to only the packages
built by the repo build process.
[1] https://review.openstack.org/549012
Change-Id: I16706d399f6b026a6d0004fd07e5f18605a7b5db
We shouldn't use virtualenv-tools, and it shouldn't appear
in our documentation.
Change-Id: I168e400ab8176bc94e48822284441368d8e73441
Partial-Bug: #1741634
The pypi server is able to use our existing built wheels. Because
the pypi-server recursivly scans the provided directory, all wheels
for from any source can be fed into the server making it a universal
solution for serving python packages; no matter the distro or
archetecture.
To ensure system performance the pypi-server package has been changed to
include the optional cache functionality it provides. As noted in the
server docs, enabling caching will greatly improve performance when
serving thousands of packages.
> The documentation for pypi-server is all contained within the
application, use `pypi-server -h`. That said, under bullet point 4,
here - https://pypi.python.org/pypi/pypiserver#table-of-contents -
an online copy exists.
Change-Id: Icc2ee264fc213b258642b5393dd78b1b26ef0542
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Currently the upstream pypi mirror is hard set to
pypi, but sometimes it is preferred to use a different
mirror. This allows the upstream mirror to be changed.
Change-Id: Icd93c0c801bfee1b4fdc8154d078067722c0640a
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
To avoid any port conflict, whether on LB or on bare metal nodes,
we should make sure each service runs on its own port.
The 8080 port is already used by swift, and opendaylight.
We keep 8080 for swift, move opendaylight to 8180, and the
pypiserver to 8280 to avoid overlaps when everything runs on metal.
Closes-Bug: 1735764
Change-Id: I69dd043efe5d2e50e83014bdbd6a848bfcc2aa39
This patch implements nginx as a reverse proxy for python
packages. The initial query will be to a local deployment
of pypiserver in order to serve any locally built packages,
but if the package is not available locally it will retry
the query against pypi and cache the response.
Depends-On: Id20a43fed833d53ca0f147f517deafba6587352d
Change-Id: Ic4fd64f4dc82121a65088f3d7f4ae53f373df608
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
The variable repo_auto_rebuild appears to have been
introduced in kilo, but never used.
The 'Init reload' handler is no longer required as
all services are now managed via systemd and the
service module.
The 'reload ssh' handler is also unused.
Change-Id: I1078ff7d2bd1acd0d1cea90fb90bc632386f0daf
Users can configure the number of worker threads however when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs.
Change-Id: Idda6e476e9e9b5842c4cc03e9853ec31d123abc5
The lsyncd service runs as the 'nginx' user such that files sync'd
from the master node to the backups will have 'nginx' as the owner.
However, the apt-cacher-ng service needs to be the owner to function
properly. This fix consolidates the pre and post sync tasks into
a script that can be called by lsyncd. The script can then change
the file owners as needed before and after the rsync. The owners
need to be 'nginx' before the rsync so that lsyncd can update
files and 'apt-cacher-ng' after the sync so the cacher service works.
Additionally, setup lsyncd to sync each service's directory separately
rather than being rsync'd all together. This avoids lsyncd bouncing
services when their respective files are not being sync'd.
Change-Id: Ifaba17b89035398917f2b3257574e18eb9027c08
Closes-bug: #1649339
The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: Iee1c61e431e93c60d9fa95e66a7fe278d0a11d3c
This change implements package caching on the repo server.
To take advantage of this a deploy will need to do nothing more
than setup an apt-proxy configuration file. This will speed up
package delivery while also providing ha capabilities within the
environment.
Change-Id: I78b2fba6a1f294751bd7098513060015cb41300c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Separate files have been created for vars and tasks related to a
specific package manager.
The 'repo_apt_packages' variable has been deprecated and renamed to
the more generalized 'repo_server_packages' to better describe its
purpose and to simplify reuse of existing install tasks between multiple
distros.
git daemon is configured to host git repositories from the repo servers
using the git protocol.
Currently, openstack-ansible uses git over http to access repositories
on servers created by this role.
fcgiwrap and its configuration within nginx should be removed in a
follow-up patch after openstack-ansible has been updated to use the git
protocol.
Change-Id: I62321a7b62dabca469eb072ddbf4e8f250ce0fb3
Limit repo_apt_packages to a more minimally viable list of requirements
for deploying Nginx servers and syncing files between them.
Change-Id: I677c78473b7f0442f8c334cd59b8c676973f4535
Depends-On: I03d5c061ec506a9dc142ff55a50fb3ecb18c238f
Closes-Bug: #1550418
The repo_server role does not require a memcached server and makes no use
of any related variables. Remove the memcached_server role dependency
and memcached variables.
Change-Id: I1c3b57ac3f25bac0e1640f5735cca7b73b573090
The pip requirements in this role are not needed to stand up the
repo-server. Being that they're not required they should be removed.
The needed bits used to build the python wheels will be re-added to
the repo-build role where it can leverage the cloned upstream upper
constraints file from the openstack global requirements repo.
Change-Id: I16cd9c3b00b4bcf886da1d31c69d19f49c46969f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Workarounding the upstream ansible apt module bug
documented here:
https://github.com/ansible/ansible-modules-core/pull/1517
For the next versions of ansible we'll be using, we should
check if the apt bug is fixed. When it's fixed, we could
abandon this change and use the standard apt module
with correct cache handling.
Change-Id: Iaff1eded0fd77ebfc69aca49b271ceaf719068a8
Both the yaprt package and the openstack-wheel-builder scripts are
no longer used for the repository building process.
This patch removes them.
Change-Id: I52073953e33948a9fdaeceea1127e46f7c820921
Review https://review.openstack.org/#/c/245966/ is currently failing
on building pillow, which requires libjpeg-dev. This commit installs
libjpeg-dev in the repo server containers so that dependency can be
successfully built.
Change-Id: I50dd1e4941bc7f57ce0d43f637b5446df6c935ce
Dmitriy Rabotyagov