As repo_build role has been succesfully replaced with
python_venv_build role, we have no need in it's futher maintaining and
releasing.
Depends-On: https://review.opendev.org/717715
Change-Id: Iaf1d6da8192afbbd18c64e2b58d54fa5804fe43c
If we sync a couple of item.paths over to the repo_all[0]. There is
definitely a change happening, so setting changed_when to false is
not correct.
During the with_items loop you can check each individual task under
the "register: _variable". After the task has completed with all
items, they're bundled up into _variable.results for future
processing.
rsync reports changes to stdout, and will remain empty if there are
none. So we can just check for stdout.
Depends-On: Ieca6ec9bf898432c76c019a1d2d97c280cd85dce
Change-Id: I2cf628d322e09ab7a0dc12557b07b1ab8d174b99
The links directory is only used by repo_build when creating requirement
wheels. The repo_build process builds wheels for it's own distro/arch combo
in serial. So in a mixed environment, where you might have ubuntu-16.04-x86_64
and ubuntu-18.04-x86_64 repo_containers, heck even centos-7-x86_64 for that
matter, copying the links pointing to pools/distro-arch over to another
repo_container creating wheels for a different distro/arch leads to bad builds
getting deployed later.
We also need to prevent the later lsync config from spreading these out
https://review.opendev.org/#/c/705390/
We can prevent this mix-up from happening even further by setting
repo_build_global_links_path: "{{ repo_build_base_path }}/links/{{ repo_build_os_distro_version }}"
as suggested in https://review.opendev.org/#/c/708115/
This is probably only relevant for stable/rocky.
Depends-On: Ieca6ec9bf898432c76c019a1d2d97c280cd85dce
Change-Id: I3bd6d3d987e32ee11c5f1fcb5c1b4b0fc797e7f9
As urlsplit is filter as well, so we need to quote it for applying
to all items. Otherwise 'urlsplit' undefined variable error is raised.
Change-Id: Ie4a51b177fb7f8e28c8c7dfb080960d6734ee21f
Related-Bug: #1820830
Co-Authored-By: Nicolas Bock <nicolasbock@gmail.com>
Remove the references to netorigin and netloc_no_port as they will
be taken out from openstack-ansible-plugins
Now using ansible 2.4's urlsplit filter as a replacement.
Change-Id: I94302ced3e5d6faa4d8748590f323e89d9b7280c
Related-Bug: #1820830
===
[DEPRECATION WARNING]: Using tests as filters is deprecated.
Instead of using `result|search` use `result is search`. This
feature will be removed in version 2.9.
===
Change-Id: If7ca8c8e749209ca3e021fd4604ff809d9e3a5db
To ensure consistency, we make sure that the venv used for the wheel
build has an up to date and the right versions of pip/setuptools/wheel.
Change-Id: Iac3e5054b5ef482bba2954e40c2c4d32a7f863df
This change creates a list of repos that are unconditionally
cloned. The list is available for a future parallel git clone
ansible module to consume.
Change-Id: If7c0b06124cfc11775ce38c900e076e3cf9b69d8
Similarly to I7204c09ed0e2dbc03aab5ce74684d88d985ce411
we can simply stop using get-pip.py any more and just
use the pip that is already in the virtualenv. This
cuts down on yet another reliance on get-pip.py and
makes offline installs simpler.
Depends-On: https://review.openstack.org/587835
Change-Id: Ia58554d222f27eac772df7d8dfb2394c1881fc82
To limit the python packages installed on the host,
all packages installed by pip are implemented into
a venv.
While many of the repo_pip_packages may be available
via distribution packages, the versions vary wildly.
We therefore opt to use a venv instead to keep things
simple.
This reduces the requirement on the host to *only*
require the virtualenv package which is installed in
the openstack_hosts role.
The 'virtualenv' package is removed from the list as
it is not used.
Change-Id: I0619f51960e23756df7010d476d70f55ceda8441
Version 3.2 has a bug with it on some systems that can result in the
following stacktrace: [ http://paste.openstack.org/show/726064 ]. To
correct this issue the version of the get-pip script has been set to 3.3
which does not suffer from the same issue.
Change-Id: Iaddb390a13d1955356e85af11ab087319808697c
Closes-Bug: #1779534
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: Idea3c1336b5dd10bd9dfc075587cc130a0a34449
As get-pip.py evolves based on pip 10, while we're still using
pip 9, changes in the way it can be used are causing problems.
For example, the ability to use --download is no longer there.
As such, let's pin to a known good version and leave it at that
until we no longer need to use this script. Version 3.2 maps to
pip 7.1.2 which fulfills our needs.
Change-Id: I24609eb753e8c041842bd3b0cc6d404ad9d7cb22
The moreutils package is needed from EPEL so we install it inside
the role. Moreover, in order to minimize the potential EPEL vs RDO
package conflicts, we only allow a small subset of packages from the
EPEL repository. Finally, we include the nodepool playbook from the
tests repository in order to populate the necessary CI facts.
Also, we don't need the LXC development headers so drop them.
Change-Id: I287a7209d7840283e374a8bb14d9caa4f714baf0
The I693b930bf337fd08931d835d57ffe9ecae904517 introduces the new
issue on environments where https://bootstrap.pypa.io not reachable
and overrided by pip_upstream_url setting.
Change-Id: I79d95f0efff4af41850de39632a95a6b6a7dc933
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
Some old Python2 pip sources still use setup.py, so when you provide
your own index sources via repo_build_pip_default_index the 'Download
requirement pip sources' task might fail while trying to reach a
default external PyPI index URL.
Change-Id: Ib7d2a9aee9b13f38b73136f22f0b51e766c8dd10
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
/opt/get-pip.py may not exist if the host uses the distribution
python-pip package. As such, we need to check if the file exists
and download it if it's missing before trying to copy it to the
repo build directory. Fixes the following problem:
==> opensuse423: TASK [repo_build : Copy get-pip script into release folder] ********************
==> opensuse423: Friday 25 May 2018 19:19:07 +0200 (0:00:01.362) 0:17:03.950 ************
==> opensuse423: fatal: [aio1_repo_container-b8bbcfc4]: FAILED! => {"changed": false, "failed": true, "msg": "Source /opt/get-pip.py not found"}
Change-Id: I693b930bf337fd08931d835d57ffe9ecae904517
The async method of running the git-clone commands has the unfortunate
side effect that git-clones are not being retried in case of failures.
This breaks deployment on hosts that either have limited network
resources or when the remote server implements some sort of rate
limiting when they are being hit by many connections from the same host.
As such, lets get rid of the async method and use the regular
retry/until method to try and recover from the various git-clone
failures.
Change-Id: Iff0c3c589f5e0a935e0e01cd820083b76d43cc92
To cater for a situation where the repo build packages are changing,
but the repo does not yet have the package built, we need to ensure
that the install task for it can fetch from pypi (or the designated
pypi mirror). To do this we try the local repo first, then fall back
to using the designated default index which defaults to pypi.
Change-Id: Iba7e9939250b24dc64fdd1883f963d0253ddad7e
When doing the wheel build we need to ensure that we make
use of the pypi mirror directly.
When upgrading, if we leave it to the pip.conf file contents,
it will only use the contents of the pypiserver which locks
it down to the wheels built for the version we're upgrading
from. The wheel build therefore fails.
Leaving the pip.conf config in place is still good for
ensuring that anything installed on the repo server later uses
the pypiserver contents.
Change-Id: I5d1f33832c9738683b49384ea5ebfb571b526a81
Due to the use of the pipe in the shell command the return code is
always zero even when the wheel build fails. This results in some
very obscure failures down the line which do not make it obvious
that the failure to build the appropriate wheel was the root cause.
In this patch we ensure that the shell will fail if the pip command
fails by setting the bash pipefail option.
Change-Id: Id539515557b439824e39ea2f693f62b913d5bdda
The installation of these packages are currently not constrained,
resulting in inconsistent experiences at deploy time.
This patch ensures that the venv preparation process properly
constrains the installation so that results are consistent.
Change-Id: I744e43d8a1b85d5bdaf7019a6420a39222911330
Partial-Bug: #1764470
The repo_build_venv_rebuild toggle is supposed to trigger
a rebuild of the venvs, but currently it does nothing.
This patch fixes that, making it possible to rebuild the
venvs without having to rebuild the wheels too.
Change-Id: I61db1f5555cba2c659ff09d8369ce3c31b011b5d
In I298dd45ada552b78ef7d6fcb79481035581bd77a timestamping
for the wheel build process was added, but it wasn't added
to the source download process. This patch adds timestamping
to the source download process too, but also changes the log
file used for each step so that they each have their own log
files.
The two tasks are also normalised to execute in the same way
to make them a little easier to follow. It thereby removes
the extra indirection of using yet another script for the
execution.
Change-Id: If571e9d290cbdd5ba02542b6bc19e4321b67ea6f
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
When preparing virtualenvs, we have an implementation which
tries to ensure that the versions of pip, setuptools and
wheel are the ones we want them to be. In order to do that
we have previously removed the virtualenv_support folder
which contains the pip/setuptools/wheel versions that
virtualenv shipped with. This was done to prevent any
leaking of the wrong version into the virtualenv and was
the only way of doing it with much earlier versions of
virtualenv.
We can now update the method of achieving the same goal
without breaking the virtualenv tooling, which this patch
does.
The virtualenv_support folder missing is causing the
repo-server role which uses virtualenv to fail in upgrades
to queens. That role needs a complete and properly working
version of virtualenv in place.
Change-Id: I2c50bb693dc5dc505bddce34ff45294ac44f7dc9
Closes-Bug: #1746935
Currently the manifest contains all the possible repositories
which would be present if a non-selective git clone was done,
rather than the actual resulting list of git folders.
This patch corrects the manifest to only contain the list of
what is actually there after the build process completes.
Depends-On: I15f993a3f80283f772bbb8ad75311f5117ff7ac0
Change-Id: I20ad8d21d4a68f8daa30c730a08a134b0cc9c0d3
Ansible 2.3 introduced a new option to the find module, but
in doing so they also implemented a breaking change.
Previously the find module would find both files and links
by default, but now it only finds files unless you tell it
specifically to find links.
As all the python wheels in the os-releases folder are links,
we tell the find module to find anything in the folder so
that the cleanup done is complete.
Change-Id: I6a63762f55782d8aeaa2a47aa55da21bc06ab431
Ansible 2.3 introduced a new option to the find module, but
in doing so they also implemented a breaking change.
Previously the find module would find both files and links
by default, but now it only finds files unless you tell it
specifically to find links.
As all the python wheels in the os-releases folder are links,
we tell the find module to find anything in the folder so
that the manifest file is complete.
Change-Id: Id4ebb2aa6de85a7a2d048bede787240493aadfec
This patch adds timestamps to pip wheel builds logs. This should
allow us to profile the wheel build runs and discover why CentOS
and OpenSUSE are so much slower than Ubuntu when building wheels.
The path also moves the wheel build process into a templated script
in /opt, which makes it easier to redirect output and prepend
timestamps to the log.
Partial-Bug: 1738424
Change-Id: I298dd45ada552b78ef7d6fcb79481035581bd77a
On top of that, it bumps the version of bindep.txt
from the osa-tests repo, because the automatic bump
cannot merge without lint tests passing, and the
other way around. Moreover, we drop the 'force'
argument from zypper since we are not switching
package vendors anymore.
[hwoarang: Add patch from https://review.openstack.org/#/c/526128/
to unblock the gates]
Depends-On: I4ba16ea660907c3fb481a9e80d73b750c2f4c138
Change-Id: I67e44434d4b3eb07b8af5c12aaa807514518be4c
When a set of global_environment_variables is not provided, Ansible
throws a warning:
[WARNING]: could not parse environment value, skipping:
[u'{{deployment_environment_variables | default({}) }}',
u'{{global_environment_variables | default(omit) }}']
This patch returns an empty dict rather than using the omit filter.
Change-Id: I5e3399b816779821e936b84a2972ca1db39c1a62
The repo does not need to carry a full git tree in order to build
the specified SHA. We can specify a small clone depth to speed up
the git clone process.
Change-Id: Ic73cd938ba27b9e0030cc158b86828d279d476e3
The venvs directory will be created during the repo server
playbook. Creating it a second time during the repo build process
is unnecessary.
This change removes the creation of the venvs directory during
the repo build process.
Depends-On: If1dd5e568e06715f4321c391727d428e2e0805ea
Change-Id: I50876f0443a958ec8433226ec7f6738347205f0a
The repo container sync task synchronises the following directories to a
wrong location.
pools
-----
Wrongly created dir on target: /var/www/repo/{{ repo_build_os_distro_version }}
Fixed dir on target: /var/www/repo/pools/{{ repo_build_os_distro_version }}
As the links and other directories are synced successfuly, they point
to invalid locations which will cause the repo build on the existing
repo master to fail.
venvs
-----
Wrongly created dir on target: /var/www/repo/{{ repo_build_os_distro_version }}
Fixed dir on target: /var/www/repo/venvs/{{ repo_build_release_tag }}/{{ repo_build_os_distro_version }}
os-releases
-----------
Wrongly created dir on target: /var/www/repo/{{ repo_build_os_distro_version }}
Fixed dir on target: /var/www/repo/os-releases/{{ repo_build_release_tag }}/{{ repo_build_os_distro_version }}
The patch achieves this by creating the directory befor rsync starts.
Using rsyncs relative option and synchronizing to the remotes root
/ dir does not work due to permissions on /var/.
See this blog post:
http://www.schwertly.com/2013/07/forcing-rsync-to-create-a-remote-path-using-rsync-path/
or the man pages of "rsync":
https://linux.die.net/man/1/rsync
[1] https://bugs.launchpad.net/openstack-ansible/+bug/1707630
Change-Id: I947ffc815e1574fe533d2a9ca6df6b2a991cff24
Closes-Bug: #1707630
The repo container sync task synchronises the following directory to a
wrong location.
pools
-----
Wrongly created dir on target: /var/www/repo/{{ repo_build_os_distro_version }}
Fixed dir on target: /var/www/repo/pools/{{ repo_build_os_distro_version }}
As the links and other directories are synced successfuly, they point
to invalid locations which will cause the repo build on the existing
repo master to fail.
venvs
-----
Wrongly created dir on target: /var/www/repo/{{ repo_build_os_distro_version }}
Fixed dir on target: /var/www/repo/venvs/{{ repo_build_release_tag }}/{{ repo_build_os_distro_version }}
os-releases
-----------
Wrongly created dir on target: /var/www/repo/{{ repo_build_os_distro_version }}
Fixed dir on target: /var/www/repo/os-releases/{{ repo_build_release_tag }}/{{ repo_build_os_distro_version }}
Change-Id: I9e5902ec91afbb2805663eb5678fc9b2b31f23b2
Closes-Bug: #1704936
If there are no local facts, the initialisation of
facts fails due to ansible_local being undefined.
This patch adds a conditional for that situation and
als makes the task set slightly more code-efficient.
Change-Id: I605280fb3ccd0f12b32bb572b064b9b04109934b
This commit passes the `global_environment_variables` dict to the git module
call in repo_clone_git.yml. This variable often includes proxy configuration,
which is now honored during the git clone process.
Prior to this commit, deployments requiring all external traffic to pass
through an HTTP proxy would fail.
Partial-bug: 1699550
Change-Id: If67f59fbf4437c2bf5fec2736952fe099a0ff342
Some tasks are no longer necessary in Pike
and the variable removed was deprecated in
Mitaka, so it's time to go.
Change-Id: Ibabe46967582929a441e66848c5267f0460a358b
This patch implements the following:
1. The ability to use a boolean variable to always force
a git clone or the wheel build to happen.
2. The use of local facts to mark whether a wheel build
or venv build is required. This ensures that if there
is a failure, the tasks will still be done when the
repo build is executed again.
3. The git clone or wheel build tasks are only actioned
if there is a change to the requirements or constraints.
This ensures that when the repo build is executed
without any changes, those processes will be skipped.
4. Re-arranges the tasks to ensure idempotency and to
make the process easier to follow. A smaller set of
tags are implemented which are intended to provide
a clear code path for each of them.
5. Log output is added to the venv build process to make
troubleshooting easier.
6. The stdout output for the wheel and venv build
processes is made minimal to reduce confusion and
make it easier to spot which item failed to build.
The log output in /var/log/repo contains the verbose
output from pip.
Change-Id: I2008926b43653edf50c284f5068160e27915c90a
Add support for SUSE based distributions. Since we grab some packages
from the OBS repositories, we need to use the 'force' option to allow
zypper switch vendor on these packages. We also update the bindep.txt,
run_tests.sh, tests-repo-clone.sh and Vagrantfile files from the
openstack-ansible-tests repository.
Change-Id: Icea249aae20156b253017c6028f0ba2bb74bd577
The venv build process currently executes on
every build, destroying all existing venvs for
the distribution/architecture of the repo server
it executes on, then rebuilds them.
It is also terribly difficult to troubleshoot
due to the fact that it is executed in parallel
through a single bash script using backgrounded
processes.
This patch breaks the build process up into two
parts - the per-venv options files, and the
script that uses the options to build the venvs.
With this breakdown we're able to do the following:
- Only execute a venv rebuild if the venv
options (indexes, requirements) have changed.
- Use the Ansible asynchonous execution to
execute parallel venv builds.
As a very welcome side-effect, this also means
that the venv build execution provides individual
output for success or failures, making it much
easier to see what went wrong when failing.
As part of the patch, the removal of the *.in,
*.txt and *.html files on each wheel build is
taken out. This is to protect the venv options
files. The removal of those files was unnecessary
anyway as they're templated and therefore replaced
by Ansible if they need to be changed.
Change-Id: I063c3addb6fbabb01d620be33aac2cab29a02750
Dmitriy Rabotyagov