As repo_build role has been succesfully replaced with
python_venv_build role, we have no need in it's futher maintaining and
releasing.
Depends-On: https://review.opendev.org/717715
Change-Id: Iaf1d6da8192afbbd18c64e2b58d54fa5804fe43c
When introducing a new repo_container with a different distro, using
the same directory for symlinks can result in weird builds. We should
prevent the links directory to be copied all willy-nilly, as suggested
here;
https://review.opendev.org/#/c/712784/https://review.opendev.org/#/c/712787/
But we can further prevent bad builds by separating under distro-arch
under links with this patch.
Depends-On: Ieca6ec9bf898432c76c019a1d2d97c280cd85dce
Change-Id: I4ae6737f1ded4cbb680a380c8c309a0f5bf64723
Add --pre to the pip install options. Otherwise, prerelease versions are
not accepted by pip to resolve dependencies and problems appear when
building the venvs. For example tacker venv build fails with:
Could not find a version that satisfies the requirement neutron>=12.0.0
and we have neutron=13.0.0.0b3.dev94
Change-Id: I0dcb390a189b9fcad4dc5990b03c99ef30b4bbe4
Signed-off-by: Manuel Buil <mbuil@suse.com>
Similarly to I7204c09ed0e2dbc03aab5ce74684d88d985ce411
we can simply stop using get-pip.py any more and just
use the pip that is already in the virtualenv. This
cuts down on yet another reliance on get-pip.py and
makes offline installs simpler.
Depends-On: https://review.openstack.org/587835
Change-Id: Ia58554d222f27eac772df7d8dfb2394c1881fc82
To limit the python packages installed on the host,
all packages installed by pip are implemented into
a venv.
While many of the repo_pip_packages may be available
via distribution packages, the versions vary wildly.
We therefore opt to use a venv instead to keep things
simple.
This reduces the requirement on the host to *only*
require the virtualenv package which is installed in
the openstack_hosts role.
The 'virtualenv' package is removed from the list as
it is not used.
Change-Id: I0619f51960e23756df7010d476d70f55ceda8441
Add an option to define a default set of python packages to install
within a virtual environment. This can be used to install a package
wihin a virtual environment that may be outside of a normal package
list but for a given service.
Change-Id: Iace13b8b4156b0aa6a919cf2b1f3a0c4376a74ee
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This reverts commit 51fb5bb898.
It didn't make any difference to the issue it was trying to solve,
so we should revert it because it changes the path of the builds
from before without adding any more value.
Change-Id: Id235766741398c54868c8b053d181c90e06f70db
Building in tmp, while functional, has a couple drawbacks due to it
being tmpfs and commonly noexec. This change moves the build path to
/opt which is a regular filesystem and a common build target. This
change also removes the copy-on-write bit on the build directory
which will assist when the underlying filesystems are BTRFS or ZFS.
Change-Id: I7184765a5902f535fe18162949dd40dd0d339062
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
When doing the wheel build we need to ensure that we make
use of the pypi mirror directly.
When upgrading, if we leave it to the pip.conf file contents,
it will only use the contents of the pypiserver which locks
it down to the wheels built for the version we're upgrading
from. The wheel build therefore fails.
Leaving the pip.conf config in place is still good for
ensuring that anything installed on the repo server later uses
the pypiserver contents.
Change-Id: I5d1f33832c9738683b49384ea5ebfb571b526a81
A new version of PBR was released which causes some services, like
gnocchi, to break in new and exciting ways. This change installs PBR
within the repo_build role which will ensure all of the correct tools
are available to the repo build process and are installed using known
constraints.
Change-Id: Ibf5e3bf0381ec40b1adbe6ce49a5cf7284d3e147
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
In I2c50bb693dc5dc505bddce34ff45294ac44f7dc9 we applied changes
to the settings to repo_build_venv_command_options so that pip,
setuptools and wheel were not copied into the new virtualend on
creation, but rather installed into it later.
Unfortunately the patch missed that CentOS and SuSE platforms
have an override for the same variable, so the issue set out in
the patch still applies to those platforms.
This patch ensures that the overrides are removed and the
variable is only declared in one place, but the same functionality
which gave rise to the override for those platforms is still
applied when executed.
Closes-Bug: #1746935
Change-Id: Ie0c8c95a0d4315f8d1f848b1309bf88b7c2644d4
When preparing virtualenvs, we have an implementation which
tries to ensure that the versions of pip, setuptools and
wheel are the ones we want them to be. In order to do that
we have previously removed the virtualenv_support folder
which contains the pip/setuptools/wheel versions that
virtualenv shipped with. This was done to prevent any
leaking of the wrong version into the virtualenv and was
the only way of doing it with much earlier versions of
virtualenv.
We can now update the method of achieving the same goal
without breaking the virtualenv tooling, which this patch
does.
The virtualenv_support folder missing is causing the
repo-server role which uses virtualenv to fail in upgrades
to queens. That role needs a complete and properly working
version of virtualenv in place.
Change-Id: I2c50bb693dc5dc505bddce34ff45294ac44f7dc9
Closes-Bug: #1746935
In [1] the intent was to have each venv build log
to its own log file. This makes troubleshooting a
lot easier due to the verbosity of the output in
the log - this at least separates each venv's
output. Unfortunately this extra CLI option was
supposed to be removed. This patch completes the
original intent.
[1] https://review.openstack.org/470757
Change-Id: I714d160a39beb870ba66db11a60cf0241ca770f3
Because of the changes in the other repos (removing
virtualenv-tools), it's expected that the repo-build process's
test don't see any mention of virtualenv-tools.
We should therefore remove anything related to it.
Change-Id: I4fc9a729b4dbac9c1101c47cdcff254a07d00435
Partial-Bug: #1741634
The repo does not need to carry a full git tree in order to build
the specified SHA. We can specify a small clone depth to speed up
the git clone process.
Change-Id: Ic73cd938ba27b9e0030cc158b86828d279d476e3
Some tasks are no longer necessary in Pike
and the variable removed was deprecated in
Mitaka, so it's time to go.
Change-Id: Ibabe46967582929a441e66848c5267f0460a358b
This patch implements the following:
1. The ability to use a boolean variable to always force
a git clone or the wheel build to happen.
2. The use of local facts to mark whether a wheel build
or venv build is required. This ensures that if there
is a failure, the tasks will still be done when the
repo build is executed again.
3. The git clone or wheel build tasks are only actioned
if there is a change to the requirements or constraints.
This ensures that when the repo build is executed
without any changes, those processes will be skipped.
4. Re-arranges the tasks to ensure idempotency and to
make the process easier to follow. A smaller set of
tags are implemented which are intended to provide
a clear code path for each of them.
5. Log output is added to the venv build process to make
troubleshooting easier.
6. The stdout output for the wheel and venv build
processes is made minimal to reduce confusion and
make it easier to spot which item failed to build.
The log output in /var/log/repo contains the verbose
output from pip.
Change-Id: I2008926b43653edf50c284f5068160e27915c90a
ansible_distribution may return a string with spaces such as "openSUSE Leap"
so we need to replace the space with underscore in order to create a more
sensible repo name for the distro.
Change-Id: I0966786226d3f534a7f326d9df2cd73e7e3749dd
The venv build process currently executes on
every build, destroying all existing venvs for
the distribution/architecture of the repo server
it executes on, then rebuilds them.
It is also terribly difficult to troubleshoot
due to the fact that it is executed in parallel
through a single bash script using backgrounded
processes.
This patch breaks the build process up into two
parts - the per-venv options files, and the
script that uses the options to build the venvs.
With this breakdown we're able to do the following:
- Only execute a venv rebuild if the venv
options (indexes, requirements) have changed.
- Use the Ansible asynchonous execution to
execute parallel venv builds.
As a very welcome side-effect, this also means
that the venv build execution provides individual
output for success or failures, making it much
easier to see what went wrong when failing.
As part of the patch, the removal of the *.in,
*.txt and *.html files on each wheel build is
taken out. This is to protect the venv options
files. The removal of those files was unnecessary
anyway as they're templated and therefore replaced
by Ansible if they need to be changed.
Change-Id: I063c3addb6fbabb01d620be33aac2cab29a02750
The repo build role will now store all wheels and venvs in directories
containg information about the OS in which the bits were build on. The
intention here is to ensure that all installation are able to support
multi-OS without running into issues caused by system packages and
different core libs. This change builds upon the multi-archetecture
support we already have.
Closes-Bug: 1641131
Change-Id: I3f36afa307e02a38d73b860fc23ed94a09882c34
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Now that the external repository management is done in pip-install
role, we don't need this wiring anymore.
Everything will be consistent across all the roles that include
pip-install.
Change-Id: I370135ab49dd75c2c3632b9f807b609c539bd4d6
Apt cannot have 2 mirrors with the same content in 2 different files.
If a deployer has an apt mirror with uca, the deployer
still need to add a repository, but will also need to define the
filename used, in order to avoid clashes.
This commit makes possible to decide the filename for the repo, and
has the same behavior as Newton, allowing a backport.
Change-Id: Ie95dc9a90ad1772dc2e938fb1432d37a9c01f131
The repo build concurrency was pulling the processor core count and using
that int as the value for determining concurrent builds. Sadly,
this value can be high, low, or simply undefined and is entirely dependent
on the Ansible fact gathering system which has additional complexities and
dependencies based on how the hardware has represented the information.
Examples:
* Should the value be low, 3 or less than, the build process would run
serially making things painfully slow.
* Should the value be high the build process may run too many tasks at
one time causing general system instability.
This commit statically defines the concurrency level as "4" which is small
enough to support vagant, and minimal resource machines, while also being
performant enough to support large scale production clouds.
Change-Id: If2ad496c31c4d0092e7856c3deb07035ebc6c8b9
This change ensures that a venv create process can work when a
dependency is crafted outside of the OpenStack global requirements
system. The change keys off of the original package url setting the
isolated perameter should "withindex=True" be found.
The requirements.txt.j2 template file has been updated to allow
packages to be ignored from wheel building should the
`ignorerequirement=True` property be set within a given remote package.
This ability is important as some packages, generally external testing
repos, do not use requirement boundries and are known to break
given enough time.
Tagged tasks that MUST always be run have had tags set to "always".
Change-Id: I5d11aca84dcb74c77f3d0e3f31ce6546458a3e84
Closes-Bug: #1631992
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Add cython to repo_pip_packages to make sure the wheel file for pandas
can be built for ppc64le architecture.
Change-Id: Ib3210cff1b07302716cd91b798b81bb890e1f372
In https://review.openstack.org/380553 a change was made to
only execute the clone script if the script changes. This
has resulted in a failure when using a pre-staged git cache.
As the script only executes clone tasks if the git repo is
broken or the SHA isn't in the clone, it's OK to execute it
multiple times.
This patch ensures that the script always runs, but also tries
to add changes to the tests to try to uncover the cause of the
integrated build failure which the previous patch caused.
Additionally, the patch also adds tasks to ensure that the git
folder has the correct owner:group.
Change-Id: I40a02c26d0abc9d17dd7fad2eab969d169c436bb
This prevents pip SSL certificate verification failures in
environments with internally signed certificates by allowing the
certificate location to be specified in the pip_install_options
variable.
The pip_install_options variable is also added to the role defaults.
Change-Id: I447e5378254f88ac4fa87a9db3d34221a21e3685
Partial-Bug: 1621596
This patch adds support for building across multiple repo servers,
with artifacts for each CPU architecture being tagged appropriately,
then synchronizing the built artifacts back to the primary repo server
for distribution out to all repo servers deployed.
Partially Implements: bp/multi-arch-repo
Needed-By: I045de3ac8b81cadbcb34102f1a2db5bff74c32a6
Change-Id: I31756f8383e6d69d2f80caf6a85c4c5021bfc46d
The current repo build process clones all git repositories for all
OpenStack services, regardless of whether they're actually going to
be used.
To reduce the repo build time, this patch will only clone the
git repositories for OpenStack services where there is one or more
members in the service inventory group.
Depends-On: I6a0508e8aaa4993bc68e72e3d2988d6db6470f6d
Change-Id: I6e5a554d6f87058ef5cb63adf7995b066cb6fd78
The current repo build process builds wheels for all OpenStack services,
regardless of whether they're actually going to be used.
To reduce the repo build time, this patch will only build wheels for
OpenStack services where there is one or more members in the service
inventory group.
An updated set of test vars is included based on a recent stable/mitaka
output of the py_pkgs lookup based on the below-mentioned dependant
patch.
Depends-On: I5e981c818a1da50aa0ae15630655a8dfac4f3db7
Change-Id: Idda16b4f382eee57c7469af898859d6d81d4eb30
The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: Ic570c4029b06918e223f21da02012f330911ad51
This patch adds a new list variable which allows a deployer to
override OpenStack's upper-constraints for the repo build
process.
Change-Id: I46ba9345a85c43f2a1f24ea1819132e7474e6909
Co-Authored-By: Jimmy McCrory <jimmy.mccrory@gmail.com>
The repo build server sometimes includes old versions
of the requests and urllib3 python packages. These
old versions may be due to outdated system libs.
This patch instructs pip to install requests and urllib3
to the latest version on the repo-build host.
Change-Id: I78dd9b1430f13e23fe153b162ef4aa38b216c279
The current repo build process builds venvs for all OpenStack services,
regardless of whether they're actually going to be used.
To reduce the repo build time, this patch will only build venvs for
OpenStack services where there is one or more members in the service
inventory group.
Change-Id: I01e6cdaa7a1a4aa07d2118c0badfea25b85b3249
There are situations where you may wish to ensure that a package
is always built from source, regardless of whether a pre-compiled
wheel is available on one of the indexes.
This patch adds the ability to set a list of packages which must
always be compiled from source.
Change-Id: I2f47d94c58503514a3f9a0a62081cbc1ca53613b
When the UCA repo is enabled for nova, a newer version of libvirt will
be installed. The repo_build role will also need to have the UCA repo
enabled to install the same version of libvirt-dev so that a
corresponding libvirt-python wheel will be built.
The test preparation implements an override of the URL for OpenStack-CI
to make use of the local mirrors.
Change-Id: I6d5532e862e98929cff868f940c202875d1ee560
This works around a bug in Ansible where ansible_processor_count is
reported as 0 on ppc64le today due to issues with /proc/cpuinfo
format differences. See this issue [1] for current status.
[1] - https://github.com/ansible/ansible/issues/16533
Change-Id: Icc26c415fb36aefc6ad22eb76073bcd1f8878466
This change enables our repo-build process to store the package sources
for all of our requirements within a given release. This change enables
the ability to have a multi-archetecture / multi-distro / multi-interpreter
deployment.
Because a python wheel can be tied to a particular archetecture, distro, or
interpreter wheels alone can not be relied on to power a mixed environment. The
source is needed to ensure pip is able to resolve a package depenedency even if
a pre-built wheel does not meet the python or system requirements. To enable
this a task has been added to the repo_build.yml file to first download all of
the sources in non-binary format, once downloaded the sources are used to build
all of the wheels locally.
Change-Id: Ib081c24f67e92c165cba14848ff86e20fe6c1530
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This is an implementation of a pattern for multi-distro support
This allows for adding support of distros other than Ubuntu
Change-Id: Ife463766563ace3a329718a8df5ad7d2b38548c6
This change updates the repo-build process to be ansible2.1
compatible. The change is needed because the if statements in the
sting cause a \\ to exist if the variables are not used. To resolve
this issue the entry to the shell call has been changed to a folding
style instead of a literal style which will no longer generate the \\.
A change was also need in the op-venv script where we rely on a bash
script to build all of the venvs based on input provided by a folding
style string. Within Ansible 2.x the folding string type may have a \n
on the end which will break direct bash bash functions.
Change-Id: I394ab60f5505a45cf39b1efce1016dad9332be78
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Currently the repo build process hard codes the use of a set of indexes
when building wheels. This overrides any existing pip configuration in
the repo container and is somewhat unintuitive as a result.
As the default index is pypi, which is the default for pip anyway, this
is a redundant hard-code which really shouldn't be required.
The specification of additional indexes should also be optional, as the
default of using pypi again just results in a duplicated check by pip.
Also, for context, OpenStack-CI gating will prefer the use of a
system-wide pip.conf which will be copied from the host to the container.
Without this patch, that simple solution to our gating process will not
work.
Change-Id: I25e4faecbffad6301509c62f55c34ba86b92a658
The repo_build process is slow and can do with some improvement.
To achieve better speed of execution some several slow tasks,
as profiled by the integrated gate have been moved into executable
scripts using jobs. These scripts are generated from the playbooks
and the Jinja2 variables created during the repo_build run time and
make it possible to perform multiple actions in parallel. In test
the results were significantly faster[0].
The test case has been updated to build all of the packages known
to exist and be required for a real OpenStack environment.
[0] - http://paste.openstack.org/show/491537/
Change-Id: I3a1872741a17dee25054a340859333b96203ca15
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The var 'repo_build_venv_force_rebuild' was made available to
allow the venv rebuilding to be optional, and defaulted to
false. This causes a mismatch between expected behaviour and the
actual behaviour.
When executing a repo-build, the deployer expectes the wheels
and venvs to be rebuilt. The actual behaviour is that the wheels
are rebuilt, but the venvs are left alone.
If a deployer/developer is updating SHA's for anything within the
same tag (or following the OSA master branch) then the services
never get deployed with updated code as the venvs never change.
This patch removes this var altogether. Deployers may still use
tag skipping to isolate tasks they wish to run if that's
required.
Change-Id: I8ef8d8c13e067223eb908fd2177cef8beb929397