As repo_build role has been succesfully replaced with
python_venv_build role, we have no need in it's futher maintaining and
releasing.
Depends-On: https://review.opendev.org/717715
Change-Id: Iaf1d6da8192afbbd18c64e2b58d54fa5804fe43c
===
[DEPRECATION WARNING]: Using tests as filters is deprecated.
Instead of using `result|search` use `result is search`. This
feature will be removed in version 2.9.
===
Change-Id: If7ca8c8e749209ca3e021fd4604ff809d9e3a5db
Similarly to I7204c09ed0e2dbc03aab5ce74684d88d985ce411
we can simply stop using get-pip.py any more and just
use the pip that is already in the virtualenv. This
cuts down on yet another reliance on get-pip.py and
makes offline installs simpler.
Depends-On: https://review.openstack.org/587835
Change-Id: Ia58554d222f27eac772df7d8dfb2394c1881fc82
Add an option to define a default set of python packages to install
within a virtual environment. This can be used to install a package
wihin a virtual environment that may be outside of a normal package
list but for a given service.
Change-Id: Iace13b8b4156b0aa6a919cf2b1f3a0c4376a74ee
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This reverts commit 51fb5bb898.
It didn't make any difference to the issue it was trying to solve,
so we should revert it because it changes the path of the builds
from before without adding any more value.
Change-Id: Id235766741398c54868c8b053d181c90e06f70db
Building in tmp, while functional, has a couple drawbacks due to it
being tmpfs and commonly noexec. This change moves the build path to
/opt which is a regular filesystem and a common build target. This
change also removes the copy-on-write bit on the build directory
which will assist when the underlying filesystems are BTRFS or ZFS.
Change-Id: I7184765a5902f535fe18162949dd40dd0d339062
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
When building a venv with no git sources, currently the venv build
executes without constraints as it uses the default pypi index.
This patch ensures that if there is no git source, then the --no-index
setting will still be applied.
Change-Id: I4468092eb5c86d6346a793ba1e2bfc6e772de3bb
The installation of these packages are currently not constrained,
resulting in inconsistent experiences at deploy time.
This patch ensures that the venv preparation process properly
constrains the installation so that results are consistent.
Change-Id: I744e43d8a1b85d5bdaf7019a6420a39222911330
Partial-Bug: #1764470
In I298dd45ada552b78ef7d6fcb79481035581bd77a timestamping
for the wheel build process was added, but it wasn't added
to the source download process. This patch adds timestamping
to the source download process too, but also changes the log
file used for each step so that they each have their own log
files.
The two tasks are also normalised to execute in the same way
to make them a little easier to follow. It thereby removes
the extra indirection of using yet another script for the
execution.
Change-Id: If571e9d290cbdd5ba02542b6bc19e4321b67ea6f
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
When preparing virtualenvs, we have an implementation which
tries to ensure that the versions of pip, setuptools and
wheel are the ones we want them to be. In order to do that
we have previously removed the virtualenv_support folder
which contains the pip/setuptools/wheel versions that
virtualenv shipped with. This was done to prevent any
leaking of the wrong version into the virtualenv and was
the only way of doing it with much earlier versions of
virtualenv.
We can now update the method of achieving the same goal
without breaking the virtualenv tooling, which this patch
does.
The virtualenv_support folder missing is causing the
repo-server role which uses virtualenv to fail in upgrades
to queens. That role needs a complete and properly working
version of virtualenv in place.
Change-Id: I2c50bb693dc5dc505bddce34ff45294ac44f7dc9
Closes-Bug: #1746935
Currently the manifest contains all the possible repositories
which would be present if a non-selective git clone was done,
rather than the actual resulting list of git folders.
This patch corrects the manifest to only contain the list of
what is actually there after the build process completes.
Depends-On: I15f993a3f80283f772bbb8ad75311f5117ff7ac0
Change-Id: I20ad8d21d4a68f8daa30c730a08a134b0cc9c0d3
Because of the changes in the other repos (removing
virtualenv-tools), it's expected that the repo-build process's
test don't see any mention of virtualenv-tools.
We should therefore remove anything related to it.
Change-Id: I4fc9a729b4dbac9c1101c47cdcff254a07d00435
Partial-Bug: #1741634
This patch adds timestamps to pip wheel builds logs. This should
allow us to profile the wheel build runs and discover why CentOS
and OpenSUSE are so much slower than Ubuntu when building wheels.
The path also moves the wheel build process into a templated script
in /opt, which makes it easier to redirect output and prepend
timestamps to the log.
Partial-Bug: 1738424
Change-Id: I298dd45ada552b78ef7d6fcb79481035581bd77a
Use dict.items() instead of dict.iteritems() for Python 3 compatibility.
Change-Id: I7270eef4b03fdd4cdf8b578edb479ecb336b6a83
Implements: blueprint goal-python35
Some tasks are no longer necessary in Pike
and the variable removed was deprecated in
Mitaka, so it's time to go.
Change-Id: Ibabe46967582929a441e66848c5267f0460a358b
This patch implements the following:
1. The ability to use a boolean variable to always force
a git clone or the wheel build to happen.
2. The use of local facts to mark whether a wheel build
or venv build is required. This ensures that if there
is a failure, the tasks will still be done when the
repo build is executed again.
3. The git clone or wheel build tasks are only actioned
if there is a change to the requirements or constraints.
This ensures that when the repo build is executed
without any changes, those processes will be skipped.
4. Re-arranges the tasks to ensure idempotency and to
make the process easier to follow. A smaller set of
tags are implemented which are intended to provide
a clear code path for each of them.
5. Log output is added to the venv build process to make
troubleshooting easier.
6. The stdout output for the wheel and venv build
processes is made minimal to reduce confusion and
make it easier to spot which item failed to build.
The log output in /var/log/repo contains the verbose
output from pip.
Change-Id: I2008926b43653edf50c284f5068160e27915c90a
The venv build process currently executes on
every build, destroying all existing venvs for
the distribution/architecture of the repo server
it executes on, then rebuilds them.
It is also terribly difficult to troubleshoot
due to the fact that it is executed in parallel
through a single bash script using backgrounded
processes.
This patch breaks the build process up into two
parts - the per-venv options files, and the
script that uses the options to build the venvs.
With this breakdown we're able to do the following:
- Only execute a venv rebuild if the venv
options (indexes, requirements) have changed.
- Use the Ansible asynchonous execution to
execute parallel venv builds.
As a very welcome side-effect, this also means
that the venv build execution provides individual
output for success or failures, making it much
easier to see what went wrong when failing.
As part of the patch, the removal of the *.in,
*.txt and *.html files on each wheel build is
taken out. This is to protect the venv options
files. The removal of those files was unnecessary
anyway as they're templated and therefore replaced
by Ansible if they need to be changed.
Change-Id: I063c3addb6fbabb01d620be33aac2cab29a02750
The repo build role will now store all wheels and venvs in directories
containg information about the OS in which the bits were build on. The
intention here is to ensure that all installation are able to support
multi-OS without running into issues caused by system packages and
different core libs. This change builds upon the multi-archetecture
support we already have.
Closes-Bug: 1641131
Change-Id: I3f36afa307e02a38d73b860fc23ed94a09882c34
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This patch switches from using the command/shell module
to using the find module to index the files.
It also adds the files that are not *.whl to the manifest
file to make it easier to implement a staging system for
deployments.
Change-Id: Ie927fb40fdf1f4cb934377320b800a506ed842e2
This change adds the release venvs to the built manifest.in file. This
change will allow a deployer to sync the constructed venv archive into
deployments from an external source.
Change-Id: Idaaae0ac5ab2e7cec94fa4aa3a008ec04acc684c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This patch makes the git clone process idempotent.
Using asynchronous Ansible tasks to achieve the goal
also resolves a race condition seen when using the
selective git clone process with a small number of
git sources.
Ansible handles the asynchronous tasks better than
the bash job control 'wait' command.
The test to validate that a missing 'origin' remote
does not break the git clone process was removed due
to https://github.com/ansible/ansible/issues/19290
Closes-Bug: #1649329
Change-Id: I3e68f155190461470adb7e46d904705d991973af
The repo server has autoindexing enabled, so the tasks
to enumerate the contents of the folder and to produce
a static index file are unnecessary.
This patch removes the tasks and ensures that the files
in an existing repository are also removed to cater for
upgrades.
Change-Id: Ic2bb537f331d14a4ddfefa5ea9ec785d0fce3835
This change ensures that a venv create process can work when a
dependency is crafted outside of the OpenStack global requirements
system. The change keys off of the original package url setting the
isolated perameter should "withindex=True" be found.
The requirements.txt.j2 template file has been updated to allow
packages to be ignored from wheel building should the
`ignorerequirement=True` property be set within a given remote package.
This ability is important as some packages, generally external testing
repos, do not use requirement boundries and are known to break
given enough time.
Tagged tasks that MUST always be run have had tags set to "always".
Change-Id: I5d11aca84dcb74c77f3d0e3f31ce6546458a3e84
Closes-Bug: #1631992
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change causes the contents of /etc/environment to be
evaluated so that deployments in environments with HTTP
proxies can be successful.
Change-Id: Ida2a0465bea8b87bcb7a71c61661852ac1417391
Closes-Bug: 1625405
This change ensures that the should the `op-clone-script.sh` find a valid
git directory it's working with a clean git history. This is done by simply
running `git clean -f -d` on the target destination. This is important because
there is a chance that a git repository may have had a force push on a given
remote or there were local changes made to the repo that break a new checkout.
Change-Id: I177871a90e08f0d29f28a0923fd8fc89fe91c15c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The recent patch merged [1] had two errors:
1) The 'requires_pip_packages' list was taken as part of the list
of packages which should go into the venv. This should not be
the case as these packages go on to the host.
2) When looping through each list set of packages for the role, each
list in turn would replace the last - therefore only the last set
would be built into the venv.
This patch ensures that the 'requires_pip_packages' list is ignored
(along with the optional and proprietary lists) and also ensures that
each list processed that should be included is properly merged into
a consolidated list before executing the build.
[1] https://review.openstack.org/360406
Change-Id: I35ca4305803d55fdad23039851a25915b2cf0100
Previously the requirements were processed multiple times - once
to build requirements.txt and another to download the pip sources.
This patch ensures that the processed requirements.txt is used to
download the pip sources to ensure that the processing only happens
once and that the downloaded sources are accurate and complete.
Change-Id: I8d056668208a09ec61d318bd67f29cbf32f3b195
The updated py_pkgs lookup in https://review.openstack.org/352695
produces a granular list of all python package requirements in
the roles and the role:host_group data mapping.
This patch updates the venv build process to make use of the new
data structure.
Change-Id: I6ea8fdb2032ff27db6240d5ec4b1427cced6687e
Depends-On: Iad00e4f1268187a29e90d7b8e6afb5abfd1e92a2
The script was previously trying to determine the group
based on the name of the role, but we already have the role name
to group mapping in the data provided by the py_pkgs lookup
Change-Id: Ife1c81f21f67f4d643d60fa5d6061e4f3a7dad0a
The updated py_pkgs lookup in https://review.openstack.org/352695
produces a granular list of all python package requirements in
the roles.
This patch updates the wheel build process to make use of the new
data structure to ensure that the resulting build includes the
optional python packages specified in each role.
It also makes use of the role:host_group data mapping in the new
structure.
Depends-On: Iad00e4f1268187a29e90d7b8e6afb5abfd1e92a2
Change-Id: I695d3f49263f759ece85d23949c9bc560ba1d756
Now that https://review.openstack.org/345986 has merged, this
compatibility is no longer required.
This reverts commit d3a4d327f2.
Change-Id: I4865b2171041f3278a0677584cc6bc858dd68ca3
Until https://review.openstack.org/345986 merges, integrated builds
cannot proceed due to the fact that the repo build is producing
venv archives which are named with the processor architecture in
the name, while the playbooks are expecting a different name.
This patch implements links to the archives using the old names as
a temporary compatibility measure.
Change-Id: Ie0f478aaad1fdddc267c5ad2d4a893701dad78b8
This patch adds support for building across multiple repo servers,
with artifacts for each CPU architecture being tagged appropriately,
then synchronizing the built artifacts back to the primary repo server
for distribution out to all repo servers deployed.
Partially Implements: bp/multi-arch-repo
Needed-By: I045de3ac8b81cadbcb34102f1a2db5bff74c32a6
Change-Id: I31756f8383e6d69d2f80caf6a85c4c5021bfc46d
The current repo build process clones all git repositories for all
OpenStack services, regardless of whether they're actually going to
be used.
To reduce the repo build time, this patch will only clone the
git repositories for OpenStack services where there is one or more
members in the service inventory group.
Depends-On: I6a0508e8aaa4993bc68e72e3d2988d6db6470f6d
Change-Id: I6e5a554d6f87058ef5cb63adf7995b066cb6fd78
The requirement aodh[mysql] is not properly interpreted by the
requirements template, and thus causes a build failure.
This patch ensures that this requirement is included in the
test process, and that it is interpreted properly and builds.
Change-Id: Ia17ee7788f3218db0c7ece95d3e11aa900f7fd98
The current repo build process builds wheels for all OpenStack services,
regardless of whether they're actually going to be used.
To reduce the repo build time, this patch will only build wheels for
OpenStack services where there is one or more members in the service
inventory group.
An updated set of test vars is included based on a recent stable/mitaka
output of the py_pkgs lookup based on the below-mentioned dependant
patch.
Depends-On: I5e981c818a1da50aa0ae15630655a8dfac4f3db7
Change-Id: Idda16b4f382eee57c7469af898859d6d81d4eb30
In https://review.openstack.org/350352 the repo_build process
implemented the test vars and changes to the constraints build
which caters properly for global pins, but the data set used
was incorrectly mapped from the py_pkgs lookup output.
This patch fixes the depth, tests and template to use the
correct depth.
Related-Bug: #1605846
Closes-Bug: #1609056
Change-Id: Ic8959fd27b4012d569aef1b9ddd1d7be40ed49e0
The current repo build process has the following issues:
1. The constraints consruction doesn't properly handle two constraints
which use the same name, but have different version specs. eg:
ovs===2.5.0;python_version=='2.7'
ovs===2.6.0.dev2;python_version=='3.4'
This is a problem in both the constraints construction, and
in the testing.
2. The pip packages installed on the repo server in order to
construct the repo do not respect the global pins or the upper
constraint overrides.
3. The constraints are selected based on the requirements specified.
This makes the process unnecessarily complex and hard to test.
4. The global pins are not applied to the constructed constraints,
so the wrong packages get built and installed.
5. The tests aren't thoroughly checking that the constraints are
correct.
This patch corrects all of these issues and hopefully makes the
mechanism for constructing constraints more readily apparent, and
therefore easier to maintain.
Change-Id: If52d073d43081468e2faf2cd063c4b211c29994f
Related-Bug: #1605846
Closes-Bug: #1609056
This patch provides the ability for the repo server to take advantage
of a pre-staged git cache on the deployment host, such as that provided
on OpenStack-CI's nodes.
This reduces the execution time of the build process by:
- only executing a repo clone for invalid or missing repositories
- only updating the clone if the existing clone doesn't already have
the version required
Due to the way OpenStack-CI stages the cache (from a file based origin
which is not accessible) it is also necessary to adjust the origin
remote instead of creating a new remote as was done previously,
otherwise the 'git fetch' execution would fail.
The patch includes changes to the tests to pre-stage two repositories
and mangle them intentionally to test the resilience of the clone
process.
Change-Id: Ibe70ba27b9a716e9385a5db46cd745bc0d538d63
The current repo build process builds venvs for all OpenStack services,
regardless of whether they're actually going to be used.
To reduce the repo build time, this patch will only build venvs for
OpenStack services where there is one or more members in the service
inventory group.
Change-Id: I01e6cdaa7a1a4aa07d2118c0badfea25b85b3249
The current constraints file is generated from a template, which
completely recreates the url used for the constraints.
Therefore, it deletes previously given arguments, like subdirectories.
This commit replaces the initial url to the local (previously fetched)
repo, so the constraints generation behavior stays consistent.
Change-Id: I44b2d1ea20e8f49744add34cf44423b8fdeea45d
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>
This change enables our repo-build process to store the package sources
for all of our requirements within a given release. This change enables
the ability to have a multi-archetecture / multi-distro / multi-interpreter
deployment.
Because a python wheel can be tied to a particular archetecture, distro, or
interpreter wheels alone can not be relied on to power a mixed environment. The
source is needed to ensure pip is able to resolve a package depenedency even if
a pre-built wheel does not meet the python or system requirements. To enable
this a task has been added to the repo_build.yml file to first download all of
the sources in non-binary format, once downloaded the sources are used to build
all of the wheels locally.
Change-Id: Ib081c24f67e92c165cba14848ff86e20fe6c1530
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Currently the build process builds all the wheels for all packages
listed in OpenStack's global requirements.
This is a wasteful process as it contains packages needed by all
OpenStack projects, instead of just the projects OpenStack-Ansible
deploys in the integrated build.
This patch removes the wheel build of all global requirements and
switches the repo build process to only building the wheels that
are actually needed.
Depends-On: I6536993523f02ba49da741b9ea21535eb33fe402
Change-Id: I5a865391803de2beef49c61b55277fad6317561f
This change updates the repo-build process to be ansible2.1
compatible. The change is needed because the if statements in the
sting cause a \\ to exist if the variables are not used. To resolve
this issue the entry to the shell call has been changed to a folding
style instead of a literal style which will no longer generate the \\.
A change was also need in the op-venv script where we rely on a bash
script to build all of the venvs based on input provided by a folding
style string. Within Ansible 2.x the folding string type may have a \n
on the end which will break direct bash bash functions.
Change-Id: I394ab60f5505a45cf39b1efce1016dad9332be78
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Currently the repo build process hard codes the use of a set of indexes
when building wheels. This overrides any existing pip configuration in
the repo container and is somewhat unintuitive as a result.
As the default index is pypi, which is the default for pip anyway, this
is a redundant hard-code which really shouldn't be required.
The specification of additional indexes should also be optional, as the
default of using pypi again just results in a duplicated check by pip.
Also, for context, OpenStack-CI gating will prefer the use of a
system-wide pip.conf which will be copied from the host to the container.
Without this patch, that simple solution to our gating process will not
work.
Change-Id: I25e4faecbffad6301509c62f55c34ba86b92a658
Dmitriy Rabotyagov