Commit Graph

58 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov f72de305b8 Retire openstack-ansible-repo_build role
As repo_build role has been succesfully replaced with
python_venv_build role, we have no need in it's futher maintaining and
releasing.

Depends-On: https://review.opendev.org/717715
Change-Id: Iaf1d6da8192afbbd18c64e2b58d54fa5804fe43c
2020-04-06 14:30:36 +03:00
Erik Berg d5e33c8afa repo_build: take distro version into account
When introducing a new repo_container with a different distro, using
the same directory for symlinks can result in weird builds. We should
prevent the links directory to be copied all willy-nilly, as suggested
here;

https://review.opendev.org/#/c/712784/
https://review.opendev.org/#/c/712787/

But we can further prevent bad builds by separating under distro-arch
under links with this patch.

Depends-On: Ieca6ec9bf898432c76c019a1d2d97c280cd85dce
Change-Id: I4ae6737f1ded4cbb680a380c8c309a0f5bf64723
2020-03-16 17:24:58 +01:00
Jesse Pretorius b1148411cc Remove venv build process
The packaged venvs are no longer used by any of the roles,
so we remove the whole process.

Change-Id: I086c7f53f90c11969f56391551a573e1bd526c50
2018-09-22 19:29:45 +01:00
Manuel Buil 88d817836e Consider prereleases when working with pip
Add --pre to the pip install options. Otherwise, prerelease versions are
not accepted by pip to resolve dependencies and problems appear when
building the venvs. For example tacker venv build fails with:

Could not find a version that satisfies the requirement neutron>=12.0.0

and we have neutron=13.0.0.0b3.dev94

Change-Id: I0dcb390a189b9fcad4dc5990b03c99ef30b4bbe4
Signed-off-by: Manuel Buil <mbuil@suse.com>
2018-08-23 15:04:08 +02:00
Jesse Pretorius caef3380fe Simplify pip/setuptools/wheel prep for venvs
Similarly to I7204c09ed0e2dbc03aab5ce74684d88d985ce411
we can simply stop using get-pip.py any more and just
use the pip that is already in the virtualenv. This
cuts down on yet another reliance on get-pip.py and
makes offline installs simpler.

Depends-On: https://review.openstack.org/587835
Change-Id: Ia58554d222f27eac772df7d8dfb2394c1881fc82
2018-08-08 09:50:33 +00:00
Jesse Pretorius d7acac1b99 Use a venv to build the python wheels
To limit the python packages installed on the host,
all packages installed by pip are implemented into
a venv.

While many of the repo_pip_packages may be available
via distribution packages, the versions vary wildly.
We therefore opt to use a venv instead to keep things
simple.

This reduces the requirement on the host to *only*
require the virtualenv package which is installed in
the openstack_hosts role.

The 'virtualenv' package is removed from the list as
it is not used.

Change-Id: I0619f51960e23756df7010d476d70f55ceda8441
2018-08-07 22:02:10 +01:00
Zuul c4af997fe5 Merge "Add option for default pip package install" 2018-07-02 09:07:42 +00:00
Kevin Carter 7a4d888d35
Add option for default pip package install
Add an option to define a default set of python packages to install
within a virtual environment. This can be used to install a package
wihin a virtual environment that may be outside of a normal package
list but for a given service.

Change-Id: Iace13b8b4156b0aa6a919cf2b1f3a0c4376a74ee
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-29 23:25:53 -05:00
Jesse Pretorius (odyssey4me) 92aea73d38 Revert "Change build path from tmp to opt and disable COW"
This reverts commit 51fb5bb898.

It didn't make any difference to the issue it was trying to solve,
so we should revert it because it changes the path of the builds
from before without adding any more value.

Change-Id: Id235766741398c54868c8b053d181c90e06f70db
2018-06-29 11:12:40 +00:00
Kevin Carter 51fb5bb898 Change build path from tmp to opt and disable COW
Building in tmp, while functional, has a couple drawbacks due to it
being tmpfs and commonly noexec. This change moves the build path to
/opt which is a regular filesystem and a common build target. This
change also removes the copy-on-write bit on the build directory
which will assist when the underlying filesystems are BTRFS or ZFS.

Change-Id: I7184765a5902f535fe18162949dd40dd0d339062
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-06 17:07:31 +00:00
Jesse Pretorius a25d1b41db Override the pip index used for the wheel build
When doing the wheel build we need to ensure that we make
use of the pypi mirror directly.

When upgrading, if we leave it to the pip.conf file contents,
it will only use the contents of the pypiserver which locks
it down to the wheels built for the version we're upgrading
from. The wheel build therefore fails.

Leaving the pip.conf config in place is still good for
ensuring that anything installed on the repo server later uses
the pypiserver contents.

Change-Id: I5d1f33832c9738683b49384ea5ebfb571b526a81
2018-04-20 14:26:43 +01:00
Kevin Carter b7aa8d4eef
Install PBR to ensure we're not breaking builds
A new version of PBR was released which causes some services, like
gnocchi, to break in new and exciting ways. This change installs PBR
within the repo_build role which will ensure all of the correct tools
are available to the repo build process and are installed using known
constraints.

Change-Id: Ibf5e3bf0381ec40b1adbe6ce49a5cf7284d3e147
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-03-28 16:05:01 -05:00
Jesse Pretorius d206e7a61b Ensure virtualenv is not broken for CentOS/SuSE too
In I2c50bb693dc5dc505bddce34ff45294ac44f7dc9 we applied changes
to the settings to repo_build_venv_command_options so that pip,
setuptools and wheel were not copied into the new virtualend on
creation, but rather installed into it later.

Unfortunately the patch missed that CentOS and SuSE platforms
have an override for the same variable, so the issue set out in
the patch still applies to those platforms.

This patch ensures that the overrides are removed and the
variable is only declared in one place, but the same functionality
which gave rise to the override for those platforms is still
applied when executed.

Closes-Bug: #1746935
Change-Id: Ie0c8c95a0d4315f8d1f848b1309bf88b7c2644d4
2018-02-20 10:33:43 +00:00
Jesse Pretorius 9a4e0b806a Prepare virtualenvs without breaking the virtualenv tooling
When preparing virtualenvs, we have an implementation which
tries to ensure that the versions of pip, setuptools and
wheel are the ones we want them to be. In order to do that
we have previously removed the virtualenv_support folder
which contains the pip/setuptools/wheel versions that
virtualenv shipped with. This was done to prevent any
leaking of the wrong version into the virtualenv and was
the only way of doing it with much earlier versions of
virtualenv.

We can now update the method of achieving the same goal
without breaking the virtualenv tooling, which this patch
does.

The virtualenv_support folder missing is causing the
repo-server role which uses virtualenv to fail in upgrades
to queens. That role needs a complete and properly working
version of virtualenv in place.

Change-Id: I2c50bb693dc5dc505bddce34ff45294ac44f7dc9
Closes-Bug: #1746935
2018-02-02 13:04:53 +00:00
Jesse Pretorius 12611a9440 Correct location of venv build logs
In [1] the intent was to have each venv build log
to its own log file. This makes troubleshooting a
lot easier due to the verbosity of the output in
the log - this at least separates each venv's
output. Unfortunately this extra CLI option was
supposed to be removed. This patch completes the
original intent.

[1] https://review.openstack.org/470757

Change-Id: I714d160a39beb870ba66db11a60cf0241ca770f3
2018-01-23 21:57:35 +00:00
Jean-Philippe Evrard f69ed50dd2 Remove all mentions of virtualenv-tools
Because of the changes in the other repos (removing
virtualenv-tools), it's expected that the repo-build process's
test don't see any mention of virtualenv-tools.

We should therefore remove anything related to it.

Change-Id: I4fc9a729b4dbac9c1101c47cdcff254a07d00435
Partial-Bug: #1741634
2018-01-15 14:23:36 +00:00
Logan V 3191800c07 Use git clone depth 1 for repo clones
The repo does not need to carry a full git tree in order to build
the specified SHA. We can specify a small clone depth to speed up
the git clone process.

Change-Id: Ic73cd938ba27b9e0030cc158b86828d279d476e3
2017-09-14 15:02:58 -05:00
Jesse Pretorius c505f88b12 Remove deprecated tasks/variable
Some tasks are no longer necessary in Pike
and the variable removed was deprecated in
Mitaka, so it's time to go.

Change-Id: Ibabe46967582929a441e66848c5267f0460a358b
2017-06-12 19:04:37 +00:00
Jesse Pretorius 8c3933c069 Make wheel build process idempotent
This patch implements the following:

1. The ability to use a boolean variable to always force
   a git clone or the wheel build to happen.
2. The use of local facts to mark whether a wheel build
   or venv build is required. This ensures that if there
   is a failure, the tasks will still be done when the
   repo build is executed again.
3. The git clone or wheel build tasks are only actioned
   if there is a change to the requirements or constraints.
   This ensures that when the repo build is executed
   without any changes, those processes will be skipped.
4. Re-arranges the tasks to ensure idempotency and to
   make the process easier to follow. A smaller set of
   tags are implemented which are intended to provide
   a clear code path for each of them.
5. Log output is added to the venv build process to make
   troubleshooting easier.
6. The stdout output for the wheel and venv build
   processes is made minimal to reduce confusion and
   make it easier to spot which item failed to build.
   The log output in /var/log/repo contains the verbose
   output from pip.

Change-Id: I2008926b43653edf50c284f5068160e27915c90a
2017-06-06 15:16:35 +01:00
Markos Chandras abf24e4bf0 defaults: Replace spaces with underscores in distro repository
ansible_distribution may return a string with spaces such as "openSUSE Leap"
so we need to replace the space with underscore in order to create a more
sensible repo name for the distro.

Change-Id: I0966786226d3f534a7f326d9df2cd73e7e3749dd
2017-05-19 11:08:12 +01:00
Jesse Pretorius 27ec4a229d Make venv build process idempotent
The venv build process currently executes on
every build, destroying all existing venvs for
the distribution/architecture of the repo server
it executes on, then rebuilds them.

It is also terribly difficult to troubleshoot
due to the fact that it is executed in parallel
through a single bash script using backgrounded
processes.

This patch breaks the build process up into two
parts - the per-venv options files, and the
script that uses the options to build the venvs.

With this breakdown we're able to do the following:
- Only execute a venv rebuild if the venv
  options (indexes, requirements) have changed.
- Use the Ansible asynchonous execution to
  execute parallel venv builds.

As a very welcome side-effect, this also means
that the venv build execution provides individual
output for success or failures, making it much
easier to see what went wrong when failing.

As part of the patch, the removal of the *.in,
*.txt and *.html files on each wheel build is
taken out. This is to protect the venv options
files. The removal of those files was unnecessary
anyway as they're templated and therefore replaced
by Ansible if they need to be changed.

Change-Id: I063c3addb6fbabb01d620be33aac2cab29a02750
2017-05-09 20:03:24 +01:00
Kevin Carter fe3ae20f74
Modified repo build for better multi-OS support
The repo build role will now store all wheels and venvs in directories
containg information about the OS in which the bits were build on. The
intention here is to ensure that all installation are able to support
multi-OS without running into issues caused by system packages and
different core libs. This change builds upon the multi-archetecture
support we already have.

Closes-Bug: 1641131
Change-Id: I3f36afa307e02a38d73b860fc23ed94a09882c34
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2017-03-15 13:05:07 -05:00
Jean-Philippe Evrard 96d4ee3694 Remove UCA/RDO in role
Now that the external repository management is done in pip-install
role, we don't need this wiring anymore.

Everything will be consistent across all the roles that include
pip-install.

Change-Id: I370135ab49dd75c2c3632b9f807b609c539bd4d6
2017-02-13 13:00:09 +00:00
Jean-Philippe Evrard 1d0493b58a Allow override of the repo filename
Apt cannot have 2 mirrors with the same content in 2 different files.
If a deployer has an apt mirror with uca, the deployer
still need to add a repository, but will also need to define the
filename used, in order to avoid clashes.

This commit makes possible to decide the filename for the repo, and
has the same behavior as Newton, allowing a backport.

Change-Id: Ie95dc9a90ad1772dc2e938fb1432d37a9c01f131
2017-01-10 13:01:32 +00:00
Logan V b309086be3 Fix selective wheel and venv build behavior
Change-Id: I9eb664308cf6f7f169dbe58b589c6784e77722e1
Partial-Bug: #1641346
2016-11-14 09:13:21 -06:00
Kevin Carter 534f40f3dd Statically define the repo build concurrency
The repo build concurrency was pulling the processor core count and using
that int as the value for determining concurrent builds. Sadly,
this value can be high, low, or simply undefined and is entirely dependent
on the Ansible fact gathering system which has additional complexities and
dependencies based on how the hardware has represented the information.

Examples:
  * Should the value be low, 3 or less than, the build process would run
    serially making things painfully slow.
  * Should the value be high the build process may run too many tasks at
    one time causing general system instability.

This commit statically defines the concurrency level as "4" which is small
enough to support vagant, and minimal resource machines, while also being
performant enough to support large scale production clouds.

Change-Id: If2ad496c31c4d0092e7856c3deb07035ebc6c8b9
2016-11-03 14:53:30 +00:00
Logan V c7bf677914 Fix linting issues for ansible-lint 3.4.1
Preparing this role for the ansible-lint version bump

Change-Id: Ia3f8c38271fc4e0315c9af7e7cc951c5e67acdf8
2016-11-02 13:28:29 +00:00
Kevin Carter baab4e2e7c
Updates to support ignored packages and external indexes
This change ensures that a venv create process can work when a
dependency is crafted outside of the OpenStack global requirements
system. The change keys off of the original package url setting the
isolated perameter should "withindex=True" be found.

The requirements.txt.j2 template file has been updated to allow
packages to be ignored from wheel building should the
`ignorerequirement=True` property be set within a given remote package.
This ability is important as some packages, generally external testing
repos, do not use requirement boundries and are known to break
given enough time.

Tagged tasks that MUST always be run have had tags set to "always".

Change-Id: I5d11aca84dcb74c77f3d0e3f31ce6546458a3e84
Closes-Bug: #1631992
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-10-27 01:42:53 -05:00
Ravi Gummadi e0eeeb039a Add cython to repo_pip_packages
Add cython to repo_pip_packages to make sure the wheel file for pandas
 can be built for ppc64le architecture.

Change-Id: Ib3210cff1b07302716cd91b798b81bb890e1f372
2016-10-12 06:32:08 -04:00
Jesse Pretorius e7baae22ae Ensure git folder rights and always run clone script
In https://review.openstack.org/380553 a change was made to
only execute the clone script if the script changes. This
has resulted in a failure when using a pre-staged git cache.

As the script only executes clone tasks if the git repo is
broken or the SHA isn't in the clone, it's OK to execute it
multiple times.

This patch ensures that the script always runs, but also tries
to add changes to the tests to try to uncover the cause of the
integrated build failure which the previous patch caused.

Additionally, the patch also adds tasks to ensure that the git
folder has the correct owner:group.

Change-Id: I40a02c26d0abc9d17dd7fad2eab969d169c436bb
2016-10-06 21:29:25 +00:00
Charles Farquhar 8fb9246ec6 Use pip_install_options when creating reqt wheels
This prevents pip SSL certificate verification failures in
environments with internally signed certificates by allowing the
certificate location to be specified in the pip_install_options
variable.

The pip_install_options variable is also added to the role defaults.

Change-Id: I447e5378254f88ac4fa87a9db3d34221a21e3685
Partial-Bug: 1621596
2016-09-08 16:09:38 -05:00
Adam Reznechek 628f01cba0 Add support for multi-repo/multi-arch repo builds
This patch adds support for building across multiple repo servers,
with artifacts for each CPU architecture being tagged appropriately,
then synchronizing the built artifacts back to the primary repo server
for distribution out to all repo servers deployed.

Partially Implements: bp/multi-arch-repo

Needed-By: I045de3ac8b81cadbcb34102f1a2db5bff74c32a6

Change-Id: I31756f8383e6d69d2f80caf6a85c4c5021bfc46d
2016-08-15 13:52:26 -05:00
Jesse Pretorius 0af26bced9 Only clone git repositories for services which are deployed
The current repo build process clones all git repositories for all
OpenStack services, regardless of whether they're actually going to
be used.

To reduce the repo build time, this patch will only clone the
git repositories for OpenStack services where there is one or more
members in the service inventory group.

Depends-On: I6a0508e8aaa4993bc68e72e3d2988d6db6470f6d
Change-Id: I6e5a554d6f87058ef5cb63adf7995b066cb6fd78
2016-08-05 07:18:11 +01:00
Jesse Pretorius fff4f1332e Only build wheels for services which are deployed
The current repo build process builds wheels for all OpenStack services,
regardless of whether they're actually going to be used.

To reduce the repo build time, this patch will only build wheels for
OpenStack services where there is one or more members in the service
inventory group.

An updated set of test vars is included based on a recent stable/mitaka
output of the py_pkgs lookup based on the below-mentioned dependant
patch.

Depends-On: I5e981c818a1da50aa0ae15630655a8dfac4f3db7
Change-Id: Idda16b4f382eee57c7469af898859d6d81d4eb30
2016-08-03 18:59:04 +01:00
Jesse Pretorius f141d6dc82 Add ability to change apt/yum package state
The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.

Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.

This patch adds the ability for a deployer to change the desired state
so that the results are predictable.

Change-Id: Ic570c4029b06918e223f21da02012f330911ad51
2016-08-02 15:34:50 +01:00
Jesse Pretorius 22b14ceb7f Provide ability to override upper-constraints
This patch adds a new list variable which allows a deployer to
override OpenStack's upper-constraints for the repo build
process.

Change-Id: I46ba9345a85c43f2a1f24ea1819132e7474e6909
Co-Authored-By: Jimmy McCrory <jimmy.mccrory@gmail.com>
2016-07-29 09:57:41 +01:00
Michael Gugino be53a01eaa Fix out-dated requests and urllib3
The repo build server sometimes includes old versions
of the requests and urllib3 python packages.  These
old versions may be due to outdated system libs.

This patch instructs pip to install requests and urllib3
to the latest version on the repo-build host.

Change-Id: I78dd9b1430f13e23fe153b162ef4aa38b216c279
2016-07-25 18:01:03 -04:00
Jesse Pretorius c391fd3eb0 Only build venvs for services which are deployed
The current repo build process builds venvs for all OpenStack services,
regardless of whether they're actually going to be used.

To reduce the repo build time, this patch will only build venvs for
OpenStack services where there is one or more members in the service
inventory group.

Change-Id: I01e6cdaa7a1a4aa07d2118c0badfea25b85b3249
2016-07-21 11:02:28 +00:00
Jenkins 3c37bc3f42 Merge "Add and enable UCA repo by default" 2016-07-19 12:50:41 +00:00
Jesse Pretorius c06679ccc3 Add --no-binary option to wheel compile task
There are situations where you may wish to ensure that a package
is always built from source, regardless of whether a pre-compiled
wheel is available on one of the indexes.

This patch adds the ability to set a list of packages which must
always be compiled from source.

Change-Id: I2f47d94c58503514a3f9a0a62081cbc1ca53613b
2016-07-19 08:30:50 +00:00
Jimmy McCrory f4f7009925 Add and enable UCA repo by default
When the UCA repo is enabled for nova, a newer version of libvirt will
be installed. The repo_build role will also need to have the UCA repo
enabled to install the same version of libvirt-dev so that a
corresponding libvirt-python wheel will be built.

The test preparation implements an override of the URL for OpenStack-CI
to make use of the local mirrors.

Change-Id: I6d5532e862e98929cff868f940c202875d1ee560
2016-07-19 00:45:26 -07:00
Jenkins 4897319e3a Merge "Work around Ansible processor fact bug on ppc64le" 2016-07-12 07:23:03 +00:00
adreznec b1bb5ecced Work around Ansible processor fact bug on ppc64le
This works around a bug in Ansible where ansible_processor_count is
reported as 0 on ppc64le today due to issues with /proc/cpuinfo
format differences. See this issue [1] for current status.

[1] - https://github.com/ansible/ansible/issues/16533

Change-Id: Icc26c415fb36aefc6ad22eb76073bcd1f8878466
2016-07-11 11:16:40 -05:00
Kevin Carter 34e3da032c Updated repo-build to store package sources
This change enables our repo-build process to store the package sources
for all of our requirements within a given release. This change enables
the ability to have a multi-archetecture / multi-distro / multi-interpreter
deployment.

Because a python wheel can be tied to a particular archetecture, distro, or
interpreter wheels alone can not be relied on to power a mixed environment. The
source is needed to ensure pip is able to resolve a package depenedency even if
a pre-built wheel does not meet the python or system requirements. To enable
this a task has been added to the repo_build.yml file to first download all of
the sources in non-binary format, once downloaded the sources are used to build
all of the wheels locally.

Change-Id: Ib081c24f67e92c165cba14848ff86e20fe6c1530
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-07-07 15:07:08 +00:00
Travis Truman 19cfa68874 Multi-distro pattern for repo_build role
This is an implementation of a pattern for multi-distro support

This allows for adding support of distros other than Ubuntu

Change-Id: Ife463766563ace3a329718a8df5ad7d2b38548c6
2016-04-27 15:18:48 -04:00
Kevin Carter d9154a3a9d update repo-build for ansible 2.1 compat
This change updates the repo-build process to be ansible2.1
compatible. The change is needed because the if statements in the
sting cause a \\ to exist if the variables are not used. To resolve
this issue the entry to the shell call has been changed to a folding
style instead of a literal style which will no longer generate the \\.

A change was also need in the op-venv script where we rely on a bash
script to build all of the venvs based on input provided by a folding
style string. Within Ansible 2.x the folding string type may have a \n
on the end which will break direct bash bash functions.

Change-Id: I394ab60f5505a45cf39b1efce1016dad9332be78
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-04-10 20:44:56 -05:00
Jesse Pretorius b769f0a03b Make the use of the repo index overrides optional
Currently the repo build process hard codes the use of a set of indexes
when building wheels. This overrides any existing pip configuration in
the repo container and is somewhat unintuitive as a result.

As the default index is pypi, which is the default for pip anyway, this
is a redundant hard-code which really shouldn't be required.

The specification of additional indexes should also be optional, as the
default of using pypi again just results in a duplicated check by pip.

Also, for context, OpenStack-CI gating will prefer the use of a
system-wide pip.conf which will be copied from the host to the container.
Without this patch, that simple solution to our gating process will not
work.

Change-Id: I25e4faecbffad6301509c62f55c34ba86b92a658
2016-03-30 16:08:35 -07:00
Kevin Carter 7348e023ac
Updated repo_build process to be faster
The repo_build process is slow and can do with some improvement.
To achieve better speed of execution some several slow tasks,
as profiled by the integrated gate have been moved into executable
scripts using jobs. These scripts are generated from the playbooks
and the Jinja2 variables created during the repo_build run time and
make it possible to perform multiple actions in parallel. In test
the results were significantly faster[0].

The test case has been updated to build all of the packages known
to exist and be required for a real OpenStack environment.

[0] - http://paste.openstack.org/show/491537/

Change-Id: I3a1872741a17dee25054a340859333b96203ca15
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-03-24 14:50:06 -05:00
Jesse Pretorius 644a33f9ba Always rebuild venvs when rebuilding wheels
The var 'repo_build_venv_force_rebuild' was made available to
allow the venv rebuilding to be optional, and defaulted to
false. This causes a mismatch between expected behaviour and the
actual behaviour.

When executing a repo-build, the deployer expectes the wheels
and venvs to be rebuilt. The actual behaviour is that the wheels
are rebuilt, but the venvs are left alone.

If a deployer/developer is updating SHA's for anything within the
same tag (or following the OSA master branch) then the services
never get deployed with updated code as the venvs never change.

This patch removes this var altogether. Deployers may still use
tag skipping to isolate tasks they wish to run if that's
required.

Change-Id: I8ef8d8c13e067223eb908fd2177cef8beb929397
2016-03-23 17:05:33 +00:00
Jenkins 852de167a6 Merge "Install packages necessary to build Python wheels" 2016-03-16 17:13:59 +00:00