In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.
Change-Id: I0c3f20032fc673a20088fb6a946c7af57d6a4841
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.
We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.
This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:
1. Replaces 'developer mode' with an equivalent mechanism
that uses the common role and is simpler to understand.
We will also simplify the provisioning of pip install
arguments when doing this.
This patch also cleans up the unnecessary tags and makes a
few changes to the role to tidy things up a bit.
Depends-On: https://review.openstack.org/598957
Change-Id: I61d005d4732bd1d6cf4125cbf1adfdb1479a4b1b
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: I7a5b4cf134a73e9dc6eda08c935696c5b8b8a034
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
With the dependent patches, the openstack_openrc role is now executed once
on the designated host, so it is no longer required as a meta-dependency for
the role.
Depends-On: https://review.openstack.org/582359
Change-Id: Ic844753c3ece4383eefa490f00be0bff441c6a7e
We reinitialize the venv to ensure that the right version of
python is in the venv, but we do not want virtualenv to also
replace pip, setuptools and wheel so we tell it not to. If we
don't do this then virtualenv will install the latest available
version, which is not what we want.
Change-Id: I8772e90fa3ff2653836870921ee0e182150715e4
Partial-Bug: #1764470
In the previous patch removing the virtualenv-tools, we've updated
paths to the proper binaries, but we don't reset the virtualenv
like what we used to do.
Change-Id: I05e02d36bed208b44f666ddb4ace8e847c88b76a
Partial-Bug: #1741634
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: If5d213bbe27c7f07508f4c20e7d3f5d911ba974d
Partial-Bug: #1741634
In order to do a developer mode that allows installation of packages
from local file, which take precedence over the developer_mode
constraints, we need to allow the order of the constraints to be
changed.
This patch adds a "pip_install_developer_constraints" var which is used
to set the developer mode constraints. By default this will leave the
same behaviour but will allow additional constraints to be added, or the
developermode constraints file to be overriden altogether.
Change-Id: I4022629a957a11ac90026a4df13855324db4c2a9
We use an SSH bastion host which we do our deployment through. The
deployment host doesn't have direct access to the same network as the
host. As a result the venv local checksum lookup fails.
I have described this here:
https://bugs.launchpad.net/openstack-ansible/+bug/1689283
This is a simple fix for this problem, assuming everything is good it
will need repeating in multiple places in the code base.
Change-Id: I5b1ac355370365a7895033e772b418b8c1db8f3f
Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server
Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.
Change-Id: Id41f3963f4be4444dca8362e6f71f537f01429f2
Reinitializes (copies python, etc binaries) into the venv when
dropping a new venv into place. This is needed because the Python
binary packaged with the venv may not match the Python running on
the host it is being installed to. (ie. in the case of a Xenial
repo container and a Trusty target host.)
Change-Id: I4ea706ac12e37c42eec41d5d734ad95cdcef9e58
Partial-Bug: #1637509
Ansible 2.2 now treats the 'name' argument for the pip module
as a list, removing the need for us to implement the join
filter to optimise the install execution.
Change-Id: Ifbeae6a808f6c8be34cd7bb4d3a8d8e7ba1e5c9d
Starting in Ansible 2.0, the get_url [1] module provides the
ability for a checksum to be provided to the get_url module
which will be verified against the local destination file
and the task skipped if it matches.
[1] http://docs.ansible.com/ansible/get_url_module.html
This patch implements the use of this functionality.
The ability to ignore a venv download failure is also removed
as this is not necessary or desirable. It is better for the
download to fail and the playbook execution to stop immediately
so that the failure point is exposed.
Change-Id: Ibe9d3a98e663cb1c1d7ae07aa10c9cee8e6aba3f
The current constraints generation for the
installation involves multiple tasks and multiple
variables.
Using multiple tasks extends the installation time
unnecessarily and the additional variables are
unnecessary.
This patch aims to simplify the mechanism and
hopes to speed it up a little.
Change-Id: I6784878e7a46560bebfb0be9d51b048da3774f70
This change removes the use of 'ignore_errors: true' because it causes deployers
to see red output and a stacktrace, which traditionally means something is broken,
even when the failure is known to have a fall back option or be intentional. This
conversion will provide a generally cleaner interface.
It should be noted that the 'failed' filter will still function normally. Tasks
with the 'failed_when: false' option will still be marked as 'failed' in any
registered variable. This change simply makes the output look cleaner.
Change-Id: Iafffee89b2629643799efd1e2fa233de9ca281f6
Closes-Bug: #1633438
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Ansible 2.1.1 introduces a regression in the way conditional
includes are handled which results in every task in the
included file being evaluated even if the condition for the
include is not met. This extends the run time significantly
for a deployment.
This patch forces all conditional includes to be dynamic.
Change-Id: Id26b3f254e68f0707c6d8b23b501868213b11224
Related-Bug: https://github.com/ansible/ansible/issues/17687
In order to make it easier to differentiate between the lists of
python packages, distribution packages, downloaded packages,
package pins and other similar variables the variable names are
being changed to ensure that they have a more explicit suffix
that defines the purpose and makes the naming more consistent.
This is to facilitate a lookup plugin which will be able to look
up all the package lists and present them as a consolidated piece
of data which may be used for artifact preparation.
Change-Id: Ib607e78bd3be2f50fbfe2c7098706df7e91cd780
The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.
Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.
This patch adds the ability for a deployer to change the desired state
so that the results are predictable.
Change-Id: I3e99a77fb53ba91367c6548523b9e0f3b910e469
Previously only role convergence was tested.
This patch paves the way to have functional testing support
the addition of Xenial support as well as adding more
thorough functional testing using the Tempest plugin
available in the zaqar source.
Change-Id: I593fe5e61df2daca0857548394e04aa2afa35499
Unlike the Ansible apt module, the Ansible pip module does not
recognise a with_items list and process all the items at once.
To optimise the pip install tasks, this patch replaces the use
of with_items with a join filter so that the pip install task
does an install with all the packages in a list, ensuring that
the execution is one action instead of many.
Change-Id: Idfb5059b60b2edb61113fecc9ad33f90a2ab9e65
Remove all tasks and variables related to toggling between installation
of zaqar inside or outside of a Python virtual environment.
Installing within a venv is now the only supported deployment.
Additionally, a few changes have been made to make the creation of the
venv more resistant to interruptions during a run of the role.
* unarchiving a pre-built venv will now also occur when the venv
directory is created, not only after being downloaded
* virtualenv-tools is run against both pre-built and non pre-built venvs
to account for interruptions during or prior to unarchiving
Change-Id: I4a67a53d3d685c1e2f937114352569c9f45a74a0
Implements: blueprint only-install-venvs
When executing the role with Ansible 2.1, the following
deprecation warning is issued in the output for some tasks.
[DEPRECATION WARNING]: Using bare variables is deprecated.
This patch addresses the tasks to fix the behaviour appropriately.
Change-Id: I7a8a0b0dc3d479834a1f5d037aae382e2d51ccc1
* The default apt packages have been moved into a var file that is only loaded when the detected OS is matched.
* The Install task file has had the apt specific tasks moved into a named install task file.
Change-Id: Id95826a9dd9e5d2ed83a95c3f02d18929735a8f2
Implements: blueprint multi-platform-host
Currently all pip install tasks only require the package to be
present. This means that when an environment undergoes a minor
upgrade the package is not upgraded to the same version that
was tested with. This ultimately results in a deployed
environment that does not match the tested environment.
While for the services installed into venvs this is not an
issue, it does affect those which do not use venvs and any
packages which are installed outside of a venv or on top
of a venv.
This patch changes the behaviour to ensure that the install
task will always use the latest available package. In
developer_mode this will mean using the version specified
in upper-constraints, and in an integrated build this will
mean the version which is available in the wheel repo's
folder for the tag.
Change-Id: I8d67e0a42b82c5502f10e9307fbfd932c1dd2476
Functional testing will follow in another patch
but we should have convergence testing in the meantime.
Change-Id: Ib3d596eea7f6ded233a3b1f9a06e3e893764c0e5
Add clarity to the role's function by
removing usages of with_items where only a
single item is present.
Change-Id: Ic057e1fff4ae4f5969662230feb14561752b8d22
Now that auth token usage is deprecated, prefer the admin
user and password for all zaqar setup tasks run against
keystone.
Default vars in keystone namespace have also been removed.
Change-Id: Ic6639d5765925ce6a50b3e9f701eafb45cd6a004
Guilherme Steinmüller