Commit Graph

43 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov d9df50f4e3 Retire pip_install role
As repo_build role has been succesfully replaced with
python_venv_build role, we have no need in it's futher maintaining and
releasing.

Depends-On: https://review.opendev.org/717715
Change-Id: Ibf9ce1d602c5cfa9ed7431ca37de02df3335b36a
2020-04-06 14:27:00 +00:00
ShangXiao 32046d423e Update the source link to the lastest one
Change-Id: I3f2f5894c9b35f7e9b4173f6e9c44332fe8020eb
2019-11-04 17:02:26 +08:00
Jesse Pretorius 79e3480a68 Allow the role to only configure pip, not install it
There are times when you may wish to just configure pip,
not install it. This adds a flag to enable that option.

Change-Id: Ibaa3c0b6b323f22dbd640562362106166bbc0a0a
2018-08-01 15:48:03 +01:00
Jesse Pretorius (odyssey4me) 889b1248ea Revert "defaults: Add virtualenv to default pip packages"
This reverts commit d148471894.

In order to install the virtualenv package using a distribution package
in the openstack_hosts role, we need to remove it from being installed
here.

Change-Id: I26046ab20cef0012a2d6c52c29827908f2d9f81c
2018-08-01 15:46:54 +01:00
Kevin Carter 79063a3e10
Update get-pip to version 3.3
Version 3.2 has a bug with it on some systems that can result in the
following stacktrace: [ http://paste.openstack.org/show/726064 ]. To
correct this issue the version of the get-pip script has been set to 3.3
which does not suffer from the same issue.

Change-Id: Iaa824cc64bd0e8bdfdda4de243d80400f1529e8c
Closes-Bug: #1779534
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-16 21:10:47 -05:00
Jesse Pretorius 6c00fab455 Pin get-pip.py to 3.2
As get-pip.py evolves based on pip 10, while we're still using
pip 9, changes in the way it can be used are causing problems.

For example, the ability to use --download is no longer there.

As such, let's pin to a known good version and leave it at that
until we no longer need to use this script. Version 3.2 maps to
pip 7.1.2 which fulfills our needs.

Closes-Bug: 1779534
Change-Id: I5a4d4096457d84fcf3a9e21a6873dd61f2a456fc
2018-07-04 08:24:15 +00:00
Markos Chandras d148471894 defaults: Add virtualenv to default pip packages
Right now, every os_* role explicitly requires and installs the
'virtualenv' package since they all deploy virtual environments for
the services. As we move towards using distribution packages as an
alternative, we can move this dependency to the pip_install role
so we can have a central place to reference the common PIP dependencies
across the hosts.

Change-Id: I4eb0a70dc6aa4437a6aa52caf510225c49ef8b11
Implements: blueprint openstack-distribution-packages
2018-04-23 16:14:22 +01:00
Zuul 670f97b4a0 Merge "Disable the pip version check" 2018-04-22 13:29:20 +00:00
Jesse Pretorius 70b8cd5737 Disable the pip version check
In our logs it is frequent that the warning to upgrade the
pip version is present, but this warning is unnecessary as
we are always using a well tested version of pip.

To reduce noise, we set disable the version check. This has
the added benefit of not adding more internet access to every
pip task than is necessary.

In the template we ensure that the var is interpreted as a
boolean (in case of user error), then force it to lower case
as that's what pip prefers in pip.conf.

We also now remove the test for the self check file, as it
will no longer be present with this option implemented.

Change-Id: Ie0427d3435270836302441b92eaff7e7980d7047
2018-04-20 14:31:55 +01:00
Markos Chandras 2c724509d1 pip: Use 'upgrade-strategy' option to make PIP upgrades less aggressive
The default upgrade-strategy could change from one version of PIP to
another and the results may be unpredictable. We can control the PIP
behavior during upgrades using the 'upgrade-strategy' option and default
that to 'only-if-required' so PIP only upgrades what's really necessary.
This also ensure that getting the latest pip passes the idempotence test
now that we only upgrade only what's really necessary.

Depends-On: I998182eca9851d2cc745930fc73ca37bfefd0951
Link: https://pip.pypa.io/en/stable/user_guide/#only-if-needed-recursive-upgrade
Co-Authored-By: Jean-Philippe Evrard <jean-philippe@evrard.me>
Change-Id: I2ba89e25c0010c9a5b515a3d0c9c731b30876e74
2018-04-18 16:15:26 +01:00
Jesse Pretorius 3133ee8384 Remove pip_required_pip_packages var and related tasks
The pip_required_pip_packages variable was introduced in
order to facilitate installing extra packages required
for SNI support. In Pike these extra package installs
were removed due to interference with distribution
packages installed, and the distro package installation
for this support was deemed better.

This variable and the related tasks therefore no longer
server any function and should be removed.

Change-Id: I76fb9ae324ff69e8fc8b23f7b610d2aca48d0949
2018-03-11 08:59:11 +00:00
Jesse Pretorius f9d65b184f Remove pip_lock_to_internal_repo variable
The ability to lock the pip configuration down has
now been extended so that it is possible to implement
a different index and to use find-links to achieve the
same goal.

The only replacement implementation could be to disable
the setting of an index if one wanted to *only* use
wheels, but that is unnecessary in OSA builds as we are
able to set the build to only use a specific set of
wheels through the pypiserver on the repo container.

As such, this variable has become useless and unused.
This patch therefore removes it.

Change-Id: If472c845036ab904ffc15ca7201ae68d282c1be0
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2018-02-18 15:49:16 +00:00
Jean-Philippe Evrard a6216fa4a8 Cleanup pip install role
With the change in openstack_hosts repo, we can remove
duplications in the pip_install role.

Depends-On: I541287f906a143ed79553c69e1a9e39ee38dc8dc
Change-Id: I1afc143579689a690eddbbf2423a4cd0501f9688
2017-12-01 03:22:36 +00:00
Jesse Pretorius 5f566318d5 Add ability to disable implementing pip.conf
On something like the repo server we do not want to
change the default pip settings so that we allow
the global pip.conf or default pip settings to do
the right things.

Change-Id: I7bd31f2d89d3fe9d48e32c79ddef7a8ef1392eb7
2017-11-24 16:53:48 +00:00
Jesse Pretorius 4606182b00 Add ability to set the default index
In this patch we allow the default index to be
changed so that we can set it to the repo server
when the pypi reverse proxy is available to be
used.

The repeated implementation of the find-links
in the install section is removed as it is
unnecessary - the global find-links option
applies there too.

Implements: blueprint python-build-install-simplification
Change-Id: I56f22f46ff849a7049ac7ae873af7a4bc526fa63
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2017-11-24 09:51:36 +00:00
Markos Chandras 0544c622b2 SUSE: Add ability to configure external mirror for openSUSE
Allow deployers to choose a specific mirror by setting the
'pip_install_opensuse_mirror_url' variable

Change-Id: I4f4cc284ab7d1c3378e01737f9fade3afe251824
2017-09-07 10:22:44 +01:00
Major Hayden 3500a26919
Allow override of RDO repo baseurl
This patch allows deployers to set pip_install_centos_mirror_url
and override the default RDO repository (mirrors.centos.org).

Change-Id: I7b0dd5d14c82a74be955071e638c122ffb45ab31
2017-08-21 07:35:28 -05:00
Jesse Pretorius 1bd49f3201 Ensure that role tests pin pip/setuptools/wheel
Currently the role tests use whatever versions of pip,
setuptools and wheel are already installed on the host.

When a version of these tools changes it often causes
problems for our testing.

This will ensure that we use a known good set of pins
which is maintained in the general SHA bumping process.

Change-Id: Ibdbe49fe6d408627762185aaa20a609f6b5e2343
2017-06-01 12:39:48 +01:00
Jesse Pretorius 793ae4d013 Resolve user_external_repo deprecation warnings
Warnings about 'user_external_repo_keys_list' and
'user_external_repos_list' being undefined are
confusing.

Closes-Bug: #1670021
Change-Id: I2fe78300dd04066515f4c5b175d56cc1a6628c9d
2017-03-29 11:37:48 +00:00
Jean-Philippe Evrard d8526a61f7 Fix documentation for external repos and keys
Currently, this lists the old method of only using a dict instead
of a list of dicts.

Change-Id: I192fa24b6ddc3c2d92140e8cf587604c913da9b0
2017-03-23 17:46:23 +00:00
Jean-Philippe Evrard 1de94eb012 Use RDO package for master and ocata
This brings more recent version of packages (needed for
liberasure).

Change-Id: Id68a300c8060561a82dbba5f9499c9da984dd969
Co-authored-by: Marc Gariepy <gariepy.marc@gmail.com>
2017-02-23 11:42:13 -05:00
Jean-Philippe Evrard 27669f193c Allow the usage of own repos instead of uca/rdo
If a deployer has a mirror of uca/rdo, it probably wants to use it.
Defining the URL currently works, but that process wouldn't work if
the mirror is using a different key.

This commit keeps the current behavior of using UCA with signing
packages from upstream everywhere, but with this change, you can also
define your own mirror (CentOS/Ubuntu) with your own key.

Change-Id: Icd9c3b6b7b824cb5cb29fd745777e4c6148e6481
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>
2017-02-13 12:50:46 +00:00
Jean-Philippe Evrard fe4fa18822 Use RDO/UCA everywhere
The pip_install role installs binary dependencies, like python-openssl
(on Ubuntu), or pyOpenSSL (on Centos).

By default, we didn't configure extra sources (like UCA or RDO)
in the pip_install role. Because the repo is built using UCA or RDO
(current default), we could have inconsistency issues:
containers that haven't UCA or RDO applied during their role execution
would run with a venv packaged for a different python-openssl
(or python-cryptography).

This commit brings the consistency by installing UCA/RDO everywhere.

Closes-Bug: 1624791

Change-Id: I9b5cd40b2972c93af348d4ddfde21a038cf9becc
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>
2017-02-07 13:36:47 +00:00
Jesse Pretorius d3d8276639 Provide toggle for get-pip.py get_url task force option
This patch provides the ability to toggle whether the get-pip.py
script is always fetched from the source when the task is
executed.

In the integrated build, the get-pip.py script is put into the
repository, so the default of 'force: yes' is appropriate.

In role tests, however, we may wish to skip always fetching the
script to reduce errors relating to SSL validation when the task
executes a second time.

Needed-By: I8b1c570c3a0713de3e84fb4e4068e270d9e5a4dd
Change-Id: Ifb877e859a302b85c877aa867a0fc630b8765a19
2016-12-16 15:11:12 +00:00
Logan V 379048a8f4 Conditional switch to install compiler
Adds a conditional switch to install the compiler and python
development libraries. The switch is enabled by default to
preserve the old behavior.

Change-Id: If484e14a49c97467a38f9ab6e4b0e3351f60d4ed
2016-12-12 12:40:51 -06:00
Logan V 8236a981a5 Remove unnecessary package installs
Xenial PIP installs do not need all of the extra packages laid
down on the host for SNI support like Trusty did.

Change-Id: Ia3fe82b78ef06d8b8d90eb5071bf0e284dba38b9
2016-12-10 12:20:25 -06:00
Jesse Pretorius 92fbfc238d Add ability to set upper constraints file
When this role executes the installation of packages
for SNI support it should also respect upper constraints.

This patch adds the ability to set an upper constraint
file URL/path.

The tox config is also adjusted to ensure that it makes
use of upper constraints when installing requirements.

Change-Id: I2ab6f55096b560a7b443de412df168e26a3eca07
2016-10-18 16:59:58 +01:00
Neill Cox 571fe1f127 Make pip_install honour pip_install_options
Add a pip_install_options variable to the extra_args parameter of
the Ínstall pip packages task.

The variable is set to a default value of "" in the roles default
variables.

Change-Id: I3ffead4f63c85215c2d6b6fec5a02837cc7f7551
Closes-Bug: 1620872
2016-09-09 11:06:30 +10:00
Jesse Pretorius ae00b93ebc Ensure that apt cache is up to date before installing packages
Change-Id: I6db566bc0a1e3a9d2ad423389522f1c1c418d07c
2016-08-15 12:49:19 +01:00
Steve Lewis c1323f2d77 Install required packages for get_url module
To prevent Ansible get_url module from generating SNI warnings
several system packages are being installed so that system python
will allow us to get pip and install it.

The strategy for using system pip to install the python packages
to prevent SNI warnings causes additional issues as system package
pip versions may be broken in spectacular ways which prevent us from
upgrading to pinned versions of each of pip, setuptools, wheel.

Change-Id: I45e3eff716dba4fb2926794bc25bfb079bb328bf
Closes-Bug: 1612377
2016-08-13 03:28:21 +00:00
Jesse Pretorius 57c5ec07df Add ability to change apt/yum package state
The current method of installing the distribution packages required is
set in the tasks and cannot be changed by a deployer.

Currently the apt task always installs the latest package. This results
in unexpected binary changes when a deployer may simply be trying to
execute a configuration change.

This patch adds the ability for a deployer to change the desired state
so that the results are predictable.

Change-Id: I7ddc32b286071b29c197b18370cf49af235527fa
2016-08-07 15:56:38 +01:00
Kevin Carter aafb6dcb81
Add pip to the default package set
This is needed so "pip" will upgrade itself if no other "pip_packages"
have been defined which is needed to unblock gating within the IRR
roles.

Change-Id: I6dd389666500c95023b8f4b76f19a32c55fcb2f7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-08-06 07:25:55 -05:00
Logan V 17e59991da Fix the pip required packages variable name
The SNI changes that merged in
Idcf773e16d62d2ad76d9341177dd4d6c3e410af3 are not being picked up
by py_pkgs and the repo build process because the variable name
does not match the *_pip_packages format.

Change-Id: Iea9857870c498d70f501aaf7d3528aad5737418e
2016-08-05 20:26:18 -05:00
Kevin Carter 30d59585e8
Add SNI support via pip
Some Linux distribution releases, such as CentOS 7 and Xenial, have
trouble validating SSL certificates when using get_url with servers
that use Server Name Indication (SNI).

This patch adds packages for those distributions that allow Python
to validate connections with servers using SNI.

Change-Id: Idcf773e16d62d2ad76d9341177dd4d6c3e410af3
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-08-05 18:44:02 -05:00
Jimmy McCrory afb88cbf2e Rename pip_packages_tmp variable
The py_pkgs lookup plugin looks for any variables containing
'pip_packages' to find names of Python packages to build.
https://github.com/openstack/openstack-ansible-plugins/blob/master/lookup/py_pkgs.py#L57

Integrated OSA gate testing is currently failing since each character
within '/tmp/pip_install/' is being considered as a package name.

Rename the 'pip_packages_tmp' variable to 'pip_tmp_packages' to avoid it
being evaluated by py_pkgs.

Change-Id: I4d0f40dbb254571b9871d407962627393083d618
2016-06-20 18:59:13 -07:00
Logan V bd9c9cd617 Refactor pip_install to support offline deployment
Optionally support the download and installation of pip and its dependencies
via the deployment host to remove the dependence of internet connectivity on
remote containers.

When pip_offline_install is true, download get-pip.py and the pip
installation wheels to the deployment host. These wheels along with get-pip.py
are copied to the remote container and the installation is performed
offline.

Change-Id: Ic6583c54976b6fb083d9c67d055897c15bce05dc
2016-06-12 16:45:10 -05:00
Travis Truman c7d3889bd8 Combine pip_install and pip_lockdown roles
Since lockdown is really just a form of specific pip
configuration and since the pip_install role already does
some pip configuration, it seems logical and desirable for
these functions to take place in the same role.

This change should pave the way for a simplification of
role dependencies and the removal of explicit pip_lockdown
role usage with various playbooks that will already have a
dependency on pip_install.

Change-Id: Ia0fc276c2b501f16d4acf73bbbcad6f80804628e
2016-05-03 11:12:18 -04:00
Jesse Pretorius f5d179a64d Provide the option to specify a list of requirements when installing pip
This patch removes the current pip pin and changes the role to provide a way
for a consumer of the role to specify any requirements they want to in a
list.

This is a more generic facility and allows the consumer to specify the
requirements in a playbook or group vars instead of being forced to
override them as they're defaulted into this role.

Change-Id: I0a78135737c40a505d77df6ed67da0ef6695bfcb
2016-02-26 10:03:01 +00:00
Hector Gonzalez Mendoza d7b4a76c83 Make Certificates validation configurable
When using proxy settings, sometimes the proxy server intercepts
the certificates making the download fail. Set the default to "yes"
may be set to "no" when using proxy settings.

Change-Id: I891a0e59833978660879bdd85f584b6da04663e9
2016-02-03 20:47:43 +00:00
Jesse Pretorius 35e41a9981 Implement pip version range
This patch implements a major version range lower and
upper bound to ensure that both the repository and any
online retrieval of pip result in the same major version
being deployed.

Also, the isolated fallback should not use the extra options to
ensure that it goes to the internet to find the pip wheel
to install.

Change-Id: I93a5abdbc001348075cb43a0913407d38ae89678
2016-01-20 21:05:50 +00:00
Jesse Pretorius 820c9bd6c3 Update pip primary and fallback URL's, and set version
This patch updates the primary and secondary locations for
downloading the get-pip.py script and also sets the
installation process to ensure that a known, stable
version of pip is installed.

Change-Id: Ib463f61b1ad44aec0e3953a89d58596c3e87f2e0
2016-01-20 16:00:54 +00:00
Paul Belanger 654e5b51ea
Change pip install to a stable version by default
The change flips the git url and the default pypa bootstrap
url. This will ensure we have a known version of pip by default.

Change-Id: If406a2321d987619de434ec0b0093f1902c5cdae
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2015-12-21 12:19:45 -06:00
Kevin Carter 2dd4df9d3d
IRR for pip_install
The change moves the role out from the main repo pip_install
repository and into its own standalone repository.

Items within this change:
  * The role has been updated to ensure it runs standalone.
  * Tests added to the role within tox.
  * Functional tests added to the role that can either be run
    via the run_tests.sh script or using tox.
  * dev requirements have been updated for testing usecases.
  * Docs added to both the README.rst file as well as the docs
    folder.

Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2015-11-03 02:34:10 -06:00