Commit Graph

59 Commits

Author SHA1 Message Date
Riccardo Pittau fb6d6d6fe9 Fix enabling epel repo for rpm distributions
The epel packages are installed but their content is purged in the
centos nodes. Removing and reinstalling them when we run tests in
CI to reenable their content.

Change-Id: Ibf1d810a37811151449f79a24bb837db6663cce5
2023-03-08 16:43:58 +01:00
Erik Berg eb5fab25a3 Remove remaining traces of Suse
It's been deprecated, time for removal.

Change-Id: I84c441f54c5793fe94b50810cda25f9eacee8c7e
2022-10-24 17:50:19 +02:00
Erik Berg 7640a231a8 Allow for almalinux and rocky installations
With the changes to bifrost-ironic-install required_defaults for
the RedHat_family, we can now try to not get in the way of
deploying on distros like almalinux and rocky.

Fedora 32 has been EOL for close to a year, so we'll just assume
firewalld for that case. Distributions RedHat, CentOS and Fedora
are in the RedHat_family, so assume firewalld all distributions
in the family.

Change-Id: I1c8b768a3daf0af0d8e30d935558b6fdd912adfd
2022-04-19 15:29:06 +02:00
Dmitry Tantsur 4cb0395d75 Make virtual environment owned by root
It's dangerous to allow a regular user to write into the source code of
the services. The repositories are still writable, so that the
development workflow can still use the ``--develop`` flag to make
modifications.

While here, remove the horribly outdated and broken ANSIBLE_INSTALL_ROOT.

Change-Id: Id2e25dd57668d24a11dc2cd16eea2b607b7a3f16
2022-01-25 15:59:42 +01:00
Riccardo Pittau 95c429b5ba Use explicit path for which
Use the /usr/bin/which path for which in rpm-based distros to avoid
issues with any aliases.

Change-Id: I3a4f2d667548f25513829579b22712a44e7c6b0f
2021-04-08 17:46:26 +02:00
Riccardo Pittau 381aa41d8f Upgrade pip version to fix pep-517 enforcing
This upgrade fixes an issue in some disitributions when the build environment
enforces pep-517 and breaks compiling some versions of python libraries.
For example see [1]

[1] https://github.com/sqlalchemy/alembic/issues/707

Story: 2008676
Task: 41973

Change-Id: If3e5d37a42de2fb25c8034d9fdebb16d8bd21e9f
2021-03-03 18:22:29 +01:00
Dmitry Tantsur d048a5f4c4 Restore the conditional verbosity in bash scripts
Became unconditional in 422bafd6ea

Change-Id: I6e9b32d1ca4651daea26330c96c63d40e63cec00
2021-02-15 17:44:54 +01:00
Riccardo Pittau 422bafd6ea Exit on all errors in bash scripts when possible
Except for the collect-test-info script, the other bash scripts should
just exit on error.

Change-Id: Ia68bef56edd7edec14f678155220b6e4e15e57f6
2021-02-09 12:05:11 +01:00
Riccardo Pittau 5360d0e869 Install at least pip version 19.1.1
This is to avoid a known issue with a recent version of cryptography [1]

[1] https://github.com/pyca/cryptography/issues/5771

Change-Id: I034bcd24031b5881ae49b8bc03bed6654cd1d335
2021-02-09 12:03:43 +01:00
Dmitry Tantsur 1c7cefa1d4 Add a workaround for openSUSE
They keep shipping broken bundled packages in their venv module.
While we may want to eventually deprecate openSUSE support, provide
a work around to unbreak people.

Change-Id: Ib5c1730f5325e726bf3aef164cfaef01ad4c7a13
Story: #2008591
Task: #41747
2021-02-03 12:14:28 +01:00
Dmitry Tantsur 71b86a3139 Recognize TOX_CONSTRAINTS_FILE as a valid constraints variable
Add-on for https://review.opendev.org/760782

Change-Id: I62413ba860d84c48c2da97ea469d3906d885e6ef
2020-11-04 14:19:54 +01:00
Mark Goddard b6624b2dbe Use upper constraints when installing Bifrost requirements
Currently the install-deps.sh script installs Bifrost's Python
requirements without upper constraints. This can result in requirements
drifting over time, with issues arising in older branches.

This change fixes the issue by using an upper constraints to install
Bifrost's requirements.

Change-Id: Icf66d0faf77837b9b48df344578823e46017b1df
2020-10-08 15:39:05 +01:00
Dmitry Tantsur a874c7ac73 Always setup environment in bifrost-cli and make it less verbose
For upgrades we need bifrost-cli to always try to setup environment
(e.g. in case of a different ansible version). To avoid polluting
the output, make `set -x` conditional on a new variable BIFROST_TRACE.

Change-Id: I8caea6be0962db0e7019d0aa1484ef2424b7629e
2020-09-08 12:47:56 +02:00
Zuul f3a4162615 Merge "Change LANG in install-deps.sh to en_US.UTF-8" 2020-08-31 15:42:16 +00:00
Michal Nasiadka 807700ce5e Change LANG in install-deps.sh to en_US.UTF-8
LANG=C causes pip installation issues in CentOS 7, observed in Kolla container
image build on stable/train and earlier, see [1] for full build log:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 25:
ordinal not in range(128)

[1]: https://zuul.opendev.org/t/openstack/build/45a5a3c3c2a046f6bd1868026e1e7803/log/kolla/build/000_FAILED_bifrost-base.log#8807

Change-Id: Iff11316a0708f5df437efecf95de47bcc4579a09
2020-08-28 14:27:30 +02:00
Dmitry Tantsur 2cf21258af Remove support for non-venv deployments
It has been deprecated and untested since ussuri, time to move on.

Change-Id: Ieddda38b12d07a2a5c51650850b150dd82f332a9
2020-08-25 11:08:09 +02:00
Dmitry Tantsur 0b58a05ac6 Do not use 'sudo pip install' when venv is used
We end up installing bindep and bifrost deps in the system otherwise.

Change-Id: I3d3b24ba5313013051e8b18d1ce4ec42a61d20f0
2020-07-07 11:44:05 +02:00
Dmitry Tantsur 27d87522f1 install-deps: install setuptools early for Debian
Seems like bindep cannot work without it but does not pull it for
some reason.

Change-Id: I6c989e2aa8e71ca2fa802ec403d34612262eef2f
2020-06-22 16:32:15 +02:00
Iury Gregory Melo Ferreira fa1b8ea9a0 Add openstack collection
- add ansible-collection-requirements.yml so we
can specify the collections we need.
- Changed env-setup to install collections unsing
ansible-galaxy.

- Moved taks "Dynamic enrollment" to use the openstack.cloud
collection

Change-Id: I1cf21a505b923f333b6853944965593170cbe8cd
2020-06-19 10:36:53 +02:00
Dmitry Tantsur 8aa8086ab2 Switch to using virtual envs by default
They're already used in the CI, and chances are high that non-venv installation
is broken on at least some supported systems.

Change-Id: Ie3c93a0a1537658eaccf69da3f64d0fc43f292d1
2020-06-05 11:46:04 +02:00
Dmitry Tantsur 7e363d4318 Drop hard requirement on debootstrap and EPEL
EPEL is only needed for debootstrap, debootstrap is only needed when
building Debian/Ubuntu images. Install it on demand and update the docs.

Change-Id: I6b90c91f20b5acd198cabbb89f21a72caaf90c63
2020-06-03 16:41:04 +02:00
Riccardo Pittau 5182388b7c Fix pip3 path
The pip3 binary can be in a different location from python3.

Change-Id: I19f32c0a8316dc0fca452b7e8d36c777effcbeb5
Story: 2007694
Task: 39806
2020-05-28 14:39:22 +02:00
Riccardo Pittau d75a07ccb2 Fix check packages command
The dpkg -l command always exits with 0, we should use the -s
option instead that exits with 1 if the package is not installed.

Change-Id: I656b111b77b637a10c9a94fd3880aee377c166f1
2020-05-28 14:39:22 +02:00
Dmitry Tantsur a381b0f9c0 install-deps: use system pip and bindep for installing dependencies
Updating pip was required long ago to work around Ubuntu and CentOS 7
problems. Now even CentOS 8 has pip 9.0 which should be enough for us.

Also moving binary dependencies to bindep and install them before
we try to install Python dependencies (to be able to compile them).

Workarounds is needed for segfault on Ubuntu because of cryptography:
remove python3-cryptography package and avoid --ignore-installed.

This change makes install-deps work on openSUSE again.

Change-Id: Ib9b81075f35068c046880de46b20f98ef71a8d8e
2020-05-26 12:17:53 +02:00
Riccardo Pittau cc415e590e Install python3-pip system package
Without the python3-pip package, installation on ubuntu bionic
without virtualenv will fail.

Change-Id: I666879c06b119920b1e925c1e2f8848caa6cead4
Story: 2007691
Task: 39803
2020-05-19 14:59:56 +02:00
Mark Goddard 21cb4ea6fc Remove --ignore-installed from CentOS 8 pip 19 install
In the kolla bifrost container image this results in a broken pip
installation. The pip 19 package is downloaded, but this message is
printed at the end:

Successfully installed pip-20.0.2

The subsequent 'pip freeze' gets the following error:

ImportError: cannot import name 'PackageFinder'

This change removes the --ignore-installed argument, which results in
the virtualenv getting pip 19 as expected.

Change-Id: Idf2b03a4772111f63f79203f63ea95f805e7a43d
2020-03-17 18:13:42 +00:00
Riccardo Pittau 19b9f685b5 Fix for PEP517 issue with Python 3.6.8
When creating a virtualenv with Python 3.6.8 using the
--system-site-packages option and pip>=19.0.0 a bug affects
the virtualenv breaking pep build isolation.
This patch downgrade pip to version 19.0.0 as a
workaround.

Also converts pip3 command to pip.

See [1] for more info.

[1] https://github.com/pypa/pip/issues/6264

Change-Id: I738d7c2e50914a773920e5381305e8c5131ccd5f
2020-03-13 14:24:56 +01:00
Zuul f259b6ce3f Merge "Prepare to switch CentOS 8 job voting again" 2020-03-09 15:04:17 +00:00
Riccardo Pittau 0c3baf04dd Prepare to switch CentOS 8 job voting again
Fixing things in scripts and playbooks in preparation of
making CentOS 8 tinyipa job vote again.

- python packages are not defined anymore
- change yum to dnf when needed
- change python2 packages to equal python3 when possible
- ipxe binary has changed to ipxe-x86_64.efi

Change-Id: I89989b6d2e35f0291ba1c1aed8b3317cfe2d691d
2020-03-06 13:58:25 +00:00
Georgy Karataev c8c4b8c90f Fix "unbound variable" error in scripts
If user runs scripts without VENV defined, then an error
"unbound variable" will be received. This patch fixes this issue

Change-Id: I22b817ad7b945509bb2e89d0467df53975434313
2020-02-05 21:04:52 +05:00
Riccardo Pittau c0d5c4ccb0 Fix variable checks in install-deps
A couple of fixes:
- We check the content of the binary dir as debug step, but this
should be necessary only when in virtualenv.
Also using the correct virtualenv path instead of hardcoding it,
and removing - character after variable name.
- EXTRA_PKG_DEPS is an array and we should treat it like that
when we check if it's empty or not.

Change-Id: I294ef9dce53d7404f4d9285506f1504afabe833f
2020-01-24 11:30:16 +01:00
Riccardo Pittau 906d7169ac Fix test for python3-venv pkg install
The test to check if the python3-venv package is needed is wrong
and it will always fail, asking to install the package in systems
where that doesn't exist, like CentOS or Fedora.
The python3-venv package is always needed for ubuntu bionic, so
installing it all the time if we're using that distro.

Change-Id: I6fe765609cc0561f359986f488268e79cca469e2
2020-01-22 12:36:21 +01:00
Riccardo Pittau 1a41d469ed Define venv package for debian based distro
On L115 we ask to install the [venv] package in case we're unable
to create the virtual environment.
For Ubuntu Bionic the package needed is python3-venv.

Change-Id: I065b515e66ced54daaa7e607b89137bd10a1e2f7
2020-01-20 09:51:27 +01:00
Julia Kreger 44196ea91b Switch to use a venv by default for testing
Also updates the method to use the python3 native
venv module instead of the virtualenv module.

Change-Id: Iae4248acec0b3bdf48be62134f64356d40375dde
2020-01-15 17:18:54 +00:00
Julia Kreger 8a6cdf7d3a Change install-scripts for py3 with Ubuntu
Reworked the pip install logic so that we run the pypi
pip installer instead of trying to navigate through running
a pip install and upgrade with setuptools additionally.

These steps were orginally only done in order to force
us to a pip version >= 9.0.1, which is now the default
for Ubuntu Bionic.

Change-Id: Ic3a95973338bbd03061d5b3e4080c0d72d6b891a
2020-01-08 11:30:13 -08:00
Radosław Piliszek c5745daf00 Fix install-deps.sh failing on bare CentOS 7 due to old pip
This change upgrades pip instead of just ensuring it meets
the lowest version constraint.

This is in line with the setuptools being upgraded just below.

This did not affect CI because it provided its own pip.

Change-Id: I36a0e31ecba1dc9d5eec1d2338dc752467be5ac0
Story: 2006833
Task: 37406
2019-11-11 18:48:59 +01:00
Radoslaw Smigielski e5e1df0cb0 Replace hardcoded yum command by package manager variable
Fix hardcoded yum command by package manager variable, PKG_MANAGER
in scripts/install-deps.sh. That was problem on Fedora where default
package manager is dnf and not yum.

Change-Id: I2dbec079adbfcab71e46e086fcd201a5175f22d5
Story: 2005451
Task: 30503
2019-04-14 01:42:35 +02:00
Victor Morales 95091a4c3e Pass global enviroment variables
The install-deps.sh script is sometimes used in a server sitting
behind a corporated network, for those scenarios is necessary to
pass proxy global environment variables. This change enables that
for fedora/centos OSes.

Change-Id: Ic327168eec892970cd2d7dd4b24d92fa0deee184
2018-11-12 23:27:51 -08:00
Markos Chandras ec88ca2c86 Add openSUSE Leap 15 jobs
We make the following changes in order to support openSUSE Leap 15.0

- Updates the bindep.txt file to match the SUSE family.
- The netstat binary has been moved to net-tools-deprecated
  package.
- We fix a bug when checking the status to extra dependencies. The
  variable is an array, but the code was only checking the first
  entry of the array.
- Replace spaces with '_' on ansible_distribution since openSUSE Leap
  15.0 is being reported as 'openSUSE Leap' by Ansible.

Change-Id: I800331e29e0d47a3121229cd66a5814277e34b84
2018-11-07 09:25:26 +00:00
Olivier Bourdon 632a5367b1 Fix CI for CentOS
Change-Id: I7d8c2b29ece07ead5f6e23706a5e51d4c1b65ad8
2018-10-17 14:18:44 +02:00
Tuan Do Anh 82f6d119d0 Pin get-pip.py to 3.2
As get-pip.py evolves based on pip 10, while we're still using
pip 9, changes in the way it can be used are causing problems.

For example, the ability to use --download is no longer there.

As such, let's pin to a known good version and leave it at that
until we no longer need to use this script. Version 3.2 maps to
pip 7.1.2 which fulfills our needs.

Change-Id: I2fa1c67ecd712364a822edcf7470e5489447d990
2018-07-18 15:52:15 +07:00
Mark Hamzy ad0cbe99be upgrade setuptools before installing package requirements
Also force an upgrade if the package is already installed.

Change-Id: Idd5a456475bc7b16936ff75542de6b4928648d6f
2018-05-13 15:26:09 -05:00
Julia Kreger 9b15ca277f Fedora 25-27 support
Update bifrost to handle fedora as minor changes between
releases have been slightly breaking, and have gone largely
un-noticed since the bulk of bifrost users seem to use
centos or ubuntu.

Added a fedora 26 job to help enforce support moving forward.

Change-Id: I9a654b91edfa52321adccfa5df0c260eb0c9e487
Closes-Bug: #1695969
Closes-Bug: #1695968
2018-02-02 12:22:30 -08:00
Markos Chandras ebdae8441c scripts: install-deps.sh: Ensure distros have package metadata available
Before trying to install any packages for the tests, we should ensure
that the package metadata is updated otherwise some packges may be
reported as missing.

Change-Id: Idc2033eee917b3024e392712aec9feb7611c097a
2017-12-07 09:29:25 +00:00
Markos Chandras 5d82414d4f scripts: install-deps.sh: Set noninteractive for DEBIAN_FRONTEND
This helps fixing the following problem when installing MySQL on
Ubuntu hosts:

Preparing to unpack .../mysql-server-5.7_5.7.19-0ubuntu0.16.04.1_amd64.deb ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs
shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
Configuring mysql-server-5.7
[...]
New password for the MySQL "root" user:

The package configuration becomes interactive and asks for configuring
the root password on MySQL and this breaks automated testing.

Change-Id: Icacd843950eee1b87bac2c3387e3562f8f584b77
2017-10-04 13:47:22 +01:00
Markos Chandras 69d6cbae25 scripts: install-deps.sh: Fix distribution detection
Relying on the installed package manager to detect the distribution
is not reliable since it's possible to install multiple package managers
at the same time. As such, lets simply use the information in the
os-release files.

Change-Id: I285c6cd94932f84bb9133b39d282c39d9a25b301
2017-10-02 07:45:52 +01:00
Jenkins ac40825d55 Merge "Install the latest version of setuptools" 2017-09-28 13:45:10 +00:00
Yolanda Robla b6a61445c1 Install the latest version of setuptools
When trying to install bifrost on CentOS , shade could not be
installed properly, because the system had an old version of
setuptools. Add this upgrade to install-deps, to have the latest
version of it.

Closes-Bug: #1695969

Change-Id: I31e2d3af04eb6fa74a03e7fa731f58dbb30bd3bc
2017-09-20 06:35:32 +00:00
melissaml 9819720f78 Fix to use "." to source script files
Adhering to coding conventions. Refer to ``Code conventions`` at
https://docs.openstack.org/contributor-guide/ for details.

Change-Id: I54b2d7d84bdd651fb96a62b3af3244c6ab484059
2017-09-20 02:58:52 +08:00
Yolanda Robla bf74574076 Set language to c to fix latest bindep problems
Newer bindep versions are not working with languages not
set to en_US. As a workaround, export the LANG setting.

Workaround for: https://storyboard.openstack.org/#!/story/2001205

Change-Id: I60be976ff151bb61014f0272cdf02a4fb4851d0a
2017-09-19 17:13:55 +02:00