Commit Graph

212 Commits

Author SHA1 Message Date
ricolin eb2ce826ef Revert "Remove uwsgi by default"
This reverts commit 39aaeb3f32.

Change-Id: Ib607f3ecd5ada46ffe724fbc0d245d08f2a0fe01
2024-01-18 21:01:12 +08:00
Johannes Kulik f4cc70548a Make fetch-wheels PEP8 compliant
This is just a little cleanup.

Change-Id: I148369869e9aac1b3f4ba25d3780d6bca380d4c9
2023-12-05 23:22:49 +00:00
liyingjun 75b269d8d1 fix(nova): enable login ability for nova resizing or migrating
Login should be enabled for nova resizing or migrating between
hypervisors[1].

[1]: https://docs.openstack.org/nova/latest/admin/ssh-configuration.html

Change-Id: I1db692c3fbb371873ebe5432495130f4ecfdc930
2023-08-21 07:21:10 +00:00
okozachenko 82082bf82d Copy default configuration files to /etc/
This will allow us to shorten the values.yaml
in many cases.

This is disabled by default. Set COPY_DEFAULT_CONFIG_FILES=yes
to enable it.

Change-Id: I244c29844d3ceb29b73e9a60b284d900bbeeb4ec
2023-08-16 00:39:53 +03:00
Vladimir Kozhukalov 2f88cf4644 Fix build of those wheels which depend on python-nss
In the https://review.opendev.org/c/openstack/loci/+/888351
we build python-nss separately because we need to
patch it before building. Then we moved the result
.whl file to the root directory.

Now let's add --find-links / to all pip wheel commands.

Change-Id: I5faca92eb229af989b4bf63e6e044e49c7bacb5a
2023-07-13 03:25:03 +03:00
Vladimir Kozhukalov ce9fea8fe6 Fix building python-nss on Ubuntu Jammy
The libnss3 headers in Ubuntu Jammy are compatible
with python-nss===1.0.1. Ubuntu Jammy itself
provides the binary package python-nss and
they apply the patch that renames RSAPublicKey/DSAPublicKey
types into PyRSAPublicKey/PyDSAPublicKey.

This change applies the same patch before
building python-nss wheel.

Change-Id: I5211aac7b6a3bbc1f1b74e8662170dc8932525f9
2023-07-12 22:45:24 +03:00
Vladimir Kozhukalov ec70407d68 Build jammy images
Change-Id: I639588a618816f6ff70a7736963589977bb36eb1
2023-07-11 19:34:19 +03:00
Vladimir Kozhukalov 7c3c53c375 Add more info to the /etc/image_info
Change-Id: I956985410f55293c998e75c9fa8960e2b9106d45
2023-06-16 20:30:51 +03:00
Vladimir Kozhukalov c152ea081b Install dist packages before pip packages
Sometimes we need dist packages before
installing pip packages. For example
we might want gcc to be able to install
some pip packages from source.

Change-Id: I820a35f61999acbbed40a83b33ea523a0c15fcc6
2023-04-03 06:05:42 +03:00
Sam Yaple 39aaeb3f32 Remove uwsgi by default
Not all deploys will use uwsgi by default but there is no way to disable
it currently. Since uwsgi itself isnt truly a python app (just a C
program with a python building shim).

The user will still be able to install binary package based uwsgi or
include uwsgi in the extra pip packages to install if they wish to.

Related-Id: I76612794c1ba8dbc45b62dff00cee43c6ba10a34
Change-Id: I15008d41633168fda31e061003ccf4681cade68e
2022-11-18 16:44:58 +00:00
Sam Yaple 2e4f6d555a Update fetch_wheels to add Accept header
A private registry run from docker.io/registry:2 will use the Accept
content type added. This allows private repos to work:

  WHEELS=192.168.1.50:5000/privateloci/requirements:ubuntu

Also remove the simple distutils function we used ahead of its removal
in python3.12

Change-Id: I7482e9097a7c0a5b332907c9e5849f2fe48af473
2022-09-21 21:28:12 -04:00
Sam Yaple 4ff506ae5c Remove PYTHON3
The python2 is long dead and the PYTHON3 flag has been noop for a while.

Change-Id: Ia7be53e84090cc8b80e1dd5753ba1460ff454851
Related-Id: I3f4f60756887ad07229cf522803b30d4e16463cd
2022-09-21 21:28:12 -04:00
Sam Yaple 361fd6e104 Do not always install libapache2-mod-oauth2
This package is unavailable for 22.04 (the current LTS version of
Ubuntu). It is in 22.10 and may be backported to 22.04 through the
backports repo, but currently there is no prebuilt trusted source for
this package on 22.04.

This package is getting deployed on all ubuntu systems. In my opinion,
it should not have been merged, but we can gate the install around a
PROFILE='mod_oauth2' rather than revert the change entirely.

Related-Id: Ib3eac4269f7ac8365623f95a81b07a6436464ed0
Change-Id: I282b7b0aefa8aabc1f5c6495564d13b70cd0ec5d
2022-09-21 21:28:12 -04:00
Zuul 8ac5d338db Merge "Remove Git repo data from NoVNC and SPICE HTML5 dirs - Story: 2010049" 2022-09-20 18:03:31 +00:00
Mohammed Naser 9e7b71c304 Reduce build targets
OpenSUSE LEAP is currently not in use by any known downstream
users and it has been broken for quite sometime, leaving the gate
in a broken state and unable to land any code.  There's no active
maintainer for it as well.

Debian support was added by us (VEXXHOST) but we decided that
we're not going to continue using it and move towards Ubuntu, so
we do not want to maintain it, as well as that it is building images
for Train only.

It also updates the README file accordingly as well as includes
focal in there which has been recently added.

Change-Id: Ideb497c169828184d301b6be4359a7c2228aa444
2022-05-23 15:59:49 +00:00
Jeremy Lamb fe9caeebbb Remove Git repo data from NoVNC and SPICE HTML5 dirs - Story: 2010049
Change-Id: I9f8a81ed160a5d09b01e846b091d5bbb2bee1656
2022-05-23 03:28:09 +00:00
Zuul effe0b875e Merge "Remove python2 references from requirements.sh" 2022-04-27 22:12:29 +00:00
Zuul ff2c43e85a Merge "Remove python2 workaround in cleanup.sh" 2022-04-27 22:12:27 +00:00
Zuul a32aca4dc5 Merge "Remove python 2 workaround in fetch_wheels.sh" 2022-04-27 22:12:02 +00:00
Gage Hugo 2ad0531bb0 Remove python 2 workaround in fetch_wheels.sh
This change removes an import workaround for python 2 in the
fetch_wheels script that is no longer needed since openstack
has moved away from python 2. This is part of the effort to
remove all the python 2 workarounds in the loci repo.

Change-Id: Icccec48297cdd18577ec3fb28efa173e3a7ef000
2022-04-27 11:02:56 -05:00
Gage Hugo 8e486b84cf Remove python 2 workaround in fetch_wheels.sh
This change removes a workaround to handle python 2 vs python 3
in the fetch_wheels.sh script. This is part of the effort to
remove the python 2 workarounds and references in the loci repo.

Change-Id: I7176d665548a14442b06214e450e7b17e54dd85d
2022-04-27 08:24:41 -05:00
Gage Hugo 6c1f0290d6 Remove python2 workaround in cleanup.sh
This change removes a workaround for python 2 in the cleanup.sh
script. This is part of the effort to remove the python 2
handling in the loci repo.

Change-Id: I7edf860a0b035d91aa9a3f8e2e3af1e71ce61065
2022-04-27 08:22:10 -05:00
Gage Hugo 23a95c5e19 Remove python2 references from requirements.sh
This change removes the python 2 references and workarounds from
the requirements.sh script. This is part of the effort to remove
the python 2 uses and workarounds from the loci repository.

Change-Id: I23271067587dff938c3bcce4f798db8dc33a428b
2022-04-27 08:19:16 -05:00
Gage Hugo 8b514187cc Remove trollius workaround
The workaround for trollius appears to be no longer needed since
openstack has moved away from python 2. This change removes the
logic to handle removing trollius since it's no longer present
in upper-constraints.

Change-Id: Icd832edffec2cea907276480fd8c67e44459ad90
2022-04-26 20:31:51 +00:00
Zuul f9676f734e Merge "Authenticate with registry on-demand" 2022-03-25 18:33:12 +00:00
Zuul 279af7f51b Merge "aarch64: stop trying to build confluent-kafka" 2022-03-01 23:37:05 +00:00
Zuul 8231314038 Merge "uwsgi: fix occasional failures" 2022-03-01 23:01:56 +00:00
Mohammed Naser a7a31b18c4 aarch64: stop trying to build confluent-kafka
This package requires a version of libfdkafka-dev that is too old inside
the distributinos to be able to build the wheel.  In amd64, it has
wheels on PyPI so it doesn't attempt to build it but it fails to build
it inside an aarch64 system.

Change-Id: If0a90919cdddaf91a67ce96646dea87cf0b9632c
2022-03-01 21:41:48 +00:00
Mohammed Naser 0dd8eb61b6 uwsgi: fix occasional failures
Without CPUCOUNT=1, uwsgi frequently fails to build in
certain environments.  Once this was implemented, the
builds were stable again.

Change-Id: If85687046297c582b39ca591b21e93c76b7de876
2022-03-01 21:41:36 +00:00
Mohammed Naser cb0650d098 keystone: add mod_oauth2 to image
mod_oauth2 is needed to enable oauth2 as mod_auth_oidc is
deprecating that feature. It's also not included by default
in the repos so we have to script installing it.

Change-Id: Ib3eac4269f7ac8365623f95a81b07a6436464ed0
2022-03-02 01:24:31 +11:00
Mohammed Naser 68388c8ac6 Authenticate with registry on-demand
This patch makes a change to allow LOCI to dynamically
determine the authentication server of the registry by
grabbing it from WWW-Authenticate if we get an unauthorized
request and then re-issue the request again with a token
afterwards.

Change-Id: Icb5eb390d64a944c6ad46c10581980076befda41
2022-01-26 13:12:37 -05:00
Andrii Ostapenko a61c73e9b2
Update gates
* Use bionic for ussuri build as it's the latest supported
release [0][1].
* Switch to centos 8 to run master [1][2].
* Switch default base image to ubuntu:focal.

[0] https://governance.openstack.org/tc/reference/runtimes/ussuri.html
[1] https://governance.openstack.org/tc/reference/runtimes/victoria.html
[2] https://governance.openstack.org/tc/reference/runtimes/train.html

Change-Id: I9a70aebd9636e170c6e04398356933668a2bfbba
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2021-10-12 18:30:09 -06:00
Andrii Ostapenko 68c3d9177f Pin virtualenv to 20.7.2
Current latest 20.8.0 have updated embedded pip 21.2.4 and setuptools
58.4.8 incompatible with some of constrainted packages.

Change-Id: I81ec48dbd123e6f24ff05d64906d47c31ae693f7
2021-09-20 19:40:29 +00:00
Andrii Ostapenko c19db02c6d
Fix setup_pip.sh and remove xenial based builds
Also fix monasca bindep.

Change-Id: I2b5b1a31165e98ce71380824e8f57af8906701c6
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2021-07-29 17:27:51 -06:00
Andrii Ostapenko 3e5730cd53 Enable Debian gate build and fix mirrors related issues
Also fixes related issue with distutils installation for Debian Stretch
and Ubuntu Xenial with py3: unlike in Bionic, distutils is installed with
python3 and python3-distutils package is not available in these distributions.

Run Debian build with py3 against stable/train, as master requires at
least python3.6, not available for distribution out of box.

Also increases job timeout to 5400 due to increased amount of builds.

Change-Id: I04c9253af80d941afb45231bee20f7bb1c4a96d4
2021-02-16 14:33:04 +00:00
Zuul 0157128488 Merge "Set DEBIAN_FRONTEND=noninteractive when installing from apt" 2021-02-16 07:29:12 +00:00
Andrii Ostapenko c1a589698e Don't use pip index if KEEP_ALL_WHEELS!=no
With requirements image built with KEEP_ALL_WHEELS pip prefers to
download wheel from index.

With this commit and KEEP_ALL_WHEELS=yes during a component image
build packages will be installed from local wheels only. This brings
a requirement to ensure all wheels are present - in the case when
PIP_PACKAGES is provided to component image build, it should be also
passed to requirements image build.

Change-Id: Iea72ee68aa59f2fea2c82a98f7f637c4476f7d2a
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-12-18 01:53:38 +00:00
Sam Morrison 3f392ed7e7 Set DEBIAN_FRONTEND=noninteractive when installing from apt
Change-Id: I8f4534304fbcc8b15561125d71c25986397214ab
2020-12-11 13:45:08 +11:00
Andrii Ostapenko 5e3fa8dd2c
Fix gates
opendev luminous ceph mirror is no longer available for xenial ocata
build, thus using original location.
Setting locale to resolve salt installation failure for master.

Change-Id: I8b25d7ddf5432de65cfd814193e10afee516fd1a
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-12-10 12:07:58 -06:00
Mohammed Naser 115396b0aa aarch64: enable building multi-arch images
At the moment, the only missing piece of being able to build
multiarchitecture images is the fact that loci does not support pulling
an image that has a 'fat manifest' (i.e. pointing at many different
images of different architectures but with the same name, built using
buildx).

This patch adds support to fetch-wheels in order for it to be able to
understand how these manifests work and download the correct one for the
architecture that is being built.

This has been tested locally here in our environments and successfully
produces multiarchitecture images using docker buildx.

Change-Id: I5ea132d12fec50b4c2cddd8b8c7c1f251ee2c30b
2020-10-08 18:31:50 +00:00
Andrii Ostapenko a26fa6fb98
Add support for unnamed constraints to loci with new pip resolver
New pip resolver is being released with pip 20.3 and does not support
unnamed constraints. Instead of using pip functionality to handle
unnamed constraints we'll instead first build corresponding wheels,
determine their versions and substitude unnamed constraints with named
ones in upper-constraints.txt. They'll be fed to general wheel build
afterwards and used by wheel bundle consumer.

Change-Id: Ie2acf93d4c91c6bbf3a3777b6ee697253207e2ef
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-08-07 18:34:38 -05:00
Andrii Ostapenko af1cdd69bf
Build/download all wheels including dependencies
If KEEP_ALL_WHEELS flag is not False, keep all packages specified in
upper-constraints.txt together with their dependencies.

Change-Id: I79051757780eabba15a2995fa8eaece0ff459f9e
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-08-07 18:33:39 -05:00
Andrii Ostapenko f041966a8c Fix KEEP_ALL_WHEELS flag usage
Change-Id: Ia1c4774aede69f83e1f43c7b511e176edc10cc9e
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-08-06 19:45:06 +00:00
Andrii Ostapenko 3e78a46caf
Workaround unicode issue with py2 and scandir==1.4
Also move manila ocata to eol.

Change-Id: I0282b57ab27a51698f55294037dd3429d6a11416
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-08-01 00:37:22 -05:00
Zuul a08b229441 Merge "Avoid storing cache in pip install for requirements image" 2020-07-24 05:33:36 +00:00
Zuul 7e9b2624b9 Merge "Add KEEP_ALL_WHEELS flag" 2020-07-24 05:33:34 +00:00
Vasyl Saienko 75bba38822 Add KEEP_ALL_WHEELS flag
The patch adds KEEP_ALL_WHEELS set to False by default.
The flag allows to do not remove wheels after built of
requirement image. This is useful where reproducibility
of builds with the same WHEEL image is needed as will allow
to keep 3rd party libs (not specified in upper-constraints.txt)
file with same versions.

Change-Id: I00fd3df2ba46072b3f19c9d08f69bb3c1f53d01f
2020-07-13 18:35:26 +00:00
Vasyl Saienko ec429cb082 Add collect_info.sh stage
Collect info about installed packages versions and
pip packages into /etc/image_info folder

Change-Id: I781457ceeb821a0cad7edc9c0921736b989e9ec7
2020-07-13 18:34:19 +00:00
Zuul 8cf631df18 Merge "Add an urlopen exception for ImcompleteRead error." 2020-04-24 05:13:22 +00:00
yuchengde 76a4b299ef Add an urlopen exception for ImcompleteRead error.
Using urllib2.request.urlopen() from python3 will meet the IncompleteRead error.
Add an exception to avoid it.

Story: 2007474
Task: 39180

Change-Id: I2b60a81168f5c47eab316d2be31a4fe3ba9c2254
2020-04-23 08:12:36 +08:00