Move most curl sources to sources.py, so it can be automatically
updated in future.
This also makes overriding the versions/locations/sha256 easier.
Change-Id: Ib3497fe96162ae190e5c113cacaaa3ef8334f590
The main goal is to be able to build ARM
images on standard x86 hardware. Add simple
functions supported by python-py and
podman-py to enable creation of container
images on emulated architectures.
Change-Id: I1526b460f6e271c2ec63ab3a3b1dc348d742e96d
Add podman as an option to choose for container engine
for kolla-build (--engine podman)
Signed-off-by: Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>
Co-Authored-By: Michal Arbet <michal.arbet@ultimum.io>
Change-Id: I068c906df97745e397408d8c3ef6af47ee037638
Change to use docker.DockerClient instead of the low level
docker.APIClient.
Signed-off-by: Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>
Change-Id: I88842c17d8e6ddbe89b984c7b5e0367ed356fa50
We move to use some components from distribution instead of 3rdparty
repos. At least for now - this may/will change later.
Packages from Debian:
- Erlang 25.2
- MariaDB 10.11
Some Ubuntu/jammy repositories are used in places where there is not
(yet) Debian/bookworm one.
openvswitch-netcontrold image fails as it needs to be converted to use
virtualenv - added into unbuildable images for now.
See https://etherpad.opendev.org/p/kolla-bobcat-ptg#L199 for more information.
Depends-On: https://review.opendev.org/891946
Change-Id: I45b5a5980fb2318f1aab23a3a6aaf0922563b171
Add support for writing the build summary to a JSON file. This is more
machine-friendly than the current approach of writing to standard output
at the end of the build.
The file is configured via [DEFAULT] summary_json_file.
Change-Id: I9868af5d00977750d0673fc48f8863a99f878dc8
Since I650fcbc8f773fad8116338f6fb0cf7b4f4f17b33 builds from git fails
on plugins with an exception: 'tarfile.ReadError: not a gzip file'
because the test checks only gzip compressed archives but plugins
created as plain tar files. This change fixes the issue using
transparent compression support and also adds some debug info.
Closes-Bug: #1990432
Change-Id: If0f9b4dd058a257d0653332d1b663e150c717304
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
Co-Authored-by: Michal Nasiadka <mnasiadka@gmail.com>
In addition to that, mark bifrost unbuildable since it's
failing on EPEL 9 enablement.
Closes-Bug: #1990432
Change-Id: I650fcbc8f773fad8116338f6fb0cf7b4f4f17b33
Add engine adapter module to allow additon of other container engines,
create one common EngineClient that would be called in all other
modules.
Based on patch by Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>
Change-Id: Ice6467086bd292af086322afc3fc4e869d89eefa
As part of kolla/image/ refactoring we move KollaWorker related classes
into own file.
Based on patch by Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>
Change-Id: I55c3f6b34631048b2463b02e6ea611d7e4c1a912
As part of kolla/image/ refactoring we move build related task classes
into own file.
Based on patch by Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>
Change-Id: I093620679016b37e1664c9fe4cf7559433e744b7
As part of kolla/image/ refactoring we move status enums/consts into
own file.
Based on patch by Konstantin Yarovoy <konstantin.yarovoy@tietoevry.com>
Change-Id: I940816c7e487f6d5c7abc4446226e2992cf9e63d
OpenStack 'zed' requires Python 3.8+ so RHEL 8 family has to go.
This changeset moves to CentOS Stream 9 while move to RockyLinux 9 is
planned as final solution.
CI moved to CentOS Stream 9 nodes.
Depends-on: https://review.opendev.org/c/openstack/kolla-ansible/+/839715
Change-Id: I113b9984294cf8663d3fc0c8840320e1d40ea731
With version 6.0.0 of the docker python module, their implementation of
versioning has changed, making our check for old versions fail. Since
version 3.0.0 is from 2018, we should be safe to assume that as a
minimum version though, so we can just get rid of the special handling
of older versions.
Change-Id: I077b7b5acf2c1f9beb6da06d3555e2ebe30831d1
mariadb - downloads.mariadb.com doesn't have jammy version, should
be there soon - using Ubuntu provided version for now
telegraf/collectd - collectd packages missing in Jammy - marked
as unbuildable
monasca - requires new librdkafka, no wheel for Python 3.10 for now
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/846070
Change-Id: I11b4fbcaba165824252902d0270ce1dbbc3d6e9d
The only place where it stays is argument parsing where it gets ignored.
Hardcoded to 'source' to make merging both kolla and kolla-ansible
changes easier.
Change-Id: I6890a7ec7baf507f8acd75b42bbd0fb17c2e4511
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/837620
We no longer support binary images. This patch removes a way to build
them.
"--install-type" argument has one available option now. Later in cycle
it will be ignored.
Change-Id: If0e881d253d6a4c89ffe27831a661df321b6d5b6
Sometimes it might be useful to disable a source for an image, e.g. for
a plugin that is not used. This may be to reduce image sizes, or
restrict the dependencies necessary to build images.
This change adds a new 'enabled' option to each source definition. This
allows sources, plugins or additions for each source image to be
disabled individually.
Change-Id: I31ee0744574fb3db96bee2e9f545600de53d9ee4
This is a non-voting job to validate py3 unittests on AArch64
Also fixed test units to use source install type with Debian as Kolla
does not support Debian/binary on !x86-64
Co-authored-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Task: 41911
Story: 2007938
Change-Id: I6889555820adcaee8a86e70b5c3ea31c15e3192c
Bullseye is now in hard freeze cycle and goes for release.
https://release.debian.org/bullseye/freeze_policy.html
Co-Authored-By: Michal Nasiadka <mnasiadka@gmail.com>
Change-Id: I543965a2741cebfa759576a4c75669a7bacd4208
Sometimes we have images failing. Not always we care about them. So let
allow them to fail and stop caring do they build or not.
Change-Id: I01cf3635814255f682d6bd161cf1d985d92a9a31
Commit 3fa5a236b4 converted status
consts into Enum. But not completely - this fix gets it done properly.
Also fixed tests.
Change-Id: Ifdacf62cd4b99c452bd09de7c6cde6fbee75560d
There is a time once every 2 years when ubuntu team releases new LTS
release. And then UCA joins with binary packages for current OpenStack
development cycle.
It is this time for Ubuntu 20.04 'focal'.
Depends-On: https://review.opendev.org/745156
Change-Id: I045aa6b4b4fd83fbe7d1fda89549f0ef1e88ec12
With the move to RHEL/CentOS 8 we no longer have Python 2 in our images
so there is no need for checking which Python version (2.x or 3.x) is
used inside of containers.
We also no longer have to support yum as a value for
distro_package_manager.
Partially-Implements: blueprint centos-rhel-8
Change-Id: Ie45cf3465fedddbde7856961527421883ba3d5c9
Now that py2 is gone, oslotest dropped dependency on mock and will
soon affect Ussuri CI [1], let's use unittest.mock built in py3.
This also fixes py38 jobs and proactively prevents py36 and py37
failing due to [1]. This is because we never included mock in
test-requirements (but in lower-constraints where it does not
really belong at all) and instead relied on oslotest to bring
it in.
[1] https://review.opendev.org/716322
Change-Id: Iceedcc7ad0d087414f6f94dfc0235fda4754be63
There are currently a few corner cases with the --skip-existing and
--skip-parents flags. Examples:
1. kolla-build --type source --skip-parents monasca-grafana
This does not try to build the monasca-grafana image. This appears to be
an issue when the image is a grandchild.
2. kolla-build --type source --skip-parents
This will build all images. Expect it to skip parents.
3. kolla-build --type source --skip-existing
This will build all images. Expect it to skip existing images.
The filter_images method does quite a lot, including handling
whether images are buildable, regex/profile matching, and which images
to skip. The matching and skipping parts are done in a single pass,
which can lead to some weird effects due to dependencies between the
images and their statuses. Also, skipping is only currently applied when
there is a regex/profile filter.
This change splits out the matching and skipping into two separate
passes. In the first pass, we mark all buildable images that match the
filter as matched. In the second, we iterate over matched images,
applying the skip existing and skip parents rules.
Change-Id: I2f895aea0cc59d808129e9fc636af0890196af33
Closes-Bug: #1867614
Related-Bug: #1810979
If there was some error during pushing then Kolla was greeting with "all went
fine" message anyway:
INFO:kolla.common.utils.aodh-api:Trying to push the image
ERROR:kolla.common.utils.aodh-api:Get http://10.101.16.1:5000/v2/: dial tcp 10.101.16.1:5000: connect: no route to host
INFO:kolla.common.utils.aodh-api:Pushed successfully
This patch changes that. Now if there is an error during push then proper
exception is raised to PushTask and image is marked with PUSH_ERROR status.
This way at the end of build it is easy to spot which images did not got
pushed to registry:
INFO:kolla.common.utils:===========================
INFO:kolla.common.utils:Images that failed to build
INFO:kolla.common.utils:===========================
ERROR:kolla.common.utils:base Failed with status: push_error
ERROR:kolla.common.utils:nova-api Failed with status: push_error
ERROR:kolla.common.utils:nova-base Failed with status: push_error
ERROR:kolla.common.utils:nova-compute Failed with status: push_error
ERROR:kolla.common.utils:nova-compute-ironic Failed with status: push_error
ERROR:kolla.common.utils:nova-conductor Failed with status: push_error
Closes-Bug: #1848019
Change-Id: Id2ab97bf4c0dc3423268a0ea435b56f4a65f7196
Currently the retry mechanism is broken for pushing because the image
state gets set to an error state, and is never reset to 'built'. This
prevents the PushTask from setting success to True.
This change sets the image state to 'built' if the push succeeds,
ensuring it overrides any previous failures.
Change-Id: I93fc0e383da8fec6b3ca31f8094321c2a0c3af71
Closes-Bug: #1844697
We have a list of images which are unbuildable for distro/buildtype/arch
combos. This patch renames it from SKIPPED_IMAGES to UNBUILDABLE_IMAGES
because 'skipped' images is something else in our code.
Now all unbuildable images are marked as such and so are their children.
At the end of build there is information provided about skipped images
(due to --skip-parents or --skip-existing options) and then information
about not buildable images. Both list should only contain entries
related to requested build.
Change-Id: If9b521339f564e483cba03d52e7c4eba271821a5
Commit 43b74ccc15 enabled use of Python 3
based packages but not switched to use Python 3.
Some of images still contain Python 2. There are two reasons:
- Ceph (ceph-common depends on Py2)
- python3-ldappool on Ubuntu 18.04
In Ceph situation Py3 packages were added. For second one we can not do
anything - Py2 dependency got dropped in Ubuntu 18.10 version.
Removed neutron-server-plugin-networking-infoblox due to being not
maintained. Once https://review.opendev.org/#/c/657578/ get merged
someone may revert that part.
Implements: blueprint debian-ubuntu-python3
Depends-on: Ie2a1077f7def0743f1403341985e2109aa490026
Change-Id: Ibfe0c2b8be98db56c61f74fb0247488ab3749ef4
RHEL7 derived distros use Python 2.7
RHEL8 uses Python 3.6
Ubuntu 18.04 uses Python 3.6, but we haven't enabled it yet.
Debian:buster uses Python 3.7, but we haven't enabled it yet.
We have several places in code where "/lib/pythonX.Y/site-packages"
is used so we need to know value of X.Y rather than py2/py3 switch.
How to check for Python 2:
{% if distro_python_version == '2.7' %}
Breaks: Iad6b70b433a0dd1b0f8ae6790fd280594517661a
Change-Id: Ie24f109af19ae0438c21e0d43cb3cc947812b4ce
Currently in all the Dockerfile.j2 files, we are doing base distro if
conditions for package names when installing via a binary. Rather than
duplicating these base_distro in [<distros>], let's provide a single
value for determining the binary installation method. This change adds a
base_package_type which we can use to lookup if 'rpm' or if 'deb' rather
than relying on specific distro names.
Change-Id: I8a0c6c8fbc32a8cfa6932fddf28a449fceda3d49
As we add support for python3 on RHEL based systems, we need to add a
flag to key off of in the Dockerfiles to swap out the python3 package
names. This change adds a distro_python3 configuration option that can
be defined to force python3 or python2 package names. If this option
is not provided, we set this to True for RHEL 8 based distros and false
for other distros.
Change-Id: I4028991bad92c0e8e21066cc4173c06ce5eba393
Related-Blueprint: python3-support
Add a decode=True option during call to dockerAPI in order
to build container against recent docker swarm.
Closes-Bug: #1779460
Change-Id: I48f041e7ed9dd5a5f410b4eaa081f6f013119d17
Rework push_image function to use docker client push function with decode=True, to decode automatically to dict.
Change-Id: I28d38e968c3809a1074347271e49dc07dc861f11
Closes-Bug: 1771514
Add the ability to specify a different network_mode when running the
image builds. Under the covers the default is to use bridge networking
but if that is unavailable or a user would like to specify a different
networking method they can now do so using the [DEFAULT]\network_mode
configuration.
Change-Id: I1d2398273b57ca4d52bbfacbc896ffecc58bc54b
The argument of 'insecure_registry' for image push has been removed
by this commit (in docker 3.0.0):
b180b8770a
Actually this argument was marked as deprecated and unfunctional
before this removing.
This commit is to remove this parameter at invoking the push() function
when docker version >= 3.0.0.
Change-Id: Ifa7d304d08a4073dcb1dff751d5a443c112cd377
Closes-Bug: #1746703
When the first build attempt fail for any reason (e.g. short network
outage, proxy outage,...) then the second attempt always fail for
git source type. Git can't clone repository because destination
directory exists and is not empty.
Current fix checks destination directory and if it exists then
it is removed before cloning.
Change-Id: I949140c49a64baea579d61047e3b2f1240da2771
Closes-Bug: #1706369
Bandit was not running in all directories
due missing -r parameter.
Also, this change fix bandit errors.
* Add nosec to subprocess module
* Change /tmp/releases to .releases
* Change /tmp/ to tmp/, tmp/ is not checked
and in unittest is not an issue, simpler that mocking tempfile
and make more complex tests.
Change-Id: I42bafa4707a9a4d1faeb01cb9b5a843f56280deb
The mocks were in the wrong order. The only reason the tests were
passing so far was because the calls to mocks were not checked, the
mocks were only used to prevent the tested function to make calls to
actual resources.
Change-Id: If7ea23ac20e3a0d584e02e31d9d60ba914efa8aa
Zuul