This provides some test coverage on the new diskimage-builder.
It also makes the logfile argument handling a bit simpler.
Change-Id: Iecba581a00ba26131248566cb3088a1566dde00d
SUSE dropped OpenStack Cloud in 2019 [1], and as a result, some
OpenStack-related repositories were removed from openSUSE Download and
root filesystem images stopped being provided. This change deprecates
Leap releases before 15.3 and employs the extract-image script. It also
moves the extract-image script to the sysprep element, since now it's
also used by openSUSE-related elements.
Additionally, revert the "Remove opensuse related funtests" change [2]
so that the opensuse element is tested again and set the default Leap
release to 15.3.
[1] https://www.zdnet.com/article/suse-drops-openstacks/
[2] https://review.opendev.org/c/openstack/diskimage-builder/+/824002
Change-Id: I73d6323aa65cee69a55e54bc53ed682f096dfc89
CentOS Stream 9 is close to be released, and official mirrors are
already poplated. This patch is adding support to centos-minimal in CS9.
Also enable centos-minimal/[8,9]-stream-build-succeeds tests.
This patch is being tested together with [1] to apply following list of elements:
vm centos-minimal simple-init growroot nodepool-base openstack-repos infra-package-needs
[1] https://review.opendev.org/c/openstack/project-config/+/811442
Change-Id: Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6
It seems this code does not consider that you might set
test-output-formats with only "qcow2" -- it currently also tries to
inspect a .tar output.
Skip if tar isn't in the output format list, and fall through to
passing so both paths are considered success, only exiting if there is
a failure.
Change-Id: Ia21dbfd298d04b04ec02dfe76a0acd7c9a7a31f2
This is consistent with the previous simplication of
build targets in the opendev environment to refer to
"opensuse15" being the alias of "latest stable openSUSE Leap 15.x".
Change-Id: I904a3ca0d6dbddd2bb1a673836ab6a0ad249526d
Use openSUSE 15.1 as default, which is the latest released stable
openSUSE release.
Remove leftovers for unmaintained openSUSE 42.2 images.
Depends-On: https://review.opendev.org/#/c/660126/
Change-Id: I0b204b7b3d7ae74b6749320b3bfe1ca89d154ebb
Use openSUSE 15.0 as default, which is the latest released stable
openSUSE release. Switch to https for accessing download.o.org
as encrypted transfers should be used by default.
Remove leftovers for definitely unmaintained openSUSE 13.x images
and split into old/new leap style versioning scheme for clarity.
Change-Id: Iab129eeee2b1a2563f0f0d2cb17bbad57c068e38
I'm not sure why, but run_ramdisk_element_test is not using the same
interface as run_disk_element_test, despite being called the same way.
Update it so we are capturing logs.
Change-Id: I41b6a3f2d6e5466b340677c6252b7646c1dc2663
Move several parts of the "install_test_deps.sh" script into the more
standard bindep.txt. This list is intentionally restricted as a first
step.
Developer documentation is updated to use bindep and clarified
slightly.
Change-Id: I7520902dc324d920a0c7c44a2d35fe49f9b05614
Rename output log files so they can be quickly identified as passing
or failing logs just by the filename. Put them all in the same
directory for similar reasons.
Change-Id: I75bbe724e0d20c6561c2bf93857499cc49474250
This option to the functional test runner outputs logs for each test
to ${LOGDIR}/${element}/${test_element}.log
Change-Id: I3bf5362bcda8acda1a03c1058728875a9a6c9711
Add a bionic test in replacement of trusty. We are already building
bionic images in the gate, so this seems like a good time to switch.
Change-Id: I20d4c25e9b79e7326c86767c36be8615ba0888a3
This adds support for a GPT label type to the partitioning code. This
is relatively straight-forward translation of the partition config
into a sgparted command-line and subsequent call.
A unit test is added based on a working GPT/EFI configuration and the
fedora-minimal functional test is updated to build a single-partition
GPT based using the new block-device-gpt override element. See notes
in the sample configuration files about partition requirements and
types.
Documentation has been updated.
Co-Authored-By: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Change-Id: I6b819a8071389e7e4eb4874ff7750bd192695ff2
I747c2b8754effbc6ec82af3bf7543fd9599a6c14 removed the
centos-openstack-octata repos from the base image. We were previously
getting debootstrap from there, so this broke the centos-7 based jobs.
Enable EPEL for the install so we use that version of debootstrap.
Additionally we noticed that dpkg wasn't installed on the build host
either, but is actually required (calls were silently failing ... a
problem for another day). Add this.
Change-Id: I1e9db5130a05a004c3c877d1444d7097023c06e2
Change I008f8bbc9c8414ce948c601e3907e27764e15a52 has shown that we
build redhat images without the "semange" tool available, which comes
from the policycoreutils-python package (see also
I3f9e2c322d042a5dddba33451c0fc21a4d32a88a).
I403e7806ae10d5dd96d0727832f4da20e34b94c7 added some of the selinux
libraries to yum-minimal for ansible support, but not to others.
Given both these changes, it seems that selinux[-targeted],
libselinux[-python] and policycoreutils[-python] can reasonably
considered part of all base images. Move the selinux related packages
into redhat-common.
This also adds it explicitly to install_test_deps.sh. It was actually
being dragged in by the docker install, but is a required component
for building (should be in bindep, but not there with that yet).
Change-Id: Idd4ae71ee6deee84604823b6b5dc4a845f316e01
Related-Bug: #1707788
This was just to avoid our initial gate crisis, and has been put into
project-config with I45b4b181369032155f8908ee11641d2327586e6f
Change-Id: I3ab57b4455b39ccc3fa94ef1be2193fa7f082fb6
The image download tests have long been too unreliable for the gate.
We need to cache the base images similar to how devstack caches it's
testing images. Let's move them to non-voting jobs for the time
being.
This means that the gate jobs are now all based on "-minimal" and are
using infra mirrors. Unfortunately, there is still some unreliability
because we currently have issues with infra mirrors being very slow
after AFS updates, leading to job timeouts. But we're on the right
path...
Also, I noticed we don't have tests of the "ubuntu" image-download
based tests, which were tacitly being tested by apt-sources before we
moved that to -minimal. Add simple tests for these.
Change-Id: Ie33ee49656872467ef68d753210032156bb6b2cb
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently. I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image. Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.
This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror. It pulls the info from the mirror_info.sh script
on each CI node.
The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it. Tests are restored
Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
This adds "openstack-ci-mirrors" element which performs various
settings to get builds using local mirrors. As a first step, we
convert ubuntu-minimal jobs
The main trick is that since infra mirrors are created with rerepo
they are not signed (they are recreated, not cloned, and not signing
is seen as a feature in that it deters external use). So we need to
instruct debootstrap to ignore signing and also turn it off for
in-chroot apt. Other than that, the existing DIB_DISTRIBUTION_MIRROR
works to redirect installs.
Remove "restricted" as it's not mirrored, and I don't think we want it
in here by default.
(I think DIB_DISTRIBUTION_MIRROR is a bit of an anti-pattern, because
it leaves the mirrors in the final image -- just because you use them
to build, doesn't mean you want them at runtime). But we don't need
to fix that now, and we don't use any created images.)
This pauses fedora testing until the next change, which moves to using
local mirrors for testing on fedora/centos
Change-Id: I778bd05a1e615c27edf1c9f0a1409119a6b3a850
The gate is currently extremley unstable, and these two issues are
causing most of the problems. We need to commit them atomically so we
can get anything moving again
---
The gate is very unstable downloading the ubuntu tarballs from
upstream at the moment. Move this to ubuntu-minimal which, in a later
change will source files from our local mirror.
We need a caching mechanism for these large files to avoid this
instability. This is future work for the various image-based jobs.
---
Move debian to default skip lists
I don't know if it's mirrors being worked hard for the Stretch
release, but this is constantly failing the gate. I will move this to
the -nv extras job
I am working on having the voting job use local mirrors for
everything. Unfortunately debian infra mirrors don't have stretch yet
and we need to do some fiddling to get "stable" available. Once we
have all this, we can consider making it voting again.
Change-Id: Iaf7b3888ef06c7aef63cbf76a94b33f96bc9c5c2
This adds a devstack-inspired output filter to standardise
timestamping.
Currently, python tools timestamp always (timestamp setup in
logging_config.py) but all the surrounding bash does not.
We have extra timestamps added in run_functests.sh for our own
purposes to get the bash timestamps; but this ends up giving us
double-timestamps for the python bits. Additionally, callers such as
nodepool capture our output and put their own timestamps on it, and
again have the double-timestamps.
This uses a lightly modified outfilter.py from devstack to standardise
this.
All output is run through this filter, which will timestamp it. I
have removed the places where we double-timestamp -- logging_config.py
and the prefix in dib-run-parts.
An env option is added to turn timestamps off completely (does not
seem worth taking up a command-line option for). For callers like
nodepool, they can set this and will just have their own timestamps as
they collect the lines.
Since all logging is going through outfilter, it's easy to add a
--logfile option. I think this will be quite handy; personally I'm
always redirecting dib runs to files for debugging.
I've also added a "quiet" option. I think this could be useful in
run_tests.sh if we were to start logging the output of each test to
individual files. This would be much easier to deal with than the
very large log files we get (especially if we wanted to turn on
parallel running...)
Change-Id: I202e1cb200bde17f6d7770cf1e2710bbf4cca64c
Add a -k flag that disables deleting of of ${destdir} for tests. This
should allow examination of the resulting images if required.
Change-Id: I107c33e70100b21495a807f10762d3b6babe9bfe
Ubuntu 12.04 LTS reached its regular End of Life on April 28, 2017.
Depends-On: I5e145095a10db112bb27516bfe652d2cdc052a61
Change-Id: I64af4c5183d77a75dcd062895d19b0a1330c8da8
On SUSE family distros the squash-tools are simply part of the
main package called "squashfs", so install that one instead. Without
this change bindep on SUSE hosts fails with:
ERROR: These requested packages were not installed:
squashfs-tools
Also adjust install_test_deps.sh to install required packages
on an openSUSE host.
Change-Id: I61dcd5314e78dbb1fb31e723799374edd456da99
Currently openSUSE 42.3 has entered feature freeze mode
so it is a good point in time to verify that 42.3 builds
are working successfully. Also test opensuse-minimal for
platforms that support it (need working zypper package)
Change-Id: I4c613e1e68cb7375c29d544bbf70b5da9bf21414
Instead, either use the bash built-in of type to ensure it exists. Since
which is an external dep, things can fail oddly in a constrained
environment.
Also add a dib-lint test for this.
Change-Id: I645029f5b5bfe1198c89ce10fd3246be8636e8af
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
I think this has gone missing on test nodes during unpuppeting. Add,
as the gentoo job grabs some bz2 tarballs for extraction.
Change-Id: Icb82930702124f7bf112b03a469d90d4163d398b
If the path is missing, unmount_dir currently exits with an error which
unintentionally aborts cleanup efforts early. This change makes
unmount_dir idempotent by exiting successfully if a directory doesn't
exist.
Change-Id: I1491b4344e8569ecb2833f44baee445a89a39d61
We somewhat discussed skipping qcow2 generation previously in
I9372e195913798a851c96e62eee89029e067baa1. As recent issues with PPC
testing have shown, we are not actually testing the "vm" element and
hence the bootloader path in the functional tests.
I don't think we need to test this on every element; it overlaps
somewhat with the testing done by the nodepool jobs which build full
images and boot them. I also didn't want to introduce a separate run
for this. Thus it seems valuable to at least have one element
enhanced to do this installation and conversion in our default tests
for basic sanity.
This disables qcow generation by default, as per the other change, but
allows an element to drop a file that will override the output
formats. The Xenial element is modified to produce a qcow2 using
this, and also introduces a dependency on the "vm" element so it tries
to install the bootloader.
We now exit if the .qcow2 fails to build as well.
Change-Id: I1a6acefe52f8c696c39b2d592fdc7ae32a87e6fe
Change ec7f56c1b2 add added unit tests
in diskimage_builder/tests/functional (this is probably misnamed).
These are found and run by testr just as part of the normal "setup.py
test" run. Don't run them as part of the functional tests (this
breaks "-h"/"-l" because it installs a virtualenv and runs tests, and
also is incorrect in the gate where it's creating a nested virtualenv
underneath the testing virtualenv).
Change-Id: I9908e080042d3026a198ba89eb653c6eff376d22
During the creation of a disk image (e.g. for a VM), there is the need
to create, setup, configure and afterwards detach some kind of storage
where the newly installed OS can be copied to or directly installed
in.
This patch implements partitioning handling.
Change-Id: I0ca6a4ae3a2684d473b44e5f332ee4225ee30f8c
Signed-off-by: Andreas Florath <andreas@florath.net>
DIB automatically includes it's base elements in the ELEMENT_PATH;
this double-set means the base elements are in the path twice, leading
to some tools that walk the element list (package-installs-squash, for
example) to do things twice.
Change-Id: Icf631427601cfd2abed436711cedb9e002fff8f9
Because we run this image in openstack-infra, we want to increase our
test coverage to help avoid potential breaks to our CI systems.
Change-Id: I26405e3f7465654075278ec35b5e0da1338bb45e
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Since we still run these 3 version of ubuntu-minimal elements in
openstack-infra, also run functional testing for them.
Trusty and xenial will be in voting gate, precise added as skipped for
non-voting.
Add the default skip/run status to the "-l" output just to confirm
this too.
Change-Id: Icfbfd0cb7d9acae824972474b77e2fe0486c4f69
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The squashfs format brings a couple of advantages over the other
formats. Image is often an order of magnitude smaller and it can
be used natively, either as an initrd, either with loop mount.
Change-Id: If72940b0c4dafb2504c52dd0429a8eb3f8305751
We now support tgz (tar.gz) as an output format.
Change-Id: Iadec92f2f96c3f904f28bd49f87ffc7d48ef7bd7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Steve Baker