Run script generate-gitmodules.py to sync with governace repository to
add new repositories and remove retired ones.
Change-Id: Ia42b717ad242958ba4a9689ac814c956f460712f
* Update diskimage-builder from branch 'master'
- Drop support for python2
python2 is EOL. OpenStack and Zuul both are python3 only.
Drop the Python-2 CentOS 7 build host testing, including the tripleo
centos-7 train job.
Drop the old openstack-python-jobs template and move to specific
version templates for what we support.
Drop upper-constraints from the py35 environment, as master of
requirements is now using python 3.6-only libraries.
Drop the Xenial tests for similar reasons (rather than making some
sort of forced install); we are keeping Python 3.5 support just for
Zuul compatability at this point so a tox run is enough.
Depends-On: https://review.opendev.org/729787
Depends-On: https://review.opendev.org/729508
Change-Id: I6d90754cb4b7217b10ae777d414c03466b751761
* Update diskimage-builder from branch 'master'
- Prepare to drop Python 2 support
Pin func test requirements to stable/train; the last stable release
with Python 2 support. Switch to the python-jobs-no-constraints
template to avoid using master constraints file.
Only build focal on bionic, other debootstraps are too old.
Remove pip-and-virtualenv testing as we are moving to plain images.
The tripleo-buildimage-jobs are unstable; see linked bug.
Add a note that this is the last Python 2 release.
Change-Id: Ibde7a564dd41cc2d6e80e2dffe5a95a57bbf8ada
* Update diskimage-builder from branch 'master'
- Revert "dib-lint: use yamllint to parse YAML files"
This reverts commit 6ee2995214ee5512c64244ce0ccc7fda639c2d36 and
e85c2a6f035a016f80de3d5aebc026f28dfb6bc1.
I missed that if you pip install and then run dib-lint, it's not going
to pick up the .yamllint file shipped here. Thus it gives spurious
errors.
The reason for this was simply better duplicate key detection in yaml
files, which caused us problems with the kernel installs. However, at
this point it seems just the old "does it load" test from pyyaml will
be enough.
Change-Id: I87a9fc9bb119cfeffad48fc0fa0df31f0181825d
* Update diskimage-builder from branch 'master'
- Merge "Pre-install xz package in opensuse chroot"
- Pre-install xz package in opensuse chroot
As of recently, opensuse-minimal images fail to build because of an
error installing the kernel-default package:
> Problem: kernel-default-5.6.12-1.3.x86_64 requires mkinitrd >= 2.7.1, but this requirement cannot be provided
> not installable providers: dracut-050+suse.61.g0fe0e854-1.1.i586[repo-oss]
> dracut-050+suse.61.g0fe0e854-1.1.x86_64[repo-oss]
The problem is there is a recently added package `busybox-links` which
provides a subpackage `busybox-xz` which provides the /usr/bin/xz
utility. Since this is available, the `aaa_base` package installs it
during the root.d base installation phase to fulfill it's dependency on
/usr/bin/xz. On the other hand, the dracut package explicitly requires
the `xz` package, and this is not co-installable with the `busybox-xz`
package, so the dracut package is not installable during the install.d
phase. This change explicitly adds the `xz` package to the initial
chroot provisioning phase so that the /usr/bin/xz requirement is already
fulfilled and `busybox-xz` does not get installed.
Change-Id: Iba8c301eb496657873963e1aa99736aacf87cb00
* Update diskimage-builder from branch 'master'
- Merge "use stage3 instead of stage4 for gentoo builds"
- use stage3 instead of stage4 for gentoo builds
The main reason for using the stage4 is now gone (kernel compile).
Install and use the distro provided binary kernel package.
In addition to this, set the locale and timezone, beyond that very
little was done in the gentoo stage4.
Change-Id: I541b7d9b807e2357398ae1c249b1978958dd1137
Signed-off-by: Matthew Thode <mthode@mthode.org>
* Update diskimage-builder from branch 'master'
- Add dependency on yamllint
dib-lint requires yamllint now, so we need to depend on it.
Change-Id: Iccd2cdbb04bd9b429d8d19b31ff3b10cdd568e15
* Update diskimage-builder from branch 'master'
- Revert "Revert "ubuntu-minimal : only install 16.04 HWE kernel on xenial""
This reverts commit 6e549c33ac1ead9e7bbb3e13f38296ee36e28ed9.
It uses the new multiple-parameter matching format from
Idff7b067ad4255e6fc4138f7eff313a81b75c8ba to actually do what it says.
Change-Id: I4656ff1a5c46bcfbd8587f2f541825f4ad08820f
* Update diskimage-builder from branch 'master'
- package-installs : allow a list of parameters
The change Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d introduced having
"when:" as a list of values. However, this was actually not
sufficient to express the logic required for arm64/x86_64/xenial
kernel matching we wanted.
Because the package name is a key, we can't have multiple entires in
the package-map YAML files. This means we can't do more advanced
matching and thus we need to be able to match through multiple
parameters. Similar to Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d we
modify the matching rules to allow a list.
A an example of using this is provided in the README.rst, and this
same example worked through by the unit tests.
This also slightly updates the matching logic to be more sequential.
After each check we either continue on or log the failure and continue
to the next check (rather than set a list of flags then check that at
the end). This makes it much easier to understand what is being
matched in the logging output from the tool.
Change-Id: Idff7b067ad4255e6fc4138f7eff313a81b75c8ba
* Update diskimage-builder from branch 'master'
- dib-lint: use yamllint to parse YAML files
This gives us better linting of YAML files that just opening them.
This would have detected the duplicate keys in
I34e27d821fbefe274e7b007f37b0bd34db2e1d26.
The .yamllint is taken from zuul-jobs where it is also used as a
fairly sane set of default rules.
A few minor newline fixes are added.
Change-Id: I96d6644ae24f7deb84fa50fefbda0f0d33e0e009
* Update diskimage-builder from branch 'master'
- Revert "ubuntu-minimal : only install 16.04 HWE kernel on xenial"
This reverts commit 14ff8f942c076936e79ee2c5fce1e9f34efe55a3.
This seems to not be installing the kernel at all, and needs further
investigation.
Change-Id: Ifd809d4b67aff5d80f979235db246a16af0375b3
* Update diskimage-builder from branch 'master'
- ubuntu-minimal: Add Ubuntu Focal test build
Add test builds for Focal on x86 and ARM64
Change-Id: Idb23f0e00d37c7447441ea002aad078e8c61f969
* Update diskimage-builder from branch 'master'
- ubuntu-minimal : only install 16.04 HWE kernel on xenial
Only install the HWE kernel by default for Xenial. This was actually
installing the 16.04 HWE kernel on Bionic by accident, since it seems
to have that package; however it was breaking Focal.
On the other distros, just install the default generic kernel. Let's
KISS for now if we can ...
Change-Id: I34e27d821fbefe274e7b007f37b0bd34db2e1d26
* Update diskimage-builder from branch 'master'
- ubuntu-minimal: fix HWE install for focal
On Focal, install the 20.04 HWE kernel if linux-image-hwe is
specified.
Change-Id: I5d536e8f64e7b987415849d8cd4da7959bba7af7
* Update diskimage-builder from branch 'master'
- package-installs: allow when filter to be a list
Allow the "when:" statements to be a list of values, which are
effectively anded together to filter the package install.
Change-Id: Ia6f10741fa6be24b11d6991c8a6b6e07951ff68d
* Update diskimage-builder from branch 'master'
- block device: update variable name
New versions of flake8 fail as "l" can be confused for "1" apparently
(E741) ... not sure I totally agree but since it's only one instance,
update it.
Change-Id: Ic5c47867facd56b53cc6534da4ae3a345c516202
* Update diskimage-builder from branch 'master'
- Merge "functests: use ensure-virtualenv"
- functests: use ensure-virtualenv
This ensures the virtualenv tool is available on all testing nodes.
Depends-On: https://review.opendev.org/723309
Change-Id: Ic52a380dab9faa0f65d3c1ebe0bff6f69736e3cd
* Update diskimage-builder from branch 'master'
- Add sibling container builds to experimental queue
These are required for the experimental container boot jobs
Change-Id: I16c27311adab09eafef747c1bba868115c000928
* Update diskimage-builder from branch 'master'
- Merge "Add a focal test"
- Add a focal test
This adds a boot test for the newly released Ubuntu Focal release.
Depends-On: https://review.opendev.org/722556
Change-Id: If8976a3f99a02c8d85cd3818d8c77c75b60816d8
* Update diskimage-builder from branch 'master'
- Merge "Restore SUSE tests to gate"
- Restore SUSE tests to gate
We can build SUSE in the container after the switch in
Ieaf07438c145ba609d4d7b9a055e91b894eca6d9.
Let's start non-voting and promote when we feel it's stable.
Change-Id: Ibe997d3167dc918fcdb28cf065c50049943dd86a
* Update diskimage-builder from branch 'master'
- Merge "Switch functional tests to containers"
- Switch functional tests to containers
This switches the functional tests to container based build jobs.
This is mostly transparent for the existing build and boot jobs.
The release test is a bit interesting here. The nodepool
container-based release job doesn't work in this context because it
doesn't have a way to also include dib from the zuul checkout into the
container -- ergo it's pointless to run in the dib gate.
We should probably reorganise the nodepool siblings job so it can
check some projects out from tags, then we could run something similar
to this with nodepool/openstacksdk/glean etc checked out from tags if
we want.
We can handle this in a future update so we can test these images
which we want to build in our production container asap.
Depends-On: https://review.opendev.org/722556
Depends-On: https://review.opendev.org/722549
Change-Id: Ieaf07438c145ba609d4d7b9a055e91b894eca6d9
* Update diskimage-builder from branch 'master'
- Merge "Bionic functional tests should be voting"
- Bionic functional tests should be voting
We now have a bionic(-ish ... buster debian image anyway) builder and
have moved some of the important tests (fedora, pip-and-virtualenv) to
here because they don't work on Xenial any more. This should now be
voting so we don't miss anything.
Change-Id: I16d313e1dfa95707d996456da244667c38d6343f
* Update diskimage-builder from branch 'master'
- Merge "pip-and-virtualenv : fix fedora 30 install"
- pip-and-virtualenv : fix fedora 30 install
This should be installing the python2 and python3 packages (that's
what pip-and-virtualenv is designed to do), but we dropped the +=
accidentally in ee9ad32b6fd766dce04d7276c6a7eb489e24741f.
However, we've moved on anyway and after
I7a6a342461d6001c25e55638ba9b7438c28f2519 F31 doesn't support this
element. fedora-latest is already updated to f31 in the opendev gate.
Remove the testing as it is no longer relevant.
Change-Id: Id696a90baa1eb05cb4c08501f8dac3665d395682
* Update diskimage-builder from branch 'master'
- Merge "yum-minimal: strip env vars in chroot calls"
- yum-minimal: strip env vars in chroot calls
This showed up with dnf in containers when TMPDIR was set; dnf started
trying to write to this directory while in the chroot.
We already do stripping like this in run_in_target -- but this is a
bit of a unique place because it's actually setting up the initial
chroot so the target doesn't actually exist yet; so we just hard-code
it in place here.
Change-Id: If7310cb820846da903bf60daa4486c8bf7cb0136
* Update diskimage-builder from branch 'master'
- Merge "Fix args to debuntu functional tests"
- Fix args to debuntu functional tests
Ia4dde1fb01da284a5e681237ab88c68fb9afcbef dropped these from the
generic job in nodepool, because they shouldn't apply to other distros
using the job like CentOS etc. Add the required args back here to the
debuntu tests.
Change-Id: I3f9ae58f3aa534d65344d79d44b0b74368e329a8
* Update diskimage-builder from branch 'master'
- Merge "pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install"
- pip-and-virtualenv: drop f31 & tumbleweed, rework suse 15 install
This is an alternative approach to commit
68bb43535e8f032890687be6a253c3505cb319e5. I think this proposes a
better overall solution that the prior change which had the Python 3
packages being installed, but did not specify the _do_py3 flag to do
the installation steps that redirect the various tool installations.
Fedora 31+ doesn't have python2, and Tumbleweed does have some Python
2 support but there seems to be no reason to bother updating this
element for either with infra very close to removing this completely
[1]. Error out on these platforms, and add a release note.
The 15 path should include the python2 and python3 packages, along
with the flags to do the "cleanup"; i.e. forced removal of distutils
packages that pip 10+ won't touch. As mentioned in the original
change, the six package causes problems here, but we can clear that
too by explicitly listing it instead of letting it come in via
dependencies. Again, this element will be removed from the infra 15
builds ASAP; but we can release with this to provide a roll-back point
if we need to revert the removal to fix things temporarily.
Add it to the testing path as well.
[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html
Change-Id: I7a6a342461d6001c25e55638ba9b7438c28f2519
* Update diskimage-builder from branch 'master'
- Remove Trusty testing
Opendev has dropped trusty from the mirrors. With no testing the
distribution is effectively unsupported, add a release note. Update a
few other random doc bits (that are not really that up to date
anyway).
Change-Id: I5bd0d0a94477cf8d84cef72f5d4b2e9e15ab9fd2
* Update diskimage-builder from branch 'master'
- Merge "Remove Babel and any signs of translations"
- Remove Babel and any signs of translations
DIB does not currently have any translations or translatable strings,
so removing local configuration from setup.cfg.
Additionally, Babel is only required if you import it directly,
removing it as well.
Change-Id: I036de97641df38eef408290b50cecfe216c69f6d
* Update diskimage-builder from branch 'master'
- Merge "Make ipa centos8 dib job voting"
- Make ipa centos8 dib job voting
The ironic-python-agent-builder dib job based on centos8 is stable
since a while now.
We should make it voting to catch any possible issue on dib side.
Change-Id: I42dd81fa17de1130a3af7c031f9479e18790c350
* Update diskimage-builder from branch 'master'
- Do not try to use MBR on AArch64
When I tried to build CentOS8 image for AArch64 I got error saying that
MBR is not supported. So make sure that it will not be used by default.
Change-Id: Ib67ab7f808d727c3c61932c540d398dbe723972f
* Update diskimage-builder from branch 'master'
- opensuse: fix python 2.x install
openSUSE Tumbleweed is dropping python2-* packages so we need
to stop intalling them. We can also stop installing those
for Leap 15. which avoids a pip uninstall issue (as python2-six
was still built with distribute).
Change-Id: Ie93c8addb26aab3d0154c4b5b52423799abede91
* Update diskimage-builder from branch 'master'
- Merge "centos 8 image build: fix mirror"
- centos 8 image build: fix mirror
We're ending up with "centoscentos" in the mirror location and the
build fails; strip out the $contentdir from the original too.
Change-Id: If09dbbd8028ea510d2ab0d3d8afe484cea611df5
* Update diskimage-builder from branch 'master'
- Merge "run_functests: handle build without tar"
- run_functests: handle build without tar
It seems this code does not consider that you might set
test-output-formats with only "qcow2" -- it currently also tries to
inspect a .tar output.
Skip if tar isn't in the output format list, and fall through to
passing so both paths are considered success, only exiting if there is
a failure.
Change-Id: Ia21dbfd298d04b04ec02dfe76a0acd7c9a7a31f2
* Update diskimage-builder from branch 'master'
- Merge "Mellanox element: removed ibutils,libibcm,libmlx4-dev"
- Mellanox element: removed ibutils,libibcm,libmlx4-dev
Those packages will be installed as part of dependencies
Change-Id: I640dc55c818dd8538bfcd199c33f9af93aae7608
* Update diskimage-builder from branch 'master'
- Merge "Add Fedora 31 support and test jobs"
- Add Fedora 31 support and test jobs
Change-Id: Iad0261ac8db001ffa4d3a4dd6df05fe923402a69
* Update diskimage-builder from branch 'master'
- Merge "Add python-stow-versions element"
- Add python-stow-versions element
This element is designed to install latest minor versions of different
python releases, like py27, py35, py36, py37, py38
into stow directory, and later easily enable them with stow.
Change-Id: Iab6d20e7643e549b53c629fb430e58b1c5e72991
* Update diskimage-builder from branch 'master'
- Use rpm -e instead of dnf for cleaning old kernels
If the running kernel of the system building the image
matches the kernel that is to be removed dnf will fail.
Repalce use use of dnf with rpm -e.
Closes-Bug: #1623409
Change-Id: Ie2481ea8a02b7b0720e46fa179f24badf4aa25c5
* Update diskimage-builder from branch 'master'
- Add support for build-only packages
Sometimes an element needs packages installed so that it can
perform tasks but those package are not appropriate for the
final image content. Add a "build-only" flag to package-install-squash
which will cause package to be installed at the beginning of the
phase and then uninstalled at the end of the phase.
Change-Id: Ie01b795991710c93f6b669c8f14b57eb4412c1d5
* Update diskimage-builder from branch 'master'
- bindep: remove lsb-release
This came in with edc06a20e57bdf0a9a03949b54395008d73520dc although
it's not totally clear why.
We found that we were sourcing devstack scripts during build in infra,
and inadvertently relying on this package being around on the host.
This was incorrect but can this type of problem is hidden if the
package is installed unnecessarily.
I do not think this actually required now; remove from bindep to
minimise chance for such false or hidden dependencies.
Change-Id: I7895d4c5019511eb2169915cd8092e707e8ee732
Story: #2007407
Task: #39068
* Update diskimage-builder from branch 'master'
- Remove hacking from requirements
putting it in requirements means it's installed for anything that
installs diskimage-builder. Not just for tests, but in actual prod
installations. Because of the way flake8 plugins work, this means
then that hacking is injected into all flake8 invocations in that
environment.
This is currently causing issues in nodepool where we do not
actually use hacking but it's still spewing spurious messages to
the screen because it is incorrect finding our apache header to be
incorrect.
Just depend on flake8, since that's what dib-lint depends on.
Change-Id: I8d2b2076b604d596906a3b12a5518e2a56c0fe88
* Update diskimage-builder from branch 'master'
- Merge "Add ensure-venv element, install glean with it"
- Add ensure-venv element, install glean with it
All the platforms we care about now have python3 with venv (even
centos7 now) packaged somehow. Add an ensure-venv element to make
sure that "python3 -m venv" works. Any other elements that wish to
install non-distribution-packaged Python utilities can use this to
keep them separate from the main system installs.
Port glean to use this, and drop its dependency on pip-and-virtualenv.
Change-Id: Ic16f134fe34293bb68e7c632dd320f523366320d
* Update diskimage-builder from branch 'master'
- Merge "Uncap hacking"
- Uncap hacking
This causes problems for other projects incorporating dib; we don't
have a specific need for a cap.
Fix a few issues, mostly spacing or regex matches. No functional
changes.
W503 and W504 relate to leaving artithmetic operators at the start or
end of lines, and are mutually exclusive and, due to "ignore"
overriding the defaults both get enabled. It seems everyone gets this
wrong (https://gitlab.com/pycqa/flake8/issues/466). Don't take a
position on this and ignore both.
Use double # around comments including YAML snippets using "# type: "
which now gets detected as PEP484/mypy type hints.
Change-Id: I8b7ce6dee02dcce31c82427a2441c931d136ef57
* Update diskimage-builder from branch 'master'
- Merge "fix iscsi-boot element exiting build even if dracut-regenerate used"
- fix iscsi-boot element exiting build even if dracut-regenerate used
in CentOS build case building an image with "iscsi-boot" and "dracut-regenerate" will exit building because of statement "[ "$found" = 0 ]"
Change-Id: I1a6d60e9ec5f5cb508866c8376465c3e73551a30
* Update diskimage-builder from branch 'master'
- Merge "Change tgt pkg-map to target-restore CentOS/RHEL-8"
- Change tgt pkg-map to target-restore CentOS/RHEL-8
In ironic-agent tgt pkg-map, scsi-target-utils was used for
redhat family which is not used in CentOS/RHEL-8.
As per https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/dib/ironic-python-agent-ramdisk/pkg-map#L5
targetcli got removed and switch to target-restore. It syncs
the same element for RHEL-8 and adds the same for CentOS-8.
Closes-Bug: #1864427
Change-Id: I95a0c09a6739af23cd1e8c88dded198bd69cc53e
Signed-off-by: Chandan Kumar (raukadah) <chkumar@redhat.com>
* Update diskimage-builder from branch 'master'
- Merge "Do not include efibootmgr and efivars for ppc architectures"
- Do not include efibootmgr and efivars for ppc architectures
These don't exist and are not needed for ppc.
Change-Id: I9ba53e22583b148b43f74cd9b8ec79402796d09b
* Update diskimage-builder from branch 'master'
- Add CentOS 8 support
* Add "centos" element, a CentOS version-independent element. This is in
line with the same work done for RHEL in Stein cycle.
* Deprecate the centos7 element. CentOS 7 support itself it not
deprecated though. The new "centos" element provides the same support
level as the "centos7" element.
* Add functional testing
The default CentOS version is 8. You can adjust it using the DIB_RELEASE
environment variable.
Change-Id: I373ba2296c4613765676e59aabd9c651345298d1
* Update diskimage-builder from branch 'master'
- Merge "Fix cache-url -f"
- Fix cache-url -f
When there is a hashsum mismatch diskimage-builder forces downloads
with the -f switch of cache-url. This is currently broken because bash
escapes the quotes in curl_opts. This tricks curl trying to download
'no-cache' instead of the url. This can be fixed by using an array for
curl_opts which does the correct thing here.
Change-Id: Id9f1579dda9a3e0a2b08dd5faaeef0e2e580d419
* Update diskimage-builder from branch 'master'
- Merge "Allow python3 to be used in Debian"
- Allow python3 to be used in Debian
Debian default Python interpreter version is 2.7, but it's
possible to install a Python 3 interpreter from the base
repository.
With this change, if we set DIB_PYTHON_VERSION to 3, we install
the python3 package from base, with python3-libs, python3-pip and
python3-setuptools, and redefine python_path, effectively allowing
Python 3 interpreter to be used in Debian.
See a result of the job for building the ipa image here:
https://review.opendev.org/705773
Change-Id: Idabfa94c2bff6e0de6daa0866084d5db14d7dcb0
Andreas Jaeger