This started a long way from here, when I noticed that "top" on centos
9-stream images wasn't working because ncurses-base wasn't installed.
This led me to the extant install of bash/glibc/ncurses-libs from
Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6. However it didn't really
explain why these are brought in here.
Reading further it became clearer that over the years of distribution
additions, Fedora updates, etc. this has grown into a bit of a mess.
Refactor the release package installs into a more logical flow,
pulling out checks/comments for Fedora's of ancient history, etc.
Remove the 9-stream package installs; this isn't the place for them,
and the should be brought in by the base packages.
Ultimately, this is intendend to a be a no-op refactor.
Change-Id: Ie7d9a6497d0d20a3303ec0da3d0668c74efa2c3d
AFAICS, use of this was removed with
I7f98a13091056809fedae8a5c8ee10b0ef8bbb2a and I can't see any other
references to it. Correct the comment to describe how it works.
Change-Id: I5123729b7457dcbd4f4a51cff49904f7bd071e9b
For centos stream, the $releasever is just the major version. Several
of our .repo files are using $releasever in their path, and I think
that 8-stream installs are actually using 8 repos to install from.
For 9-stream, which doesn't have a corresponding 9, we're getting
errors enabling some of the aarch64 tests.
Replace all the $releasever expansions in the .repo files with the
exact version they are being installed for. They don't need to be
generic; we are installing these specific repos for each DIB_RELEASE,
so they don't mix-and-match.
Change-Id: I48d438d8f51280cd060433fc8a67358d8345287f
CentOS Stream 9 is close to be released, and official mirrors are
already poplated. This patch is adding support to centos-minimal in CS9.
Also enable centos-minimal/[8,9]-stream-build-succeeds tests.
This patch is being tested together with [1] to apply following list of elements:
vm centos-minimal simple-init growroot nodepool-base openstack-repos infra-package-needs
[1] https://review.opendev.org/c/openstack/project-config/+/811442
Change-Id: Iecf7f7e4c992bb23437b6461cdd04cdca96aafa6
The latest Debian bullseye release doesn't provide yum any more, only
DNF. This breaks the minimal builds that are using on-host yum tools
to start the chroot. Probe for yumdownloader, and if it's not there,
use DNF.
Note this requires "dnf download" which may not be packaged. See
I21cfbd3935e48be4b92591ea36c7eed301230753 for a sample work-around
that installs this plugin in the nodepool-builder container.
Change-Id: Ia7f1e4d115cc67c378d865d91af94a07b8cdc6cc
Add openeuler-minimal element and add CI functional tests for both
x86_64 and arm64.
OpenEuler is an open source community driven YUM/DNF distro like
Fedora. It references Fedora and CentOS a lot for the rpm packages
building. So somewhat it can be treated as a redhat family distro
and reuse the YUM/DNF related elements to help build openEuler images.
For more info about openEuler, see: https://openeuler.org/en
Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/803413
Change-Id: I3e06e49b524364c3a4edeba8bce7a8c06b9c7b76
This change permits the yum-minimal element to be used in downstream
custom distributions, which may have additional packages containing repo
config or GPG keys needed.
This could also be utilized at a later time to move the
distribution-specific logic in this method to each distribution element
separately.
Change-Id: Ic1434bb2fe7301086cf11ba6bd7f2ee187c5e6c8
As noted inline, this works around potential issues by being a strong
indication you are in a container (e.g. [1]). Since nothing should be
changing anything on the host/build system, this is a generically
safer way to operate.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1975588
Change-Id: Ic6802c4ffc2e825f129af10717860a2d1770fe80
This package doesn't exist in the stream base repo, and neither does
centos-linux-repos.
These are presumably replaced by centos-stream-release and
centos-stream-repos. This change adds an else block to handle the
non-stream base packages.
Change-Id: I32249199c3dfa44fc24fba28d24f314112c2e200
With Centos 8.3, centos-repos package has been replaced by
other packages [1].
[1] https://lists.centos.org/pipermail/centos-devel/2020-September/056069.html
Also Increase flake8 and pyflakes version in lower-constraints.txt as
this was already broken.
Change-Id: Ife139fcaff0c2d944098ea353259971d2d3f18b8
centos-repos package is not available on centos-8-stream:
2020-11-02 15:35:20.962 | No Match for argument centos-repos
2020-11-02 15:35:20.992 | No Match for argument centos-release-stream
The build fails later on install_pkg_manager sed command:
2020-11-02 15:35:22.903 | sed: can't read
/var/cache/nodepool/dib_tmp/dib_build.iPSHptNW/mnt/etc/yum.repos.d/*repo: No
such file or directory
This change ensure centos-stream-repos package is installed to provide
/etc/yum.repos.d/*repo files
This change replace 'centos-release-stream' package with the new
'centos-stream-release' package [1]
[1] https://lists.centos.org/pipermail/centos-devel/2020-September/056069.html
Change-Id: I6c397bf7b5797a02e5f006c18ee63c9cdf66b38c
This patch adds support for CentOS 8 Stream [1] to the centos-minimal
element. Users should set DIB_RELEASE=8-stream.
[1] https://www.centos.org/stream/
Change-Id: Id0825de735ab957c10daf35fb3c641f850cc6847
This showed up with dnf in containers when TMPDIR was set; dnf started
trying to write to this directory while in the chroot.
We already do stripping like this in run_in_target -- but this is a
bit of a unique place because it's actually setting up the initial
chroot so the target doesn't actually exist yet; so we just hard-code
it in place here.
Change-Id: If7310cb820846da903bf60daa4486c8bf7cb0136
CentOS 8.1 split repositories and GPG keys out into subpackages. This
broke DIB support for CentOS 8.
7e41cef418?branch=c8
26a0d73ced?branch=c8
Change-Id: If3de6efa6074e059dc9fdd47c7bdc19d26d4d7f2
Don't install the "yum" package, which is a backwards compat around
dnf. With 687003f we should not need the backwards compat links any
more.
Add libcurl to avoid conficts with in the curl "-minimal" packages
that happens on CentOS 8. But skip it on Fedora, because it seems to
create more problems there (not going to pretend it isn't all a
hack ... but it seems to work).
Change-Id: I1de2703eb5075a0a22837b6898bd8eb960d080dd
CentOS 8 has the "new" split-up locales packages. Fedora 24 is now
long gone, so take out the old branch and apply the lang package
install to Centos 8 as well.
The manual locale cleanup is not necessary on Centos 8; skip it.
Change-Id: Ib65fc15fe471348793fd6efb034517f11abd905e
The repo format has slightly changed for CentOS 8 (s/os/baseos/).
Make the chroot builder look for a more specific repos.d directory
first named for the distro variable, then fall back to to top-level
dir (this avoids having to constantly change fedora).
Update the gate mirror setup and roles for new Centos 8 paths too.
Change-Id: I5b7f0c3624cac1d7aa7ed8bf6286b85d808b9c9a
This patch adds option DIB_YUM_MINIMAL_EXTRA_REPOS to yum-minimal to
allow DIB users to include extra repositories to their final image.
Change-Id: I89549f4b0f4c9470143b5064817acab5043e31c5
Something (possibly [1], but that change is at best cryptic) has
changed such that we don't get correct /etc/os-release files
installed. This flows on to grub half-installing itself, enough to
not fail the build but not enough to make something bootable.
Installing the -cloud release package gets it back, and seems like a
sane choice for dib.
[1] 617b1bed34
Change-Id: Iff0413887fad798273b2bfcb140cc07f36d54a04
It looks like fedora-release on fedora 30+ has been split into sub
packages. Use fedora-release-common to avoid package conflicts.
Change-Id: I8f8711044fc4074b91939e0a6dfdac4d7a14a35b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
In fedora-30 is when we migrate to dbus-broker, fedora-29 is still using
dbus-daemon.
Change-Id: I1e1d3a3826157b8b22386c211eaa58b6439b5f3c
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
As described in the comments, it seems the transition between
dbus-daemon -> dbus-broker in Fedora 29 has made it so the packages
can get into a state where neither service is enabled.
Explicitly install and enable dbus-broker for F29
Change-Id: I06753043a75be2f635653899c6c251b9fbdd7c67
As described, Fedora 27 has a curl-minimal package that comes in to
satisfy the rpm package dependency. It conflicts with the "real" curl
package -- which is so commonly installed (by infra elements, etc)
that this becomes an annoying problem. Just pre-install the full curl
package.
Fedora 24 is old enough to not worry about, so remove some old
workarounds to make the flow a little simpler.
Change-Id: I67baf96377109ac4521ba00243a0d91b35fafba0
The current implementation - as introduced in
Iee44703297a15b14c715f4bfb7bae67f613aceee - has some shortcomings / bugs,
like:
* the 'grep' check is too sloppy
* when /dev/pts is already mounted multiple times the current implementation
fails:
$ mount | grep devpts | sed 's/.*(\(.*\))/\1/'
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
* code duplication
* Using the undocumented and non-robust output
of 'mount'.
This patch fixed the above problems.
Change-Id: Ib0c7358772480c56d405659a6a32afd60c311686
Signed-off-by: Andreas Florath <andreas@florath.net>
This is a continuation for f2cc647dae ("diskimage_builder: lib:
common-functions: Fix options for devpts mount"). We also need to
respect the devpts mount options when the dib elements are mounting
this virtual filesystems themselves.
Change-Id: Iee44703297a15b14c715f4bfb7bae67f613aceee
In a couple of places we use flock for critical sections, but we leave
lockfiles around in various locations which can be confusing.
Introduce DIB_LOCKFILES global (under ~/.cache/dib/lockfiles) and
write lockfiles in there.
Fix up removal of the lockfile in the yum path; we just want to make
sure we cleanup the .rpmmacros file, but we don't need to remove the
lockfile as well.
Co-Authored-By: Andreas Florath <andreas@florath.net>
Change-Id: Ie810b2836be521325afe923708d046112e1e1e20
Currently a bind is used when mounting /dev/pts in chroot.
This leads to problems - especially when running DIB in parallel:
It was observed that the /dev/pts mount vanishes from the host
system.
This patch uses '-t devpts' - as it is done for /sys and /proc -
for handling /dev/pts.
Change-Id: Id7775ae6fca6502af800e7b73a00862ef320206b
Signed-off-by: Andreas Florath <andreas@florath.net>
As described in the referenced bug, the dependency solver in yum
doesn't handle weak dependencies well and in some cases, such as
Fedora 26, can end up choosing coreutils-single (the busybox-esque
single binary) instead of actual coreutils, which then causes problems
with conflicting packages later.
Change-Id: I2907bf3b74c146986b483d52cc6ac437036330b4
fedora/centos-minimal don't obey DIB_DISTRIBUTION_MIRROR currently. I
don't really want them too -- we want to be able to separate the
mirrors used during the build process from those embedded into the
final image. Add DIB_YUM_MINIMAL_BOOTSTRAP_REPOS which is a directory
with repo files to use during the install.
This introduces setup-gate-mirrors.sh which is intended to setup
repo/sources/whatever files in the openstack gate that point to the
local region mirror. It pulls the info from the mirror_info.sh script
on each CI node.
The openstack-ci-mirrors element is updated to export these variables.
elements are updated to depend on it. Tests are restored
Change-Id: I7604fc4d41cb1483be16b8d628a24e8fc764f515
Currently we have all our elements and library files in a top-level
directory and install them into
<root>/share/diskimage-builder/[elements|lib] (where root is either /
or the root of a virtualenv).
The problem with this is that editable/development installs (pip -e)
do *not* install data_files. Thus we have no canonical location to
look for elements -- leading to the various odd things we do such as a
whole bunch of guessing at the top of disk-image-create and having a
special test-loader in tests/test_elements.py so we can run python
unit tests on those elements that have it.
data_files is really the wrong thing to use for what are essentially
assets of the program. data_files install works well for things like
config-files, init.d files or dropping documentation files.
By moving the elements under the diskimage_builder package, we always
know where they are relative to where we import from. In fact,
pkg_resources has an api for this which we wrap in the new
diskimage_builder/paths.py helper [1].
We use this helper to find the correct path in the couple of places we
need to find the base-elements dir, and for the paths to import the
library shell functions.
Elements such as svc-map and pkg-map include python unit-tests, which
we do not need tests/test_elements.py to special-case load any more.
They just get found automatically by the normal subunit loader.
I have a follow-on change (I69ca3d26fede0506a6353c077c69f735c8d84d28)
to move disk-image-create to a regular python entry-point.
Unfortunately, this has to move to work with setuptools. You'd think
a symlink under diskimage_builder/[elements|lib] would work, but it
doesn't.
[1] this API handles stuff like getting files out of .zip archive
modules, which we don't do. Essentially for us it's returning
__file__.
Change-Id: I5e3e3c97f385b1a4ff2031a161a55b231895df5b
Ian Wienand