Remove yum chroot caching

Every run we are doing a full tar.gz of the chroot environment that
never gets used.

It's not suitable for CI since we use fresh images each time there.

The cache in general isn't really isn't a very safe thing to have
around, because there's no invalidation procedure and no real way to
make one -- we've no guarantee that a new chroot build even moments
after a previous one wouldn't bring in or different packages, etc (of
course this is *unlikely*, but the longer you go between builds the
worse the problem becomes.  Also, tons of packages get installed after
this not from any cache, so potential speed-up is rather marginal.

Debian turned this off with I58fc485aacacaa17243bf9ce760ed91256d1f182.
However, given the reasons above and it's complete lack of testing, I
don't see this as useful.

If we really want this type of thing, I think we should come up with a
way to use a persistent external yum/dnf cache that yum/dnf keeps in
sync with it's usual invalidation rules.

Change-Id: I66789c35db75c41bc45ea1ad2e26f87456de4e4d
This commit is contained in:
Ian Wienand 2016-12-09 14:41:24 +11:00
parent 8c74c8e409
commit 4585955a8b
5 changed files with 106 additions and 121 deletions

View File

@ -6,13 +6,9 @@ Create a minimal image based on CentOS 7.
Use of this element will require 'yum' and 'yum-utils' to be installed on
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
variables can be set to prefer the use of a pre-cached root filesystem
tarball.
By default, `DIB_YUM_MINIMAL_CREATE_INTERFACES` is set to enable the
creation of `/etc/sysconfig/network-scripts/ifcfg-eth[0|1]` scripts to
enable DHCP on the `eth0` & `eth1` interfaces. If you do not have
By default, ``DIB_YUM_MINIMAL_CREATE_INTERFACES`` is set to enable the
creation of ``/etc/sysconfig/network-scripts/ifcfg-eth[0|1]`` scripts to
enable DHCP on the ``eth0`` & ``eth1`` interfaces. If you do not have
these interfaces, or if you are using something else to setup the
network such as cloud-init, glean or network-manager, you would want
to set this to `0`.
to set this to ``0``.

View File

@ -11,10 +11,6 @@ Due to a bug in the released version of urlgrabber, on many systems an
installation of urlgrabber from git is required. The git repository
can be found here: http://yum.baseurl.org/gitweb?p=urlgrabber.git;a=summary
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
variables can be set to prefer the use of a pre-cached root filesystem
tarball.
This element sets the `DIB_RELEASE` var to 'fedora'. The release of fedora
to be installed can be controlled through the `DIB_RELEASE` variable, which
defaults to '21'.
This element sets the ``DIB_RELEASE`` var to 'fedora'. The release of
fedora to be installed can be controlled through the ``DIB_RELEASE``
variable, which defaults the latest supported release.

View File

@ -9,10 +9,6 @@ or fedora-minimal elements to get an actual base image.
Use of this element will require 'yum' and 'yum-utils' to be installed on
Ubuntu and Debian. Nothing additional is needed on Fedora or CentOS.
The `DIB_OFFLINE` or more specific `DIB_YUMCHROOT_USE_CACHE`
variables can be set to prefer the use of a pre-cached root filesystem
tarball.
If you wish to have DHCP networking setup for eth0 & eth1 via
/etc/sysconfig/network-config scripts/ifcfg-eth[0|1], set the
environment variable `DIB_YUM_MINIMAL_CREATE_INTERFACES` to `1`.

View File

@ -31,8 +31,6 @@ if [ $ARCH = amd64 ]; then
ARCH=x86_64
fi
# Calling elements will need to set DISTRO_NAME and DIB_RELEASE
DIB_YUMCHROOT_EXTRA_ARGS=${DIB_YUMCHROOT_EXTRA_ARGS:-}
YUMCHROOT_TARBALL=$DIB_IMAGE_CACHE/yumchroot-${DISTRO_NAME}-${DIB_RELEASE}-${ARCH}.tar.gz
# TODO Maybe deal with DIB_DISTRIBUTION_MIRROR
http_proxy=${http_proxy:-}
YUM=${YUM:-yum}
@ -189,107 +187,97 @@ function _install_pkg_manager {
$TARGET_ROOT/etc/yum.repos.d/*repo
}
if [ -n "$DIB_OFFLINE" -o -n "${DIB_YUMCHROOT_USE_CACHE:-}" ] && [ -f $YUMCHROOT_TARBALL ] ; then
echo $YUMCHROOT_TARBALL found in cache. Using.
sudo tar -C $TARGET_ROOT --numeric-owner -xzf $YUMCHROOT_TARBALL
# Note this is not usually done for root.d elements (see
# lib/common-functions:mount_proc_dev_sys) but it's important that
# we have things like /dev/urandom around inside the chroot for
# the rpm [pre|post]inst scripts within the packages.
sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys
sudo mount -t proc none $TARGET_ROOT/proc
sudo mount --bind /dev $TARGET_ROOT/dev
sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts
sudo mount -t sysfs none $TARGET_ROOT/sys
# initalize rpmdb
sudo mkdir -p $TARGET_ROOT/var/lib/rpm
sudo $_RPM --root $TARGET_ROOT --initdb
# this makes sure that running yum/dnf in the chroot it can get
# out to download stuff
sudo mkdir $TARGET_ROOT/etc
sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf
# Bind mount the external yum cache inside the chroot. Same logic
# as in the yum element to provide for yum caching copied here
# because the sequencing is wrong otherwise
sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum
sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum
_install_repos
# install dnf for >= f22
if [ $DIB_RELEASE -ge 22 ]; then
_install_pkg_manager dnf dnf-plugins-core yum
else
# Note this is not usually done for root.d elements (see
# lib/common-functions:mount_proc_dev_sys) but it's important that
# we have things like /dev/urandom around inside the chroot for
# the rpm [pre|post]inst scripts within the packages.
sudo mkdir -p $TARGET_ROOT/proc $TARGET_ROOT/dev $TARGET_ROOT/sys
sudo mount -t proc none $TARGET_ROOT/proc
sudo mount --bind /dev $TARGET_ROOT/dev
sudo mount --bind /dev/pts $TARGET_ROOT/dev/pts
sudo mount -t sysfs none $TARGET_ROOT/sys
# initalize rpmdb
sudo mkdir -p $TARGET_ROOT/var/lib/rpm
sudo $_RPM --root $TARGET_ROOT --initdb
# this makes sure that running yum/dnf in the chroot it can get
# out to download stuff
sudo mkdir $TARGET_ROOT/etc
sudo cp /etc/resolv.conf $TARGET_ROOT/etc/resolv.conf
# Bind mount the external yum cache inside the chroot. Same logic
# as in the yum element to provide for yum caching copied here
# because the sequencing is wrong otherwise
sudo mkdir -p $TMP_MOUNT_PATH/tmp/yum
sudo mount --bind $YUM_CACHE $TMP_MOUNT_PATH/tmp/yum
_install_repos
# install dnf for >= f22
if [ $DIB_RELEASE -ge 22 ]; then
_install_pkg_manager dnf dnf-plugins-core yum
else
_install_pkg_manager yum
fi
# we just installed yum/dnf with "outside" tools (yum/rpm) which
# might have created /var/lib/[yum|rpm] (etc) that are slighlty
# incompatible. Refresh everything with the in-chroot tools
sudo -E chroot $TARGET_ROOT rpm --rebuilddb
sudo -E chroot $TARGET_ROOT ${YUM} clean all
# populate the lang reduction macro in the chroot
echo "%_install_langs C:en_US:en_US.UTF-8" | \
sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null
# bootstrap the environment within the chroot; bring in new
# metadata with an update and install some base packages we need.
sudo -E chroot $TARGET_ROOT ${YUM} -y update
sudo -E chroot $TARGET_ROOT ${YUM} -y \
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
install systemd passwd findutils sudo util-linux-ng
# This package is split out from systemd on >F24, dracut is
# missing the dependency and will fail to make an initrd without
# it; see
# https://bugzilla.redhat.com/show_bug.cgi?id=1398505
if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then
sudo -E chroot $TARGET_ROOT ${YUM} -y \
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
install systemd-udev
fi
# Put in a dummy /etc/resolv.conf over the temporary one we used
# to bootstrap. systemd has a bug/feature [1] that it will assume
# you want systemd-networkd as the network manager and create a
# broken symlink to /run/... if the base image doesn't have one.
# This broken link confuses things like dhclient.
# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204
echo -e "# This file intentionally left blank\n" | \
sudo tee $TARGET_ROOT/etc/resolv.conf
# set the most reliable UTF-8 locale
echo -e 'LANG="en_US.UTF-8"' | \
sudo tee $TARGET_ROOT/etc/locale.conf
# default to UTC
sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \
/etc/localtime
# cleanup
# TODO : move this into a exit trap; and reconsider how
# this integrates with the global exit cleanup path.
sudo umount $TMP_MOUNT_PATH/tmp/yum
sudo umount $TARGET_ROOT/proc
sudo umount $TARGET_ROOT/dev/pts
sudo umount $TARGET_ROOT/dev
sudo umount $TARGET_ROOT/sys
# RPM doesn't know whether files have been changed since install
# At this point though, we know for certain that we have changed no
# config files, so anything marked .rpmnew is just a bug.
for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do
sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//')
done
echo Caching result in $YUMCHROOT_TARBALL
sudo tar --numeric-owner \
-C $TARGET_ROOT \
-zcf $YUMCHROOT_TARBALL --exclude='./tmp/*' .
_install_pkg_manager yum
fi
# we just installed yum/dnf with "outside" tools (yum/rpm) which
# might have created /var/lib/[yum|rpm] (etc) that are slighlty
# incompatible. Refresh everything with the in-chroot tools
sudo -E chroot $TARGET_ROOT rpm --rebuilddb
sudo -E chroot $TARGET_ROOT ${YUM} clean all
# populate the lang reduction macro in the chroot
echo "%_install_langs C:en_US:en_US.UTF-8" | \
sudo tee -a $TARGET_ROOT/etc/rpm/macros.langs > /dev/null
# bootstrap the environment within the chroot; bring in new
# metadata with an update and install some base packages we need.
sudo -E chroot $TARGET_ROOT ${YUM} -y update
sudo -E chroot $TARGET_ROOT ${YUM} -y \
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
install systemd passwd findutils sudo util-linux-ng
# This package is split out from systemd on >F24, dracut is
# missing the dependency and will fail to make an initrd without
# it; see
# https://bugzilla.redhat.com/show_bug.cgi?id=1398505
if [ $DISTRO_NAME = "fedora" -a $DIB_RELEASE -ge 24 ]; then
sudo -E chroot $TARGET_ROOT ${YUM} -y \
--setopt=cachedir=/tmp/yum/$ARCH/$DIB_RELEASE \
install systemd-udev
fi
# Put in a dummy /etc/resolv.conf over the temporary one we used
# to bootstrap. systemd has a bug/feature [1] that it will assume
# you want systemd-networkd as the network manager and create a
# broken symlink to /run/... if the base image doesn't have one.
# This broken link confuses things like dhclient.
# [1] https://bugzilla.redhat.com/show_bug.cgi?id=1197204
echo -e "# This file intentionally left blank\n" | \
sudo tee $TARGET_ROOT/etc/resolv.conf
# set the most reliable UTF-8 locale
echo -e 'LANG="en_US.UTF-8"' | \
sudo tee $TARGET_ROOT/etc/locale.conf
# default to UTC
sudo -E chroot $TARGET_ROOT ln -sf /usr/share/zoneinfo/UTC \
/etc/localtime
# cleanup
# TODO : move this into a exit trap; and reconsider how
# this integrates with the global exit cleanup path.
sudo umount $TMP_MOUNT_PATH/tmp/yum
sudo umount $TARGET_ROOT/proc
sudo umount $TARGET_ROOT/dev/pts
sudo umount $TARGET_ROOT/dev
sudo umount $TARGET_ROOT/sys
# RPM doesn't know whether files have been changed since install
# At this point though, we know for certain that we have changed no
# config files, so anything marked .rpmnew is just a bug.
for newfile in $(sudo find $TARGET_ROOT -type f -name '*rpmnew') ; do
sudo mv $newfile $(echo $newfile | sed 's/.rpmnew$//')
done
sudo rm -f ${TARGET_ROOT}/.extra_settings

View File

@ -0,0 +1,9 @@
---
deprecations:
- The ``DIB_YUMCHROOT_USE_CACHE`` variable has been removed and the
Fedora and CentOS ``-minimal`` initial chroot will always be
created by the package manager. The default creation of a chroot
tarball is stopped for these elements. This unused option was
unsafe; there is no guarantee that the base system will not change
even between runs. Getting the package manager to reuse the cache
for the initial chroot install is future work.