Commit Graph

213 Commits

Author SHA1 Message Date
Zuul 72513f6bdf Merge "Remove cloud-init when using simple-init" 2023-12-14 19:05:33 +00:00
Charalampos Kominos b197d2c641 feat: Add new fail2ban elemenent
This patch proposes a new element which installs fail2ban on the final
image. More crucially, a custom jail.local is injected during built
time which is a useful feature for cloud admins.

Change-Id: I47b90bbf3809cd6f90148b848b2afe4233be79d7
Signed-off-by: Charalampos Kominos <hkominos@gmail.com>
2023-11-25 20:23:31 +00:00
Julia Kreger 6df7921cb7 Remove cloud-init when using simple-init
When using simple-init, we are making an explicit choice
along the lines of "I want the simple tool to do the simple needful"
which works well, except when cloud-init tries to run because it
is already baked into the source image diskimage-builder started
with.

So what would happen is Glean would execute from simple-init,
and then cloud-init would get launched by default, and cloud-init
in some cases everything is DHCP, so suddenly any static
configuration, such as what might be in an attached configuration
drive, is stomped upon resulting in an unreachable instance if
DHCP is just not available.

If DHCP is available, generally this is not an issue and goes
un-noticed, yet can add a substantial amount of time to the
boot sequence "waiting" for meta-data endpoints which may
not exist.

Change-Id: I380b9638cd28f5771530089c558ef5ab638c0173
2023-11-01 09:41:44 -07:00
Zuul 0576fadab8 Merge "Stop creating default user for cloud-init" 2023-10-12 20:53:57 +00:00
Zuul 87e2321d55 Merge "Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE" 2023-10-12 20:53:55 +00:00
Dmitriy Rabotyagov cff37ce502 Stop creating default user for cloud-init
All relatively modern cloud-inits are capable of creating default user
as well as granting root privileges for them. Currently
cloud-init creates pretty much the same sudoers file.
So running steps under the new DIB_DEBIAN_CLOUD_INIT_HELPER
does not make sense for last couple of Debian releases.

Change-Id: I3cebd318f1f0313bba00ecf639328978d3ad0f32
2023-10-10 16:04:31 +00:00
Maksim Malchuk 4fbf564615 Set grub timeout style
Set the grub timeout style to display the menu. By default it set to
'hidden' but can be changed to 'menu' to display the menu and then
wait for the timeout expire before booting the default entry.

Change-Id: I8c58407ef645d528dd77efe866bfe0389cbbbd33
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2023-09-15 20:31:26 +00:00
Dmitriy Rabotyagov 335f8dc6fd Change default value of DIB_DEBIAN_ALT_INIT_PACKAGE
For quite a while Debian is shipped with systemd-sysv
by default. However, default value of DIB_DEBIAN_ALT_INIT_PACKAGE
is not in sync across elements. We change a default now for
the `debian` element along with removing `apt_get_bp_extra_opts`
that is not defined or used anywhere else.

Change-Id: If5d3f0a21467f926c23bb39a1853be73befa768e
2023-08-31 19:20:29 +02:00
Julia Kreger a692673b90 Deprecate legacy deployment elements
Legacy elements deploy-tgtadm, deploy-targetcli, and deploy-baremetal
have not seen use in ages. Another element seems to date back to this,
deploy-kexec, but appears to see no actual use as the underlying methods
leveraged by these elements were long moved away from. iSCSI based
deployment being the last, and even then it required the
ironic-python-agent.

Change-Id: Ib5b3a7690c35d6859e2e0fdac2326dcd16c051d3
2023-08-29 08:02:32 -07:00
Zuul b8bda7455c Merge "Add nm-dhcp-ib-interfaces element" 2023-07-17 15:15:36 +00:00
Maksim Malchuk 43e47f1912 Extend the checksum files generation procedure
The usage of the DIB_CHECKSUM variable is extended to have an
ability generate the only one checksum file, for example only 'sha256'
(by setting an environment variable DIB_CHECKSUM='sha256'), and to
retain the backward compatibility (DIB_CHECKSUM=1 will generate
both 'sha256' and 'md5' supported at this moment). As an additional
feature we have the simple way to completely deprecate 'md5' later,
and add new methods, for example, 'sha512' etc.

Change-Id: I2dd1c60e3bfd9c823a7382b1390b1d40c52a5c97
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2023-07-08 17:25:23 +00:00
waleedm 33c11e1e9c Add nm-dhcp-ib-interfaces element
Currently, NetworkManager can't automatically create default
connection profiles for InfiniBand interfaces.

So, as a workaround, we are adding nm-dhcp-ib-interfaces element to
install NetworkManager-system-connections-infiniband.nmconnection
to NetworkManager to create a wildcard InfiniBand connection profile.

The content of NetworkManager-system-connections-infiniband.nmconnection
is generated by running this command:
`nmcli --offline connection add type infiniband connection.multi-connect multiple`

Closes-Bug: #2016965
Change-Id: Ic972b90e4df9c4aa36cfe3c8631db3e4533045f4
2023-06-04 05:45:12 +00:00
Zuul f8733f729b Merge "Allow custom console=tty0 argument" 2023-04-20 04:27:09 +00:00
Steve Baker d56dd56881 Allow custom console=tty0 argument
The bootloader element now has variable
DIB_BOOTLOADER_VIRTUAL_TERMINAL to customize or suppress the
console=tty0 kernel argument.

This is proposed to allow console=tty0 to be removed entirely as it is
causing significant performance degredation in DPDK environments.

Change-Id: Iba2ee5b8a6b4acdd236a770550dffd29c784ce11
Related: rhbz#2179366
2023-04-11 17:15:12 +12:00
Harald Jensås dd0ee23989 Update satellite_repo labels + add env var
Red Hat changed the repository names/labels for
Satellite Client repository in Satellite 6.11 and
above, See: https://access.redhat.com/solutions/7004377

This change updates the satellite_repo URL's to use the
new labels.

Also adds environment variable REG_SAT_REPO to allow the
user to override the repository label.

Closes-Bug: #2013451
Change-Id: I6c2a93658213644140caf0e4a8c910b1af22cd1c
2023-03-31 09:59:30 +02:00
Zuul f8b0468e06 Merge "Add a FIPS element" 2023-03-22 21:39:17 +00:00
Michael Johnson 45544d5038 Fix ubuntu-minimal to run autoremove
The "ubuntu" target had a post-install 99-autoremove task that removed
unnecessary dependency packages, but the "ubuntu-minimal" target does not.
This patch moves the 99-autoremove post-install task from the "ubuntu" target
to the "ubuntu-common" target so that both will run an autoremove at the end of
the image build.
For the Octavia amphora image, this saved about 1GB in the image by removing
build only package dependencies.

Closes-Bug: #2012406
Change-Id: I4592e3bd502045fa89203c075d3ea8f632e77177
2023-03-21 15:19:37 +00:00
Julia Kreger c217956079 Add a FIPS element
Adds an element whose purpose is to set the stage
in the resulting image so that a user can generate an
image utilizing DIB which can be used in a FIPS
configuration without doing so with the input image
or after the fact.

Change-Id: Ia8a45584a56f6e06856fc2920c333351935dcd9d
2023-03-21 13:07:02 +00:00
Zuul 9c1ee6dcd8 Merge "Correct boot path to cover FIPS usage cases" 2023-03-21 06:39:00 +00:00
Zuul 950ad3324d Merge "Add swap support" 2023-03-21 06:38:57 +00:00
Julia Kreger 4633da7750 Correct boot path to cover FIPS usage cases
When your booting a Linux system using dracut, i.e. with any
redhat style distribution, dracut's internal code looks to validate
the kernel hmac signature in before proceeding to userspace.

It does this by looking at the /boot/ folder file for the kernel
hmac file.

And it normally does this with the root filesystem. Except if the
kernel is not on the root filesystem and is instead on a /boot
filesystem, this breaks horribly. This is compounded because
DIB enables the operator to restructure the OS image/layout
to fit their needs. In order for this to be navigated, as dracut
is written, we need to pass a "boot=" argument to the kernel.

So now we attempt to purge any prior boot entry in the disk image
content, which is good because any filesystem operations invalidate
it, and then we attempt to identify the boot filesystem, and save a
boot kernel command line parameter so the resulting image can
boot properly if FIPS was enabled in the prior image.

Regex developed with https://sed.js.org utilizing stdin:

VAR="quiet boot=UUID=173c759f-1302-48a3-9d51-a17784c21e03 text"
VAR="quiet boot=PARTUUID=173c759f-1302-48a3-9d51-a17784c21e03"
VAR="quiet boot=PARTUUID=173c759f-1302-48a3-9d51-a17784c21e03 reboot=meow"
VAR="quiet boot=UUID=/dev/sda1 text"
VAR="quiet boot=/dev/sda1"
VAR="quiet boot=/dev/sda1 reboot=meow"
VAR="quiet after_boot=1 reboot=meow boot=/dev/sda1"
VAR="quiet after_boot=1 reboot=meow"

Which resulted in stdout:

VAR="quiet text"
VAR="quiet"
VAR="quiet reboot=meow"
VAR="quiet text"
VAR="quiet"
VAR="quiet reboot=meow"
VAR="quiet after_boot=1 reboot=meow"
VAR="quiet after_boot=1 reboot=meow"

Change-Id: I9034c21e84deda2ba2c0ec0d1d6d6595ed10bed4
2023-03-15 11:25:21 -07:00
Steve Baker 7e38f85724 A new diskimage-builder command for yaml image builds
The `diskimage-builder` command provides a yaml file based interface
to `disk-image-create` and `ramdisk-image-create`. Every argument to
these scripts has a YAML equivalent. The command has the following
features:
- Environment values can be provided from the calling environment as
  well as YAML
- All arguments are validated with jsonschema in the most appropriate
  YAML type
- Schema is self-documenting and printed when running with --help
- Multiple YAML files can be specified and each file can have multiple
  images defined
- Entries with duplicate image names will be merged into a single
  image build, with attributes overwritten, elements appended, and
  environment values updated/overwritten. A missing image name implies
  the same image name as the previous entry.
- --dry-run and --stop-on-failure flags

A simple YAML defintion would resemble:

- imagename: centos-minimal
  checksum: true
  install-type: package
  elements: [centos, vm]
- imagename: ironic-python-agent
  elements:
  - ironic-python-agent-ramdisk
  - extra-hardware

The TripleO project has managed image build options with YAML files
and it has proved useful having git history and a diff friendly
format, specifically for the following situations:
- Managing differences between distros (centos, rhel)
- Managing changes in major distro releases (centos-8, centos-9-stream)
- Managing the python2 to python3 transition, within and across major
  distro releases

Now that the TripleO toolchain is being retired this tool is being
proposed to be used for the image builds of TripleO's successor, as
well as the rest of the community.

Subsequent commits will add documentation and switch some tests to
using `diskimage-builder`.

Change-Id: I95cba3530d1b1c6c52cf547338762e33738f7225
2023-03-13 09:01:49 +13:00
Maksim Malchuk 601dc0387f Add swap support
Adds swap as a valid "filesystem"

Closes-Bug: #1816136
Change-Id: Ie50834a9834815b1dfacafd283f505f3323d35c8
Co-Authored-By: luke.odom <luke.odom@dreamhost.com>
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2023-03-06 14:54:35 +03:00
Ian Wienand 41aa936fa2
tox jobs: pin to correct nodesets; use host networking for containerfile
These must have broken when we switched the base nodes to Jammy.
Update to use compatible versions of distros.

We need to squish another gate-breaking change in here to update the
containerfile "podman build" calls to use "--network host".  We added
this with Ia885237406bf4c7b9d49b349f374558ae746401f and the only
external user I can find is kayobe, which is setting this anyway.

I honestly haven't 100% root-caused what changed to require this; the
last time our containerfile jobs ran and worked has unfortunately been
purged so I can't compare versions to try and pinpoint something;
i.e. this may be a podman bug or feature.  At first I thought it
related to the networking plugin package from the Depends-On (which is
still useful for the right packages) but that didn't help get the
bridge networking working.

Depends-On: https://review.opendev.org/c/zuul/nodepool/+/867590
Change-Id: I23f091654cb212e8bdd908664b262de9bfe98cef
2022-12-16 09:52:46 +11:00
Zuul 9ce3358fde Merge "changed release check logic in lvm element" 2022-09-16 12:30:03 +00:00
Zuul 0c323755bc Merge "added elrepo element" 2022-09-16 09:50:21 +00:00
Rafal Lewandowski 35c0c2c6db added elrepo element
Change-Id: I7bc144afa96f45122857ff634c8c19c1fd759450
2022-09-02 15:09:08 +02:00
Rafal Lewandowski f9287fe2ae changed release check logic in lvm element
Change-Id: I83b3c2dad3b0d6006ae23307ae7a8a83b12806e7
2022-09-02 12:37:15 +02:00
Steve Baker 833c5b8ceb Support LVM thin provisioning
This change extends the block device lvs attributes to allow creating
a volume which represents a thin pool, and to create volumes which are
allocated from this pool.

Change-Id: Ic58f55c36236cc8c6279fbcb708e27dc2982f2d5
2022-08-24 10:34:42 +12:00
Zuul 6745279243 Merge "Upgrade openEuler to 22.03 LTS" 2022-08-02 11:21:59 +00:00
wangxiyuan 934a65bc34 Upgrade openEuler to 22.03 LTS
openEuler 20.03-LTS-SP2 was out of date in May 2022. 22.03 LTS
is the newest LTS version. It was release in March 2022 and
will be maintained for 2 years. This patch upgrades the LTS
version. It'll be used in Devstack, Kolla-ansible and so on
in CI jobs.

This patch also enables the YUM mirror to speed up the package
download.

Change-Id: Iba38570d96374226b924db3aca305f7571643823
2022-08-01 19:22:35 +08:00
Steve Baker d090126c66 Parse block device lvm lvs size attributes
The block device lvm lvs `size` attribute was passed directly to
lvcreate, so using units M, G means base 2. All other block device
size values are parsed with accepted conventions of M, B being base 10
and MiB, GiB being base 2.

lvm lvs `size` attributes are now parsed the same as other size
attributes. This improves consistency and makes it practical to
calculate volume sizes to fill the partition size. This means existing
size values will now create slightly smaller volumes. Previous sizes
can be restored by changing the unit to MiB, GiB, or increasing the
value for a base 10 unit.

The impact on this change should be minimal, the only known uses of lvm
volumes (TripleO, and element block-device-efi-lvm) uses extents
percentage instead of size. The smaller sizes can always be increased
after deployment.

Requested sizes will also be rounded down to align with physical
extents (4MiB). Previously specifying a value which did not align on
4MiB would consume an extra extent which could unexpectedly consume
more than the partition size.

Change-Id: Ia109cc5105071d82cc895d8d9cb85bc47da20a7a
2022-07-06 11:27:42 +12:00
Dr. Jens Harbott 931f5b0a33 Revert "Remove py 3.6 support and update jobs"
This reverts commit fe0e5324d4.

Reason for revert: Python3.6 is still being used on Centos 8 based
platforms.

This is a partial revert, since the py36 job is currently failing, it
will be restored in a follow-up patch.

Change-Id: Idc0373f9a639cd66925543376fb1e2e3398666da
2022-06-09 08:51:01 +02:00
Jens Harbott fe0e5324d4 Remove py 3.6 support and update jobs
Although we're not on the OpenStack release schedule as such, Zed
cycle is dropping 3.6/3.7 support.  This means it seems like as good a
time as any to also update ourselves to this regime.  One important
dependency to think about is nodepool, but that is already >3.8 only
so we will be in sync there.

This also changes dib jobs to run using the zed template and adapts
the bindep file to handle Ubuntu Jammy.

[1] https://governance.openstack.org/tc/reference/runtimes/zed.html

Change-Id: Ibdbcf459608711ac64e7fefb1707f6708d68e750
Co-Authored-By: Jay Faulkner <jay@jvf.cc>
Co-Authored-By: Jens Harbott <frickler@offenerstapel.de>
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
2022-06-03 08:43:37 -07:00
Maksim Malchuk b97dfb8fbd Revert "Fallback to persistent netifs names with systemd"
This reverts commit 8401290976.

We are reverting this because some users may want to use predictable
device names and may not even use Debian. However, after some
investigation we have found a couple of bugs in dhcp-all-interfaces on
Debuntu distros. The parent change corrects those bugs. Additionally new
Linux kernels emit "move" events to udev when interfaces are renamed to
their predictable name. Support this "move" in the dhcp-all-interfaces
udev rules. Making these changes appaers to produce functional images
for Debian users using predictable device names. If predictable device
names are not desired turning them off is straightforward and release
notes are updated to give users the info they need to do that outside of
this element.

Change-Id: I125f1a0c78a103b51bda961528c3e66c345bf604
Co-Authored-By: Clark Boylan <clark.boylan@gmail.com>
Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2022-04-27 16:29:58 +00:00
Zuul 53524dec59 Merge "dhcp-all-interfaces: opt let NetworkManager doit." 2022-02-21 18:55:51 +00:00
Zuul 354417f3e6 Merge "Fallback to persistent netifs names with systemd" 2022-02-14 21:30:06 +00:00
Ian Wienand 79ea63f525 Futher bootloader cleanups
GRUB_OPTS has never been documented as externally available, and is
not used.  Assume it's value to simplify the code.

Move the grub version check separately, as we only support grub2

Remove references to buliding i386 images.  I don't image it works in
any way.

Remove ci.md, which is no longer relevant.

Refactor the test for "building BIOS image on EFI system" consiberably
after these changes.

Change-Id: Ia99687815667c3cf5e82cf21d841d3b1008b8fa9
2022-02-10 15:08:56 +11:00
Riccardo Pittau 8401290976 Fallback to persistent netifs names with systemd
The dhcp-all-interfaces element does not work with the predictable names
scheme, fallback to the persistent names scheme as workaround.

Bug: 1960301

Change-Id: I117964a60615a5b7e9984f52f02cd018d1a48ed0
2022-02-09 10:17:49 +01:00
Zuul b713c7fe6c Merge "Revert "Use rpm -e instead of dnf for cleaning old kernels"" 2022-02-09 07:50:24 +00:00
Zuul 2c159985a3 Merge "General improvements to the ubuntu-minimal docs" 2022-02-09 05:19:32 +00:00
Steve Baker 19ecc16d97 Revert "Use rpm -e instead of dnf for cleaning old kernels"
Using rpm -e to remove old kernels fails when other packages also
depend on the removed kernel.

This change reverts back to using dnf to remove the kernel, but also
sets the config value protect_running_kernel=False to avoid the issue
where the build host kernel version matches the version of the package
being deleted.

reverts commit 1ac31afd62.

Change-Id: Ie58630c23a34f2db34f3934abbd0c1076ab9d835
2022-02-09 03:33:34 +00:00
Eduardo Santos 0f430664a2 Fix openSUSE images and bump them to 15.3
SUSE dropped OpenStack Cloud in 2019 [1], and as a result, some
OpenStack-related repositories were removed from openSUSE Download and
root filesystem images stopped being provided. This change deprecates
Leap releases before 15.3 and employs the extract-image script. It also
moves the extract-image script to the sysprep element, since now it's
also used by openSUSE-related elements.

Additionally, revert the "Remove opensuse related funtests" change [2]
so that the opensuse element is tested again and set the default Leap
release to 15.3.

[1] https://www.zdnet.com/article/suse-drops-openstacks/
[2] https://review.opendev.org/c/openstack/diskimage-builder/+/824002

Change-Id: I73d6323aa65cee69a55e54bc53ed682f096dfc89
2022-01-28 02:18:47 -03:00
Harald Jensås e7c52139aa dhcp-all-interfaces: opt let NetworkManager doit.
NetworkManager is quite capable to do automatic
interface configuration. NetworkManager will by default
try to auto-configure any interface with no configuration.
It will use DHCP for IPv4 and Router Advertisements to
decide how to initialize IPv6.

It will most likely do it just as good, or better than the
dhcp-all-interfaces.sh script.

Since dhcp-all-interfaces clean out all ifcfg files in
60-remove-cloud-image-interfaces it means NetworkManager will
by default attempt auto configuration for all interfaces.

This change add's and environment variable:
  DIB_DHCP_NETWORK_MANAGER_AUTO (default: false)

When DIB_DHCP_NETWORK_MANAGER_AUTO is set to `true` only the
NetworkManager config will be written. The dhcp-all-interfaces
service will not be installed. Hence dhcp-all-interfaces will
not write any config files, allowing NetworkManager to just do
it's thing.

Change-Id: Id6f8d6aaaf52a78175bb6c065ec88274c364834e
2022-01-24 01:45:49 +01:00
Eduardo Santos 5779fa8525 General improvements to the ubuntu-minimal docs
This change:
- adds a note regarding an error when building focal ubuntu-minimal
  images on operating systems with older versions of debootstrap
- adds a reference to where the DIB_RELEASE variable definition can be
  found

Closes-Bug: #1941831
Change-Id: Ibc1e04dba0562c4f4909a8cb8af041d9b8ac45c4
2022-01-21 19:21:00 -03:00
Harald Jensås 9c5f5a56d4 Remove centos 9 and rhel 8 block in grub2 pkg-map
In the grub2 element the grub2-efi-x64-modules package
is missing in the centos 9 section, this cause a failure
because grub2 cannot find the neccecary files when
installing the bootloader on EFI systems.

It seems grub2-efi-x64-modules was not included in release
9, this is likely why the block was added initially without
this package. Since it is now there, the Centos 9 specific
block is no longer needed.

Removing the rhel 8 block as well, as it is identical to the
family "redhat" block i.e it is redundant.

Closes-Bug: #1957169
Change-Id: Ia6b0ecf0cd15fb23c6740543940ee513a8602afe
2022-01-17 23:40:24 +01:00
Zuul a32b969d95 Merge "Bump Ubuntu release to focal" 2022-01-11 11:21:13 +00:00
Ian Wienand 3833c2e59c containerfile: fix tar extraction
Ic68e8c5b839cbc2852326747c68ef89f630f26a3 removed the sudo from the
tar extraction here, meaning that production is failing to create the
chroot.  This is hidden in testing because
DIB_CONTAINERFILE_PODMAN_ROOT is set.  Make the sudo here
unconditional.

Change-Id: I6e36e3fc65981f85fad12ea2cd10780fde9c37da
2021-11-10 11:42:49 +11:00
Zuul 82aa8c516d Merge "Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF" 2021-10-04 06:51:31 +00:00
Steve Baker 296c81b9ca Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF
A custom yum repository can now be configured by defining
`DIB_YUM_REPO_PACKAGE` as a yum available package or a URL to an rpm file.
This package can install repo files with any associated keys and
certificates.

A good example of such a package upstream is rdo-release[1] which
includes multiple repo files, the repo keys, and a root certificate.
This makes these repos impractical to install via DIB_YUM_REPO_CONF.

Downstream, repo packages like this a frequently used to bootstrap
development builds of RHEL with development repos.

[1] https://www.rdoproject.org/repos/rdo-release.rpm

Change-Id: I2832e723998c9bd7635cdf7541a4c20eff6294d2
2021-09-13 09:32:53 +12:00