Commit Graph

426 Commits

Author SHA1 Message Date
Jeremy Stanley 16ddb49e48 Drop libvirt-python from suse in bindep fallback
The bindep fallback list includes a libvirt-python package for all
RPM-based distros, but it appears that OpenSuse Leap has recently
dropped this (likely as part of removing Python 2.7 related
packages). Exclude the package on that platform so that the
opensuse-15 job will stop failing.

Change-Id: I0bb7d9b7b34f4f6c392374182538b7e433617e13
2023-09-06 15:15:03 +00:00
Dr. Jens Harbott 3df7459924 Fix unbound setup for debian-bookworm
dns-root-data has been demoted to a "Recommends" dependency of unbound,
which we don't install. Sadly the default unbound configuration is
broken without it.

Change-Id: I93e6928d30db8a90b45329ca00f066b4ec1b4ae7
2023-07-04 09:37:49 +02:00
Dr. Jens Harbott 4c16313ad2 Build debian bookworm images
Release is done, mirror is in place, ready to go.

Adopt using systemd-timesyncd like we do for recent Ubuntu releases.

Change-Id: I3fbdc151177bf2dba81920a4a2e3966f271b50ad
2023-07-03 06:05:36 +00:00
Dr. Jens Harbott 6b1cfbe079 Cache new cirros images
The cirros project has released new images, add them to our cache prior
to actually using them in the CI. We can remove the old images once the
migration is completed and not too many stable branches using the old
images are still active, but comparing the size of these in relation to
the total size of our images, the impact of this shouldn't be too large
in comparison to the benefit in CI stability.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I6d6bcc0e9cfef059de70bbb19e4254e8d29d415b
2023-06-01 16:26:54 +00:00
Jeremy Stanley 5262094f9e Stop caching infrequently-used CirrOS images
According to Ic8b3e790fe332cf68bad7aaa3d5f85229600380b review
comments, OpenSearch indexing indicates jobs aren't often using
CirrOS 0.3.4, 0.3.5, 0.4.0 or 0.5.1 images any longer. If jobs
occasionally used them and have to retrieve them from the Internet
then that's fine, we really only need to cache images which are used
frequently. Remove the rest in order to shrink our node images
somewhat.

Change-Id: Ibada405e0c1183559f428c749d0e54d0a45a2223
2023-02-14 17:25:45 +00:00
yatinkarel 10abfbe573 Cache Cirros 0.6.1 images
0.6.1 is the latest cirros release and with [1][2]
is being used in neutron jobs.

Add these to nodepool images to avoid pulling it
in jobs and hit external connectivity issues.

[1] https://review.opendev.org/c/openstack/neutron/+/869154
[2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/869152

Change-Id: Ic8b3e790fe332cf68bad7aaa3d5f85229600380b
2023-02-09 17:03:31 +05:30
Ian Wienand 75a6a641b1
nodepool: infra-package-needs; cleanup tox installs
The package-maps install of tox is only defined for gentoo, and that
came in with the original image build parts.  We don't need that any
more.

10-pip-packages I didn't trace down, but it hasn't been doing anything
for a long time, since we removed pip-and-virtualenv.  We can remove
that.

The install done in 40-install-tox I can not see being used anywhere.
It came in with If5397d731e9fb04431482529aed23cd9fdaecc1d but I can't
see the venv actually referenced anywhere.  I think this has all been
replaced by the ensure-tox role (or, indeed, jobs migrating away from
tox).  Remove it.

Change-Id: If3fddd79dde56f4087e465ed8b8013f0f337e0cb
2023-02-02 11:46:16 +11:00
Ian Wienand 5a6b14875f
nodepool: infra-package-needs; remove lvm2
This came in via Ie1a0aba57390c9c0b269b4cbb076090ae1de73a9 many years
ago, when it was copied from old puppet.  I can't see that we need to
be installing this for any infra reason.

I guess there is a small posibility things are relying on this, but
they would be better to install it themselves anyway.

Change-Id: I0b8908a79a5dcbe2a5bf5bf72986ea28e17c95fa
2023-02-02 11:24:24 +11:00
Ian Wienand 4437dcd0fd
nodepool: infra-package-needs; cleanup python
We don't need to pull in Python 2 python-xml or python-dev packages
any more

python3 is always installed by DIB (it needs python3 on the image to
run elements).  So we don't explicitly need to pull that in.

Change-Id: I36942435a709c25097cb57d336c45c2884a0103c
2023-02-02 11:24:21 +11:00
Ian Wienand 90fcb99cf6
nodepool: infra-package-needs; drop curl
c.f. I9ccebe2dbf3a8682dab60c2070c5f78849e01446

The RedHat platforms vary if they come pre-installed with curl or
curl-minimal, and if curl-minimal is installed, it causes conflicts
when you try to install "curl" (without removing it first, or using
"swap").

pkg-map is not designed to deal with this at all; it can't say "curl |
curl-minimal".  But all our base images come with curl, because we're
using cache-url which uses it.

So, in short, drop it here to avoid this conflict.

Change-Id: I4e930080f89fe833702f7cafef09642e0638960f
2023-02-02 10:15:25 +11:00
Cédric Jeanneret 916b043710 Ensure NetworkManager doesn't override /etc/resolv.conf
Until now, whenever NetworkManager gets reloaded/restarted/lease
refresh, it would override the /etc/resolv.conf file with the
nameservers and related it gets from the network.

This patch ensures this won't happen ever again.

Note: this is a corrected version of
I92bc12b8f712e28962d24dd6474cfce22b81222c
that was reverted due to indentation + use of ConfigParser issues.

Change-Id: I48560641238911154cc9f353f707a9374613e51a
2022-12-02 15:10:41 +01:00
Jeremy Stanley 7a513f0036 Revert "Ensure NetworkManager doesn't override /etc/resolv.conf"
Bug in the inline Python's indentation levels. Please correct and
resubmit with more testing included where possible.

This reverts commit 368bb77ee3.

Change-Id: I20ed2c48bc223bd13d04c297e877f47c02141e4d
2022-12-02 13:42:42 +00:00
Cédric Jeanneret 368bb77ee3 Ensure NetworkManager doesn't override /etc/resolv.conf
Until now, whenever NetworkManager gets reloaded/restarted/lease
refresh, it would override the /etc/resolv.conf file with the
nameservers and related it gets from the network.

This patch ensures this won't happen ever again.

Change-Id: I92bc12b8f712e28962d24dd6474cfce22b81222c
2022-11-28 09:25:23 +01:00
Clark Boylan a6d4fae070 Tune sshd connections settings on test nodes
Update the sshd_config on our test nodes to accomodate what appears to
be an increase in ssh scanner traffic. In particular LoginGraceTime
defaults to 120 seconds. We reduce that to 30 seconds to cycle
connections more quickly. Then we also increase the maximum number of
connection startups to 30 from the default of 10. We also reduce the
random fail rate from 30% to 10% between 31 and 100 connections.

I'm not entirely certain this will fix things, but based on what we've
seen from logs it may be what we need to make ssh to test nodes more
reliable.

Change-Id: Ifacf7d00de157ab2fb60cde990f0b49f03f71415
2022-08-17 12:40:46 -07:00
Ian Wienand 4fbc3e1b30 infra-package-needs: blank out coreutils for Rocky 9
Rocky 9 has coreutils-single package installed, so trying to install
coreutils package conflicts.  Just blank this out for this platform
like 8.

Change-Id: I48933a61a065cee9402cb803b0da214eafe2cd8a
2022-08-11 13:37:55 +10:00
Ian Wienand c043b9c2b6 nodepool: update package maps for Rocky 9
Update the package maps.  It also seems like matching just "9" will
cover 9-stream and Rocky, which just uses DIB_RELEASE "9".  Also fixup
the 80-enable-haveged to skip on rocky.

Change-Id: Ia352d217d00e10068a463b62f7d9aca72cb88a8c
2022-08-11 10:17:10 +10:00
wangxiyuan e37ee42a41 Update openeEuler pkg mapping
openEuler 22.03 supports python3 only. There is no
python-devel package.

This patch update the pkg mapping to fix the package
install problem in nodepool.

Change-Id: I11750048841ec49c893b4c9332a6029b329b54cb
2022-08-05 09:17:42 +08:00
Ian Wienand efbb9b8961 nodepool elements: fix pip upgrade venv
Change I316e9587b6e290cd421b47f506c91dbebe0975c0 had a rather
embarrasing oversight in that it copied the /usr/bindep-env/bin/pip
invocation for upgrading pip to all the other venv's.

i.e. we were upgrading hte bindep-env pip over and over, and not
actually the pip in the working venv.  The os-testr install on older
platforms has now broken because it still tries to install with the
ancient inbuilt pip -- local testing has confirmed that it works with
the updated pip.

Change-Id: I22c549b5f9b9e3882fcd2340946d2850b0b2f86b
2022-07-06 13:27:58 +10:00
Clark Boylan 3e57097f50 Install setfacl on test nodes
Ansible v5 appears to rely on setfacl more than ansible 2.9 did when
running tasks as a different unprivileged user than the one currently
running ansible. Without setfacl installed we get errors like:

  Failed to set permissions on the temporary files Ansible needs to
  create when becoming an unprivileged user (rc: 1, err: chmod:
  invalid mode: ‘A+user:stack:rx:allow’ Try 'chmod --help' for more
  information.}). For information on working around this, see
  https://docs.ansible.com/ansible-core/2.12/user_guide/become.html#risks-of-becoming-an-unprivileged-user

Installing setfacl makes the error go away as ansible get use setfacl
instead of chown/chmod.

Ubuntu, Debian, Fedora, CentOS, and OpenSUSE all appear to call the
package 'acl'. We assume that openeuler and rocky inherit this package
name. That means we only need to override the package name for Gentoo.

Change-Id: I71736578dbd5e0683b18023e73ab44255eb6eb18
2022-05-24 13:02:25 -07:00
Michael Johnson 4ceb4adc83 Fix selinux context for unbound.log
The previous patch[1] applied a partial context to the unbound.log file.
This patch applies a full context to resolve the "partial context" error, using semanage to make the file label persistent.

[1] https://review.opendev.org/c/openstack/project-config/+/841546

Change-Id: Ic15957fa4ef58355efd2e96f143386f393b0a59d
2022-05-12 19:25:06 +00:00
Ian Wienand 4be8c66630 Set context for unbound.log on selinux systems
I4f3265c16320613d4ba74a02df1361c5d9cf2fb1 moved this file to
/var/lib/unbound on selinux systems, as it was getting permissions
errors trying to write into /var/log.

This turns out to make it harder to collect the logs from projects
like devstack.  It's simpler if we just have the log file in a
consistent place.  On selinux systems, set the context, and revert
things to just writing into /usr/log/unbound.log

Change-Id: I6bb58ef0d6bf4cbbb7fd4066e01b7a01d05009c3
2022-05-12 15:16:08 +10:00
Dr. Jens Harbott a694b41c74 Fix apt-phased-updates
Missing then

Change-Id: Ifb446e44a788583c6ab7b93747927b921932aba6
2022-05-04 17:56:39 +02:00
Dr. Jens Harbott 37a934aacb Always include phased updates for Jammy
Our images builds always install the newest updates,
so we need this in order to avoid conflicts.

Change-Id: I7e14968f2aa7beed88e806ac1eef9370a5388bbb
2022-05-04 15:16:14 +00:00
Dr. Jens Harbott f5b01d71b7 Start bulding ubuntu-jammy images
Jammy was released yesterday, we can start building images for it.

Change-Id: I5fd16df8bf0e3b74711875ad0573be69db12791b
2022-04-24 15:23:28 +02:00
Ian Wienand 39d2332299 infra-package-needs: don't require coreutils for Rocky Linux 8
coreutils comes in two variants now, 'coreutils-single' which is a
busybox-like single binary called through symlinks and the regular
coreutils.  Both satisfy the dependency for coreutils for any other
packages, but if you explicitly ask to install coreutils over
coretuils-single you get an error.

Since coreutils-single is already in the base-image, just skip
installing it on Rocky 8.

Change-Id: I89f8cb49b0cd373e454dd37439bf6efd971233e5
2022-02-21 08:13:11 +11:00
Neil Hanlon cc6b6de7a0
Add Rocky Linux to nodepool elements tooling
Change-Id: I2a94a34b1518c90ef5f0d91c9131482520c38c62
2022-02-15 17:31:34 -05:00
Dr. Jens Harbott eebe545182
Add py36 variant of get-pip.py
The common version of get-pip.py no longer supports python3.6 or older.
Devstack has amended it's use of this file to use a stable version of
the installer for python3.6 [0]. Pre-cache this version in our images,
too, so that it can be consumed in the CI.

[0] https://review.opendev.org/#/q/Iab2c391d5388461fe9e9037cee81884ce8032e72

Change-Id: Ied14dc7188e27e2a4b144d597b696ee85f25721d
2022-02-02 16:42:04 +01:00
wangxiyuan c28cce5d75 Add openEuler disto support for elements
This patch add openEuler disto support for the diskimage-builder
elements in project-config

Change-Id: I7d7804376c4579b550c865a3ad5719fea1ec703b
2021-12-16 09:37:19 +08:00
Ian Wienand d54c2b46a7 cache-devstack: use dstat_graph from opendev
We have incoporated outstanding changes into a new opendev.org hosted
repository.

Change-Id: Ie4d31fa380573eddf6a5c31eeb6403d7c3b00159
2021-12-07 13:07:35 +11:00
Alfredo Moralejo 42fa5a7e73 Fix haveged installation in CentOS7
When adding support for CentOS Stream 9 [1], I made dib to install haveged
in centos8 or centos8s only. This broke centos7 images.

This patch should get haveged installed in all centos releases != 9-stream and fix
centos7 one.

[1] https://review.opendev.org/c/openstack/project-config/+/811442

Change-Id: I5a33160c6272ee4e452b83599ca3ed552422c6d2
2021-11-05 11:06:08 +01:00
Ian Wienand 7df74a82fa infra-package-needs: skip haveged start on 9-stream
This package is not installed (see
I9b88baf422d947d5209d036766a86b09dca0c21a) so we can't enable this
service on 9-stream.

Change-Id: Ie42d73e7cd12c80b076429a643d95778ff5665b8
2021-11-05 17:40:46 +11:00
Ian Wienand 04b3eb3d43 nodepool elements: use yaml.safe_load
The recent pyyaml 6 release, incorporated in the latest
nodepool-builder/dib images, requires a specific loader now.  Use
safe_load as there's nothing crazy going on here.

Change-Id: I26e5e1cf6778cb51872d6a65b2ac683335201bb4
2021-11-05 11:25:17 +11:00
Alfredo Moralejo 4e88e748bb Add support for CentOS Stream 9 in nodepool elements
This patch is adding support for CentOS Stream 9 in elements
infra-package-needs and nodepool-base which are used in nodepool images.

- Remove installation of ntpdate (it has been removed in CS9).
- It skips installation of haveged until it's available in EPEL9.
- It maps package iptables to iptables-service.

Note that this patch has been tested together with dib patch in Depends-On.

Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/811392
Change-Id: I25d9bb7272edc3215840a53e5d79efe5d1fd7210
2021-11-03 12:47:06 +01:00
Ian Wienand 30fd4b4549 dstat graph: update to version with fixes
This updates the on-disk cache version to the same as the default
version in zuul-jobs from I0d5e1b567c364a9e6c7aa0b95de17abffaef0434

Note we have a pull request open on this original project to
incorporate these changes, but the project has been dormat for a long
time.  If we don't have a response in the medium-term future, we can
bring this project into opendev.org git and maintain our enhancments
there.

Change-Id: I26b1a7d0dde12be7c493d44d754b9ea8f72e6e66
2021-10-29 09:36:33 +11:00
Ian Wienand 00d29d829b infra-package-needs: install latest pip
The pip installed in the venv with "python3 -m venv" on Xenial is 8.X
-- this does not understand python_requires metadata on packages and
can thus pull in requirements that won't actually run inside the
virtualenv.

Avoid this by upgrading pip in the venv before installing.

While this is the immediate need; do the same on the other venv's we
create for general sanity.

Change-Id: I316e9587b6e290cd421b47f506c91dbebe0975c0
2021-10-20 09:56:06 +11:00
Ian Wienand c554a6b3b3 infra-package-needs: don't start ntp for Fedora
See I361059c6b62ea240b6fef5a61d254959622199d7 where we modified Fedora
to not install the deprecated ntp package.

Change-Id: I9147f16a4e67b15ac7cc0bc4684ad8390718525f
2021-06-16 14:08:14 +10:00
Ian Wienand 20d69256ff infra-package-needs: stub ntp for Fedora
ntp/ntpdate isn't a package on Fedora any more [1].  Make this like
centos 8 above and install chrony for time services.

[1] https://fedoraproject.org/wiki/Changes/NtpReplacement

Change-Id: I361059c6b62ea240b6fef5a61d254959622199d7
2021-06-15 19:43:31 +10:00
Ian Wienand f7a8f02462 nodepool elements: create suse boot rc directory
As noted inline, the /etc/init.d directory appears to have been
somehow remove/no longer created with a recent update.  I've added
this manually and the image builds, and the rc-local.service still
runs.  Do this for now to unblock other builds.

Change-Id: I0b0b2e38951bad656bcfdb47b6470e033564db59
2021-03-17 13:37:12 +11:00
Lee Yarwood 388d227704 Add Cirros 0.5.2 to cache
Recently released [1] and including a fix [2] required by the
Iad1adbc23b31dd54a96299e7a8a4b622c15eed8d change introducing q35 testing
in the nova-next job this image needs to cached before we update
devstack.

[1] https://github.com/cirros-dev/cirros/releases/tag/0.5.2
[2] https://github.com/cirros-dev/cirros/pull/65

Change-Id: I064f1e4a8ba0bab5b759fb419fa9f9c52e122baa
2021-03-08 08:46:15 +00:00
Lee Yarwood cf2ba9c2e9 Revert "Add custom cirros image with ahci module enabled to cache"
This reverts commit 5ee0780486.

0.5.2 [1] was cut after another colleague asked for a release. I guess their release build issues have been resolved since I asked a few weeks ago. As a result this build is no longer required once we've bumped to 0.5.2.

[1] https://github.com/cirros-dev/cirros/releases/tag/0.5.2

Change-Id: I5332d0e47ad863ca9795a8b0b86b73156621622d
2021-03-08 08:43:53 +00:00
Lee Yarwood 5ee0780486 Add custom cirros image with ahci module enabled to cache
As discussed on the ML [1] the nova-next job is looking to start testing
the q35 machine type. In order to do this *before* the next Cirros
release a custom dev build of the Cirros image has been built with the
ahci module included, as is now required to allow for SATA based config
drives to work.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020823.html

Change-Id: I67912064487598c0e5b4ce3001276f42e0ebcad1
2021-03-03 22:28:05 +00:00
Zuul e707d93f2b Merge "zuul-worker: remove additional install of apt-transport-https" 2021-02-04 18:20:14 +00:00
Zuul fb00c509f8 Merge "add gnupg2 to package map for gentoo" 2021-01-17 14:47:18 +00:00
Matthew Thode a76ab5745b
add gnupg2 to package map for gentoo
Change-Id: Ia0967f8669a6cdd4606f0b10aae3a224b2cf4cee
2021-01-17 02:41:06 -06:00
Matthew Thode d29d9313d7
allow create-repo-list to be run without six
Some distros (gentoo) do not come with six in the base image (stage3),
it is installed later in the image build.  Use the native
urlopen/URLError if the six version is not available.

Change-Id: Id1c52aa17e565c16369645508e43e4f485defa66
2021-01-17 02:38:37 -06:00
Tristan Cacqueray 66cb2317fd infra-package-needs: add support for centos-8-stream
This change fix the centos version check to work with both '8' and '8-stream'
value for DIB_RELEASE. The current test is failing with:

  $ export DIB_RELEASE=8-stream
  $ set -u
  $ [[ $DIB_RELEASE -gt 8 ]]
  bash: stream: unbound variable

The fix is lifted from
https://review.opendev.org/#/c/734083/14/diskimage_builder/elements/simple-init/environment.d/15-simple-init-networkmanager

Change-Id: I23dca12eef1c3cc2aacf6ac50029e2bc9fde72dc
2020-10-02 18:19:07 +00:00
Clark Boylan c6660ecfcb Revert "Pin setuptools<50 in our image venvs"
This reverts commit 6f992efbc5.

Setuptools 50.1.0 has been released which reverts the breaking behavior.
https://review.opendev.org/#/c/749766/ tests that these fixes work in
other venvs that exhibited the same problems. That change looks happy so
I think this revert is ready to go.

Change-Id: I31b62be4f85f40f4d99e463cd961dec0a3542f47
2020-09-03 09:19:52 -07:00
Clark Boylan 6f992efbc5 Pin setuptools<50 in our image venvs
setuptools==50 doesn't work with python older than python3.8. There are
a number of issues [0][1][2] the first of which we are tripping over. Be
conservative here and install older setuptools until these issues on
python3.5, 3.6, and 3.7 are sorted out.

[0] https://github.com/pypa/setuptools/issues/2352
[1] https://github.com/pypa/setuptools/issues/2363
[2] https://github.com/pypa/setuptools/issues/2357

Change-Id: I1ae7251c8a4c214544524871fdfe51e947b638f8
2020-08-31 14:02:24 -07:00
Hervé Beraud 95c01abf38 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: Icc115b2271a6fd84f3952b03406250367babd273
2020-06-02 20:44:50 +02:00
Monty Taylor 0a4e2aab9a Don't install python-dev on focal
Also, install yamllint in the dib env, as it's a requirement
of dib-lint now but is only in test-requirements for dib.

Change-Id: I083bca901ca51438099d1d3bbbd0076ac3d7da07
2020-05-27 10:24:51 -05:00