Commit Graph

144 Commits

Author SHA1 Message Date
Dr. Jens Harbott 3df7459924 Fix unbound setup for debian-bookworm
dns-root-data has been demoted to a "Recommends" dependency of unbound,
which we don't install. Sadly the default unbound configuration is
broken without it.

Change-Id: I93e6928d30db8a90b45329ca00f066b4ec1b4ae7
2023-07-04 09:37:49 +02:00
Cédric Jeanneret 916b043710 Ensure NetworkManager doesn't override /etc/resolv.conf
Until now, whenever NetworkManager gets reloaded/restarted/lease
refresh, it would override the /etc/resolv.conf file with the
nameservers and related it gets from the network.

This patch ensures this won't happen ever again.

Note: this is a corrected version of
I92bc12b8f712e28962d24dd6474cfce22b81222c
that was reverted due to indentation + use of ConfigParser issues.

Change-Id: I48560641238911154cc9f353f707a9374613e51a
2022-12-02 15:10:41 +01:00
Jeremy Stanley 7a513f0036 Revert "Ensure NetworkManager doesn't override /etc/resolv.conf"
Bug in the inline Python's indentation levels. Please correct and
resubmit with more testing included where possible.

This reverts commit 368bb77ee3.

Change-Id: I20ed2c48bc223bd13d04c297e877f47c02141e4d
2022-12-02 13:42:42 +00:00
Cédric Jeanneret 368bb77ee3 Ensure NetworkManager doesn't override /etc/resolv.conf
Until now, whenever NetworkManager gets reloaded/restarted/lease
refresh, it would override the /etc/resolv.conf file with the
nameservers and related it gets from the network.

This patch ensures this won't happen ever again.

Change-Id: I92bc12b8f712e28962d24dd6474cfce22b81222c
2022-11-28 09:25:23 +01:00
Ian Wienand efbb9b8961 nodepool elements: fix pip upgrade venv
Change I316e9587b6e290cd421b47f506c91dbebe0975c0 had a rather
embarrasing oversight in that it copied the /usr/bindep-env/bin/pip
invocation for upgrading pip to all the other venv's.

i.e. we were upgrading hte bindep-env pip over and over, and not
actually the pip in the working venv.  The os-testr install on older
platforms has now broken because it still tries to install with the
ancient inbuilt pip -- local testing has confirmed that it works with
the updated pip.

Change-Id: I22c549b5f9b9e3882fcd2340946d2850b0b2f86b
2022-07-06 13:27:58 +10:00
Michael Johnson 4ceb4adc83 Fix selinux context for unbound.log
The previous patch[1] applied a partial context to the unbound.log file.
This patch applies a full context to resolve the "partial context" error, using semanage to make the file label persistent.

[1] https://review.opendev.org/c/openstack/project-config/+/841546

Change-Id: Ic15957fa4ef58355efd2e96f143386f393b0a59d
2022-05-12 19:25:06 +00:00
Ian Wienand 4be8c66630 Set context for unbound.log on selinux systems
I4f3265c16320613d4ba74a02df1361c5d9cf2fb1 moved this file to
/var/lib/unbound on selinux systems, as it was getting permissions
errors trying to write into /var/log.

This turns out to make it harder to collect the logs from projects
like devstack.  It's simpler if we just have the log file in a
consistent place.  On selinux systems, set the context, and revert
things to just writing into /usr/log/unbound.log

Change-Id: I6bb58ef0d6bf4cbbb7fd4066e01b7a01d05009c3
2022-05-12 15:16:08 +10:00
Dr. Jens Harbott a694b41c74 Fix apt-phased-updates
Missing then

Change-Id: Ifb446e44a788583c6ab7b93747927b921932aba6
2022-05-04 17:56:39 +02:00
Dr. Jens Harbott 37a934aacb Always include phased updates for Jammy
Our images builds always install the newest updates,
so we need this in order to avoid conflicts.

Change-Id: I7e14968f2aa7beed88e806ac1eef9370a5388bbb
2022-05-04 15:16:14 +00:00
Neil Hanlon cc6b6de7a0
Add Rocky Linux to nodepool elements tooling
Change-Id: I2a94a34b1518c90ef5f0d91c9131482520c38c62
2022-02-15 17:31:34 -05:00
wangxiyuan c28cce5d75 Add openEuler disto support for elements
This patch add openEuler disto support for the diskimage-builder
elements in project-config

Change-Id: I7d7804376c4579b550c865a3ad5719fea1ec703b
2021-12-16 09:37:19 +08:00
Ian Wienand 00d29d829b infra-package-needs: install latest pip
The pip installed in the venv with "python3 -m venv" on Xenial is 8.X
-- this does not understand python_requires metadata on packages and
can thus pull in requirements that won't actually run inside the
virtualenv.

Avoid this by upgrading pip in the venv before installing.

While this is the immediate need; do the same on the other venv's we
create for general sanity.

Change-Id: I316e9587b6e290cd421b47f506c91dbebe0975c0
2021-10-20 09:56:06 +11:00
Ian Wienand f7a8f02462 nodepool elements: create suse boot rc directory
As noted inline, the /etc/init.d directory appears to have been
somehow remove/no longer created with a recent update.  I've added
this manually and the image builds, and the rc-local.service still
runs.  Do this for now to unblock other builds.

Change-Id: I0b0b2e38951bad656bcfdb47b6470e033564db59
2021-03-17 13:37:12 +11:00
Clark Boylan c6660ecfcb Revert "Pin setuptools<50 in our image venvs"
This reverts commit 6f992efbc5.

Setuptools 50.1.0 has been released which reverts the breaking behavior.
https://review.opendev.org/#/c/749766/ tests that these fixes work in
other venvs that exhibited the same problems. That change looks happy so
I think this revert is ready to go.

Change-Id: I31b62be4f85f40f4d99e463cd961dec0a3542f47
2020-09-03 09:19:52 -07:00
Clark Boylan 6f992efbc5 Pin setuptools<50 in our image venvs
setuptools==50 doesn't work with python older than python3.8. There are
a number of issues [0][1][2] the first of which we are tripping over. Be
conservative here and install older setuptools until these issues on
python3.5, 3.6, and 3.7 are sorted out.

[0] https://github.com/pypa/setuptools/issues/2352
[1] https://github.com/pypa/setuptools/issues/2363
[2] https://github.com/pypa/setuptools/issues/2357

Change-Id: I1ae7251c8a4c214544524871fdfe51e947b638f8
2020-08-31 14:02:24 -07:00
Ian Wienand 7602343b6a nodepool-elements: Use venv for utilities
Since all platforms have Python 3, use the new ensure-venv element
from the dependent change to install bindep and os-testr.

Since we are no longer using pip to install anything during the
builder, this drops the dependency on pip-and-virtualenv from
nodepool-base.  Avoiding this element is our long-term goal, as it's
modification to system state are problematic in a number of ways.  To
maintain the status-quo, the pip-and-virtualenv element is added
explicitly to each build's element list, with a note on it's future.

The current plan for backwards compatability is to replicate the
environment pip-and-virtualenv provides in a base role/job that can be
optionally included.  To test this, provide a new node type
"ubuntu-bionic-plain" that will not include the pip-and-virtualenv
element.  This is put on just one provider (rax) to minimise impact.

The dependent-change (and a dib release) is required before merge so
the ensure-venv element is available.

Depends-On: https://review.opendev.org/707513
Change-Id: I85438baf5bb31790a56fe5b38327361f0a2398e9
2020-03-11 10:12:00 +11:00
Ian Wienand 67cbecdfb4 Restore blank nodepool/scripts
I was a little too hasty in I76acbd08acda20c88ff9fd9148e3918b78d6c6c6
removing the scripts/ directory.  It has broken the puppeting of old
hosts.

Restore the directory with a blank file explaining the situation.

Also, we don't need to copy this in the nodepool elements; remove that.

Change-Id: I8b82950237ef69c4941866900cac9bda42f58ca2
2020-03-03 15:01:23 +11:00
Alfredo Moralejo e17d562000 Revert "Use virtualenv --seeder=pip so that libs are accessible"
--seeder=pip is breaking images where pip is not installed from source.

New virtualenv upstream release 20.0.2 has fixed the issue seen when
using sudo [1] by copying instead of symlinking and does not need this
fix anymore.

[1] f4fd6a0991

This reverts commit be9530ae16.

Change-Id: I799982d9c023141cf612901084d4ecbd4447e969
2020-02-12 09:42:54 +00:00
Clark Boylan be9530ae16 Use virtualenv --seeder=pip so that libs are accessible
New virtualenv will by default use a common location for seed libs like
setuptools, wheel, and pip. Unfortunately this breaks if root installs
the virtualenv and other users are expected to use it because these
other users cannot access /root/.local (where the files are stashed).

We fix this by using --seeder=pip which will install all of those libs
into the virtualenv itself.

Change-Id: I4922ea50e31dceda96f545a0d409c0d7dc022e19
2020-02-11 08:41:49 -08:00
Andreas Jaeger 398adb791f Bye, Bye, Trusty
This removes trusty from the repo and thus from OpenDev.

Afterwards the AFS volume mirror.wheel.trustyx64 can be deleted.

Depends-On: https://review.opendev.org/702771
Depends-On: https://review.opendev.org/702818
Change-Id: I3fa4c26b0c8aeacf1af76f9046ea98edb2fcdbd0
2020-01-19 16:00:55 +01:00
Monty Taylor 66b12be818 Update nodepool element for disk cache update
Things are in /opt/git/opendev.org/ now

Change-Id: Ib82552520b99ffcf213962e917315b0b82c3f1b0
2019-10-19 05:35:46 +09:00
Ian Wienand c5f807e699 nodepool/elements : use abstracted commands
On platforms such as CentOS 8 which are python3 first, "virtualenv"
and "pip" may not exist (removed to avoid any confusion over them
being v2 or v3 commands).

The referenced dib change exports new variables that should be correct
on all platforms for creating virtualenv's and pip installs.

(note will require DIB release to be active on builders)

Depends-On: https://review.opendev.org/684462
Change-Id: I3414fb9e503f94ff744b560eff9ec0f4afdbb50e
2019-10-09 03:07:08 +00:00
Colleen Murphy 587b85ba36 Use iptables for openSUSE
In openSUSE Tumbleweed, the SuSEfirewall2 package was removed in favor
of firewalld[1]. This commit updates the openSUSE nodeset to use plain
iptables rather than injecting iptables rules into the SuSEfirewall2
service. This will work on both Tumbleweed and Leap nodesets.

openSUSE provides no iptables-service package the way the RHEL family
does, so we can't fall back to that. Rather than try to convert iptables
rules to firewalld syntax, this change leverages init.d to ensure
iptables rules are loaded at boot. The 89-unbound script has been
coopted for this purpose since it already creates
/etc/init.d/boot.local. Switched from `dd` to `cat` which makes
conditionally composing the file more natural.

[1] https://lists.opensuse.org/opensuse-factory/2019-01/msg00490.html

Change-Id: Ia2b72e25078efa68019f1bf7c7a0b77e6ff702fd
2019-09-20 12:05:09 -07:00
Jeremy Stanley 0fffcef14b Switch from 1.1.1.1 to 1.0.0.1
CloudFlare's public recursive DNS resolvers are available at
multiple anycast addresses. For some reason 1.1.1.1 is unreachable
from parts of OVH's BHS1 region, but 1.0.0.1 seems to be
consistently reachable. Swap this for improved reliability.

Change-Id: I9a264282ea6c8239883d252f52e004deebca3edc
2019-04-25 13:14:53 +00:00
Andreas Jaeger 14eb58501b Fix nodepool after OpenDev changes
Fix a couple of places where orgs where renamed so that dib finds the
repos.

Change-Id: Ic52dcffcd9323c12410e05b1a3598911a28cbcaa
2019-04-21 17:05:34 +00:00
Andreas Jaeger f43766da84 Update nodepool for OpenDev
Use new locations following OpenDev changes.

Node: This changes the on-disk repo path to /opt/git/openstack/devstack

Change-Id: I7042913fefa64dcec4044779dbeb13f86daea858
2019-04-20 13:40:41 +00:00
Zuul 926b1e7948 Merge "Cleanup ozj confingure-unbound usage" 2019-02-11 18:26:14 +00:00
Clark Boylan 947c2fd409 Cleanup ozj confingure-unbound usage
This role now lives in opendev/base-jobs but we still have some usage
from ozj. We can clean that up by removing what appears to be the only
job doing that since it was tempoarary anyway.

Additioanlly point the nodepool element comments at the right role in
opendev/base-jobs

Change-Id: I1d73f543006d94a52fa1cfe38870391da959ae74
2019-02-08 12:27:22 -08:00
Clark Boylan 82e14dee9c Stop using OpenDNS
Ianw noticed problems on fedora29 with unbound. That resulted in a bug
filed upstream,
https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=4226. In this bug
the helpful unbound maintainers point out that OpenDNS servers are
having trouble with RRSIG records which leads to not validating dnssec
which we require in our unbound config.

Address this by switching to CloudFlare DNS which is suppsoed to be
super localized (aka responsive), and not record queries against it.
Also if we want to we can update our config to do dns over tls against
these servers.

Change-Id: I08ef6a6fba2706803d2e9de6197e0ef8d695e313
2019-02-08 09:32:38 -08:00
Matthew Thode 77535ddcec
allow local.d to preexist
Change-Id: If045a9188c365efb96db6e53e6dfe9083965ad1c
2019-01-23 00:50:27 -06:00
Ian Wienand 6565b3c140 elements/ndoepool-base: only initially populate ipv4 nameservers
We are seeing a problem on Fedora where it appears on hosts without
configured ipv6 unbound chooses to send queries via the ipv6
forwarders and then returns DNS failures.

An upstream issue has been filed [1], but it remains unclear exactly
why this happens on Fedora but not other platforms.

However, having ipv6 forwarders is not always correct.  Not all our
platforms have glean support for ipv6 configuration, nor do all our
providers provide ipv6 transit.

Therefore, ipv4 is the lowest common denominator across all platforms.
Even those who are "ipv6 only" still provide ipv4 via NAT --
originally it was the unreliability of this NAT transit that lead to
unbound being used in the first place.  It should be noted that in
most all jobs, the configure-unbound role [2] called from the base-job
will re-write the forwarding information and configure ipv4/6
correctly during the base job depending on the node & provider
support.  Thus this only really affects some of the
openstack-zuul-jobs/system-config integration jobs, where we start out
without unbound configured because we're actually *testing* the
unbound configuration role.

An additional complication is that we want to keep backwards
compatability and populate the settings if
NODEPOOL_STATIC_NAMESERVER_V6 is explicitly set -- this is sometimes
required if you building infra-style images and are within a corporate
network that disallows outbound DNS queries for example.

Thus by default only populate ipv4 forwarders, unless explicitly asked
to add ipv6 with the new variable or the static v6 nameservers are
explicitly specified.

[1] https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=4188
[2] http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/configure-unbound

Change-Id: If060455e163266b2c3e72b4a2ac2838a61859496
2018-09-27 14:27:13 +10:00
Clark Boylan 4b72a48a32 Disable unbound chroot on test node images
On centos this seems to be part of the built in config, on ubuntu this
seems to not be configured but is unbounds default behavior there, on
gentoo it chroots per the default described in the manpage.

In order to force consistent behavior across all systems disable the
chroot when we configure our logfile path (so that the logfile's dir is
present and writeable) by setting chroot: "" across the board.

This should fix the unbound service on gentoo and be a noop for our
other platforms.

Change-Id: Ic1b66c1982c14759e8fd8370452df21d2b0d9510
2018-09-17 16:34:42 -07:00
Matthew Thode 34725b45a6
include conf files for unbound via glob
Change-Id: I27af4b501add096d9983932b95fcb3c8557155d4
2018-09-17 13:27:43 -05:00
Matthew Thode 997486eff6
remove sudo from pip and virtualenv calls
Change-Id: I2f8ea4260c82d8623c74b8ac6355d9eb5f6698e9
2018-09-10 18:11:19 -05:00
Zuul 78b163fb14 Merge "Ensure rhel7 image could be build" 2018-05-24 21:23:40 +00:00
Nicolas Hicher e0f471889c Ensure rhel7 image could be build
Add rhel7 distrib in unbound and iptables script for nodepool-base, and
add haveged installation for infra-package-needs to ensure users could
use these elements to build rhel based image.

Change-Id: Ib0ad877369bafab64a1fd25cc331363d771d5753
2018-05-22 14:14:50 -04:00
Matthew Thode 228ddbf808
Add missing directory for gentoo rclocal
Change-Id: Ia4eb7a47313856fe3ef7c7691f9ecf20b1ba81c4
2018-05-16 15:31:59 -05:00
Matthew Thode 79180bc18f
enable the right iptables services on gentoo
Change-Id: I7e1008aa48a8e42ffdb422e5f26aa7f2aed57cb6
2018-05-14 11:04:56 -05:00
Ian Wienand 74fe242f26 Remove apply of iptables rules for gentoo
This is applying the iptables rules during the build on the builder.
This will need to be converted into a script to run on boot or
similar.

Change-Id: Icc753013f218c3e4f04031c2fdbc7b32a534d887
2018-04-16 07:22:34 +10:00
Zuul 0332c33dd1 Merge "Revert "Stop systemd-resolve from booting on our images"" 2018-04-10 18:52:32 +00:00
Zuul 1ffbce58c7 Merge "Ensure /etc/resolv.conf is a regular file in final image" 2018-04-09 16:36:38 +00:00
Clark Boylan d655ecc488 Ensure /etc/resolv.conf is a regular file in final image
As described in the inline comment, this ensures that on Bionic (and
any similar platform using systemd-resolved) we use a regular
/etc/resolv.conf file for nameserver configuration.

Depends-On: https://review.openstack.org/#/c/558362/
Change-Id: I1e1d285787f88370fec6e9e21701164cc232e153
2018-04-09 17:31:45 +10:00
Ian Wienand b403efdbb9 Revert "Stop systemd-resolve from booting on our images"
This reverts commit da15523595.

systemd-resolved was not really involed in this.  What seems to be
happening is that during the initial chroot creation there is no
/etc/resolv.conf file, and so the systemd postinst script creates one
pointing to the compatability files created by systemd-resolved.

This is not what we want, but dib doesn't really provide a way for us
to overwrite the file.  That is covered in
Ie0e97d8072e2b21a54b053fa6fb07b62960c686d

We actually want systemd-resolved running -- it provides the dbus
nameserver api stuff that some tools may use.  If /etc/resolv.conf is
a file, systemd-resolved leaves it alone and uses it for resolution as
you would expect.  (we do, however, want to wait for the depends-on to
ensure the image has a correct /etc/resolv.conf before merging this).

Depends-On: https://review.openstack.org/557842/

Change-Id: Ie3cdc323bf8fb4fcee725f9a52bf53a4a6a01bbf
2018-04-03 15:32:27 +10:00
Paul Belanger 26ec98b7dc
Use AFS mirrors to build ubuntu-bionic DIB
Like we do with ubuntu-trusty / ubuntu-xenial, pin our DIB builds for
ubuntu-bionic to our AFS mirrors. This will prevent issues with bionic
installing newer packages when we have mirroring issues with reprepro.

Change-Id: I036f90d25dfc6c244e1bd010a9f2db7e295a439f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-02 12:45:24 -04:00
Paul Belanger da15523595
Stop systemd-resolve from booting on our images
It seems with ubuntu-bionic, systemd-resolve boots by default. This is
then overwriting our /etc/resolv.conf file with its nameserver
address.

Change-Id: I3de217505cc0ba18a233e8712be49c5347a8e29b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-03-29 12:18:11 -04:00
Paul Belanger c919e1e782 Remove jenkins-slave element from DIB images
There is no jenkins, only zuul! With zuulv3 running in production,
there is no more need to add jenkins user or scripts to our images.

Move chmod of /opt/cache/files from
nodepool/elements/jenkins-slave/install.d/20-jenkins-slave to
nodepool/elements/zuul-worker/install.d/60-zuul-worker

We also leave the jenkins-slave elements for now, in case 3rd party CI
use them.

Change-Id: Ia9750877fbc1a17ec467ca4ac685afdb9c1627f8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-03-05 14:29:26 +01:00
Andreas Jaeger 91400af8ba Good bye jenkins/scripts
We now only have afs-slug.sh in jenkins/scripts/ and it is only
used by nodepool/scripts/configure_mirror.sh. Move the file into the
nodepool/scripts directory and adjust paths.

Remove copying of jenkins/scripts from nodepool, the directory does not
exist anymore.

Remove 89-jenkins-scripts, we don't need it anymore.

Depends-On: https://review.openstack.org/548393
Change-Id: I13cac791d70a2e5bcb0de1ba0adcd79e1b0aee9f
2018-03-05 14:28:42 +01:00
Dirk Mueller b0b73ea971 Stop using connection tracking for ssh connections
There is an issue with newer kernels (it seems to happen with 4.15.x)
that when conntrack is reloaded while a connection has packets in
flight, this connection going forward is neither considered INVALID
nor RELATED nor ESTABLISHED by the stateful tracking. While this is
certainly a bug somewhere in the kernel, we can be easily avoiding
this by just not using stateful filtering for ssh connections, as
we accept any connection from anywhere anyway.

Change-Id: I1b20644ce888930cd28d6eaf2c23787315e8199c
2018-03-01 20:09:50 +01:00
Andreas Jaeger 2c8541a21c Move jenkins/data/bindep-fallback.txt (2/2)
Remove using old location, we can switch now to the new location
everywhere.

Note: We can only merge this after new images are build with the new
location build in.

This needs a recheck once change 1 is in since project-config is
not self-testing. So, we can remove jenkins/data/bindep-fallback.txt in
this change.

Change-Id: I08f23e96defb152946ade1490628f37f0215300d
Depends-On: https://review.openstack.org/543141
Depends-On: https://review.openstack.org/543142
2018-02-20 20:52:47 +01:00
Andreas Jaeger 9863abbc72 Move jenkins/data/bindep-fallback.txt (1/2)
Let's get rid of jenkins in the name and use a new place to place
bindep-fallback.txt in.

Create new nodepool element to copy the file over.

This is a first of two changes for this repo, we also need to update
the few users in bindep and openstack-zuul-jobs before we can remove the
old location.

We can remove jenkins/data/bindep-fallback.txt only in the followup
change since project-config is not self-testing.

Change-Id: I5c94dcbbca25db7dd998bfbccfbb25c1c4342a2b
2018-02-20 20:52:17 +01:00