Commit Graph

279 Commits

Author SHA1 Message Date
Julia Kreger f15485b1ac Fix glean docs build
Change-Id: Ic9a5516a7424736ecb8e31862c81866f5d20ac98
2023-11-02 13:29:45 -07:00
Dr. Jens Harbott 528f7216c6 Drop opensuse jobs
diskimage-builder wants to drop them, they are hardly passing anymore,
so drop them from our experimental list.

Change-Id: I9b34e2f10060fad138808708db0227da94246b7e
2022-06-08 10:16:12 +02:00
Ian Wienand 69d724a6f3 redhat-ish platforms: refactor simplification of interface writing
It was pointed out in reviews of
Iac0760ed1dd33c06b81cdd2ea3885279d2aab878 (thanks clarkb) that a
single network is just a case of a multiple network list len()==1,
thus we can simplify this path.

This is a refactor to implement this, removing _write_rh_interfaces()
and combining it into write_redhat_interfaces().  I was probably a bit
deep into it when I wrote a lot of these comments, rework them a bit.

Change-Id: Iede6054d84cffa28e1365501918264a779e5db76
2022-06-02 16:59:36 +10:00
Ian Wienand cbbad59484 tests: stub out fields
Make this look more like testing data

Change-Id: Ia71d646db25cc89865fad9982788d0547ec40ccf
2022-06-01 08:01:26 +10:00
Ian Wienand b685f6cdb1 testing: add vexxhost sample config
This adds a sample setup for vexxhost, which specifies DHCP and IPv6
SLAAC.

Honestly, I think this reveals that Debian/Ubuntu aren't handling this
combo quite right.  They don't seem to be correctly specifying the
ipv6 autoconfig options for the interface.  None of this path has been
touched in a long time, so I am inclined to leave well enough alone.
his at least provides a basis for tweaking in the future.

This has some slight reworking of things in the test generator.  The
sample configs are all using the more modern stable-name "ens3"
instead of "eth0", so we need to parameterize this to some bits of the
testing.  Also makes sure it sets up the ipv6 address as a hostname.

Change-Id: If87bb8f981cb2e7e12918919210d0d58349f17a2
2022-06-01 05:08:27 +10:00
Ian Wienand 0e39e82c6c redhat-ish platforms: handle ipv6_slaac metadata
This wires in support for ipv6_slaac metadata to the RedHat like
platforms.  This is fairly straight-forward, but a full test for an
environment that provides this (vexxhost) will follow-on in
If87bb8f981cb2e7e12918919210d0d58349f17a2.

Change-Id: Ieca0706c0a27485a155cd35ab0200f22309c61bb
2022-05-31 16:42:22 +10:00
Ian Wienand 6bdb47354e redhat-ish platforms: use IPV6_DEFAULTGW
It turns out that prior to NetworkManager commit [1] you can not
specify default routes in the routes[6]-<interface> file.  So
Iac0760ed1dd33c06b81cdd2ea3885279d2aab878 works OK for modern
releases, for maximum compatability we should walk the routes and put
the default ipv6 route in the ifcfg file directly, instead of in the
separate route file.

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=c167e0140babcf1a045cee34ce4938f5087f8fe6&utm_source=anzwix

Change-Id: I41381bb65d980002c4ccfd2eca643ec51529360a
2022-05-31 16:28:06 +10:00
Ian Wienand 0b250ab85e redhat-ish platforms: write out ipv6 configuration
Handle static ipv6 information in configdrive for RedHat like
platforms.

Change-Id: Iac0760ed1dd33c06b81cdd2ea3885279d2aab878
2022-05-31 16:28:06 +10:00
Ian Wienand e40e7c599a tests: symlink same files together
Many of the test fixture output comparision files are the same.
Updating them all when you change something can get quite annoying.

Symlink files that are expected to be the same together.  The general
pattern is to use the most "generic" version -- so centos/fedora/rocky
symlink back to the "redhat" version, "ubuntu" symlinks back to the
"debian" version, etc.

Change-Id: I6d46cc076ed8571521a1edec06d4064c424c59f5
2022-05-31 16:28:06 +10:00
Ian Wienand 588effad3c _network_info: refactor to add ipv4 info at the end
These config file are not ordered, so we don't need to add the ip info
between any particular header/footer section.  Similar to
I09a79359bdc13973f67f0503eee9fa9fd7586ead I think this was mostly done
in I20bffabd333ea290d8712ec2a467f2b2d5678f3a to make the output fit
the extant test fixtures.

This will help make combining ipv6 information easier in the future.

This reorganises the output, but is intended to have no operational
effect.

Change-Id: I3f15ed78f75079e2f7cbc25629a6fff9f150cefa
2022-05-31 16:28:06 +10:00
Ian Wienand 7ef87bd12b _network_info: Clean up TYPE=Ethernet handling
This handling for "TYPE=Ethernet" came in with the suse split in
I20bffabd333ea290d8712ec2a467f2b2d5678f3a.

I think this is a case of over-fitting the test data incorrectly.  If
"TYPE=Ethernet" is not specified, it is assumed (e.g. [1]).  For
whatever reason this was not put in some of the test fixtures, which
have been copied over-and-over.  When the referenced change moved to
more "generic" handling it was written to fit this pattern, but really
we can/should specify the type.

This is a cleanup to help make combining ipv6 information easier in
the future.  This should have no operational effect, since this was
the default anyway.

[1] 6a68008e44/src/core/settings/plugins/ifcfg-rh/nms-ifcfg-rh-reader.c (L6446)

Change-Id: I09a79359bdc13973f67f0503eee9fa9fd7586ead
2022-05-31 16:28:06 +10:00
Ian Wienand cecc4b6658 write_redhat_interfaces: pass multiple networks to output functions
This is a follow-on to I0aac3674fd17ff2bfcf3542e27097e171036c837

Prior to this we are walking the list of networks and setting them up
one-by-one.  As noted in the prior change, this fails when two
networks refer to the same interface (such as configuring ipv4 &
ipv6).

This builds a dict of interface -> [networks] mapping.  For the most
part, interfaces will only have one entry in their networks.  However,
some will have ipv4 & ipv6.

For now, we filter out the ipv6 networks; this will be addressed in a
follow-on.  This means we are effectively keeping the status-quo;
which is to ignore ipv6.  This change is intended to have no
operational changes.

Change-Id: I7eb3a4e33ae1d924a46e85035ac283a6755a74a4
2022-05-31 16:28:06 +10:00
Ian Wienand 035c808c12 write_redhat_interfaces: refactor to walk interfaces first
We currently go through each network an assume that we're writing out
one ifcfg-<interface> file for each entry we see.

This fails when we have an ipv4 and and ipv6 entry for the same
interface.  In this case, we need to combine the details into the same
ifcfg-<interface> config file.

This is a small refactor of write_redhat_interfaces to filter out the
interfaces list before we start writing the config files.  This is the
first step in reworking things so an output function can take multiple
networks to configure.

This is intended to be only a refactor and have no functional change.

Change-Id: I0aac3674fd17ff2bfcf3542e27097e171036c837
2022-05-31 16:28:06 +10:00
Ian Wienand db6d29eb53 Revert "Add option to ignore config drive interfaces info"
This reverts commit 0908c99be9.

The config added and tested here is testing the override where we
ignore the metadata and just write out basic DHCP config.

OVH isn't configured like this and we haven't used this in production
coming up on 3 years since Id97aceb78019b7b71bc231778d7ea7e0f3964e0d.

I'm proposing removing this path to avoid us having to maintain this
as we move forward.

Change-Id: I2407dda8d126f2ef297de01817f8136db901f19d
2022-05-31 16:28:02 +10:00
Clark Boylan f07e8dd438 Handle udev move events in addition to add events
It seems on CentOS 9 (maybe others?) instance boot with an eth0
interface that is renamed to something like ens3. When that renaming
happens it seems to short circuit eth0 udev event handling so glean
never sees an add event for eth0 (which wouldn't be correct anyway since
it has been renamed). Then the ens3 event is a move event and our udev
rule doesn't match that which means we never run glean for ens3 either.

Fix this by handling add|move in the ACTION list of our udev rule for
glean so that we run glean for ens3 when it has been moved.

Change-Id: I5369afb425dcf17e7539a4dd000231830862802b
2022-04-01 08:39:05 -07:00
Clark Boylan 3ced516ec1 Install older voluptuous on py27 due to import error
Voluptuous is a transitive test requirement and latest breaks on py27
due to enum not being importable. Address this by capping Voluptuous on
python27.

Change-Id: I4301557247a447d0c0271474fc21cf9c6a1c5c4a
2022-04-01 08:39:05 -07:00
Ian Wienand 72c04e4608 Remove rebuild-test-output.sh
I'm not really sure what it is doing, but it seems to mess up all the
files.  I think we are better copying and modifying existing fixutres.

Change-Id: Idf58e18976b023366c787796e50b0221c6f01d14
2022-02-23 12:46:10 -08:00
Ian Wienand 3cf6e55a67 Add Rocky Linux support
Imports [1] into vendored distro.py as we can't update past 3.6
release due to it dropping Python 3.5, which we still need for Xenial.

[1] 87cbecda2c

Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/830689
Change-Id: I682ae6dc941a6b69254d3c7d1807ce9195257cf1
2022-02-23 12:45:57 -08:00
Ian Wienand 2bf8e130ed distro: sync to 3.6
This is the latest version that supports Python 3.5, which we need
until we drop Xenial.

Change-Id: I8ded08ac8a408c5521dff6b6c4e8ed174edc6ff3
2022-02-23 14:24:55 +11:00
Ian Wienand a85fd43824 testing: drop centos-8, add 8-stream and 9-stream
Drop centos-8 testing as it is now EOL.  Add 8-stream and 9-stream as
we use them in the OpenDev gate.

Change-Id: I84b6ca8480b58460467f41e18a89ed76f2761e52
2022-02-01 11:03:55 +11:00
Ian Wienand d67766f2bb Remove debian-stable, add focal/bullseye testing
OpenDev infra will soon no longer build Debian Stretch (oldoldstable)
images; remove testing here.

While we are here, do an update and add Ubuntu Focal testing and put a
Debian bullseye test in the gate; both are stable production OpenDev
image.

Recent issues with the wheel mirrors have revealed that we were
relying on pre-built wheels in testing.  Add a bindep.txt so that our
testing binary dependencies are listed.

Change-Id: Ifafeed6777e85efb3f71acc79a5ea9e8864a8cf6
2021-10-21 10:11:29 +11:00
Ian Wienand 18b924f142 Remove Fedora 32 job
Working on updating this job for later Fedora, this allows us to merge
changes to the DIB job while this is in progress.

Change-Id: Ib8b83501f59668571877512730b75f40cf6197cf
Needed-by: https://review.opendev.org/c/openstack/diskimage-builder/+/790365
2021-05-10 13:34:33 +10:00
Ian Wienand dfbb27b886 Cleanup glean.sh variable names
Rename things to be slightly clearer by removing references to
glean.sh from various variable names and subsitutions.

Change-Id: Ib05220cd2de5a59b608c3e94e26ea44a89a77c49
2021-04-28 08:18:02 +10:00
Ian Wienand 3cb334dbbb Run a glean-early service to mount configdrive
Currently for the systemd/udev path, every device activated by udev
runs the "glean.sh" script, which attempts to mount the config drive
and set the ssh keys/hostname.

We should run an early service that mounts the config drive and does
this common setup.  Then each interface activated by udev only needs
to configure it's own network settings by calling the glean tool
directly.

This modifies things to run a glean-early.service, which does the
mounting, etc.  This runs the "glean-early.sh" script, which replaces
the no-longer necessary generic "glean.sh" script (an earlier change
moved legacy users depending on glean to iterate interfaces to use
"glean-legacy.sh").

Each of the udev-activated services is updated to depend on this early
configuration.  These now call "python-glean", which is our small
wrapper to call the glean python tool under the interpreter it was
installed with.

Change-Id: I4b36e99ff8ee10e0b855733d97ec4ee12f941c11
2021-04-28 08:18:02 +10:00
Ian Wienand 2bfa1c440b Create "legacy" script path
This updates all the init scripts that call glean and expect it to
iterate the network interfaces to call a "glean-legacy.sh" script.

This is opposed to the udev path; where glean gets called for each
interface activated by the kernel.  Follow-on changes will modify this
path to do less work for each invocation.

Change-Id: I789890b3d55838f3b70c65e519991fed0eb74e6e
2021-04-28 08:18:02 +10:00
Ian Wienand 0eefc7321e Stop requiring /usr/local/bin links for glean.sh
Since Ic16f134fe34293bb68e7c632dd320f523366320d simple-init has
installed glean into a virtualenv.  Despite this, it still needs to
setup a range of global symlinks (in /usr/local/bin) so glean can
blindly call "glean" or "glean.sh".

This means that glean installation isn't actually stand-alone.  Unless
simple-init sets up these global symlinks the glean install doesn't
work.  This makes it very annoying to try and update the way the
scripts are working, because we have to merge changes into simple-init
as well.

We can make the installation self-consistent by using the install
tool.  The init scripts call glean.sh -- we can find the full path to
glean.sh using pkg_resources and write that into the files.

glean.sh wants to call the python tool "glean".  This is slightly
harder, because at runtime the script doesn't really know how to
invoke this (i.e. in a virtualenv, it should call
/path/to/virtualenv/bin/glean).  To allow for this, write a sibling
file next to "glean.sh" to invoke glean with the interpreter the
install is running under patched in.  This way, glean.sh can call this
wrapper relative to itself and get the right thing.  Add a __main__ to
allow glean to be called like this, and update the glean.sh script.

Change-Id: I1adfecf0d5c53648ee857be502216fd8d6cc5e16
2021-04-28 08:18:02 +10:00
Ian Wienand ae571bfed8 Move to Zuul standard hacking rules
These seem as good as any.  Excluded vendored code.  Fix one small
line error.

Change-Id: I7208758ea131363f305863388d06a08a15304907
2021-04-28 08:18:02 +10:00
Dmitry Tantsur 4557d573bd Update hacking and fix pep8 violations
Using version 2.0.0 which was the last to support Python 2.

Change-Id: Ie7028516f41445791125b2f670a309c40150058b
2021-04-28 08:17:52 +10:00
Ian Wienand 32a7409814 Fix Gentoo "is" comparisons
We should not be using "is" to equate literals.  It must have been
working by accident but breaks in recent Python 3.  Appears to been
like this since initial commit with
Iaa17b6ce2444dd48a3f5fcf8eeb9f46ce6077bf7.

Change-Id: Iae85431e03d2db93376798e10cddb8f43ec67be2
2021-04-28 05:02:17 +10:00
Zuul c2e2a920e8 Merge "Only force DNS handling if there is DNS data" 2021-04-27 12:32:10 +00:00
Zuul 4ffa21de8d Merge "Do not require external mock on Python 3" 2021-04-27 02:40:57 +00:00
Zuul c3d4aa94ed Merge "Fix a typo in a log message" 2021-04-27 02:40:04 +00:00
Zuul ee93fde849 Merge "Allow disabling DHCP fallback" 2021-04-27 01:21:34 +00:00
Dmitry Tantsur 27c511b5f1 Fix a typo in a log message
Change-Id: I39620b2efb32c25744a90db9ebc5d2dd8f804eb1
2021-03-24 13:57:01 +01:00
Dmitry Tantsur 6af0a1bd1f Do not require external mock on Python 3
Change-Id: Ia2c5fedbbf9b273e89b4741e124265b28688949c
2021-03-22 17:39:28 +01:00
Zuul 4ef76474b8 Merge "Override NetworkManager to wait for udev-settle" 2021-03-22 02:29:19 +00:00
Zuul e906085422 Merge "Add Python 3.6 to supported versions" 2021-03-22 02:29:16 +00:00
Zuul 4bfbbba06f Merge "Run glean fewer times in glean.sh" 2021-03-22 02:16:59 +00:00
Dmitry Tantsur ab686d9749 Only force DNS handling if there is DNS data
Currently Glean rewrites resolve.conf and disables DNS handling in
NetworkManager even if there is no DNS information provided, making it
impossible to e.g. build DNS servers into the image. This behavior
also does not play way with the --no-dhcp-fallback flag.

Change-Id: I59cf2ece4e665d569d1db08d1df19b0892c6ba9d
2021-03-19 15:17:47 +01:00
Dmitry Tantsur 2fba753e8d Add Python 3.6 to supported versions
This is the version that CentOS/RHEL 8 and Ubuntu Bionic have.
I've tested Glean on CentOS 8, and it seems to work well.

Change-Id: Ie06a9c11b140728509a69b03366e910702fd0d3d
2021-03-19 13:56:39 +01:00
Dmitry Tantsur a469de6eea Allow disabling DHCP fallback
The way we're using Glean in ironic, we'd rather have it not do anything
is there is no explicit configuration. This option allows it.

Change-Id: I4554ecf50ea03c1f56a4447befd498c5af5849e2
2021-03-19 12:51:50 +01:00
Ian Wienand 169bed877e Override NetworkManager to wait for udev-settle
As described inline, we think that NetworkManager is starting before
glean because the udev events have not been processed and the glean
services have not started.  Since there is nothing blocking it, the
network-pre.target is reached and systemd is free to start
NetworkManager.

Although it is not reccommended to wait on udev-settle because it
could result in long timeouts [1], I don't think we have a choice here
because the fundamental nature of this tool is to override settings
before NetworkManager goes off and does its own thing.

[1] https://www.freedesktop.org/software/systemd/man/systemd-udev-settle.service.html

Co-Authored-By: Dmitry Tantsur <dtantsur@protonmail.com>
Change-Id: I1c9c68d3eb5fbeb42901f2ed95860641cb2d676f
2021-03-19 12:50:48 +01:00
Ian Wienand ac70cfda76 Reduce metadata read/parsing overhead
There's not need to open and parse this file multiple times.  Do it
once and pass the json decoded variable to the various functions.  The
functions are already probing for data and handling it if keys are not
there, so we pass a blank dictionary if there is no meta-data.

Change-Id: I28f5cd72004fcb1cdc5ba0a1255376da99e47140
2021-03-18 10:39:38 +11:00
Clark Boylan b57f0be4a7 Run glean fewer times in glean.sh
Resource constrained environments like those used in testing may not be
happy running multiple python glean processes in succession. This can
take many seconds and cause race conditions.

Avoid this as much as possible by running glean once in glean.sh
depending on whether or not the config drive is present.

Change-Id: I8be30ebb3a3ef48beeeaebc4ff25a125fe4d946a
2021-03-17 10:19:14 -07:00
Zuul 8dc38e4ae4 Merge "Do not fail if routes are missing from a network" 2021-01-07 18:38:31 +00:00
Zuul 47e59ab5cd Merge "add hosts entries and ssh keys only once" 2021-01-06 23:52:53 +00:00
Matthew Thode 4ef3e3b0e7
add hosts entries and ssh keys only once
Some users add a newline to their key, strip the key when checking if
it's already added.

Stop using regex (that didn't work) when checking if we already added a
hosts entry.  Use a simple startswith() instead.

Switch to use MagicMock as it has support for iteration.

Change-Id: Ibeabbf1c4448933cdf274e6667c5a9493f3e5e37
Signed-off-by: Matthew Thode <mthode@mthode.org>
2021-01-06 15:28:43 -06:00
Daniel Pawlik dd0ec95018 Replace Fedora 31 functional tests with Fedora 32
Fedora 31 is EOL and should not be used.

Change-Id: I2b1a7b5d8075bd3db5743a2a743fe58885514ce1
2020-12-21 14:41:14 +01:00
Dmitry Tantsur 75d81fd342 Do not fail if routes are missing from a network
Routes are optional in the schema, let's not treat them as required.

Change-Id: Id3130634fa61ff2e3bd7483ba4b5452dca4de909
2020-11-11 13:49:37 +01:00
Matthew Thode a507b71117
switch glean.sh path in gentoo openrc init
Change-Id: Ibc601616537c652bee82cd3be37250e4741606bf
Signed-off-by: Matthew Thode <mthode@mthode.org>
2020-06-22 12:56:28 -05:00