The ML2 and OVN metadata agents have almost identical
code, as the former was copied to the latter and modified.
Instead, combine all the common parts and just have
each do any driver-specific operations separately.
Change-Id: Iff8bc8de16a8afc7c0195bf301d1b0643e17d7c6
In both neutron-metadata and neutron-ovn-metadata agents we should
ensure that haproxy service spawned for network/router is actually
active before moving on.
This patch adds that check and this is similar to what was already
implemented some time ago for the dnsmasq process spawned by the dhcp
agent.
Related-Bug: #2052787
Change-Id: Ic58640d89952fa03bd1059608ee6c9072fbaabf5
All of these exceptions and constants have been in
neutron-lib for a while, start using them.
While looking for other things to consume found an old
neutron-lib TODO that did not apply anymore, so removed.
TrivialFix
Change-Id: I9431075a50ba05be872db422125e6af6266ccb99
Both metadata agents (OVN and non-OVN) should handle
process exceptions when spawning haproxy processes
such that the agent can continue its operation for
other haproxy processes.
Closes-Bug: #2033305
Change-Id: I6da1b135c83ecfc41ec91e907ebf8500325a7a80
External processes, such as radvd, can refuse to start
and throw an exception such as:
"Unable to convert value in $pidfile"
because the given pidfile has more than one PID in it.
The situation can happen when the neutron node is reset
and the obsolete PID files are not cleaned before neutron
is started.
This commit adds PID file cleanup before external
process start.
Closes-bug: #2033980
Change-Id: Id62bf18067d0b144c3e8825c7603cc1e51dca052
A recent change suppressed the IPv6 DAD failure and
removed the address when multiple DHCP agents were
configured on the same network,
https://review.opendev.org/c/openstack/neutron/+/880957
But it also changed the behavior to not enable IPv4
metadata in this case. Restore the old behavior by
not returning early in the DAD failure case. The callback
that builds the config file was moved until after
the address was bound to make the two steps more obvious.
Related-bug: #1953165
Change-Id: I8436c6c9da9a2533ca27ff7312f5b2c7ea41e94f
Requests handled by the metadata-agents can now be rate-limited by
source-ip. This is done to protect the OpenStack control plane against
VMs querying the metadata endpoint in an overly enthusiastic way.
Co-authored-by: Miguel Lavalle <mlavalle@redhat.com>
Related-Bug: #1989199
Change-Id: I748ccfa8b50496dcbcbe41fd22f84249a4d46b11
IPv4 DAD is non-existent in Linux or its failure is silent, so we
never needed to catch and ignore it. On the other hand IPv6 DAD
failure is explicit, hence comes this change.
This of course leaves the metadata service dead on hosts where
duplicate address detection failed. But if we catch the
DADFailed exception and delete the address, at least other
functions of the dhcp-agent should not be affected.
With this the IPv6 isolated metadata service is not redundant, which
is the best we can do without a redesign.
Also document the promised service level of isolated metadata.
Added additional tests for the metadata driver as well.
Change-Id: I6b544c5528cb22e5e8846fc47dfb8b05f70f975c
Partial-Bug: #1953165
Move common Exception class to one place. Move haproxy same
configuration to one place.
Partially-Implements: blueprint distributed-metadata-datapath
Change-Id: I3a0fc72da4520d6bc7193fb32a1bcf9a5585fbf4
This patch switches over to callback payloads for ROUTER
AFTER_CREATE, AFTER_UPDATE and AFTER_DELETE events.
Change-Id: Ie818ffbb1a291faa80501157b46ff6671d5c26ba
Router_info class and metadata agent's driver are using same
instance of the iptables manager class and it could happend that
sometimes e.g. nat rule which packets send to 169.254.169.254:80
redirects to the port 9697 so haproxy can process them, can be missed as
they will be overwritten by the Router_info class manipulating other
rules in the same 'nat' rules list.
This patch fixed that by adding lock for methods which are changing
rules in iptables_manager's nat table in both router_info and
the metadata agent's driver.
Closes-Bug: #1920778
Change-Id: Ic3a324c0e608c7afc4b15dbc8becd33b75ee78f6
HAProxy supports hard stop [1] via SIGTERM signal. From the
documentation:
"""
... when the SIGTERM signal is sent to the haproxy process,
it immediately quits and all established connections are
closed.
"""
In case the process does not finish, the SIGKILL signal is sent.
The PID file created by the process is deleted.
[1]https://cbonte.github.io/haproxy-dconv/2.0/management.html#4
Closes-Bug: #1910691
Change-Id: Ifa3734e8eb4e52b1a132c3351ecc2e15463298bb
Both files cannot be deleted with the default permissions because
those files are created by the "root" user.
Change-Id: I73dd37b3104fac8d3172f520f71cffd85d040c4b
Closes-Bug: #1907695
We push a v6 host route to make the guest send its metadata requests
in the direction of our router. We redirect it to haproxy which
mangles the headers and sends the request along to metadata-agent.
Apparently the supported list of dhcp options for dhcpv6 is quite
short in dnsmasq (cf. dnsmasq --help dhcp6) - not including anything
like classless-static-route for dhcpv4. So we must rely solely on
radvd to push host routes to the guest.
Metadata access over IPv6 is supposed to work both on dual-stack and
v6-only networks.
The following v6 subnet modes are supposed to work:
--ipv6-ra-mode slaac --ipv6-address-mode slaac
--ipv6-ra-mode dhcpv6-stateless --ipv6-address-mode dhcpv6-stateless
--ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful
Change-Id: I28f2914b1b67659af2db7240eae730ac43daccd2
Partial-Bug: #1460177
Send IPv6 metadata traffic (dst=fe80::a9fe:a9fe) to the metadata-agent.
When running on IPv6 enabled system bind haproxy (i.e. the
metadata-proxy) to 169.254.169.254 and to fe80::a9fe:a9fe also.
We do not introduce new config options. The usual config options
(enable_isolated_metadata, force_metadata, enable_metadata_proxy)
now control the metadata service over both IPv4 and IPv6.
This change series only affects the guests' access to the metadata
service (over tenant networks). They change nothing about how the
metadata-agent talks to Nova's metadata service.
Metadata access over IPv6 is supposed to work both on dual-stack and
v6-only networks.
In order to enable the metadata service on pre-existing isolated
networks during an upgrade, this change makes each dhcp-agent restart
trigger a quick restart of dhcp-agent-controlled metadata-proxies,
so they can pick up their new config making them also bind to
fe80::a9fe:a9fe.
Change-Id: If35f00d1fc9e4ab7e232660362410ce7320c45ba
Partial-Bug: #1460177
If a user specifies a header in their request for metadata,
it could override what the proxy would have inserted on their
behalf. Make sure to remove any headers we don't want, and
override something that might be present in the request.
If the agent somehow gets a request with both headers it will
silently drop it.
Change-Id: Id6c103b7bcebe441c27c6049d349d84ba7fd15a6
Closes-bug: #1865036
This patch adds possibility to configure kill hooks used to kill
external processes, like dnsmasq or keepalived.
Change-Id: I29dfbedfb7167982323dcff1c4554ee780cc48db
Closes-Bug: #1825943
To fix bug 1722584 we inserted a checksum-fill rule for
metadata proxy replies. Recent kernels have disabled
this support for TCP because it was invalid, and
supposedly not doing anything, so let's get ahead of
things and remove the code.
Kernel mailing list discussion is at
https://lore.kernel.org/patchwork/patch/824819/
Partially reverts ed1c3b0217
Change-Id: Ib7cc8f82a91972f17987fb95130edc4069d9423f
Related-bug: #1722584
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
Currently the metadata proxy binds to default 0.0.0.0, which does not
add any advantage (metadata requests are not sent to random IP
addresses), and may allow access to cloud information from
third parties.
This changes the generated configuration to bind to METADATA_DEFAULT_IP
address instead.
This is not enabled in other metadata proxy configuration (in the L3
agent), as this would require net.ipv4.ip_nonlocal_bind everywhere
(currently only enabled for DVR) or transparent mode in haproxy (which
requires net.ipv4.ip_nonlocal_bind anyway)
Changed set_ip_nonlocal_bind_for_namespace() to support setting the
value in both the given and root namespace correctly, since it was
only used from inside the neutron codebase according to codesearch.
Change-Id: I388391cf697dade1a163d15ab568b33134f7b2d9
Co-Authored-By: Andrey Arapov <andrey.arapov@nixaid.com>
Closes-Bug: #1745618
It was added as temporary helper during migration process
and was marked to delete in Queens cycle.
Now we are in Rocky so I think we are fine to remove it
finally.
Change-Id: Iacf592841559d392b59864d507dc89ef028cbf05
This patch switches callbacks over to the payload object style events
[1] for ROUTER and ROUTER_GATEWAY BEFORE_DELETE based notifications. To
do so a DBEventPayload object is used with the publish() method to pass
along the related data.
NeutronLibImpact
[1] https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html#event-payloads
Change-Id: I3ce4475643f4f0afed01f2e9956b3bf84714e6f2
By adding a log-tag line to the haproxy config file that contains
the network or router id, we will be able to differentiate which
proxy is logging what. This should help with debugging.
Change-Id: I5bb57b7682c00645e20cce69847dbb3b72165aa8
Partial-bug: #1744359
Move the iptables metadata marking rule earlier in
router init, that way any stray metadata requests
that arrive before the filter metadata redirect rule is
installed will just be dropped. We do this irregardless
of whether we will be running the metadata proxy.
Partial-bug: #1735724
Change-Id: I8982523dbb94a7c5b8a4db88a196fabc4dd2873f
Sometimes a proxied metadata reply can be dropped by
the hypervisor because of an invalid checksum. Always
fill-in the checksum just like we do for DHCP replies.
Change-Id: I46987da3bf05577ff0a51a490f26cf2be3c3c266
Closes-bug: #1722584
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
Without this commit, the run_as_root parameter is always True when
stopping a process, which leads to the usage of unnecessary sudo such as
in some functional tests, like the keepalived ones.
This commit fixes the aforemetioned problem by taking run_as_root into
account when stopping a process. However, run_as_root will still always
be True if the process is spawned in a netns.
Closes-Bug: #1491581
Change-Id: Ib40e1e3357b9a38e760f4e552bf615cdfd54ee5a
Signed-off-by: Hunt Xu <mhuntxu@gmail.com>
Instead, use is_debug_enabled public function from oslo.log. This
ensures we won't break if library owners for some reason decide to
rename or move the option.
The new function is available since 3.21.0. We depend on >= 3.22.0 so we
should be fine.
TrivialFix
Change-Id: Id00f9ba200173d2b33c1c926a971f0bb1c836706
For a HA router, when it's updated, the l3 agents which are standby
always call the after_router_added method, then duplicate metadata
rules are added to iptables table. Althrough these rules will not be
applied to system because of the _weed_out_duplicates method, they will
grow linearly with router update operations.
Because these metadata rules are added once router is added to the agent
and will not be cleaned until router is removed, calling the add_rule
method in after_router_updated is a waste.
This patch removes adding metadata rules in after_router_updated.
Change-Id: I6650f1071499ed6cabd936bb0fb36b32a4b60bca
Closes-Bug: #1658460
Due to the high memory footprint of current Python ns-metadata-proxy,
it has to be replaced with a lighter process to avoid OOM conditions in
large environments.
This patch spawns haproxy through a process monitor using a pidfile.
This allows tracking the process and respawn it if necessary as it was
done before. Also, it implements an upgrade path which consists of
detecting any running Python instance of ns-metadata-proxy and
replacing them by haproxy. Therefore, upgrades will take place by
simply restarting neutron-l3-agent and neutron-dhcp-agent.
According to /proc/<pid>/smaps, memory footprint goes down from ~50MB
to ~1.5MB.
Also, haproxy is added to bindep in order to ensure that it's installed.
UpgradeImpact
Depends-On: I36a5531cacc21c0d4bb7f20d4bec6da65d04c262
Depends-On: Ia37368a7ff38ea48c683a7bad76f87697e194b04
Closes-Bug: #1524916
Change-Id: I5a75cc582dca48defafb440207d10e2f7b4f218b
This comment should have been cleaned up as part of
Id0517758d06e036a36dc8b8772e41af55d986b4e but we missed it during
review. Now the comment only makes readers wonder how it's relevant to
METADATA_SERVICE_NAME constant that it seems to describe, though it
doesn't.
TrivialFix
Change-Id: Ibdd811f21c6a921e309a172e26c713c964bdffa5
This patch implements the callback handler for router update events;
This checks if the proxy process monitor is active, and if not, starts
the proxy.
This is particularly important if the metadata driver misses to receive
a create notification due to failures, which in turn generates an update
event because of a resync step.
Closes-bug: #1623732
Change-Id: I296a37daff1e5f018ae11eb8661c77ad346b8075
This fixes the iptables rules generated by the L3 agent
(SNAT, DNAT, set-mark and metadata), and the DHCP agent
(checksum-fill) to match the format that will be returned
by iptables-save to prevent excessive extra replacement
work done by the iptables manager.
It also fixes the iptables test that was not passing the
expected arguments (-p PROTO -m PROTO) for block rules.
A simple test was added to the L3 agent to ensure that the
rules have converged during the normal lifecycle tests.
Closes-Bug: #1566007
Change-Id: I5e8e27cdbf0d0448011881614671efe53bb1b6a1
Optimize the execution of router delete in the L3 agent
by not performing operations in the router namespace that
will be removed as the last step of the workflow.
Comparing the slopes of the least squares fit of delete
execution time shows that this optimization decreases that
slope by over half.
Change-Id: Ic425ab1f8c3afe882c9deaa72a814ac0d2053f3b
Signed-off-by: Ryan Moats <rmoats@us.ibm.com>
Closes-Bug: 1512450
This was a todo left over from refactoring. The one place where it
was used was in the metadata driver which can just as easily check
isinstance.
The is_ha property in the HA class only returned True if self.router
was True. The agent sets this on creation and so it will always be
True. Additionally, it didn't really make sense to me anyway given
the way this was being used in the metadata driver.
Change-Id: Idf91ee5f8b72ff9a1740dd60090327d07ee8e16e
Currently iptables rules set on L3 agent with metadata_proxy enabled
mark all packets coming from all interfaces including external interfaces.
This change updates PREROUTING rules from MANGLE table to mark packets
only from internal interfaces.
Change-Id: I01549df7b99be84cd46b6f97a5fd62aec1f43275
Closes-Bug: #1477553
Since a packet can only have one mark, and we will need to mark a
packet for multiple purposes, we need to use a coordinated bitmask for
the two cases of simple marking that we currently do in Neutron
leaving the other bits for address scopes.
DocImpact
Change-Id: Id0517758d06e036a36dc8b8772e41af55d986b4e
Partially-Implements: blueprint address-scopes
It's not always possible/convenient to get namespace name
when need to disable some process (like metadata process for stale
router, see related bug). Since namespace name is not required
for process manager to disable process we can remove this parameter
from disable_isolated_metadata_proxy()
Change-Id: I0e0da01d9640aa9920f41989804fc6f320c1c1eb
Related-Bug: #1455042
Currently metadata proxy cannot run with nobody user/group as metadata
proxy requires to connect to metadata_proxy_socket when queried.
This change allows to run metadata proxy with nobody user/group by
allowing to choose the metadata_proxy_socket mode with the new option
metadata_proxy_socket_mode (4 choices) in order to adapt socket
permissions to metadata proxy user/group.
This change refactors also where options are defined to enable
metadata_proxy_user/group options in the metadata agent.
In practice:
* if metadata_proxy_user is agent effective user or root, then:
* metadata proxy is allowed to use rootwrap (unsecure)
* set metadata_proxy_socket_mode = user (0o644)
* else if metadata_proxy_group is agent effective group, then:
* metadata proxy is not allowed to use rootwrap (secure)
* set metadata_proxy_socket_mode = group (0o664)
* set metadata_proxy_log_watch = false
* else:
* metadata proxy has lowest permissions (securest) but metadata proxy
socket can be opened by everyone
* set metadata_proxy_socket_mode = all (0o666)
* set metadata_proxy_log_watch = false
An alternative is to set metadata_proxy_socket_mode = deduce, in such
case metadata agent uses previous rules to choose the correct mode.
DocImpact
Closes-Bug: #1427228
Change-Id: I235a0cc4f0cbd55ae4ec1570daf2ebbb6a72441d
This commit completes the refactoring of the L3 agent callback mechanism.
The goal here is to also use the neutron/callbacks/ mechanism for L3 agent
notifications, instead of have two mechanisms.
[1] modified the L3 agent to send notifiactions for router create, udpate,
and delete events, using the neutron/callbacks/ mechanism.
[2] modified VPN to use this new mechanism, instead of the L3EventObservers
mechanism. Note:
[3] modified FW repo to no longer depended on the L3EventObserver and
related objects (it doesn't currently use the event notifications).
This commit removes the notifications for the L3EventObservers mechanism,
removed the related modules and tests, and adds in tests to verify that the
new notifications are called for the different events.
Once [1] and [2] are upstreamed, this commit can proceed.
Refs:
[1] https://review.openstack.org/#/c/164466/
[2] https://review.openstack.org/#/c/165226/
[3] https://review.openstack.org/#/c/167275/
Change-Id: I7c4b4ea5f9fb19abb812665cdae5fb70c84fe3ec
Depends-On: If5040a827a6903cc7cb5e59cdb7fb95f61b13d47
Closes-Bug: #1433552
The goal of this refactoring is to reduce duplication by
replacing the L3EventObservers mechanism (a specific
mechanism for L3 agent notifications), with the
CallbacksManager mechanism (a more general mechanism
currently in use), so that there is one method
used.
This is the first part of refactoring the L3 agent so that
it uses the new neutron.callbacks mechanism. To do this,
duplicate calls will be made for notifications related to
the router, only using the new callback mechanism.
This commit does two things. First, it puts in place the
notifiers for the new callback mechanism. Second, it updates
the metatdata proxy agent (which is in the same repo) to
use the new callback mechanism.
Later commits will update other repos from the old to new
callback mechanism, and to then remove the old callback
mechanism, once no longer used.
Change-Id: If134947957fd671aa99a0b2d2b37f7ec65e37766
Partial-Bug: #1433552
Currently metadata proxy cannot run with nobody user/group as
metadata proxy (as other services) uses WatchedFileHandler handler to
log to file which does not support permissions drop (the process must
be able to r/w after permissions drop to "watch" the file).
This change allows to enable/disable log watch in metadata proxies with
the new option metadata_proxy_log_watch. It should be disabled when
metadata_proxy_user/group is not allowed to read/write metadata proxy
log files. Option default value is deduced from metadata_proxy_user:
* True if metadata_proxy_user is agent effective user id/name,
* False otherwise.
When log watch is disabled and logrotate is enabled on metadata proxy
logging files, 'copytruncate' logrotate option must be used otherwise
metadata proxy logs will be lost after the first log rotation.
DocImpact
Change-Id: I40a7bd82a2c60d9198312fdb52e3010c60db3511
Partial-Bug: #1427228
Currently the metadata service can be requested on 169.254.169.254:80
from all interfaces including external interfaces. This change updates
PREROUTING rules to allow request on 169.254.169.254:80 only from
internal interfaces.
Change-Id: I44a9e03992f9e2a7bd4d798ae69d8aa7d75d3078
Closes-Bug: #1187102
Brian Haley