Currently, agents in DVR mode requesting a router update fetch all the
FIPs on a network from the DB rather than just the FIPs that are
relevant to the specific host requesting the update.
While not noticable in smaller networks with a limited number of
floating IPs, this can add significant overhead in larger networks
with many FIPs and hosts.
That overhead comes from Python mapping the responses from the DB into
objects, making extra DB calls per FIP returned and adding additional
iterations to the loop in _get_dvr_sync_data. These objects are mostly
discarded later on and not updated nor included in the RPC response.
This change ensures that we only fetch FIPs from the DB that are bound
to the host requesting the update or those which are in a pre-live
migration state (as they may be migrated to the host in question).
Closes-Bug: #2028185
Change-Id: I199b0b1456aa15dadcc24cafc89db1072d224efd
This patch imitates the ML2/OVS Trunk driver behaviour. When the
trunk parent port is bound:
* A new trunk cannot be created using this parent port.
* If the port is assigned as parent port of a trunk, this
trunk cannot be deleted.
Closes-Bug: #2022059
Change-Id: I8cfa7e67524a42224cbb4b3c3cec3cfa49b795fd
The table "router_extra_attributes" is a child of "router" table.
Each register contains extra information that completes the router
description. When using ML2/OVS mechanism driver, the methods that
create and populate the "router_extra_attributes" register are always
called from the L3 DVR, L3 HA and availability zones extensions.
When using ML2/OVN, those extensions are not loaded and therefore the
"router_extra_attributes" register is not created.
Despite this register is currently not used in ML2/OVN (it will be in
future features), there are some project expecting the
"router_extra_attributes" register to be always created (for example,
neutron-dynamic-routing [1]).
This patch enforces the child register creating always when a router is
created. This register is populated with the default values. This new
register does not affect any current operation related to ML2/OVN nor
ML2/OVS.
There is a 1:1 relationship between "routers" and
"router_extra_attributes". The child register is deleted by the database
engine when the "routers" register is deleted (ondelete="CASCADE").
[1]https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/863713
Closes-Bug: #1995974
Change-Id: Ic546e40513402fa101c9687acce382cd6b84356c
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/db.
Trivialfix
Change-Id: I9311cfe5efc51552008072d84aa238e5d0c9de60
The goal of this series of patches is to make the Neutron code
compliant with SQLAlchemy 2.0.
This patch adds the missing database contexts in the execution of
some queries in the L3 code, reported in the following bugs. This
patch also refactors the ``neutron.db.l3_db`` methods, using the
method decorators instead of the inline context builders.
Closes-Bug: #1974144
Closes-Bug: #1974142
Related-Bug: #1964575
Change-Id: I7e18db7f1fa6b8878c13df09895c993704401815
Change Ia462ca4b340cd3d9a27341632b24926c3290a4b2 modified some
notification payload field names.
Because of that _create_dvr_floating_gw_port fails with missing
'fixed_port_id' field since it's now 'port_id'.
Closes-Bug: #1943846
Change-Id: I37d8722c74dfc122030175f54404703780c4d8b2
Change FloatingIP AFTER_ notifications in l3_db to use publish calls.
Move association_event field into Payload metadata.
Closes-Bug: #1933502
Change-Id: Ie4c0f4a63a87c32026c49b03068e5f461deb38b6
This patch switches over to callback payloads for ROUTER
AFTER_CREATE, AFTER_UPDATE and AFTER_DELETE events.
Change-Id: Ie818ffbb1a291faa80501157b46ff6671d5c26ba
This patch switches over to callback payloads for ROUTER
BEFORE_CREATE, PRECOMMIT_CREATE, BEFORE_UPDATE and
PRECOMMIT_DELETE events.
Change-Id: I4a52c773d3f753c918df0986f1d261083156651c
Floating IP agent gateway ports are created for each external network
for each node where DVR L3 agent is running and where there is some FIP
from the ext_net.
But even, if L3 agent is removed (e.g. when scaling down the cluster),
such floating IP gateway port is never removed so it consumes IP address
from the external network.
With this patch when the DVR L3 agent is deleted, all such fip gateway
ports owned by that agent will be deleted.
When new L3 agent is created (registered in the DB), Neutron will check
if there are any floating IPs on that host and will recreate such FIP
gateway ports for it.
Closes-Bug: #1891360
Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
This reverts commit 062336e59b.
Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.
Related-Bug: #1920001
Conflicts:
neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
neutron/common/utils.py
neutron/db/address_group_db.py
neutron/services/segments/db.py
Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
This patch switches the code over to the payload style of callbacks [1]
for ROUTER_INTERFACE events for those that are not using them yet.
The unit tests are also updated where needed to account for the
payload style callbacks and publish() method. In addition, a few
callback methods that use the retry_if_session_inactive() decorator are
separated out from the callback so that the context can still be
passed and detected by retry_if_session_inactive logic.
NeutronLibImpact
[1]
https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html
Change-Id: I8d9f8296952dfb10fcccd6afd72e90a5d4f379eb
When new dvr router is going to be created on the node, L3 agent
asks server for list of ports plugged to the subnets, to populate
arp entries for all fixed IPs from those ports.
There was missing info about allowed address pairs there, so those
IPs were not populated in the qrouter namespace.
Now it's added and L3 agent can add those arp entries to the qrouter
namespaces too.
Closes-Bug: #1928466
Change-Id: I5d6c72c271ff450d9e43b3e33a99dd59d727882d
This patch switches over to the payload style of callbacks for
NETWORK based events. As part of this change a few shims are needed
to handle cases where some callbacks don't yet use payloads and others
do. Once we move over to payloads for all callbacks the shims can be
removed.
NeutronLibImpact
Change-Id: I889364b5d184d47a79fe6ed604ce13a4b334acfa
In case when enforce_new_defaults is set to True and new policy rules
are used, context.is_admin flag isn't really working as it was with old
rules.
But in case when elevated context is needed, it means that we need
context which has full rights to the system. So we should also set
"system_scope" parameter to "all" to be sure that system scope queries
can be done with such elevated context always.
It is needed e.g. when elevated context is used to get some data from
db. In such case we need to have db query which will not be scoped to
the single project_id and with new defaults to achieve that system_scope
has to be set to "all".
Proper fix for that should be done in neutron-lib and it is proposed
in [1] already but as we are have frozen neutron-lib version for
stable/wallaby already this patch for neutron is temporary fix for that
issue.
We can revert that patch as soon as we will be in Xena development cycle
and [1] will be merged and released.
[1] https://review.opendev.org/c/openstack/neutron-lib/+/781625
Related-Bug: #1920001
Change-Id: I0068c1de09f5c6fae5bb5cd0d6f26f451e701939
The extension in neutron-lib was released in 1.29.0 so it is time to
use the extension from neutron-lib.
Change-Id: Id2bd872646feede7179affe8c7d124b4530afc9d
Related-Bug: #1811166
fip agent gw ports may be left in DB after router removal due to
race condition between l3 agent and server: when server processes
"router delete" - l3 agent is still processing "router add" and creates
fip agent gw port after server already removed the router.
The patch also adds handling of external network delete event
to cleanup fip namespaces left on agents due to same race condition.
Change-Id: Ib2f3aca08946e584156d092c37e1ea5ed5ca81a6
Closes-Bug: #1902998
This is the code behavior aligning for dvr related logical. The
L3 dvr DB will remove all related FIP agent gateway port after there
is no real use of it. But the DvrFipGatewayPortAgentBindings remain,
it will cause the issue of new floating IP failed to bind. This
patch adds the binding deleting action.
Related-bug: #1883089
Change-Id: I62c29e172bc8705dade11d37bb347241ef8ad5f8
In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError. This patch adds a simple check
for that gateway port, if it is not created, new one.
[1] https://review.opendev.org/#/c/702547/
Closes-Bug: #1883089
Change-Id: Ia90f2ee435b0a3476dbea028d3200cefe11e35e4
In create_fip_agent_gw_port_if_not_exists() method in
neutron.db.l3_dvr_db module, there was unnecessary check if
object returned from _get_agent_by_type_and_host() method
is not None or not empty dict.
But in fact this method will always return not empty dict or
raise an AgentNotFoundByTypeHost exception which is already
properly handled.
TrivialFix
Change-Id: I71379e21a307326d6fca5798a588630ed3ff5263
With python 3.x, six.text_type and six.string_type
are just str.
Also removed a six.integer_type since it was the only
one left in a file.
Another step in removing all of six usage from neutron.
Change-Id: I5208dc41bff1983ecd323286f427296b722da62a
When user is using keepalived on their instances, he often creates
additional port in Neutron to allocate some IP address which will
be then used as VIP in keepalived and will be configured in
allowed_address_pair of other ports plugged to instances with
keepalived.
This is e.g. Octavia's use case.
This together with DVR caused problems with connectivity to such VIP
as it was populated in router's arp cache with MAC address from
Neutron db.
As this port isn't bound, it is only Neutron db entry so there is no
need to set it in arp cache of the router.
This patch is doing exactly that to filter such "unbound" and
"binding_failed" ports from the list.
Change-Id: Ia885ce00dbb5f2968859e8d0850bc511016f0846
Closes-Bug: #1869887
In [1] there was introduced new db table which stored information about
which DVR L3 agent has got already floating ip gateway port. It was to
avoid race conditions and ensure that there is always only one such port
per network and per agent (host).
Unfortunately in [1] there was no added removal of correct record from
this db table so it was causing problems when such port had to be
recreated after it was already on the host and was deleted.
This patch adds removal of such entry from db when needed.
Closes-Bug: #1866336
[1] https://review.opendev.org/#/c/702547/
Change-Id: I56efd1b9f09c0449ce531a185fcf4db353f99fe1
In patch [1] there was introduced simple lock for creation of
DVR agent's floating IP gateway ports for network to avoid races
and creation of duplicated ports for one agent and one network.
This fix from [1] works in simple examples with only one neutron-server,
so it helped e.g. in CI but it wasn't proper fix for production
deployments which are much bigger and have more neutron server api
workers.
So this patch introduces constraint on database level so this works even
across cluster with multiple neutron-server api workers.
[1] https://review.opendev.org/#/c/673331/
Change-Id: Id55b8a21d6ecf5e029d1ca267b2cbd2ed91cca4c
Closes-Bug: #1830763
This reverts commit 7b81c1bc67.
It isn't needed anymore with new solution with lock "on db level"
which is introduced in follow-up patch.
Change-Id: Ibf15ee1969f902e8a266825934d9ac963353f0a0
Related-Bug: #1830763
This patch will remove all the control plane
arp updates for DVR routers. Based on the recent
patch that merged[1], DVR routers will no longer
require the control plane ARP update but will
depend on the ARP Responder for ARP updates.
[1] https://review.opendev.org/#/c/651905/
Change-Id: I538aa6d68fbb5ff8431f82ba76601ee34c1bb181
Implement the Floating IP association logic only in one single place,
L3_NAT_dbonly_mixin._update_fip_assoc(). The dictionary returned will
include a new key, "association_event", with values:
- None: there is no association event. The internal port does not
change.
- True: a new internal port is added to the FIP register. An
association event can imply a disassociation event if the FIP register
had an existing internal port.
- False: the previous internal port is removed and no one is added.
Change-Id: I775aee178cf56f842b3c0a375eda01577840e227
Related-Bug: #1842327
Save order by in port query when not require fixed_ips,
and save some useless query for dvr subnet mac.
Closes-Bug: #1834308
Change-Id: I6836840edcaa5a21fd2ba9f65ffd24f7e5038fa3
This patch switches the code over to the payload style of callbacks [1]
for PORT ROUTER_GATEWAY events for those that are not using them yet.
The unit tests are also updated where needed to account for the
payload style callbacks and publish() method. Finally the patch
normalizes the passing of gateway IPs which are currently referred to
as 'gw_ips' and 'gateway_ips' depending on the event; now all events use
'gateway_ips'.
NeutronLibImpact
[1] https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html
Change-Id: Iacd9e2d00838b402c8ab385044a7e294831a1ddc
In case when new external network is set as gateway network for
dvr router, neutron tries to create floating IP agent gateway port.
There should be always max 1 such port per network per L3 agent but
sometimes when there are 2 requests to set external gateway for 2
different routers executed almost in same time it may happend that
there will be 2 such ports created.
That will cause error with configuration of one of routers on L3 agent
and this will cause e.g. problems with access from VMs to metadata
service.
Such issues are visible in DVR CI jobs from time to time. Please check
related bug for details.
This patch adds lock mechanism during creation of such FIP gateway port.
Such solution isn't fully solving exising race condition as if 2
requests will be processed by api workers running on 2 different nodes
than this race can still happend.
But this should mitigate the issue a bit and solve problem in U/S gates
at least.
For proper fix we should probably add some constraint on database level
to prevent creation of 2 such ports for one network and one host but
such solution will not be easy to backport to stable branches so I would
prefer first to go with this easy workaround.
Change-Id: Iabab7e4d36c7d6a876b2b74423efd7106a5f63f6
Related-Bug: #1830763
When a DVR router is migrated from distributed to
centralized, we are unbinding the router from the
agents, but the ml2 distributed portbindings for
the router port still remains intact.
This patch will fix the issue by deleting the
binding entry for multiple hosts.
Closes-Bug: #1718345
Change-Id: If139790eb336ff13b07b094151946af30322ad3e
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
Enforce that a user updates the admin state of a router before modifying
the distributed state. The API currently allows setting admin state to
false concurrently with changing the distributed state.
This is fine for a transition of centralized->distributed, but the
distributed->centralized transition could leave other nodes configured
as distributed until an audit is performed.
Commit adds shim api extension which should be replaced by neutron-lib
shim extension once https://review.openstack.org/#/c/634509/ is merged.
New method 'is_admin_state_down_necessary' checks that shim extension
is loaded.
Set extension as standard by adding to _supported_extension_aliases in
neutron/services/l3_router/l3_router_plugin.py
Closes-Bug: #1811166
Co-Authored-By: Allain Legacy <allain.legacy@windriver.com>
Co-Authored-By: Enyinna Ochulor <enyinna.ochulor@intel.com>
Change-Id: Ie624aeb3f3aeb4db176d2ca0b22020208d4b408a
Signed-off-by: Matt Welch <matt.welch@intel.com>
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
port_list is already an empty list if no SNAT ports were found,
no reason to re-initialize or print it.
Trivialfix
Change-Id: I0589266646dfa0bc42ef6b7e6929f4dc2237f2b6
In method _generate_arp_table_and_notify_agent in neutron.db.l3_dvr_db
module notifiations about arp table was send only to one router
connected to subnet.
Now it will check if subnet is connected to more than one
dvr router and will send same notification to all such routers.
Closes-Bug: #1815913
Change-Id: I6a7d7f6645a8a7b5219788d51e17d54844d145bc
Adam Oswick