After updating pylint, it started emitting additional "R"
warnings in some cases, fix some of them.
use-a-generator,
unnecessary-lambda-assignment,
consider-using-max-builtin,
consider-using-generator,
consider-using-in,
use-list-literal,
consider-using-from-import
Trivialfix
Change-Id: Ife6565cefcc30b4e8a0df9121c9454cf744225df
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/db.
Trivialfix
Change-Id: I9311cfe5efc51552008072d84aa238e5d0c9de60
On router update, skip subnet of gateway router.
This will prevent a massive RPC call toward huge number of L3 agents
(all agents in same big public subnet ID).
As a side effect, it will save CPU time on database because L3 agent
receiving such events are then doing a RPC call (sync_routers) even if
router is not used/deployed on this agent.
Closes-Bug: #1992950
Change-Id: Iafa9d43614d528f230cf034103b54f73303ac815
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
If the database "port" child register "ml2_port_bindings" has been
manually deleted from the database, now is possible to delete the
Neutron Port resource (that implies the "port" database register and
all other child registers)
Before this patch, the port deletion CLI command succeeded (no exception
was raised) but the Port resource was not deleted.
Closes-Bug: #1988323
Change-Id: I02de276d0cd8e4ae27355d4aee5f48e92634f318
Delete the DVR port record on the node when the router is deleted from
the node. This patch fixes an issue where the DB actions have no context
writer which will actually update nothing duing the following
`delete_distributed_port_binding_if_stale`.
As well as fixing that when the dvr router was deleted from a node,
only the ml2_distributed_port_bindings were cleaned up and not the
ml2_port_binding_levels records for the dvr port.
Remove the last VM under one router in one host, the
ml2_distributed_port_bindings and ml2_port_binding_levels will remain
the record. So if VMs under one router had spread on many hosts,
and router (router ports) still exists, even there is only one VM under it,
the binding entries will still be equal to amount of hosts where this
router resided before. The result is a large amount of redundant data
in the database, even if some nodes are no longer in the cluster,
there are still records for that node.
Closes-Bug: #1976439
Change-Id: I320ac2306e0f25ff933d8271203e192486062d61
The goal of this series of patches is to make the Neutron code
compliant with SQLAlchemy 2.0.
This patch adds the missing database contexts in the execution of
some queries in the L3 code, reported in the following bugs. This
patch also refactors the ``neutron.db.l3_db`` methods, using the
method decorators instead of the inline context builders.
Closes-Bug: #1974144
Closes-Bug: #1974142
Related-Bug: #1964575
Change-Id: I7e18db7f1fa6b8878c13df09895c993704401815
During port delete Neutron checks if host still handle some ports
from subnets that are used by router.
It may happen that the same subnet that is used for router gateway
port may be used for another VM port. Then we skip notification
to update L3 agent about related to router port deletion.
Related-Bug: #1957794
Change-Id: Iacc7d4e543ac261a6f566b49834b6865f6eded95
This patch switches over to callback payloads for PORT
AFTER_DELETE events.
Some shims were removed.
Change-Id: If69e37b84fe1b027777b1d673b3d08a6651a979e
This reverts commit 062336e59b.
Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.
Related-Bug: #1920001
Conflicts:
neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
neutron/common/utils.py
neutron/db/address_group_db.py
neutron/services/segments/db.py
Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
This patch switches the code over to the payload style of callbacks [1]
for PORT AFTER_CREATE events. In addition it adds a branch/shim to the
dhcp_rpc_agent_api to support both payload and kwarg style callbacks.
NeutronLibImpact
[1]
https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html
Change-Id: I25d43d4f8f2390b07e0d11c631f894d88669bbe0
In case when enforce_new_defaults is set to True and new policy rules
are used, context.is_admin flag isn't really working as it was with old
rules.
But in case when elevated context is needed, it means that we need
context which has full rights to the system. So we should also set
"system_scope" parameter to "all" to be sure that system scope queries
can be done with such elevated context always.
It is needed e.g. when elevated context is used to get some data from
db. In such case we need to have db query which will not be scoped to
the single project_id and with new defaults to achieve that system_scope
has to be set to "all".
Proper fix for that should be done in neutron-lib and it is proposed
in [1] already but as we are have frozen neutron-lib version for
stable/wallaby already this patch for neutron is temporary fix for that
issue.
We can revert that patch as soon as we will be in Xena development cycle
and [1] will be merged and released.
[1] https://review.opendev.org/c/openstack/neutron-lib/+/781625
Related-Bug: #1920001
Change-Id: I0068c1de09f5c6fae5bb5cd0d6f26f451e701939
In case when related dvr router is configured by L3 agent, it is first
added to the tasks queue and then processed as any other router hosted
on the L3 agent.
But if L3 agent will ask neutron server about details of such router,
it wasn't returned back as this router wasn't really scheduled to the
compute node which was asking for it. It was "only" related to some
other router scheduled to this compute node. Because of that router's
info wasn't found in reply from the neutron-server and L3 agent was
removing it from the compute node.
Now _get_router_ids_for_agent method from the l3_dvrscheduler_db module
will check router serviceable ports for each dvr router hosted on the
compute node and will then find all routers related to it. Thanks to
that it will return routers which are on the compute node only because
of other related routers scheduled to this host and such router will not
be deleted anymore.
Change-Id: I689d5135b7194475c846731d846ccf6b25b80b4a
Closes-Bug: #1884527
When _notify_l3_agent_port_update is called, there may be the case that
portbindings.HOST_ID key is not in new_port or original_port dict. It is
like that e.g. in some unit tests of api extensions. So lets use get()
method to get it. The case when host_id is None is already handled
properly in this method so that is fine to do.
Change-Id: I490506cc6c9b18a3ca12e474f9ee9f8f3712ce7b
This patch will remove all the control plane
arp updates for DVR routers. Based on the recent
patch that merged[1], DVR routers will no longer
require the control plane ARP update but will
depend on the ARP Responder for ARP updates.
[1] https://review.opendev.org/#/c/651905/
Change-Id: I538aa6d68fbb5ff8431f82ba76601ee34c1bb181
We observe an excessive amount of routers created on
compute node on which some virtual machines got a fixed
ip on floating network.
Rpc servers should filter out those unnecessary routers
during syncing.
Change-Id: I299031a505f05cd0469e2476b867b9dbca59c5bf
Partial-Bug: #1840579
[0] introduced the concept of connected routers: routers that are
connected to the same subnets. When a L3 agent is synching a router
with connected routers, the data of the entire set should be returned
to the agent by the Neutron server.
However, if an agent tries to synch a router with
no connected routers when the same agent has other routers that are
connected among them, the Neutron server returns the former and the
latter. For details of how this bug can manifest itself, please see [1].
This change prevents this situation: only the synched router is
returned.
[0] https://review.opendev.org/#/c/597567
[1] https://bugs.launchpad.net/neutron/+bug/1838449/comments/15
Change-Id: Ibbf35d0f4a0bf9281f0bc8c411e8527eed75361d
Closes-Bug: #1838449
For large scale deployment, the dvr router will be installed to
the scheduled DHCP host. This will definitely increase the l3
agent service pressure, especially in large number of concurrent
updates, creation, or agent restart.
This patch adds a config ``host_dvr_for_dhcp`` for the DHCP port
device_owner filter during DVR host query. Then if we set
``host_dvr_for_dhcp = False``, L3-agent will not host the DVR router
namespace in its connected networks' DHCP agent hosts.
Closes-Bug: #1609217
Change-Id: I53e20be9b306bf9d3b34ec6a31e3afabd5a0fd6f
When new DVR serviceable port appears on new node we need
to update node's l3 agent with all routers which have the
port's subnets, including connected routers.
We don't need to update all nodes hosting these routers.
It costs us much as all l3 agents then go back to neutron server
and request routers info for no good reason.
This was one of the main issues with DVR at scale fixed in Mitaka.
Change-Id: I99d01d7bf29f236eff0f80d1ae8659f64ac55d39
Related-Bug: #1830456
update port may takes an excessive number of seconds
to complete if dvr routers are running on more than 100
compute nodes. This patch tries to save some time by removing
unnecessary calls inside looping through hosts.
Change-Id: Ide740e0c5c43c2d2b842460a37c8ce125da12b28
Closes-Bug: #1830456
This prevents agent gw ports created in cases where the fip is not
serviced by a dvr enable router. Also, when the router is not DVR
enabled, deleting the gw port becomes a problem even after the router
attachments to the external network are deleted. This fix creates the
agent gw port only if the router associated with the fip is distributed.
Closes-Bug: #1810349
Co-Authored-By: Allain Legacy <allain.legacy@windriver.com>
Co-Authored-By: Matt Peters <Matt.Peters@windriver.com>
Co-Authored-By: Matt Welch <matt.welch@intel.com>
Change-Id: Ibcf087136e30535678c7600bac6ce4d621afe830
Signed-off-by: Enyinna Ochulor <enyinna.ochulor@intel.com>
Due to the class inheriting order the dvr local router will
be counted to the l3 agent restart query. This will cause
l3 agent run the ha port down action. This patch renames
the function to avoid that.
Closes-Bug: #1825787
Change-Id: Id965a6140ae9a06e0b6707800933cbc14bf5bf3f
In case of live migration, binding of port on src host is first switched
to be INACTIVE and then L3 agent should remove qrouter namespace from
compute node if that was last port connected to this router on host.
Because of bug which didn't check if port which is on host has ACTIVE
or INACTIVE binding, router namespace wasn't cleaned properly.
Now it is cleaned as only ports with ACTIVE binding are count
as serviceable ports on compute node.
Change-Id: I01173ad64a19fc05a888c601b81ec097d579a7cf
Closes-Bug: #1825205
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
With DVR routers, if a port is associated with a FloatingIP,
before it is used by a VM, the FloatingIP will be initially
started at the Network Node SNAT Namespace, since the port
is not bound to any host.
Then when the port is attached to a VM, the port gets its
host binding, and then the FloatingIP setup should be migrated
to the Compute host and the original FloatingIP in the Network
Node SNAT Namespace should be cleared.
But the original FloatingIP setup in SNAT Namespace was not
cleared by the agent.
This patch addresses the issue.
Change-Id: I55a16bcc0020087aa1abe76f5bc85cd64ccdaecd
Closes-Bug: #1796491
In case when 2 dvr routers are connected to each other with
tenant network, those routers needs to be always deployed
on same compute nodes.
So this patch changes dvr routers scheduler that it will create
dvr router on each host on which there are vms or other dvr routers
connected to same subnets.
Co-Authored-By: Swaminathan Vasudevan <SVasudevan@suse.com>
Closes-Bug: #1786272
Change-Id: I579c2522f8aed2b4388afacba34d9ffdc26708e3
During the ha router state change event, the gateway port only
changed the L2 binding host. So l3 agent has the entire gateway
port information. It is not necessary to send a router_update
message to l3 agent again.
Closes-Bug: #1795127
Change-Id: Ia332421aff995f42e7a6e6e96b74be1338d54fe1
A recent change to _notify_l3_agent_port_update() was
making a call to is_distributed_router() via the l3plugin,
but since that method lives in l3_dvr_db.py it was
failing. This was not caught by the unit test since it
was mocking the return value.
Had to modify things to get the router first since the
object is required for the check.
Change-Id: I2289f882b692b375d1b40d7a50d206d72da66155
Closes-bug: #1785848
As a consequence of implementing multiple bindings for ports, [1] made
the following attributes lists:
- 'port_binding' in the in the SQLAlchemy Port model
- 'binding' in the Port OVO
This patch pluralizes their names to 'port_bindings' and 'bindings'
respectively
[1] Ie31d4e27e3f55edfe334c4029ca9ed685e684c39
Change-Id: I4ebe47cf9d51a700310aad8dcccc82fea3f00a16
Although no DVR router is hosted on controller node,
neutron-server notifies l3-agent on compute node.
As a result, the following bug happen when VM migrate or resize.
1) L3-agent on compute node output error logs
when VM connect no DVR and HA router.
2) Unused namespace is appeared on compute node
when VM connect no DVR and no HA router.
So, I modify logic of l3-agent to be notified
after migrating or resizing.
Change-Id: I53955b885f53d4ee15377a08627c2c25cb1b7041
Closes-Bug: #1775310
When DVR scheduling in live-migration, the current implementation in DVR
serviceable ports checking on host performs a "contains" operation of
sub-string match which checks the wrong port binding profile for host
(i.e. compute-1 will also match compute-10).
Add quotes to force an exact match of the host name in the port binding
profile dictionary to address this issue.
Closes-Bug: #1776255
Change-Id: I0d2bd9b9ff0aa58a7cce1b8da2a5f21ac6b38c57
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
The is_extension_supported function now lives in neutron-lib. This patch
removes the function from neutron and uses lib's version instead.
NeutronLibImpact
Change-Id: Iccb72e00f85043b3dff0299df7eb1279655e313e
The L3_AGENT_MODE_DVR_NO_EXTERNAL and DVR_SNAT_BOUND constants were
rehomed into neutron-lib with Ieb9374f5483a0ab2306592ab901686ca374db1c8
This patch consumes them by removing them from neutron and using the
constants from neutron-lib instead.
NeutronLibImpact
Change-Id: Ib63a523721a2fa3d1a978a729de28e6a2e560ef6
With the current change in allowing the unbound fip
to be associated with the snat node, we are seeing
that all floating IPs that are associated with an
unbound port are created at the snat node.
This is also applicable for floating IPs that are
created just before associating the port to a VM.
We have seen such scenarios in the test cases.
This is the right behavior as per design. But when
the port is bound to a host, the floating IP should
be migrated to the respective host.
This patch fixes the issue by sending notification to
the respective node, when the port is bound and also
clear the fip from the snat node.
Closes-Bug: #1718788
Change-Id: I6b1f3ffc3c3336035632f6a82d3a87b3be57b403
DVR supports both East/West and North/South routing. While the
SNAT is centralized the DNAT is mostly distributed. There are
certain circumstances where the DNAT might be centralized when
the ports are unbound.
In order to have a well defined behavior and when there are
no external network connectivity available in the compute host,
the DNAT functionality is centralized.
In order to achieve this we are introducing a new agent type
option 'dvr_no_external' to centralize the DNAT.
This new L3 agent type ('dvr_no_external') would only allow the East/West
routing to occur in the compute host and the DNAT or Floating IP will be
configured in the centralized network node.
Change-Id: Ia5d7336e478e0fa5ba62b7ae5ed0c56656116d94
Partial-Bug: #1667877
Unbound ports that are associated with a Floating IP and connected to
DVR Routers will not be serviced by the DVR Routers, unless we bind it
to a valid host.
This server side patch allows the neutron server to schedule the
unbound port Floating IP on the network node or the node with dvr_snat
agent where the SNAT functionality resides.
The DNAT rules for the unbound ports will be configured in the SNAT
namespace on the network node.
Related-Bug: #1583694
Change-Id: I05d0bfb3fa275b1e4e479928000cf8494da858f6
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.
Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
This patch integrates the OVO created for RouterL3AgentBinding into
the code base.
Change-Id: I0af665a97087ad72431d58f04089a804088ef005
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
This query can be moved to base l3_db since there isn't
anything DVR-specific about it and the l3_db code can use
a similar helper function.
Related: blueprint multi-l3-backends
Change-Id: Idcc982c0aa19d4685975ddaa974f42ee7cba1b4a
Neutron-lib 1.1.0 is now out and contains the portbindings
API definition (as per commit [1]). This patch moves neutron
references over to the neutron-lib version.
NeutronLibImpact
- Consumers using the public constants within neutron's
portbindings API extension must now use the values
from neutron-lib.
[1] 87e42f993c07ae320159d5123662ee9f3bd4d903
Change-Id: I669af9b4c712877772d91a03857ab108714001d4
Brian Haley