We saw auto_schedule_routers took over 40 seconds
for a DVR enabled environment with option
auto_schedule_routers enabled.
Adding new arguments to get_router_agents_count and
dealing with routers separately depending on whether
it is a regular router or HA. The benefits are
we do not need to loop over every router available in
environment. Another reason for doing this is that
get_router_agents_count is used solely to heal
routers with less than required agents so number of
routers with less agents is small for most of the times.
Related-Bug: #1973656
Change-Id: Ic29275815a8c32cee7a6470509687a18fa594514
This patch is covering an edge case that could happen when the number
of DHCP agents ("dhcp_agents_per_network") or L3 agents
("max_l3_agents_per_router") has been reduced and there are more agents
assigned than the current number. If the user removes any agent
assignation from a L3 router or a DHCP agent, it is possible to remove
first the lower binding assigned registers.
Now the method ``get_vacant_binding_index`` calculates the number of
agents bound and the number required. If a new one is needed, the
method returns first the lower binding indexes not used.
Closes-Bug: #2006496
Change-Id: I25145c088ffdca47acfcb7add02b1a4a615e4612
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/db.
Trivialfix
Change-Id: I9311cfe5efc51552008072d84aa238e5d0c9de60
The goal of this patch is to make the Neutron code compliant
with SQLAlchemy 2.0.
All SQL transactions must be executed inside an explicit
writer/reader context. SQLAlchemy no longer will create an
implicit transaction if the session has no active transaction.
A warning message, only available in debug mode, is added. When
an ORM session calls "do_orm_execute", if there is no active
transaction, a warning message with a traceback will be logged
to help to debug the regression introduced.
Related-Bug: #1964575
Change-Id: I3da37fee205b8d67d10673075b9130147d9eab5f
When retrieving a vacant L3 agent binding index, if
"is_manual_scheduling" is set, the method "get_vacant_binding_index"
should always return a valid binding index. If the existing binding
indexes are sequentially aligned, the method will return a new one
on top; if there is a gap in the binding indexes list, the first
free index will be returned.
Closes-Bug: #1884906
Change-Id: I0a89bca0734d3e735fb357e488f85589e81d709f
This option was deprecated since couple of releases already.
In Stein we removed 'external_network_bridge' option from L3 agent's
config so now it's time to remove also this one.
There is also new upgrade check introduced to check and warn
users if gateway_external_network_id was used in the deployment.
This patch also removes method _check_router_needs_rescheduling() from
neutron/db/l3_db.py module as it is not needed anymore.
This patch also removes unit tests:
test_update_gateway_agent_exists_supporting_network
test_update_gateway_agent_exists_supporting_multiple_network
test_router_update_gateway_no_eligible_l3_agent
from neutron/tests/unit/extensions/test_l3.py module as those
tests are not needed when there is no "gateway_external_network_id"
config option anymore.
Change-Id: Id01571cd42cfe9c5ce91e90159917c7d3c963878
Due to the class inheriting order the dvr local router will
be counted to the l3 agent restart query. This will cause
l3 agent run the ha port down action. This patch renames
the function to avoid that.
Closes-Bug: #1825787
Change-Id: Id965a6140ae9a06e0b6707800933cbc14bf5bf3f
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
If l3-agent was restarted by a regular action, such as config change,
package upgrade, manually service restart etc. We should not set the
HA port down during such scenarios. Unless the physical host was
rebooted, aka the VRRP processes were all terminated.
This patch adds a new RPC call during l3 agent init, it will try to
retrieve the HA router count first. And then compare the VRRP process
(keepalived) count and 'neutron-keepalived-state-change' count
with the hosting router count. If the count matches, then that
set HA port to 'DOWN' state action will not be triggered anymore.
Closes-Bug: #1798475
Change-Id: I5e2bb64df0aaab11a640a798963372c8d91a06a8
Change to do a direct return of
_check_dvr_serviceable_ports_on_host() by moving distributed
check earlier.
Note: _check_dvr_serviceable_ports_on_host() can be called
with subnet_ids=[]
Trivialfix
Change-Id: I50987858b04fb1cf57109517b82624760ce089dc
The current "agent remove-add" method to reschedule a router is not
friendly for DVR. In DVR cases, the old agent may still need to keep the
router in a non-master role to service any VM ports that exit on that
host while the new agent handles the master role of the router.
This patch proposes to send "update" instead of "remove" notification
for DVR rescheduling, which aligns with the retain_router check in
remove_router_from_l3_agent as well.
Closes-Bug: #1781179
Change-Id: I23431f9f46f72e6bce91f3d1fb0ed328d55930fb
During listing router_ids on host it is possible that on some hosts
there are no L3 agents.
In such case AgentNotFoundByTypeHost exception is raised in
neutron.db.agents_db module in _get_agent_by_type_and_host() method.
Now this exception is properly handled during listing routers on host.
Change-Id: Ia5ff1b57ef63c98b4ada4f2d46c45336e413be3d
Closes-Bug: #1737917
The is_extension_supported function now lives in neutron-lib. This patch
removes the function from neutron and uses lib's version instead.
NeutronLibImpact
Change-Id: Iccb72e00f85043b3dff0299df7eb1279655e313e
The L3_AGENT_MODE_DVR_NO_EXTERNAL and DVR_SNAT_BOUND constants were
rehomed into neutron-lib with Ieb9374f5483a0ab2306592ab901686ca374db1c8
This patch consumes them by removing them from neutron and using the
constants from neutron-lib instead.
NeutronLibImpact
Change-Id: Ib63a523721a2fa3d1a978a729de28e6a2e560ef6
The router_ids argument to auto_schedule_routers() is
unused, and was marked for deprecation in Queens.
Change-Id: Ie97b1ad05e294b5fe763ae8d7319800eb16ea3dc
Agent object has been merged [1].
This patch uses Agent object in l3_agentschedulers_db.
[1] https://review.openstack.org/#/c/297887/
Co-Authored-By: Nguyen Phuong An <AnNP@vn.fujitsu.com>
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I7c652919a258b2e0b7e068a759ff6bfa996b0c4d
Refactoring neutron db config opts to be in neutron/conf/db/ and
neutron/conf/agent/database/ so that all the configuration options
reside in a centralized location. This simplifies the process of
looking up the config opts and provides an easy way to import.
Change-Id: I4da9bb48d49b99e8c2b34a5c1b83e7eb95b70b82
Partial-Bug: #1563069
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
DVR supports both East/West and North/South routing. While the
SNAT is centralized the DNAT is mostly distributed. There are
certain circumstances where the DNAT might be centralized when
the ports are unbound.
In order to have a well defined behavior and when there are
no external network connectivity available in the compute host,
the DNAT functionality is centralized.
In order to achieve this we are introducing a new agent type
option 'dvr_no_external' to centralize the DNAT.
This new L3 agent type ('dvr_no_external') would only allow the East/West
routing to occur in the compute host and the DNAT or Floating IP will be
configured in the centralized network node.
Change-Id: Ia5d7336e478e0fa5ba62b7ae5ed0c56656116d94
Partial-Bug: #1667877
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.
Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
This patch integrates the OVO created for RouterL3AgentBinding into
the code base.
Change-Id: I0af665a97087ad72431d58f04089a804088ef005
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
according to https://wiki.openstack.org/wiki/Python3, now we should avoid
using six.iteritems and replace it with dict.items.
Change-Id: I58a399baa2275f280acc0e6d649f81838648ce5c
Closes-Bug: #1680761
This patch integrates Router Extra Attributes OVO and uses proper
context in calling methods for object operations.
The other integration parts of this OVO in l3_agentschedulers_db.py
and l3_attrs_db.py are being done in patch [1] and [2] respectively.
[1] - I0af665a97087ad72431d58f04089a804088ef005
[2] - Id5ed0a541d09cd1105f9cb067401e2afa8cd9b83
Change-Id: Ib6e59ab3405857d3ed4d82df1a80800089c3f06e
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Those were moved under neutron.db.models start of Ocata. We should be
able to clean them up now.
NeutronLibImpact
Change-Id: Iadbf44d52ee8e30712807384152a29ce1a8b8f72
This patch proposes a (rather major) refactor to the L3 scheduler.
Basically, the auto_schedule_routers() code-path was split to 2
different code-paths, each dealing with a different case (unscheduled
routers vs underscheduled routers), in addition to the API-initiated
schedule() logic. This patch removes the 2 code-paths in favor of moving
most of the logic into schedule(). While the result is a slightly
longer schedule(), the benefit is that a lot of the previous
unmaintainable code-paths of auto_schedule_routers() are now removed.
Yay! :D
Related-Bug: #1609738
Change-Id: I227ca60422545e40d3bbb8baf2b41a8ce14f4294
This reverts commit 9c3c19f07c.
Following the merge of Ie98d5e3760cdb17450aea546f4b61f5ba14baf1c, the
creation of new router uses RouterL3AgentBinding and its' new
binding_index attribute to ensure correctness of the resources. As such,
the ALLOCATING state (which was used to do just that) is no longer
needed and can be removed.
Closes-Bug: #1609738
Change-Id: Ib04e08df13ef4e6b94bd588854a5795163e2a617
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Change I3447ea5bcb7c57365c6f50efe12a1671e86588b3 added a binding_index
to the RouterL3AgentBinding table. In certain (concurrent) cases, a row
with the same binding_index might be used twice, which will raise
DBDuplicateEntry. However, that change didn't retry on this case at all
code-paths, so this patch rectifies this issue.
Closes-Bug: #1633042
Change-Id: I228b0084a8e7c48e78bc2ea6a3fccf7437210e73
Change I7be4ce2513e49e6da46a7bdffb8538613f0be7c7 relocated the Agent
model (and a few other functions), but some references to old
functions/model still remain. These cause a considerable amount of
warnings when running unit tests and the code itself.
Change-Id: Id026cae75bfa56b1023f8a1c4e5db750cf0bff5f
Partial-Bug: #1597913
This patch set is for breaking the circular dependency between
Agent/AgentVersionedObject.
See:https://review.openstack.org/#/c/297887/ for details.
Change-Id: I7be4ce2513e49e6da46a7bdffb8538613f0be7c7
Partial-Bug: #1597913
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Co-Authored-By: Sindhu Devale <sindhu.devale@intel.com>
As there would be issue of cyclic imports while the implementation
of Oslo-versioned Objects for Router Extra Attributes which has DB
models definition in same file, this patch will move Router Extra
Attributes model.
Change-Id: I8c90e35ef08ef74fb5309707673796e587a7289e
Partial-Bug: #1597913
As there would be issue of cyclic imports while
implementation of objects for l3agentbinding which has
db models definition and mixins in same file, this patch will
relocate l3agentbinding models.
Change-Id: Idef2fe3e16b245da849e2d29c5578e5f5d081dc4
Partial-Bug: #1597913
This adjusts the L3 scheduler framework to ask the L3 plugin if
a given router should be scheduled before scheduling it. To maintain
backwards compatibility, this new method is implemented to return True
in the base class.
The L3 plugin with flavor support overrides this to lookup the driver
associated with a router and check if the driver requires the L3
scheduling framework. This will allows the coexistence of flavors that
needs scheduling and flavors that don't.
Change-Id: I17a64c59eaf5d8605ff8ec2a29e491673be960e7
Implements: blueprint multi-l3-backends
The patch proposes adding a new binding_index to the
RouterL3AgentBinding table, with an additional Unique Constraint that
enforces a single <router_id, binding_id> per router. This goes a long
way into fixing 2 issues:
1. When scheduling a non-HA router, we only use binding_index=1. This
means that only a single row containing that router_id can be
committed into the database. This in fact prevents over-scheduling of
non-HA routers. Note that for the HA router case, the binding_index
is simply copied from the L3HARouterAgentPortBinding (since they are
always created together they should always match).
2. This sets the ground-work for a refactor of the l3 scheduler - by
using this binding and db-based limitation, we can schedule a router
to agents using the RouterL3AgentBinding, while postponing the
creation of L3HARouterAgentPortBinding objects for the agents until
they ask for it (using sync_routers). This will be a major
improvement over todays "everything can create
L3HARouterAgentPortBinding" way of things).
Closes-Bug: #1535557
Change-Id: I3447ea5bcb7c57365c6f50efe12a1671e86588b3
There were several patches attempting to move threading after forking
workers. By the time of development it was assumed to land in Mitaka but
it eventually landed in Newton [1], deprecating some methods but leaving
wrong name of cycle where methods should be removed.
[1] I3a32a95489831f0d862930384309eefdc881d8f6
Change-Id: Ic30014c0f94da4b6dd0323663d98be3b1cdd77da
This reverts commit b1cdba1696
Original patch was reverted because it broke neutron plugin's
backward compatibility and needed more work.
This patch fixes that problems:
1) original behaviour of add_agent_status_check,
start_periodic_l3_agent_status_check and
start_periodic_dhcp_agent_status_check methods is deprecated but kept
for using in third part plugins for backward compatibility
2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
and add_periodic_dhcp_agent_status_check method are implemented
instead and are used for implementing plugins in neutron codebase
Closes-Bug: #1569404
Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6
The logic of reschedule_routers_from_down_agents() can be useful for
additional types of resource rescheduling such as loadbalancers, as described at
Id8d3218bf1e52722cc10ddcd34e3e734eef90658
Related-Bug: #1565511
Change-Id: I7871d68246d82f343e99730c09f81bcc7800bcce
ushen