As part of the Secure RBAC community goal, we should switch options
"enforce_new_defaults" and "enforce_scope" to be True by default.
It will be still possible to fallback to old policy rules by configuring
those config options to False in Neutron config.
Change-Id: I09c0026ccf87e6c0bb1fa59165c03dc508fba6fa
In case when 2 neutron servers are trying to remove network from the
same DHCP agent it may happend that one of them will not be able
to find binding object anymore thus deletion of that binding object
will raise ObjectNotFound exception.
This patch adds proper handling of such case to not raise ugly
stacktrace in neutron logs.
Closes-Bug: #1970759
Change-Id: I67d516c4583aa0c20416114b92a6d69ece5b970c
It has been reported by bug #1959750 that for a given network the
agent is scheduling all segments reported by agent.
This commit is fixing the issue by ensuring scheduling segments per
network that they belong to only.
A test has been updated to demonstract the fact that for a given
network we only schedule the related segments.
Closes-bug: #1959750
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@industrialdiscipline.com>
Change-Id: Icd57570004055903c6817a75d814ed65db3fa53c
Config option allow_overlapping_ips is deprecated to removal now and
will be removed in the Z cycle.
Default value for that option is now set to True as this is supported by
IPAM module in Neutron.
Related-Bug: #1942294
Change-Id: I17bf5e4483025e9cc4ee04dd3e7c925f7bddc3db
DHCP notification is done after each create/update/delete for
network, subnet and port.
This notification currently has to retrieve network from DB almost
every time, which is a quite heavy DB request and hence affects
performance of port and subnet CRUD.
This patch suggests 2 optimizations:
- do not fetch network if not needed (only fetch when schedule needed)
- for port and subnet AFTER_CREATE event pass network dict from plugin
According to Rally tests these changes improve performance:
- port create ~20%
- port update ~20%
- subnet create ~15%
- port delete and subnet update/delete - not tested
Closes-Bug: #1923161
Change-Id: I0ab836ac09225f4f3ad435e9ceaf315018855d52
In case when subnet in segment X was created before hosts from
that segment was mapped in the Neutron DB, network wasn't scheduled
to the DHCP agents in that new segment.
Now DHCP scheduler is triggered in such case.
Closes-bug: #1917811
Change-Id: Ic9e64aa4ecdc3d56f00c26204ad931b810db7599
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
We observe an excessive amount of routers created on
compute node on which some virtual machines got a fixed
ip on floating network.
Rpc servers should filter out those unnecessary routers
during syncing.
Change-Id: I299031a505f05cd0469e2476b867b9dbca59c5bf
Partial-Bug: #1840579
Currently, if dhcp mapping from a network is removed, it is reassigned
to the network. This is because of the Network Scheduler's schedule
function, which considers balancing the networks with the agents, whether
enable_dhcp is set on its subnets or not. It does not take into account
the network_auto_schedule config option. This is particularly disturbing
when considering backends which have their provide their own DHCP.
With this patch, if network_auto_schedule is set to False, networks wont
be automatically scheduled to DHCP Agents. If DHCP is to be mapped to a
network, it can be mapped using the CLI itself.
While it may seem that this change is breaking what is already working,
but as mentioned earlier, if there are network backends which provide DHCP
support themselves, they wont need the automatic mapping, which the term
"network_auto_schedule" actually stands for.
Closes-Bug: #1647421
Change-Id: If1a6a2a174d0f737415efa2abce518722316a77b
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
NeutronLibImpact
Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
Currently, the dhcp Provisioning of ports is the crucial bottleneck
of that concurrently boot multiple VM.
The root cause is that these ports will be processed one by one by dhcp
agent when they belong to the same network, And the 'Provisioning complete'
port is still blocked other port's processing in other dhcp agents. The
patch aim to optimize the dispatch strategy of the port cast to agent to
improve the Provisioning process.
In server side, I classify messages to multi levels. Especially, I classify
the port_update_end or port_create_end message to two levels, the high-level
message only cast to one agent, the low-level message cast to all agent. In
agent side I put these messages to `resource_processing_queue`, with the queue,
We can delete `_net_lock` and process these messages in order of priority.
Additonally, I modified the `resource_processing_queue` for my demand. I update
`_queue` from LIST to PriorityQueue in `ExclusiveResourceProcessor`, by this
way, we can sort all message which cached in `ExclusiveResourceProcessor` by
priority.
Related-Bug: #1760047
Change-Id: I255caa0571c42fb012fe882259ef181070beccef
The common rpc and exceptions were rehomed into
neutron-lib with [1]. This patch shims those rehomed
modules in neutron to switch over to neutron-lib's
versions under the covers.
To do so:
- The rpc and common exceptions are changed to
reference their counterpart in neutron-lib effectively
swapping the impl over to neutron-lib.
- The fake_notifier is removed from neutron and lib's
version is used instead.
- The rpc tests are removed; they live in lib now.
- A few unit test related changes are required
including changing mock.patch to mock.patch.object,
changing the mock checks for a few UTs as they don't
quite work the same with the shim in place.
- Using the RPC fixture from neutron-lib rather than
that setup in neutron's base test class.
With this shim in place, consumers are effectively using
neutron-lib's RPC plumbing and thus we can move consumers
over to neutron-lib's version at will. Once all
consumers are moved over we can come back and remove
the RPC logic from neutron and follow-up with a consumption
patch.
NeutronLibImpact
[1] https://review.openstack.org/#/c/319328/
Change-Id: I87685be8764a152ac24366f13e190de9d4f6f8d8
The current "agent remove-add" method to reschedule a router is not
friendly for DVR. In DVR cases, the old agent may still need to keep the
router in a non-master role to service any VM ports that exit on that
host while the new agent handles the master role of the router.
This patch proposes to send "update" instead of "remove" notification
for DVR rescheduling, which aligns with the retain_router check in
remove_router_from_l3_agent as well.
Closes-Bug: #1781179
Change-Id: I23431f9f46f72e6bce91f3d1fb0ed328d55930fb
When a network is removed from a dhcp agent, in some scenarios if the
agent releases its port concurrently, there is chance that the
unscheduling will fail due to that the target port is not found.
Catch the PortNotFound exception as an expected error under this type of
concurrent circumstance and logs it to move forward.
Closes-Bug: #1775496
Change-Id: Ib51b364f6ced0de7685c8ee07c1d292308d919f5
Signed-off-by: Kailun Qin <kailun.qin@intel.com>
The dhcpagentscheduler extension's API definition was rehomed into
neutron-lib with https://review.openstack.org/#/c/520751/
This patch consumes it by using the API definition and it's constants
where applicable.
NeutronLibImpact
Change-Id: Ib0c97268f01885f6daacb3d1cdbbd94bb6020d60
During listing router_ids on host it is possible that on some hosts
there are no L3 agents.
In such case AgentNotFoundByTypeHost exception is raised in
neutron.db.agents_db module in _get_agent_by_type_and_host() method.
Now this exception is properly handled during listing routers on host.
Change-Id: Ia5ff1b57ef63c98b4ada4f2d46c45336e413be3d
Closes-Bug: #1737917
Agent object has been merged [1].
This patch uses Agent object in agents_db and test_agents_db.
We also introduce a new function (get_agents_object) and keep
the old function (get_agents_db) for backward compatibility.
[1] https://review.openstack.org/#/c/297887/
Co-Authored-By: Nguyen Phuong An <AnNP@vn.fujitsu.com>
Change-Id: I4c4283cb1aa05d52dca00cc249e094ea7d55b1d3
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Today our unit test code uses various ways to "patch" the global
RESOURCE_ATTRIBUTE_MAP as well as extension specific maps in some cases.
This patch consolidates such patching whereby tests should use neutron's
AttributeMapMemento in their setup() chain (only once) if they update
the global map and they should individually handle backup/restore of per
extension map updates. This change will simplify the code and make it
easier to phase-in API definition usage with neutron-lib where we have
some as API definitions and others not. Longer term the
AttributeMapMemento will be replace with neutron-lib's fixture as we
move all extension maps to API definitions in neutron-lib.
Change-Id: I2586f0b11b107d7f57214a0d65bcf7c38a5f0ebb
The neutron-lib commit I360545b6ee4291547e0c5c8e668ad03d3efa4725 moved
the externally consumed globals from neutron.common.constants into lib.
With the exception of PROVISIONAL_IPV6_PD_PREFIX all other constants
in neutron.common.constants should only be used in neutron, and will
hopefully remain that way. External consumers needing access to other
common constants should move them into lib first.
NeutronLibImpact
Change-Id: Ie4bcffccf626a6e1de84af01f3487feb825f8b65
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.
Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
This patch integrates the OVO created for RouterL3AgentBinding into
the code base.
Change-Id: I0af665a97087ad72431d58f04089a804088ef005
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
This prevents all of the warnings in unit tests about
no DHCP agents being available for scheduling during
network and port creation.
TrivialFix
Change-Id: I06cb626496866b90f60b406d1141ecad6e1a47e1
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Renames the AgentStatusCheckWorker class to PeriodicWorker
and moves it into the worker module since there isn't anything
agent-specific about it and it can be used for other periodic
jobs server side.
TrivialFix
Change-Id: Ic7a55ef534f64e6bfc60ae38bb0e139a0078510b
This worker would fail to start again if stop() or reset()
was called on it because of some bad conditional logic. This
doesn't appear to impact the current in-tree use case but
it should behave correctly.
Closes-Bug: #1641788
Change-Id: Id6334c1ef6c99bd112ada31e8fe3746d7e035356
This patch set is for breaking the circular dependency between
Agent/AgentVersionedObject.
See:https://review.openstack.org/#/c/297887/ for details.
Change-Id: I7be4ce2513e49e6da46a7bdffb8538613f0be7c7
Partial-Bug: #1597913
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Co-Authored-By: Sindhu Devale <sindhu.devale@intel.com>
As there would be issue of cyclic imports while
implementation of objects for l3agentbinding which has
db models definition and mixins in same file, this patch will
relocate l3agentbinding models.
Change-Id: Idef2fe3e16b245da849e2d29c5578e5f5d081dc4
Partial-Bug: #1597913
This makes the notifier subscribe to core resource events
and leverage them if they are available. This solves the
issue where internal core plugin calls from service plugins
were not generating DHCP agent notifications.
Closes-Bug: #1621345
Change-Id: I607635601caff0322fd0c80c9023f5c4f663ca25
The callback manager was indexing callbacks based on
a callback ID generated from oslo utils reflection.
This presented two problems.
The first was that in py34 get_callable_name would use
__qualname__ which returns the class name the function
is defined on rather than the class of the object itself.
So two classes defined in the same module inheriting from
the same parent class could not both subscribe a method
defined on the parent.
The second more general problem is that two objects which
are instances of the same class cannot subscribe the same
method because they have the same ID.
This adds the hash of the method to the ID to prevent these
issues. The hash by itself could have been used but it's not
very user-friendly so the name is left on for nice log
messages.
Change-Id: Iff1ca8c4ddb58ca5907d21fa0de7f0f292b6fc0e
Stevedore documentation suggest that full import paths are not supposed
to be user visible. Since unit tests emulate users when configuring
oslo.config, we better off relying on well known plugin aliases than
internal details.
For in-tree that may be not a big deal, but with it we set a bad example
for third parties that may later find their tests broken eg. when we
decide to move code around.
TrivialFix
Change-Id: I7bd036ac3df7e7f4c678356d0a793e7d38599dda
This is the initial support for flavors and multiple service
providers with the built-in L3 service plugin.
This patch handles a few key components:
* Adds an optional flavor_id to the router data model
* Adds a new driver controller that performs the following tasks:
* Loads up the configured drivers and 4 default drivers representing
the current matrix of ha/dvr options (single node, ha, dvr, and ha+dvr)
* Associates every router with a driver based on ha/dvr attributes
or the flavor_id if specified
Note that the current drivers are very limited because they don't do anything.
All of the complex logic for the in-tree drivers is still tied up in the giant
mixin the service plugin inherits. Breaking that apart will be in follow-up
patches.
Partially-Implements: blueprint multi-l3-backends
Change-Id: Idce75bf0fc1375dcbbff9b9803fd2fe97d158cff
In this patch, auto schedule router will be removed from sync_routers,
so that the reported bug can be fixed. And potential race can be avoid
accoridng to [1]
The result of patch will make the l3 agent can't get the router info
when the router is not bound to the l3 agent. And router in agent will
be removed during the agent processing. This makes sense, since, in
neutron server, the router is not tied to the agent. For DVR, if there
are service port in the agent host, the router info will still be
returned to l3 agent.
[1] https://review.openstack.org/#/c/317949/
Change-Id: Id0a8cf7537fefd626df06064f915d2de7c1680c6
Co-Authored-By: John Schwarz <jschwarz@redhat.com>
Closes-Bug: #1593653
Bug 1591766 unveiled an issue where calling the plugin API does not trigger
DHCP notifications. This is required by the auto-allocated-topology service
plugin that calls core_plugin.update_network(), and expect notifications
to be sent out on state changes. To accomplish this, the logic has been
encapsulated in the DHCP module, and leveraged via callback mechanisms.
For this reason, new events have been introduced, AFTER_REQUEST, and
BEFORE_RESPONSE. The latter in particular is the one needed to hook up
dhcp notifications in order to preserve backward compatibility.
More precisely, core plugins that use DHCP as is or implement their own,
(with or without an agent) should already instantiate their own notifier,
and if they do not, this should be rectified.
A search on codesearch.openstack.org reveals that out-of-tree plugins
already specify their own notifiers, and the default initialization is
clearly redundant now.
Related-bug: #1591766
Change-Id: I7440becb6d30af7159ecaeba09d7a28eceb71bea
It was deprecated at [1].
Remove the deprecated config 'router_id' and its related tests.
[1] https://review.openstack.org/#/c/248498
DocImpact: All references of 'router_id' configuration option
and its description should be removed from the docs.
UpgradeImpact: Remove 'router_id' configuration option from the
l3_agent.ini.
Change-Id: Ic9420191e8c1a333e4dcc0b70411591b8573ec7c
Closes-Bug: #1594711
This reverts commit b1cdba1696
Original patch was reverted because it broke neutron plugin's
backward compatibility and needed more work.
This patch fixes that problems:
1) original behaviour of add_agent_status_check,
start_periodic_l3_agent_status_check and
start_periodic_dhcp_agent_status_check methods is deprecated but kept
for using in third part plugins for backward compatibility
2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
and add_periodic_dhcp_agent_status_check method are implemented
instead and are used for implementing plugins in neutron codebase
Closes-Bug: #1569404
Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6
get_active_networks() has no longer been used since havana. This patch
migrates the existing tests calling this method to use the current method
that is being used: get_active_networks_info().
Change-Id: Ibd6871b4884a2694983c4a572e00647674df67ae
This patch introduces retry(func, max_attempts) method
which wraps the original func in such a way,
that if execution results in MessagingException,
the given function will be retried for max_attempts
till success or MessagingException is raised.
The function is in the utils module and can be reused
by different agentnotifiers if necessary.
Change-Id: I0d0c17e500e44c1a17438c29a0e76a9ef00872e8
Brian Haley