Update hacking to a more recent version, along with
flake8-import-order.
Remove N347 (import mock library) check as that is the
default with later hacking versions.
Update the builtins override of '_' to be the neutron.i18n
version due to the code triggering a false positive. This
is done in a couple of other projects as well.
Fix a number of new warnings it found.
Added some ignore directives for new whitespace issues
found in the test tree, can fix later.
TrivialFix
Change-Id: I5923255af86cf1fa11ab8e3b03bb9efac7dd7b58
When segment plugin is enabled, we should return segments details as
they are part of network.
Partial-Bug: #1956435
Partial-Bug: #1764738
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@industrialdiscipline.com>
Change-Id: I1dab155bc812f8764d22e78ebb7d80aaaad65515
Host parameter is needed there to filter subnets per segment when
segments plugin is enabled.
When dhcp agent requests informations about networks, and segments
plugin is enabled, subnets which belongs to the network are filtered out
based on the host passed as argument to the get_network_info() method.
But we never passed host to that method, even when we should e.g.
during the full sync of the DHCP agent, when it requests details about
each network.
This patch fixes that issue by passing host parameter to that method.
Closes-Bug: #1958955
Change-Id: Ib5eef501493f6735a47ea085196242a5807c4565
Add enable_dhcp, to make a filter to avoid unnecessary
net_info data transfer through rpc.
Change-Id: Ibcef366f5b1f4b7da4f47f1f538a17111da0faa1
Closes-Bug: #1552614
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
https://review.opendev.org/#/c/236983/https://review.opendev.org/#/c/606383/
The above patchs that resolve the race condition of DHCP agent will
result in neutron-server raise DhcpPortInUse ERROR log. And, the
second patch may result in old dhcp agent create a redundant port.
Closes-Bug: #1829332
Change-Id: If7a7ac2f88ce5b0e799c1104c936735a6cc860aa
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
Current DHCP port management in Neutron makes the server to clear the
device_id while the agent is responsible for setting it.
This may cause a potential race condition, for example during network
rescheduling. The server aims to clear the device_id on a DHCP port and
assign the network to another agent while the old agent might just be
taking possession of the port. If the DHCP agent takes possession of the
port (i.e., update port...set the device_id) before the server clears
it, then there is no issue. However, if this happens after the clear
operation by server then the DHCP port would be updated/marked to be
owned by the old agent.
When the new agent takes over the network scheduled to it, it won't be
able to find a port to reuse so that an extra port might need to be
created. This leads to two issues:
1) an extra port is created and never deleted;
2) the extra port creation may fail if there are no available IP
addresses.
This patch proposes a validation check to prevent an agent from updating
a DHCP port unless the network is bound to that agent.
Co-authored-by: Allain Legacy <Allain.legacy@windriver.com>
Closes-Bug: #1795126
Story: 2003919
Change-Id: Ie619516c07fb3dc9d025f64c0e1e59d5d808cb6f
The remainder of the neutron.plugins.common.utils were rehomed into
neutron-lib with [1][2]. This patch consumes them by using the functions
from neutron-lib, and removing the neutron.plugins.common.utils module
all together as it's fully rehomed now.
NeutronLibImpact
[1] https://review.openstack.org/#/c/560950/
[2] https://review.openstack.org/#/c/554546/
Change-Id: Ic0f7b37861f078ce8c5ee92d97e977b8d2b468ad
According to [1], when a network contains more that one IPv4
subnet, they are returned in the 'classless-static-routes'
DHCP option, regardless of whether DHCP is enabled for them
or not.
However, the get_active_networks_info() method used for
synchronizing networks after the dhcp agent restarts filters
subnets with "enable_dhcp=True", which differs from the
get_network_info() method. This will block VM access to
other VMs in the dhcp disabled subnets, even though they are
in the same network. This is visible by looking at the "opts"
file before and after a restart.
Change the dhcp agent to ask for all subnets in its
get_active_networks_info() RPC call by adding an
enable_dhcp_filter argument to toggle the behavior, with the
default being True to not break backwards compatibility.
Based on https://review.openstack.org/#/c/352530/ by Quan Tian.
[1] https://review.openstack.org/#/c/125043/
Change-Id: I11ca1d1a603d02587f3b8d4a5a52a96b0587d61f
Closes-Bug: #1652654
The neutron-lib commit I360545b6ee4291547e0c5c8e668ad03d3efa4725 moved
the externally consumed globals from neutron.common.constants into lib.
With the exception of PROVISIONAL_IPV6_PD_PREFIX all other constants
in neutron.common.constants should only be used in neutron, and will
hopefully remain that way. External consumers needing access to other
common constants should move them into lib first.
NeutronLibImpact
Change-Id: Ie4bcffccf626a6e1de84af01f3487feb825f8b65
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.
Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
In order to allow the DHCP agent to service other subnets
on the network in other segments via DHCP relay, we need to
return all subnets regardless of segment association.
However, this behavior will break older DHCP agents that
then try to get IPs on the subnets belonging to other segments.
This patch adds a new subnet attribute, 'non_local_subnets'
that will be returned in the DHCP RPC calls, so that agents
that can deal with off-link subnets can handle them
accordingly.
Change-Id: I9cce7b8a19c1201435df0c6baac7be57c57847e6
Partial-Bug: #1692486
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
Neutron-lib 1.1.0 is now out and contains the portbindings
API definition (as per commit [1]). This patch moves neutron
references over to the neutron-lib version.
NeutronLibImpact
- Consumers using the public constants within neutron's
portbindings API extension must now use the values
from neutron-lib.
[1] 87e42f993c07ae320159d5123662ee9f3bd4d903
Change-Id: I669af9b4c712877772d91a03857ab108714001d4
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Capture port not found exceptions from port updates of DHCP ports
that no longer exist. The DHCP agent already checks the return
value for None in case any of the other things went missing
(e.g. Subnet, Network), so checking for ports disappearing makes
sense. The corresponding agent-side log message for this has also
been downgraded to debug since this is a normal occurrence.
This also cleans up log noise from calling reload_allocations on
networks that have already been torn down due to all of the subnets
being removed.
Closes-Bug: #1621650
Change-Id: I495401d225c664b8f1cf7b3d51747f3b47c24fc0
Commit 85ed7017ff removed the DBError
handling to let the retry decorator do its magic, however the
full implications of this change were not evaluated. As a result,
DBReferenceError (which derives from DBError) is not processed
correctly and that caused a regression of the existing logic.
Rather than bloat the retry's responsibility even further, this
patch partially reverts commit 85ed7017f by narrowing down the
exception handling to DBReferenceError only.
Related-bug: #1618216
Change-Id: Icf4e5e4145dcdcdc710b8e42044467913ed01ec1
The DHCP port action handler has been catching DBErrors
since f1b9ac5a54, which is
well before we had the retry decorator to deal with these.
With the port action handler catching these, it means there
will not be retries on deadlocks or connection errors so
transient situations can result in a permanently broken
DHCP service for a network.
This removes the catch for DBError so the decorator can retry
the operation.
Closes-Bug: #1618216
Change-Id: I42031b481958bbfdb8f52902c294022717af7adf
The exception handler checking for concurrently deleted subnets
was missing InvalidInput, which comes from IPAM when a port creation
request references a subnet that doesn't exist.
This patch just catches that exception.
Change-Id: I14f9e4bddde845e3fef2b0d8649d3954ba5c93bd
Closes-Bug: #1618187
The dhcp agent will request network information from neutron-server.
neutron-server will give the dhcp agent the entire network tree back
to the agent including subnets. If that agent is on a host that
is mapped to a segment, subnets that do not belong to that
segment are also returned and the dhcp agent will attempt to create
or update the dhcp port based on those subnets. These requests
will fail because the ipam logic forbids it for multiple segments.
This patch strips out those subnets that do not belong to the host's
segment. If the segments plugin is not enabled or the network is not
a routed network then the neutron-server will return back to the agent
what it normally would without routed networks.
Change-Id: I379b56f510853c60bb52c2d710117fecfb9aca2e
Partially-Implements: blueprint routed-networks
This method was originally left for backwards compatibility during
an upgrade of neutron. It stopped being used in havana so I think it
should be safe to remove now.
Closes-bug: #1583835
Change-Id: I503d10ee59ba2fa046b5030e8281fd41a49ee74e
When a new subnet is added to a network, the network cache
is updated with the list of subnets regardless of which ones
have DHCP enabled. This changes the index order of the subnet
list which means that the tags used for each subnet change.
This means we must restart the process because the opts file
will be using different tags than the process args. This patch
implements that change. It also sorts the subnets on the RPC
side so the agent indexes don't change if subnets aren't
added/deleted.
The previous logic was only restarting the process when DHCP
enabled subnets changed, which meant that adding a DHCP disabled
subnet would break the association between the opts file tags and
the process arg tags, which led to the reported bug.
Closes-Bug: #1581918
Change-Id: If1452c0e8fe95eb94cd78c7a05b57aead75662b5
Sometimes an object requires multiple disjoint actors to complete
a set of tasks before the status of the object should be transitioned
to ACTIVE. The main example of this is when a port is being created.
The L2 agent has to do its business to wire up the VIF, but at the same
time the DHCP agent has to setup the DHCP reservation. This led to
Nova booting the VM when the L2 agent was done even though the DHCP
agent may have been nowhere near ready.
This patch introduces a provisioning blocks mechansim that allows the
entities to be tracked that need to be involved to make a transition
to ACTIVE happen. See the devref in the dependent patch for a high-level
view of how this works.
The ML2 code is updated to use this new mechanism to prevent updating
the port status to ACTIVE without both the DHCP agent and L2 agent
reporting that the port is ready.
The DHCP RPC API required a version bump to allow the port ready
notification.
This also adds a devref doc for the provisioning_blocks
module with a high-level overview of how it works in addition
to a detailed description of how it is used specifically with
ML2, the L2 agents, and the DHCP agents.
Closes-Bug: #1453350
Change-Id: Id85ff6de1a14a550ab50baf4f79d3130af3680c8
This patch makes use of the constant defined in the extension.
In addition to this having value of debing defined in one place it
also enables the caller to understand that the portbindings
extension is required.
Note: the constant is not used in the API tests. This has import
issues so it is not relevant.
TrivialFix
Change-Id: I7bfe2528dbbd8017ddbdcf949dbb6264ce1eb5d8
This patch introduces mechanism similar to compare-and-swap
for updating reserved DHCP port.
This addresses a case when two DHCP agents that start nearly at
the same time are assigned to one network and there is a reserved
DHCP port in the network. Then each of agents will try to use it
because agents don't check if reserved port is still available.
Reserved DHCP port can be acquired by different agent between calls to
get_active_networks and update_port, so this patch adds a check for
this case.
Change-Id: I0277ab537ff9d3a664c03ea291b9ec2b0e784dbb
Closes-Bug: #1425402
When API controller calls method create_{network, subnet, port),
it made sure that the necessary default values for attrs are filled properly
according to attr mapping.
However, internal calls to these methods do not follow the convention,
when extension codes miss these values, exceptions will be thrown.
This patch introduces helper functions to fix up arguments and replaces
the direct callers of those methods.
Co-Authored-By: gong yong sheng <gong.yongsheng@99cloud.net>
Co-Authored-By: yalei wang <yalei.wang@intel.com>
Change-Id: Ibc6ff897a1a00665a403981a218100a698eb1c33
Closes-Bug: #1383546
Specify which resources should be tracked when initializing
the ML2 and l3_router plugins. This will enable usage tracking
for the following resources:
- Networks
- Ports
- Subnets
- Subnet pools
- Security groups
- Security group rules
- Routers
- Floating IPs
Partially implements blueprint: bp/better-quotas
Change-Id: I57287b15ffdadc30297651a01e9f05110e9ce6b6
This reverts commit 1b60df85ba.
The new version of mock is now released which has the fix to
make mock.patch.stopall behave deterministically. The code to
stop double-mocking is no longer required.
Other fixes:
'assert_has_calls' that have changed behavior in the new version
(no longer accepts single calls).
Calls to non-existent assert methods that did not exist and were
silently passing.
Use of autospec on a class with decorated functions.
Closes-Bug: #1473369
Change-Id: I164a9af2a7f9ac0f0229ec3c5071f7a470445c98
When a dhcp server is moved to new host on failover,
tunnel is not created.
When a dhcp server is moved to new host,
1) network-hostagent binding is updated(properly i.e with new host).
2) dhcp port-hostagent binding is not updated
( dhcp port is still bound to old host)
If dhcp port-bound agent is different from the new dhcp agent
(which is now taking care of this dhcp port), neutron plugin won't
notify the l2pop, and hence tunnel is not created.
As after failover, the new agent is taking care of this dhcp port,
update portbinding with the new host. This will allow neutron plugin
to notify l2pop(which will create tunnel).
Change-Id: Ib7d7dcddee005395af116ccd31a43853332ae317
Closes-bug: #1411163
This method was last used in Icehouse. I think we can safely
remove all of its code and tests. Icehouse to Liberty rolling
upgrades are in no way expected to work so I just bumped
the RPC version and removed all traces of the code.
Change-Id: Ia545c5be7da80c919ad5bae6074bc1cc3aa89fa0
Closes-Bug: #1314534
This change ensures that the structure of the unit test tree matches
that of the code tree to make it obvious where to find tests for a
given module. A check is added to the pep8 job to protect against
regressions.
The plugin test paths are relocated to neutron/tests/unit/plugins
but are otherwise ignored for now.
Change-Id: If307593259139171be21a71c58e3a34bf148cc7f
Partial-Bug: #1440834
Brian Haley