Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/api.
Trivialfix
Change-Id: I1258b04f64a18036407e1d9de9ddca7472af0d11
When segment plugin is enabled, we should return segments details as
they are part of network.
Partial-Bug: #1956435
Partial-Bug: #1764738
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@industrialdiscipline.com>
Change-Id: I1dab155bc812f8764d22e78ebb7d80aaaad65515
Host parameter is needed there to filter subnets per segment when
segments plugin is enabled.
When dhcp agent requests informations about networks, and segments
plugin is enabled, subnets which belongs to the network are filtered out
based on the host passed as argument to the get_network_info() method.
But we never passed host to that method, even when we should e.g.
during the full sync of the DHCP agent, when it requests details about
each network.
This patch fixes that issue by passing host parameter to that method.
Closes-Bug: #1958955
Change-Id: Ib5eef501493f6735a47ea085196242a5807c4565
In patch [1] method get_network_info was refactored and that causes
NameError in the DHCP agent when there is "network object passed in
kwargs and there are subnets with segments in network. See related bug
for details.
[1] https://review.opendev.org/c/openstack/neutron/+/820190
Closes-Bug: #1958955
Change-Id: Iad8d85c79f8b11a24b1bb1ca44c776e909b610c3
Add enable_dhcp, to make a filter to avoid unnecessary
net_info data transfer through rpc.
Change-Id: Ibcef366f5b1f4b7da4f47f1f538a17111da0faa1
Closes-Bug: #1552614
When setting up the DHCP agent of a network, the DHCP namespace external
port is configured. If this port already exists and the fixed IP
addresses are correctly configured (in the DHCP subnets range), the port
is used as is.
Sometimes, because of 1627480 or 1841636, the port information is not
correctly retrieved. This patch does not solve it but mitigates the
process of resynchronizing the network DHCP. If the stored DHCP port
does not have the correct information, the agent calls the RPC plugin to
retrieve from the server the DHCP port updated information, including
the fixed IP address and the subnets.
Change-Id: Iff40e7bba645ee12c2001d7ce735a36e0ddc81e9
Related-Bug: #1627480
Related-Bug: #1841636
- Added get_networks() RPC call for DHCP agent
- Added get_networks() RPC call for L3 agent
This change is required in order to support out of tree
MultiInterfaceDriver and IPoIBInterfaceDriver interface drivers
as they require information on the network a port is being plugged
to.
These RPCs will be passed as kwargs when loading the relevant
interface driver.
get_networks() keyword args map to the keyword arguments of:
neutron.neutron_plugin_base_v2.NeutronPluginBaseV2.get_networks()
Change-Id: I11d82380aad8655a4fdc9656737b912b16e2859b
Partial-Bug: #1834176
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
https://review.opendev.org/#/c/236983/https://review.opendev.org/#/c/606383/
The above patchs that resolve the race condition of DHCP agent will
result in neutron-server raise DhcpPortInUse ERROR log. And, the
second patch may result in old dhcp agent create a redundant port.
Closes-Bug: #1829332
Change-Id: If7a7ac2f88ce5b0e799c1104c936735a6cc860aa
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
Current DHCP port management in Neutron makes the server to clear the
device_id while the agent is responsible for setting it.
This may cause a potential race condition, for example during network
rescheduling. The server aims to clear the device_id on a DHCP port and
assign the network to another agent while the old agent might just be
taking possession of the port. If the DHCP agent takes possession of the
port (i.e., update port...set the device_id) before the server clears
it, then there is no issue. However, if this happens after the clear
operation by server then the DHCP port would be updated/marked to be
owned by the old agent.
When the new agent takes over the network scheduled to it, it won't be
able to find a port to reuse so that an extra port might need to be
created. This leads to two issues:
1) an extra port is created and never deleted;
2) the extra port creation may fail if there are no available IP
addresses.
This patch proposes a validation check to prevent an agent from updating
a DHCP port unless the network is bound to that agent.
Co-authored-by: Allain Legacy <Allain.legacy@windriver.com>
Closes-Bug: #1795126
Story: 2003919
Change-Id: Ie619516c07fb3dc9d025f64c0e1e59d5d808cb6f
The externally consumed APIs from neutron.db.api were rehomed into
neutron-lib with https://review.openstack.org/#/c/557040/
This patch consumes the retry_db_errors function from lib by:
- Removing retry_db_errors from neutron.db.api
- Updating the imports for retry_db_errors to use it from lib
- Using the DB API retry fixture from lib in the UTs where applicable
- Removing the UTs for neutron.db.api as they are now covered in lib
NeutronLibImpact
Change-Id: I1feb842d3e0e92c945efb01ece29856335a398fe
The remainder of the neutron.plugins.common.utils were rehomed into
neutron-lib with [1][2]. This patch consumes them by using the functions
from neutron-lib, and removing the neutron.plugins.common.utils module
all together as it's fully rehomed now.
NeutronLibImpact
[1] https://review.openstack.org/#/c/560950/
[2] https://review.openstack.org/#/c/554546/
Change-Id: Ic0f7b37861f078ce8c5ee92d97e977b8d2b468ad
According to [1], when a network contains more that one IPv4
subnet, they are returned in the 'classless-static-routes'
DHCP option, regardless of whether DHCP is enabled for them
or not.
However, the get_active_networks_info() method used for
synchronizing networks after the dhcp agent restarts filters
subnets with "enable_dhcp=True", which differs from the
get_network_info() method. This will block VM access to
other VMs in the dhcp disabled subnets, even though they are
in the same network. This is visible by looking at the "opts"
file before and after a restart.
Change the dhcp agent to ask for all subnets in its
get_active_networks_info() RPC call by adding an
enable_dhcp_filter argument to toggle the behavior, with the
default being True to not break backwards compatibility.
Based on https://review.openstack.org/#/c/352530/ by Quan Tian.
[1] https://review.openstack.org/#/c/125043/
Change-Id: I11ca1d1a603d02587f3b8d4a5a52a96b0587d61f
Closes-Bug: #1652654
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.
Trivialfix
Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
The is_extension_supported function now lives in neutron-lib. This patch
removes the function from neutron and uses lib's version instead.
NeutronLibImpact
Change-Id: Iccb72e00f85043b3dff0299df7eb1279655e313e
The neutron-lib commit I360545b6ee4291547e0c5c8e668ad03d3efa4725 moved
the externally consumed globals from neutron.common.constants into lib.
With the exception of PROVISIONAL_IPV6_PD_PREFIX all other constants
in neutron.common.constants should only be used in neutron, and will
hopefully remain that way. External consumers needing access to other
common constants should move them into lib first.
NeutronLibImpact
Change-Id: Ie4bcffccf626a6e1de84af01f3487feb825f8b65
Changing rpc_api.rst file path from doc/source/devref/rpc_api.rst
to /doc/source/contributor/internals/rpc_api.rst. Because rpc_api.rst
file is located at this path
doc/source/contributor/internals/rpc_api.rst.
Closes-Bug #1722072
Change-Id: Ic243aab9e3428bfec69db61a94b4129cd768e233
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
In order to allow the DHCP agent to service other subnets
on the network in other segments via DHCP relay, we need to
return all subnets regardless of segment association.
However, this behavior will break older DHCP agents that
then try to get IPs on the subnets belonging to other segments.
This patch adds a new subnet attribute, 'non_local_subnets'
that will be returned in the DHCP RPC calls, so that agents
that can deal with off-link subnets can handle them
accordingly.
Change-Id: I9cce7b8a19c1201435df0c6baac7be57c57847e6
Partial-Bug: #1692486
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
Neutron-lib 1.1.0 is now out and contains the portbindings
API definition (as per commit [1]). This patch moves neutron
references over to the neutron-lib version.
NeutronLibImpact
- Consumers using the public constants within neutron's
portbindings API extension must now use the values
from neutron-lib.
[1] 87e42f993c07ae320159d5123662ee9f3bd4d903
Change-Id: I669af9b4c712877772d91a03857ab108714001d4
This is a runtime state users can easily create by creating tiny subnets
with no room for DHCP agent addresses. It's not something we need to
dump tracebacks on the server for because it's not a bug or unexpected
outcome.
These traces can be seen all over the API test logs where small subnets
are created without the need to actuall put things on them.
TrivialFix
Change-Id: I663b0df07608d8511a40c82e589875c50c7a6066
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
With multiple concurrent DHCP operations, the server log may be
filled with warning traces about the inability to complete the
DHCP action correctly. These should be emitted only when reraising
the underlying exception, namely when the underlying error is
not an expected nor a handled one.
Related-bug: 1621345
Related-bug: 1620864
Change-Id: I1f00609f8181e3765238550f45e6e114d2df7e6e
Capture port not found exceptions from port updates of DHCP ports
that no longer exist. The DHCP agent already checks the return
value for None in case any of the other things went missing
(e.g. Subnet, Network), so checking for ports disappearing makes
sense. The corresponding agent-side log message for this has also
been downgraded to debug since this is a normal occurrence.
This also cleans up log noise from calling reload_allocations on
networks that have already been torn down due to all of the subnets
being removed.
Closes-Bug: #1621650
Change-Id: I495401d225c664b8f1cf7b3d51747f3b47c24fc0
Commit 85ed7017ff removed the DBError
handling to let the retry decorator do its magic, however the
full implications of this change were not evaluated. As a result,
DBReferenceError (which derives from DBError) is not processed
correctly and that caused a regression of the existing logic.
Rather than bloat the retry's responsibility even further, this
patch partially reverts commit 85ed7017f by narrowing down the
exception handling to DBReferenceError only.
Related-bug: #1618216
Change-Id: Icf4e5e4145dcdcdc710b8e42044467913ed01ec1
The DHCP port action handler has been catching DBErrors
since f1b9ac5a54, which is
well before we had the retry decorator to deal with these.
With the port action handler catching these, it means there
will not be retries on deadlocks or connection errors so
transient situations can result in a permanently broken
DHCP service for a network.
This removes the catch for DBError so the decorator can retry
the operation.
Closes-Bug: #1618216
Change-Id: I42031b481958bbfdb8f52902c294022717af7adf
The exception handler checking for concurrently deleted subnets
was missing InvalidInput, which comes from IPAM when a port creation
request references a subnet that doesn't exist.
This patch just catches that exception.
Change-Id: I14f9e4bddde845e3fef2b0d8649d3954ba5c93bd
Closes-Bug: #1618187
The dhcp agent will request network information from neutron-server.
neutron-server will give the dhcp agent the entire network tree back
to the agent including subnets. If that agent is on a host that
is mapped to a segment, subnets that do not belong to that
segment are also returned and the dhcp agent will attempt to create
or update the dhcp port based on those subnets. These requests
will fail because the ipam logic forbids it for multiple segments.
This patch strips out those subnets that do not belong to the host's
segment. If the segments plugin is not enabled or the network is not
a routed network then the neutron-server will return back to the agent
what it normally would without routed networks.
Change-Id: I379b56f510853c60bb52c2d710117fecfb9aca2e
Partially-Implements: blueprint routed-networks
This method was originally left for backwards compatibility during
an upgrade of neutron. It stopped being used in havana so I think it
should be safe to remove now.
Closes-bug: #1583835
Change-Id: I503d10ee59ba2fa046b5030e8281fd41a49ee74e
When a new subnet is added to a network, the network cache
is updated with the list of subnets regardless of which ones
have DHCP enabled. This changes the index order of the subnet
list which means that the tags used for each subnet change.
This means we must restart the process because the opts file
will be using different tags than the process args. This patch
implements that change. It also sorts the subnets on the RPC
side so the agent indexes don't change if subnets aren't
added/deleted.
The previous logic was only restarting the process when DHCP
enabled subnets changed, which meant that adding a DHCP disabled
subnet would break the association between the opts file tags and
the process arg tags, which led to the reported bug.
Closes-Bug: #1581918
Change-Id: If1452c0e8fe95eb94cd78c7a05b57aead75662b5
Sometimes an object requires multiple disjoint actors to complete
a set of tasks before the status of the object should be transitioned
to ACTIVE. The main example of this is when a port is being created.
The L2 agent has to do its business to wire up the VIF, but at the same
time the DHCP agent has to setup the DHCP reservation. This led to
Nova booting the VM when the L2 agent was done even though the DHCP
agent may have been nowhere near ready.
This patch introduces a provisioning blocks mechansim that allows the
entities to be tracked that need to be involved to make a transition
to ACTIVE happen. See the devref in the dependent patch for a high-level
view of how this works.
The ML2 code is updated to use this new mechanism to prevent updating
the port status to ACTIVE without both the DHCP agent and L2 agent
reporting that the port is ready.
The DHCP RPC API required a version bump to allow the port ready
notification.
This also adds a devref doc for the provisioning_blocks
module with a high-level overview of how it works in addition
to a detailed description of how it is used specifically with
ML2, the L2 agents, and the DHCP agents.
Closes-Bug: #1453350
Change-Id: Id85ff6de1a14a550ab50baf4f79d3130af3680c8
Python 3 deprecated the logger.warn method, see:
https://docs.python.org/3/library/logging.html#logging.warning
so we prefer to use warning to avoid DeprecationWarning.
Closes-Bugs: #1529913
Change-Id: Icc01ce5fbd10880440cf75a2e0833394783464a0
Co-Authored-By: Gary Kotton <gkotton@vmware.com>
This is a normal occurence as networks are created and
deleted. It's not something that an operator should be
warned about.
Change-Id: I3bb498a29d93a88b059a669d510d21b4c65ea014
Closes-Bug: #1554695
Wrong usage of "a" in the messages:
"'%s' is not a integer"
"Create a address scope"
"Return True if port has as a allowed address pair"
"But if a agent does not report its status"
"a ipv4 link-local address"
"Retrieve and return a extended information about a network"
"It could be a eui-64 address, a random IPv6 address"
"push a elastic-recheck query for it (see below)"
"is not a enforced requirement"
"a ovs_lib.VifPort object"
Should be:
"'%s' is not an integer"
"Create an address scope"
"Return True if port has as an allowed address pair"
"But if an agent does not report its status"
"an ipv4 link-local address"
"Retrieve and return extended information about a network"
"It could be an eui-64 address, a random IPv6 address"
"push an elastic-recheck query for it (see below)"
"is not an enforced requirement"
"an ovs_lib.VifPort object"
Totally 9 occurrences in Neutron base code.
Change-Id: I0f980fc8c98524db3d194ecb779f76abb37eb31c
Brian Haley