Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/plugins.
Trivialfix
Change-Id: Id9138652f5f07ef12fa682e182fe210019e8f975
The goal of this patch is to make the Neutron code compliant
with SQLAlchemy 2.0.
All SQL transactions must be executed inside an explicit
writer/reader context. SQLAlchemy no longer will create an
implicit transaction if the session has no active transaction.
A warning message, only available in debug mode, is added. When
an ORM session calls "do_orm_execute", if there is no active
transaction, a warning message with a traceback will be logged
to help to debug the regression introduced.
Related-Bug: #1964575
Change-Id: I3da37fee205b8d67d10673075b9130147d9eab5f
Relocate "_delete_expired_default_network_segment_ranges" to avoid
code redundancy.
Now instead of searching all objects and deleting one by one,
"delete_objects" is used.
Trivial-Fix
Change-Id: I1753263cb15ce2988ac4ccae03b7395069f2c4e9
Now all Neutron ML2 drivers use OVO as input parameters to define the
allocation and the endpoints.
Reference of objects, inheritance, modifications made and presence in
other projects:
ML2TypeDriver (neutron-lib)
-> BaseTypeDriver
-> FlatTypeDriver
-> SegmentTypeDriver [1]
-> VlanTypeDriver
-> _TunnelTypeDriverBase
-> TunnelTypeDriver [2]
-> ML2TunnelTypeDriver
-> EndpointTunnelTypeDriver [3]
-> GeneveTypeDriver
-> GreTypeDriver
-> VxlanTypeDriver
[1] networking-avaya project inherits from SegmentTypeDriver, passing
the DB object instead of the OVO. This project was retired.
[2] TunnelTypeDriver class is not used anymore. networking-cisco
project imports this class if the version <= Ocata.
[3] networking-fujitsu project inherits from EndpointTunnelTypeDriver,
passing the DB object instead of the OVO. This project was retired.
Change-Id: If23d52e7839edf065619c327dc0cf47b5b560bfb
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
With python 3.x, classes can use the metaclass= logic
to not require usage of the six library.
One step in removing all of six usage from neutron.
Change-Id: I2f815e412d9a96eb5faf2b3bb3a1e393a9db9309
Just to make it clear in the message, add the
tunnel_ip_version config option in the error
message, else the user has to consult the startup
message to know what the value is set to.
Trivialfix
Change-Id: Ic8d8b0d454e8202d1bfae52b77137eb9071508da
When a tunnel with an invalid or unsupported type is to be created an
exception is raised but does not contain info for the offending host.
Add the host info and tunnel ip address to the exception error message
for better debugging
Change-Id: Id989f20043c252ff61ab9a6ee227e8c9c7c5029b
Closes-Bug: #1863888
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
This patch makes necessary changes to ML2 type drivers and plugin
manager for network segment range extension support when it is loaded.
When the network segment range extension is not loaded, no impact to the
current flow.
When the extension is loaded,
- populating a range that is managed from the configuration file [1]_,
such as "VLAN IDs", "VXLAN VNI IDs", "GRE tunnel IDs",
"Geneve VNI IDs" to the network segment range DB table as a "default"
and "shared" entry to maintain backward compatibility;
- reloading the "default" segment ranges when Neutron server
starts/restarts;
- creating a set of "default" network segment ranges out of the
ML2-config-file-defined ranges [1]_ and the segment allocation
operations are always retrieving the information from the DB to have
the network segment ranges fully administered via API;
- when a tenant allocates a segment, it will first allocate from an
available segment range assigned to the tenant, and then a shared
range if no tenant specific allocation is possible.
[1] /etc/neutron/plugins/ml2/ml2_conf.ini
Co-authored-by: Allain Legacy <Allain.legacy@windriver.com>
Partially-implements: blueprint network-segment-range-management
Change-Id: I522940fc4d054f5eec1110eb2c424e32e8ae6bad
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.
NeutronLibImpact
Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
The externally consumed APIs from neutron.db.api were rehomed into
neutron-lib with https://review.openstack.org/#/c/557040/
This patch consumes the retry_db_errors function from lib by:
- Removing retry_db_errors from neutron.db.api
- Updating the imports for retry_db_errors to use it from lib
- Using the DB API retry fixture from lib in the UTs where applicable
- Removing the UTs for neutron.db.api as they are now covered in lib
NeutronLibImpact
Change-Id: I1feb842d3e0e92c945efb01ece29856335a398fe
The get_reader_session and get_writer_session functions are available
in neutron-lib. This patch consumes them by removing the functions
from neutron and using lib's instead.
NeutronLibImpact
Change-Id: I4055172799a0d9a0ceb387b6d47e421e1c270ada
A bulk of the public APIs that are part of neutron.plugins.common.utils
were rehomed into neutron-lib with [1] and the remaining with [2].
This patch consumes [1] by:
- Removing the rehomed code from neutron.
- Removing the UTs that are no longer applicable.
- Leaving the functions in [2][3] in neutron.plugins.common.utils until
we release [2][3] and can consume it at which point we should be able to
remove the utils module.
NeutronLibImpact
[1] Iabb155b5d2d0ec6104ebee5dd42cf292bdf3ec61
[2] I2c0e4ef03425ba0bb2651ae3e68d6c8cde7b8f90
[3] I73f5e8ad7a1a83392094db846d18964d811b8bb2
Change-Id: I1d63cbea463e92e1d2e053f8e1a564ed52cb84f8
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.
Trivialfix
Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.
NeutronLibImpact
Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
neutron-lib contains a number of the plugin related constants from
neutron.plugins.common.constants. This patch consumes those constants
from neutron-lib and removes them from neutron. In addition the notion
of the dummy plugin service type is moved strictly into the test
package of neutron since it's not a real service plugin.
NeutronLibImpact
Change-Id: I767c626f3fe6159ab3abd6a7ae3cb9893b79bf66
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
This patch integrates the Oslo-Versioned Objects created for
VlanAllocation, VxlanAllocation, VxlanEndpoints, GreAllocation,
GreEndpoints, GeneveAllocation and GeneveEndpoints into the
code base.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I0d592bae452876b24c28ca4cc4bf6392b5ab6493
Co-Authored-By: Victor Morales <victor.morales@intel.com>
The ml2 MechanismDriver is now in neutron-lib along with its associated
constants. This patch switches over to the lib versions of those, but
leaves a shim of the MechanismDriver that just ref's the driver from
lib. This shim allows our broad consumer base of the driver to switch
over at their leisure.
NeutronLibImpact
Change-Id: I99e3de6d933a1bb341394f85415fb07306a82a01
Use reader and writer for db operations.
Partially-Implements blueprint: enginefacade-switch
Depends-On: Iba3520ac6cfb6b82b2013df9b8e1aee64b10a11c
Change-Id: I50be115ea69f805b48b02aebe4259ec2c839830e
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I8fb91bcb8ae4bfbfa68ddf7c38a4f1f5cf548b36
Just for consistency, get rid of the last direct call to
wrap_db_retry and instead use neutron.db.api.retry_db_errors
in the sync_allocations code so we don't potentially get
bitten by slight inconsistencies between the two.
TrivialFix
Related-Bug: #1673243
Change-Id: I975c7d0ec84698d277acbbec6d9cfec5c03439aa
This eliminates with_lockmode update in the VLAN and tunnel
type drivers in ML2. This allows us to avoid carrying with_lockmode
logic into OVO.
Removing with_lockmode exposed us to two potential races. One is
two servers inserting duplicate records at the same time at startup.
The other is deleting an allocation right as it was allocated by
another server.
The first race is simply solved by the retry decorators already
present on the functions because they will catch the duplicate
entry exception and start over.
The second is solved in this patch for VLANs by issuing an UPDATE
WHERE query inside the transaction to update 'allocated' values
to False where allocated is already False. This effectively blocks
any unallocated records we are attempting to delete from being
concurrently allocated while the transaction is open [1] before
the DELETE is committed.
The solution for the tunnel driver is more explicit since it just
emits a bulk delete statement which we can just attach a WHERE
statement to in order to ensure we don't remove allocated records.
1. Once the SQL server has given us a row count for an update
statement, it will not allow that count to be made invalid
by another query. An attempt to change their values will
result in directly blocking the changing query in a non-Galera
deployment or a deadlock error on commit in Galera which
we will retry.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I4fa553ef07c3cb1d05554629e121e86527982349
Introduce get_reader_session() and get_writer_session()
and replace get_session() with them.
Mark get_session as depricated.
Stop using get_engine from legacy facade. Use writer engine for
places where it is required.
Partially-Implements blueprint: enginefacade-switch
Change-Id: I28b741bfa27bf04cbe273586e6e3e00e14fbe683
We have a number of functions that expect to get session as one
of the arguments. Passing session is not correct and this prevents
of using new enginefacade which expects context to be passed and
session will be injected in current context.
NeutronLibImpact
Partially-Implements blueprint: enginefacade-switch
Change-Id: I68ac442f11a6698bc43210fae877ba3809b1c02d
The IPv6 header is twice the size of the IPv4 header, 40 vs 20
bytes, but the tunnel overhead constants are static, only
accounting for an IPv4 header in all cases. In order to be
correct it needs to treat the tunnel overhead different from
the IP overhead at L3.
This required removing the 20 byte IP overhead from the tunnel
type overhead constants and creating a new option,
ml2.overlay_ip_version, in order for the server to know which
version will be used, since it calculates the MTU for the network.
A version mis-match will now cause a tunnel sync to fail on
the server.
Moved all MTU tests to a common location to remove duplication.
DocImpact
Change-Id: Ia2546c4c71ff48b9fe2817fbad22b1fbf85f325b
Closes-bug: #1584940
If an object is updated in a compare and swap fashion, it
can fail when the change is flushed and raise a
StaleDataError if another process has updated the SQL record
at the same time. We need to catch these with the retry
decorator to restart the update process in a new transaction
with a fresh read of the latest state.
Partially-Implements: bp/push-notifications
Change-Id: I151ffcf47926f5ac66e452974f87e8bc2a906151
Add decorations where required.
There are roughly a dozen classes in Neutron that define abstract
methods or properties but are not decorated with
@six.add_metaclass(abc.ABCMeta). Without this decoration, children
can be created without defining the required methods or properties.
Decorating RBACColumns in db/rbac_db_models.py causes failures and will
be reported separately.
Decorating unit tests is of dubious benifit and should be addressed
separately (if at all).
There are also several more places where metaclassing isn't correct, to
be taken care of in follow-on patches.
For example, BaseScheduler is using the fact the None doesn't have a
filter_agents() method, which gives developers confusing error messages
when they incorrectly implement the interface, and they won't see any
error until schedule() is called. As an aside, the docstring for this
base class is also incorrect.
Change-Id: I2b2cce37d9b0d40559a715a7d510a969b8ba9963
Closes-Bug: #1577648
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
Change I75c6581fcc9f47a68bde29cbefcaa1a2a082344e introduced
a bug where host name changes broke tunneling endpoint updates.
Tunneling endpoint updates roaming a hostname from IP to IP
are a common method for active/passive HA with pacemaker and
should happen automatically without the need for API/CLI calls [1].
delete_endpoint_by_host_or_ip is introduced to allow cleanup of
endpoints that potentially belonged to the newly registered agent,
while preventing the race condition found when deleting ip1 & ip2
in the next situation at step 4:
1) we have hostA: ip1
2) hostA goes offline
3) hostB goes online, with ip1, and registers
4) hostA goes online, with ip2, and registers
[1] https://bugs.launchpad.net/python-neutronclient/+bug/1381664
Change-Id: I04d08d5b82ce9911f3af555b5776fc9823e0e5b6
Closes-Bug: #1464178
The oslo db retry decorator logs a traceback everytime a deadlock
is encountered even though it is being retried. With multiple workers
and a Galera cluster, deadlocks are common occurences due to our use
of with_lockmode update so we should not be polluting the logs.
This patch adjusts our usage of the retry decorator to catch deadlocks
with the exception checker which does not log them until the retries
are exhausted.
Change-Id: I433fbbad61070e20ebe934b9247e36fc190fa3e0
Currently sync_allocations[1] uses with_lockmode('update) which implies
possible deadlocks with Galera multi-writers. This change decorates the
method in order to catch and retry sync_allocations.
[1] neutron.plugins.ml2.drivers.type_tunnel
Change-Id: Ic01614cb5daf174848cf14a6aa4b38c4ed40fe1e
Currently gre/vxlan type drivers have specific sync_allocations
implementations to perform the same feature. This change abstracts
(vxlan) implementation and moves it to TunnelTypeDriver[1] in order to
share implementation between gre/vxlan; this change also adds some
unittests.
Current gre/vxlan and new implementations of sync_allocations are not
Galera-compliant with multi-writers, a follow-up will update it to
fully support Galera.
[1] neutron.plugins.ml2.drivers.type_tunnel
Change-Id: I188a7cf718d811084475f6783d844588de5d60ea
The pass is useless because there is a docstring in the methods.
Generally considered as uncovered by coverage tool.
Change-Id: Id1275c51e9adb865a3da9f0db007f3092b55b140
The latter only works in Python 2.
Also define a __next__ method in the classes that define a next method.
Change-Id: Iaa1a1e500facab50d8bcdffda39ccad3f2e4e9bb
Blueprint: neutron-python3
gre and vxlan type drivers have similar implementations for multiple
methods:
* get_endpoint_by_host
* get_endpoint_by_ip
* delete_endpoint
* get_endpoints
* add_endpoint
This change abstracts these methods and moves the abstractions to the
new class EndpointTunnelTypeDriver.
Change-Id: Iab97f8283b6bf5586334958de950664f6e74202a
ML2 will check the config parameters for the MTU settings.
It will check the segment, path and physnet mtu settings.
Change-Id: I58b57e01ec9bcafd7cdcfbf03149e98c3a1291ed
Partially-Implements: blueprint mtu-selection-and-advertisement
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
Description:
Stale GRE and VXLAN tunnel endpoints persists in neutron db this should be
deleted from the database. Also, if local_ip of L2 agent changes the
stale tunnel ports and flows persists on br-tun on other Compute Nodes and
Network Nodes for that remote ip this should also be removed.
Implementation
Plugin changes:
The plugin side changes are covered in following patch-set
https://review.openstack.org/#/c/121000/.
Agent changes:
Added tunnel_delete rpc for removing stale ports and flows from br-tun.
tunnel_sync rpc signature upgrade to obtain 'host'.
Added testcases for TunnelRpcCallbackMixin().
This patch-set agent deals with agent side changes.
Closes-Bug: #1179223
Closes-Bug: #1381071
Closes-Bug: #1276629
Co-Authored-By: Aman Kumar <amank@hp.com>
Co-Authored-By: phanipawan <ppawan@hp.com>
Change-Id: I291992ffde5c3ab7152f0d7462deca2e4ac1ba3f
Description:
Stale GRE and VXLAN tunnel endpoints persists in neutron db this should be
deleted from the database. Also, if local_ip of L2 agent changes the
stale tunnel ports and flows persists on br-tun on other Compute Nodes and
Network Nodes for that remote ip this should also be removed.
Implementation
Plugin changes:
Added host column in 'ml2_vxlan_endpoints' and 'ml2_gre_endpoints' table.
Added delete_endpoint method for deleting the stale endpoints from db.
Modified tunnel_sync() method to accommodate these changes.
Modified testcases in test_type_vxlan.py
Modified testcases in test_type_gre.py
Agent changes:
Added tunnel_delete rpc for removing stale ports and flows from br-tun.
tunnel_sync rpc signature upgrade to obtain 'host'.
Added testcases for TunnelRpcCallbackMixin().
This patch-set only deals with plugin side changes.
Partial-Bug: #1179223
Change-Id: I75c6581fcc9f47a68bde29cbefcaa1a2a082344e
Remove usage of the RpcProxy compatibility class from the ml2
AgentNotifierApi. The equivalent oslo.messaging APIs are now used
instead. A couple of other mixin APIs had to be converted at the same
time.
Note that there is one very minor functional change here. The base
rpc version is set to '1.0' now instead of '1.1'. The right pattern
to use is to always set the base to be N.0. Any method that needs a
newer version should specify it.
Part of blueprint drop-rpc-compat.
Change-Id: I640568e2d73c9eb7a9505db640dc1427a1ae2abe
Mostly trivial import changes.
- oslo.i18n no longer provide install() method to inject _() into
globals(), so removed all calls to it;
- removed Babel from dependencies (it will now be grabbed by oslo.i18n);
- updated tox.ini to ignore import violations for oslo.i18n.
Change-Id: I6623d551f512fb7fe9bf35ee734ed6d4c6cbc287
All the existing LOG.info, LOG.warning, LOG.error and LOG.critical
messages should have _LI, _LW, _LE and _LC respectively.
Also, debug level log shouldn't be translated.
This patch set will cover the ml2 directory under neutron/plugins.
Partial-Bug: #1320867
Change-Id: I9d78d23bbc14e7c536c6ddf2dc4f52c67faeb667
Brian Haley