Since [1], the physical network VLAN ranges parser populates the
ranges for those entries without a defined range, allowing all
valid VLAN ranges ([1, 4094]).
Some VLAN driver tests, relying on the previous implementation,
considered that the physical network without a defined VLAN
range does not have segments to allocated (those segments
are created on the fly by
"SegmentTypeDriverallocate_fully_specified_segment).
Since [1], all physical network segments are stored in the
"ml2_vlan_allocations" table and set as non allocated.
This patch also reverts [2]. When the physical networks are defined
in "network_vlan_ranges", there is no distinction between tenant
and provided networks; the physical network segments are assigned
by the user. It is possible to create a provider network without
defining the segmentation ID, it will be provided by the Neutron
VLAN driver, if there are free segments for the required physical
network.
[1]https://review.opendev.org/c/openstack/neutron-lib/+/779515
[2]1376df7873c2ac77c256ab2fed928de41a2c1d58
Closes-Bug: #1919280
Related-Bug: #1918274
Related-Bug: #1649750
Change-Id: I191e020ddb97dcf8fb41139d35bfd699e125379b
Relocate "_delete_expired_default_network_segment_ranges" to avoid
code redundancy.
Now instead of searching all objects and deleting one by one,
"delete_objects" is used.
Trivial-Fix
Change-Id: I1753263cb15ce2988ac4ccae03b7395069f2c4e9
A new pep8 style library must have been released which
is causing some new errors, E471 among them. Clean-up
on aisle 8.
Change-Id: I153abada74e8c522fe9866a239a36dbb8365a29e
In order to reduce the number of elements retrieved from the DB, this
patch, before processing the VLAN allocations per physical network,
deleted those registers belonging to any unconfigured physical network.
The VLAN registers per physical network are deleted using a bulk delete
operation, to speed up the process.
Those missing VLAN registers per network are now created using a bulk
insert operation, available in the ORM. This bulk operation speeds up
the sync process.
Change-Id: I8568e2277e157754aaff87a059a40e34e6a43e2b
Partial-Bug: #1862178
In case when vlan network was created with segmentation_id=0 and without
physical_network given, it was passing validation of provider segment
and first available segmentation_id was choosen for network.
Problem was that in such case all available segmentation ids where
allocated and no other vlan network could be created later.
This patch fixes validation of segmentation_id when it is set to value 0.
Change-Id: Ic768deb84d544db832367f9a4b84a92729eee620
Closes-bug: #1840895
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
This patch makes necessary changes to ML2 type drivers and plugin
manager for network segment range extension support when it is loaded.
When the network segment range extension is not loaded, no impact to the
current flow.
When the extension is loaded,
- populating a range that is managed from the configuration file [1]_,
such as "VLAN IDs", "VXLAN VNI IDs", "GRE tunnel IDs",
"Geneve VNI IDs" to the network segment range DB table as a "default"
and "shared" entry to maintain backward compatibility;
- reloading the "default" segment ranges when Neutron server
starts/restarts;
- creating a set of "default" network segment ranges out of the
ML2-config-file-defined ranges [1]_ and the segment allocation
operations are always retrieving the information from the DB to have
the network segment ranges fully administered via API;
- when a tenant allocates a segment, it will first allocate from an
available segment range assigned to the tenant, and then a shared
range if no tenant specific allocation is possible.
[1] /etc/neutron/plugins/ml2/ml2_conf.ini
Co-authored-by: Allain Legacy <Allain.legacy@windriver.com>
Partially-implements: blueprint network-segment-range-management
Change-Id: I522940fc4d054f5eec1110eb2c424e32e8ae6bad
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.
NeutronLibImpact
Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
The externally consumed APIs from neutron.db.api were rehomed into
neutron-lib with https://review.openstack.org/#/c/557040/
This patch consumes the retry_db_errors function from lib by:
- Removing retry_db_errors from neutron.db.api
- Updating the imports for retry_db_errors to use it from lib
- Using the DB API retry fixture from lib in the UTs where applicable
- Removing the UTs for neutron.db.api as they are now covered in lib
NeutronLibImpact
Change-Id: I1feb842d3e0e92c945efb01ece29856335a398fe
A bulk of the public APIs that are part of neutron.plugins.common.utils
were rehomed into neutron-lib with [1] and the remaining with [2].
This patch consumes [1] by:
- Removing the rehomed code from neutron.
- Removing the UTs that are no longer applicable.
- Leaving the functions in [2][3] in neutron.plugins.common.utils until
we release [2][3] and can consume it at which point we should be able to
remove the utils module.
NeutronLibImpact
[1] Iabb155b5d2d0ec6104ebee5dd42cf292bdf3ec61
[2] I2c0e4ef03425ba0bb2651ae3e68d6c8cde7b8f90
[3] I73f5e8ad7a1a83392094db846d18964d811b8bb2
Change-Id: I1d63cbea463e92e1d2e053f8e1a564ed52cb84f8
neutron-lib contains a number of the plugin related constants from
neutron.plugins.common.constants. This patch consumes those constants
from neutron-lib and removes them from neutron. In addition the notion
of the dummy plugin service type is moved strictly into the test
package of neutron since it's not a real service plugin.
NeutronLibImpact
Change-Id: I767c626f3fe6159ab3abd6a7ae3cb9893b79bf66
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
This patch integrates the Oslo-Versioned Objects created for
VlanAllocation, VxlanAllocation, VxlanEndpoints, GreAllocation,
GreEndpoints, GeneveAllocation and GeneveEndpoints into the
code base.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I0d592bae452876b24c28ca4cc4bf6392b5ab6493
Co-Authored-By: Victor Morales <victor.morales@intel.com>
The ml2 MechanismDriver is now in neutron-lib along with its associated
constants. This patch switches over to the lib versions of those, but
leaves a shim of the MechanismDriver that just ref's the driver from
lib. This shim allows our broad consumer base of the driver to switch
over at their leisure.
NeutronLibImpact
Change-Id: I99e3de6d933a1bb341394f85415fb07306a82a01
When an operator configured multiple network VLAN ranges, we
had no guarantee that we would actually allocate in the order
specified like we do with segmentation types. In *most* cases,
the natural database order would be that the returned available
segment IDs would come back in the order of the physnets because
the sync_allocations code inserted them in that order. So it
became something some deployments would rely on and operators
would get a surprise when suddenly an allocation would happen
from the later physnets while many VLANs were still available
in the first one.
This patch just strictly makes allocation attempts in the order
of the network_vlan_ranges so operators are always guaranteed that
the later physnets will not automatically be allocated from unless
the earlier physnets are exhausted.
Change-Id: I14ca9b7e1141199f3ec221184fbbe156f1f9e18b
Use reader and writer for db operations.
Partially-Implements blueprint: enginefacade-switch
Depends-On: Iba3520ac6cfb6b82b2013df9b8e1aee64b10a11c
Change-Id: I50be115ea69f805b48b02aebe4259ec2c839830e
This eliminates with_lockmode update in the VLAN and tunnel
type drivers in ML2. This allows us to avoid carrying with_lockmode
logic into OVO.
Removing with_lockmode exposed us to two potential races. One is
two servers inserting duplicate records at the same time at startup.
The other is deleting an allocation right as it was allocated by
another server.
The first race is simply solved by the retry decorators already
present on the functions because they will catch the duplicate
entry exception and start over.
The second is solved in this patch for VLANs by issuing an UPDATE
WHERE query inside the transaction to update 'allocated' values
to False where allocated is already False. This effectively blocks
any unallocated records we are attempting to delete from being
concurrently allocated while the transaction is open [1] before
the DELETE is committed.
The solution for the tunnel driver is more explicit since it just
emits a bulk delete statement which we can just attach a WHERE
statement to in order to ensure we don't remove allocated records.
1. Once the SQL server has given us a row count for an update
statement, it will not allow that count to be made invalid
by another query. An attempt to change their values will
result in directly blocking the changing query in a non-Galera
deployment or a deadlock error on commit in Galera which
we will retry.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I4fa553ef07c3cb1d05554629e121e86527982349
Those were moved under neutron.db.models start of Ocata. We should be
able to clean them up now.
NeutronLibImpact
Change-Id: Iadbf44d52ee8e30712807384152a29ce1a8b8f72
This patch raises InvalidInput to users when they create provider net
without seg_id if ml2_vlan_ranges just sets the physical net name. This
must be more friendly.
Closes-Bug: #1649750
Change-Id: I9831d1a5b5764dcb8ee94641a3afc3991e56f8c2
Introduce get_reader_session() and get_writer_session()
and replace get_session() with them.
Mark get_session as depricated.
Stop using get_engine from legacy facade. Use writer engine for
places where it is required.
Partially-Implements blueprint: enginefacade-switch
Change-Id: I28b741bfa27bf04cbe273586e6e3e00e14fbe683
We have a number of functions that expect to get session as one
of the arguments. Passing session is not correct and this prevents
of using new enginefacade which expects context to be passed and
session will be injected in current context.
NeutronLibImpact
Partially-Implements blueprint: enginefacade-switch
Change-Id: I68ac442f11a6698bc43210fae877ba3809b1c02d
Refactoring neutron ml2 drivers configuration options to be in
neutron/conf/plugins/ml2/drivers. This would allow centralization of
all configuration options and provides an easy way to import.
Change-Id: I825bfed769bff92de010733b16f4a4b5171a052c
Partial-Bug: #1563069
This patch will relocate the VlanAllocation db model from
plugins/ml2/drivers to db/models and separate it from mixins for OVO
implementation.
Change-Id: Ib544b22c9e1acaf4025e8cdace694b2000a4da90
Partial-Bug: #1597913
When multiple controllers are initially built, the ml2_vlan_allocations
will be empty and the neutron servers under different controllers will
try to populate the entry at the same time. This will cause this
DBDeadlock error as they try to access the db at the same time.
This patch set is to add db_api.retry_db_error decorator in the
_sync_vlan_allocations method to avoid this race condition.
Change-Id: I7c0e3ae515f592a5852e6decf6820b103f146761
Closes-bug: 1617499
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
This works with both Python 2 and 3, and should not have any performance
impact.
Change-Id: I2a14945c60de513b91c6f022ff5dcc503ce2a8ad
Blueprint: neutron-python3
The function `xrange` was renamed to `range` in Python 3.
* Remove `xrange` occurences so that Python 3 tests can pass. Use
`six.moves.range` instead to get the right function in both cases.
* Generalize the use of the efficient `range` (ex-`xrange`) in
critical sections (when iterating over large lists).
* Simplify code.
* Add a hacking check to prevent future usage of `xrange`.
Change-Id: I080acaaa1d4753619fbbb76dddba6d946d84e73f
Partially implements: blueprint neutron-python3
It is quite confusing to have values for network type in common.constants.py
instead of having in plugins.common.constants.py.
Currently, the plugins/common/constants.py consists network_type constants
like VLAN, VXLAN, GRE etc. but values for network type like ranges
are defined in common.constants.py which is not good, it is better to have
both things at the same place.
This patch set addresses the same.
Moved out few methods which are predominantly used in plugins
from common.utils.py to plugins.common.utils.py.
Removed constants which were used in neutron-fwaas from
plugins.common.constants.py: https://review.openstack.org/#/c/168709/
Closes-Bug: #1441043
Change-Id: Iecfb15c541ed5d3cce95ba48f072af7fa60ac6f1
ML2 will check the config parameters for the MTU settings.
It will check the segment, path and physnet mtu settings.
Change-Id: I58b57e01ec9bcafd7cdcfbf03149e98c3a1291ed
Partially-Implements: blueprint mtu-selection-and-advertisement
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
ml2_vxlan_allocations, ml2_gre_allocations, ml2_vlan_allocations tables
have the 'allocated' field.
There are a lot of similar queries to these tables which look
like the following:
SELECT ml2_vxlan_allocations.vxlan_vni
AS ml2_vxlan_allocations_vxlan_vni,
ml2_vxlan_allocations.allocated
AS ml2_vxlan_allocations_allocated
FROM ml2_vxlan_allocations
WHERE ml2_vxlan_allocations.allocated = 0 LIMIT 1;
Performing such selects can take quite a lot of time and if a transaction
which performs allocation is executed in parallel, it can lead to
allocation failure and retry.
Adding an index on "allocated" column significantly improves
the performance. For ml2_vlan_allocations table created
an index on (physical_network, allocation) together.
Example for MySQL for execution of query
select * from ml2_vxlan_allocations where allocated = 0;
when on the table with ~3 mln entries, ~500K of which
have allocated = 0:
+-----------------------+---------------------+
|No index on "allocated"| Index on "allocated"|
+---------------------------------------------+
| 2.02 sec | 0.43 sec |
+-----------------------+---------------------+
Closes-Bug: #1412348
Change-Id: Ie90ba611dcae6bd0cb7686a0c7b29b9484eae693
Mostly trivial import changes.
- oslo.i18n no longer provide install() method to inject _() into
globals(), so removed all calls to it;
- removed Babel from dependencies (it will now be grabbed by oslo.i18n);
- updated tox.ini to ignore import violations for oslo.i18n.
Change-Id: I6623d551f512fb7fe9bf35ee734ed6d4c6cbc287
All the existing LOG.info, LOG.warning, LOG.error and LOG.critical
messages should have _LI, _LW, _LE and _LC respectively.
Also, debug level log shouldn't be translated.
This patch set will cover the ml2 directory under neutron/plugins.
Partial-Bug: #1320867
Change-Id: I9d78d23bbc14e7c536c6ddf2dc4f52c67faeb667
Validate to make sure empty physical networks
are not permitted. Added unit test as well.
Change-Id: Ie55fab3c53276ed48d5e9d603e8c1dc59fa9cc32
Closes-Bug: 1325664
ML2 provider networks partial specs let admins choose some provider
network attributes and let neutron choose remaining attributes. This
change provides the implementation for VLAN provider networks.
In practice, for VLAN provider networks provider:physical_network
and provider:segmentation_id choices can be delegated to neutron,
in such case neutron will try to find a network in tenant network
pools which respects provided provider attributes.
DocImpact
Related to blueprint provider-network-partial-specs
Partial-Bug: #1330562
Change-Id: I2c52c71167edaa153b2e04681273e2f1be8d03aa
SELECT FOR UPDATE expression, which is triggered with the use of the
SQLAlchemy Query object's with_lockmode('update') method, is
detrimental to performance and scalability of the database
performance code in Neutron due to the lock contention it produces.
SELECT FOR UPDATE can be entirely avoided in release_segment methods
with the use of single-shot UPDATE and DELETE expressions, and this
patch clears a number of uses of SELECT FOR UPDATE by consolidating
multiple SQL expressions into one.
Partial-Bug: #1330562
Change-Id: I29ffcafc8d4d73ac1cb50c94df5da85514d47a3f
H302 violation is reported by flake8 when importing separated objects from
modules instead of importing the whole module.
e.g. from package.module import function
function()
is changed to
from package import module
module.function()
Change-Id: I83372124f4fba7b94bbfb4a56a0c0ef779ee237f
Partial-Bug: #1291032
in python3.x, there is no method named xrange(),
it has replaced by range(),which is equal to xrange() in python2.x.
so we must fix this issue.we use six module to fix this issue.
Change-Id: I9cee3f0df2d8abbf6f66d7c5a3a0fed98df622e8
Closes-Bug: #1268439
Rodolfo Alonso Hernandez