In order to decide if a port is a hardware offloaded port by just
reading the OVN Logical_Switch_Port register, it is needed the
VNIC type and the binding profile cababilities.
The maintenance task will only update those LSP ports that belong
to direct Neutron ports to avoid an unnecessary database load.
Related-Bug: #1998608
Change-Id: I5febe9d3eeef6c5b5f6d972b9e8ebfef541458ac
The ``neutron-remove-duplicated-port-bindings`` script removes the
duplicated port binding registers ("ml2_port_bindings" table) that
have status=INACTIVE.
This patch also removes the corresponding port binding levels
("ml2_port_binding_levels" table) associated to those inactive port
bindings.
Closes-Bug: #2000078
Change-Id: I12fa0764cd0ff509f1859b61060d64cc5a54a7b9
A new script to remove the duplicated port bindings was added. This
script will list all ``ml2_port_bindings`` records in the database,
finding those ones with the same port ID. Then the script removes
those ones with status=INACTIVE. This script is useful to remove
those leftovers that remain in the database after a failed live
migration.
"dry_run" mode is possible if selected in "[cli_script] dry_run"
boolean config option. The duplicated port bindings are printed in
the shell but not deleted.
Related-Bug: #1979072
Change-Id: I0de5fbb70eb852f82bd311616557985d1ce89bbf
SR-IOV agent can handle ports with same MAC address (located in
different networks). The agent can retrieve, from the system, the
MAC address and the PCI slot; because the PCI slot is unique per
port in the same host, this parameter is used to match with the
Neutron port ID stored in the database (published via RPC).
RPC API bumped to version 1.9.
Closes-Bug: #1791159
Change-Id: Id8c3e0485bebc55c778ecaadaabca1c28ec56205
Added a new port extension: device profile (``port_device_profile``).
This extension adds the "device_profile" parameter to the "port" API
and specifies the device profile per port. This parameter is a
string.
This parameter is passed to Nova and Nova retrieves the requested
device profile from Cyborg. Reference:
https://docs.openstack.org/api-ref/accelerator/v2/index.html#
device-profiles
For backwards compatibility, this parameter will be "None" by
default.
Closes-Bug: #1906602
Depends-On: https://review.opendev.org/c/openstack/neutron-lib/+/767586
Change-Id: I1202a8388e64ae4270ef4ca118993504ae7c1731
Added a new port extension: NUMA affinity policy. This extension adds
the "numa_affinity_policy" parameter to the "port" API and specifies
the NUMA affinity policy per port.
This parameter is passed to Nova when a virtual machine is created.
Nova will use this information to schedule the virtual machine.
For backwards compatibility, this parameter will be "None" by default.
Depends-On: https://review.opendev.org/#/c/740058/
Closes-Bug: #1886798
Change-Id: Ie3d68c098ddb727ab8333aa1de4064e67a4f00a7
Subnet delete triggers dhcp port deletion but asynchronously,
therefore in the condition described in the bug report we may
get a conflict when deleting the segment too fast after the subnet.
Here we follow the example of how we auto-delete ports in net delete.
Please also find a fullstack test in Related-Change below.
Change-Id: Iba02f5a2211b18c2deb9097daad6be5e7d21faf8
Closes-Bug: #1878632
Related-Change: https://review.opendev.org/728904
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
Improve port retrieval in method
"_validate_auto_address_subnet_delete". Instead of requesting each
port individually, a single DB query is executed to retrieve all
the ports with IP allocation in a in a subnet.
Change-Id: I7875142ebecd17663e17847fb14997200d7ae5c8
Related-Bug: #1865138
Filter neutron_lib.objects.utils.NotIn, designed to work with OVO objects
was used in db query and because of that ports belong to network wasn't
filtered by vif_type thus if there were any ports in network, it was not
possible to change its segmentation id.
Change-Id: I6ad2047c310e111bffe0942af12af1f642963b18
Closes-Bug: #1832985
Added field "qos_network_policy_id" to Port OVO. This parameter
will be used to retrieve the QoS policy bound to the port network.
This reduces the number of calls to the database by creating a join
between the QosNetworkPolicyBinding table and the Port table, based
on the network ID.
This backref association is not persistent (marked as "viewonly").
This relationship is using for loading the QoS policy ID of the
port network in the Port OVO.
Related-Bug: #1834484
Change-Id: I219a925d5e269b8c73a0481daa879d72c399fd8f
This patch switches the code over to use neutron-lib's test tools module
where appropriate rather than using neutron's.
This includes removing the following functions/classes from neutron and
using them from lib instead:
- get_random_EUI
- get_random_ip_network
- reset_random_seed
- OpenFixture
Change-Id: I0fbfcc7919f1b17b6bb0026fa9b98f157168255e
In case of Smart NIC vNIC type neutron should mimic nova-compute
that plug the port to the ovs bridge.
Extend the Neutron OVS mechanism driver and Neutron OVS Agent to bind
the Neutron port for the baremetal host with Smart NIC. This will allow
the Neutron OVS Agent to configure the pipeline of the OVS running on
the Smart NIC and leverage the pipeline features such as: VXLAN,
Security Groups and ARP Responder.
Story: #2003346
Closes-Bug: #1785608
Change-Id: I6d520d3bac2e9ceb30b5b6197c6eb0f958cc3659
When code needs to load a list of all ports that are not router
interfaces, they tend to fetch the port information into memory and then
iterate over the ports, inspecting each port's device_owner. For large
numbers of ports this is presumably inefficient. This came up in a
discussion of just such a part in the code [1].
A more efficient solution is to perform this filtering in SQL. This
change adds an argument to an existing method in the Port object to do
that.
[1] https://review.openstack.org/580721
Change-Id: I742d8a79175209e42f8bd6319956e960c73201ee
This patch integrates Port Binding Level OVO in /plugin/ml2/db.py
and introduces context instead of session for usage in object
operations.
Change-Id: Ifa779f5f70a7502bd96b34d64a84d272af2a6886
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Co-Authored-By: Anindita Das <anindita.das@intel.com>
Co-Authored-By: Slawek Kaplonski <slawek@kaplonski.pl>
As a consequence of implementing multiple bindings for ports, [1] made
the following attributes lists:
- 'port_binding' in the in the SQLAlchemy Port model
- 'binding' in the Port OVO
This patch pluralizes their names to 'port_bindings' and 'bindings'
respectively
[1] Ie31d4e27e3f55edfe334c4029ca9ed685e684c39
Change-Id: I4ebe47cf9d51a700310aad8dcccc82fea3f00a16
Functionality is added to the ML2 plugin to handle multiple port
bindings
Co-Authored-By: Anindita Das <anindita.das@intel.com>
Co-Authored-By: Miguel Lavalle <miguel.lavalle@huawei.com>
Partial-Bug: #1580880
Change-Id: Ie31d4e27e3f55edfe334c4029ca9ed685e684c39
When an object field has the value of corresponding model attribute set to
None, it means that it's indeed unset, and there is no need to refetch it from
the database (it will still be None).
This becomes a more noticeable problem when we introduce security field to
Network (part of I57395d0f646ffa3089c1ac6c5a68d952ccd0b42c) that is exactly
this type of field, because with the auto-expiry patch included
(I0d65d19204da8ce30addfa5faff68544534b7853) those redundant fetches trigger
actual SELECT statements that affect performance and break
test_get_objects_queries_constant regression test case.
The patch also changes the type of 'distributed_binding' Port object field from
ObjectField to ListOfObjectFields, and also renames the field into
'distributed_bindings' to reflect actual possibilities allowed by database
schema. Note this patch is NOT expected to enable actual support for multiple
binding values, which would belong to a separate patch, but just attempts to
apply the minimal needed changes required because of the other changes included
while making sure that consequent changes for the field don't need to change
object definition.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I833b07fdbb4a57bdf8bd4255e61098ec512d1a5b
The test class was not inheriting from testlib_api.SqlTestCase which
means that the test cases were never executed.
Once it was executed, it became obvious that the object has an issue
with SQL query scalability. Specifically, it was failing the
test_get_objects_queries_constant test case that validates that the
number of queries to fetch a single object is the same as for multiple
objects. This happens because PortBindingLevel's 'segment' synthetic
field is not constructed from db model relationship but through issuing
a get_object OVO call that triggers a query per fetched PortBindingLevel
object.
To fix the issue, the patch added a new 'segment' relationship to
PortBindingLevel db model, and made OVO use attributes from the
relationship to construct 'segment' synthetic field.
Another issue revealed by enabling the test class is that foreign_key
definition for the 'segment' synthetic field located in NetworkSegment
object refers to 'segment_id' field but the field with this name is not
defined on PortBindingLevel object. The easiest thing we can do to
resolve the discrepancy is adding the 'segment_id' UUID typed field to
PortBindingLevel, which is what the patch does. (An alternative could be
hacking our way around the lack of the field, building another layer of
complexity in the base OVO class for this special case; I figured it's
probably counterproductive, and we can live with 'segment_id' field even
though it's arguably somewhat redundant - 'segment' object field already
carries the ID.)
Adding a new field to PortBindingLevel object means that we need to bump
its version, as well as versions of all objects that use the newly
modified field type for their own fields. The only object that refers to
the type is Port. That's why we also bump the version of Port object.
To make older agents that use RPC callbacks work with the change, we
also provide corresponding obj_make_compatible methods for both Port and
PortBindingLevel objects.
Change-Id: I1ec56c57f87218520c2080c0b3450f1eabc13224
New facade is enabled by setting new_facade = True for the object of
interest. With new_facade on, all OVO actions will use the new reader /
writer decorator to activate sessions.
There are two new facade decorators added to OVO: db_context_reader and
db_context_write that should be used instead of explicit
autonested_transaction / reader.using / writer.using in OVO context.
All neutron.objects.db.api helpers now receive OVO classes / objects
instead of model classes, since they need to know which type of engine
facade to use for which object. While it means we change signatures for
those helper functions, they are not used anywhere outside neutron tree
except vmware-nsx unit tests, and the latter pass anyway because the
tests completely mock out them disregarding their signatures.
This patch also adds several new OVO objects to be able to continue
using neutron.objects.db.api helpers to persist models that previously
didn't have corresponding OVO classes.
Finally, the patch adds registration for missing options in
neutron/tests/unit/extensions/test_qos_fip.py to be able to debug
failures in those unit tests. Strictly speaking, this change doesn't
belong to the patch, but I include it nevertheless to speed up merge in
time close to release.
There are several non-obvious changes included, specifically:
- in neutron.objects.base, decorator() that refreshes / expunges models
from the active session now opens a subtransaction for the whole span of
call / refresh / expunge, so that we can safely refresh model regardless
of whether caller opened another parent subtransaction (it was not the
case for create_subnetpool in base db plugin code).
- in neutron.db.l3_fip_qos, removed code that updates obj.db_model
relationship directly after corresponding insertions for child policy
binding model. This code is not needed because the only caller to the
_process_extra_fip_qos_update method refetches latest state of floating
ip OVO object anyway, and this code triggers several unit test failures.
- unit tests checking that a single commit happens for get_object and
get_objects are no longer valid for new facade objects that use reader
decorator that doesn't commit but close. This change is as intended, so
unit tests were tweaked to check close for new facade objects.
Change-Id: I15ec238c18a464f977f7d1079605b82965052311
Related-Bug: #1746996
Filtering of port OVO based on ids of security groups which
are used by ports is now available.
Closes-Bug: 1744447
Change-Id: Ie5a3effe668db119d40728be5357f0851bdcebbe
This is needed to retrieve all ports in a given set of
security groups.
Partially-Implements: blueprint push-notifications
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: Iffa1bd341d9d20277ec153aa1dac6f61f05ec5bd
Added QoS policy port binding OVO. Direct access to 'qos_port_policy_bindings'
DB table is removed.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I40268f84bef808dcebd5c123fafc7a50a4d036f4
This patch implements the "default" behaviour for QoS policies.
If this flag is enabled for a QoS policy in a project, all
new networks created will have this QoS policy assigned by default.
If a new QoS policy is created or updated with this flag and another
QoS policy in the same project is set as the default policy, the new
one won't be created or updated. To set another QoS policy as default,
the current one must be unset.
DocImpact: A "default" flag is introduced for QoS policies. If this flag
is enabled in a QoS policy (attached to a project), then all
networks created in this project would have this QoS policy
assigned, unless an explicit policy is specified.
APIImpact
Closes-Bug: #1639220
Change-Id: If5ff2b00fa828f93aa089e275ddbd1ff542b79d4
Most of the _create_test methods create a property and only uses
its id during their execution. This change pretends to simplify
the returned data and reduce the number of lines used for the
creation of test data used during the execution of Unit Tests for
Oslo-Versioned Objects.
Change-Id: Icfc251a7d460246e91e260a4ac503d10c4031170
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Implements the port data plane status extension. Third parties
can report via Neutron API issues in the underlying data plane
affecting connectivity from/to Neutron ports.
Supported statuses:
- None: no status being reported; default value
- ACTIVE: all is up and running
- DOWN: no traffic can flow from/to the Neutron port
Setting attribute available to admin or any user with specific role
(default role: data_plane_integrator).
ML2 extension driver loaded on request via configuration:
[ml2]
extension_drivers = data_plane_status
Related-Bug: #1598081
Related-Bug: #1575146
DocImpact: users can get status of the underlying port data plane;
attribute writable by admin users and users granted the
'data-plane-integrator' role.
APIImpact: port now has data_plane_status attr, set on port update
Implements: blueprint port-data-plane-status
Depends-On: I04eef902b3310f799b1ce7ea44ed7cf77c74da04
Change-Id: Ic9e1e3ed9e3d4b88a4292114f4cb4192ac4b3502
The creation of security group objects are duplicated in several
places. This patch pretends to consolidate this in a single function
where it can be reused in the future.
Change-Id: I3c2b30c258e665f47ea7d3cffdb6e7333b2944a8
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Add new field acting as full Pythonic dict equivalent for OVO.
Usage of DictOfStringField is not handling the case, when values should
be boolean, integer, float or list/dict.
Also use new field for Agent, PortBinding and DistributedPortBinding.
Change-Id: Iea3a6de89325401892b4e60ae1eb3a1d75b3ef31
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
This patch is to introduce a modified get_random_fields() method to
set the attributes of an object as currently get_random_fields() returns
the modified fields which are suitable to be stored in the database,
however when fields of the objects are assigned with these values
the type won't match. For example, if an object has a custom type field
like DictOfStringField, get_random_fields() generates dict of string
values, however while returning the values it converts the dict to
a string but the object still expects a DictOfStringField type.
Also, everytime get_random_fields() is used for initializing the fields
of an object, modify_fields_to_db() gets called twice, once in the
get_random_fields() method and once while creating the object.
Change-Id: I8a5fc6603e625b0b98d330f6abd8c6588b1a8c31
This method can be used by subclasses to inject specific values into
randomly generated values for objects, for example, when we need to link
those automatic test objects to other objects through foreign keys.
Change-Id: Ic32ebf5ef055b25e4c765a9bf71ca4657d3b3d70
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
This patch introduces Agent model class to Oslo Versioned Object.
Given the number of dependencies and methods that use the model class,
the integration patch will be created in separated patch.
Co-Authored-By: Tony Tan <tonytan198211@gmail.com>
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I4145b7f59b58a55c1a76b91618097ff69aee1568
vif_details is represented as an empty string in our DB schema
when it's not defined. The 'modify_fields_from_db' method was
incorrectly attempting to feed this directly into the json decoder.
Closes-Bug: #1638703
Change-Id: I7b13fb30de71bb4dce4dbab4ffe430e1d0ef8f3a
This patch will replace the creation of network with
Network object, previously object_db_api was directly
used given now object is introduced we can use it.
Change-Id: I11a73f033171151e97c868108eda39fe178722f4
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for networks.
Segments are part of the network object. Since we need to reduce the
number of SQL queries executed per resource, and we want to include
segmentation information for networks, Network model was extended with
segments relationship that makes the information available on every
network resource fetch from the database. This change required some
change in test_dhcp_agent_scheduler to expire a session used in the
tests to avoid obsolete segment state to be reused to validate
scheduling.
This implementation of the object is not complete for the job of
updating the resource in database. For example, tags are not yet exposed
on the object; also attributes like availability_zones, or external
network attributes, are not covered. Those attributes are hopefully
needed on server side only, so until we adopt the object for server
side, it should be ok to live without them.
Another database related thing still missing in this patch is lack of
RBAC support for the object. To complete this support,
a real get_bound_tenant_ids should be put onto the object, instead of
the current no-op stub.
This patch also includes some rearrangements that simplified the work.
Specifically, all network related objects are consolidated in the
neutron.objects.network module, instead of being scattered through the
code base. Also, some setup code from test_policy relevant to RBAC was
moved into the base test class so that it can be utilized by other RBAC
enabled objects, like network.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Change-Id: I5160d0ab9e8042c356229420739db0ce42842368
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for security groups and ports.
The object embeds segmentation information in addition to what's
available through the model. Specifically, binding_levels field exposes
all ml2 binding levels, that from their side load corresponding network
segment object. The order for level objects in binding_levels list field
is guaranteed to be in the order of level. So the consumers can eg.
access the bottom binding info with:
port_obj.binding_levels[-1].segment
For PortBindingLevel object, we want to expose segmentation info. This
is achieved through a 'segment' ObjectField. The database model itself
contains segment_id too. There is no reason though to expose it for
Level object in two places (one as a model field, another one through
the ObjectField), so we avoid adding ID field. The base class that
handles loading for ObjectField based synthetic fields was assuming that
objects always have a field per model attribute, so it needed a slight
adjustment to support this case, where we extract foreign_keys
attributes from the model itself if the field is not present on the
object.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Change-Id: I25de14e42e345d9235dbf4097c298ef5d606de51
Co-Authored-By: Martin Hickey <martin.hickey@ie.ibm.com>
Co-Authored-By: Rossella Sblendido <rsblendido@suse.com>
Co-Authored-By: Manjeet Singh Bhatia <manjeet.s.bhatia@intel.com>
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Bence Romsics