This patch implements the last code bits pending to
conclude the new DB engine facade migration.
Due to the resultant interactions in the modified code, is
not possible to submit smaller patches; this code must be
migrated at once.
Partially-Implements blueprint: enginefacade-switch
Signed-off-by: Slawek Kaplonski <skaplons@redhat.com>
Co-Authored-By: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
Change-Id: Id3f09b78c8d0a8daa7ec4fa6f5bf79f7d5ab8f8b
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
The functionality from model_query is already in neutron-lib and
consumers are using it. This patch removes the _model_query module
from neutron and updates all imports to use neutron-lib's version of
it instead.
NeutronLibImpact
Change-Id: Ib2eae9edb009a93e60b3b0d63ca365056138566b
There is a analysis [1] suggested to run queries against specific
columns rather than full ORM entities to optimize the performance.
Right now, it is impossible to execute such optimization because
OVO doesn't support fetching specific column yet.
This commit introduces a new method 'get_values' in the base
neutron object class. Subclass of neutron object can leverage
this method to fetch specific field of a OVO. It supports fetching
non-synthetic fields only as syntheic fields are not directly backed
by corresponding DB table columns.
neutron-lib patch: https://review.openstack.org/#/c/619047/
[1] https://review.openstack.org/#/c/592361/
Needed-By: https://review.openstack.org/#/c/610184/
Change-Id: Ib90eae7738a5d2e4548fe9fed001d6cdaffddf3b
Partial-Implements: blueprint adopt-oslo-versioned-objects-for-db
The neutron.object.utils module was rehomed into neutron-lib with
https://review.openstack.org/#/c/557809/
This patch consumes it by removing the neutron.objects.utils module
and corresponding test module, updating the contributor internals for
objects and using lib's version of the module.
NeutronLibImpact
Change-Id: If53d0ad660851275462d2641ed1829cdb4c32d05
Except test_subnet, there are no more object tests that would require
the plugin configured, so let's not go through the hassle of setting it
up. (For the record, this setup was not needed since
If1252c42c49cd59dba7ec7c02c9b887fdc169f51).
Change-Id: Iafb886c17dbf9a91a3514b0caa1bcd14093a796c
New facade is enabled by setting new_facade = True for the object of
interest. With new_facade on, all OVO actions will use the new reader /
writer decorator to activate sessions.
There are two new facade decorators added to OVO: db_context_reader and
db_context_write that should be used instead of explicit
autonested_transaction / reader.using / writer.using in OVO context.
All neutron.objects.db.api helpers now receive OVO classes / objects
instead of model classes, since they need to know which type of engine
facade to use for which object. While it means we change signatures for
those helper functions, they are not used anywhere outside neutron tree
except vmware-nsx unit tests, and the latter pass anyway because the
tests completely mock out them disregarding their signatures.
This patch also adds several new OVO objects to be able to continue
using neutron.objects.db.api helpers to persist models that previously
didn't have corresponding OVO classes.
Finally, the patch adds registration for missing options in
neutron/tests/unit/extensions/test_qos_fip.py to be able to debug
failures in those unit tests. Strictly speaking, this change doesn't
belong to the patch, but I include it nevertheless to speed up merge in
time close to release.
There are several non-obvious changes included, specifically:
- in neutron.objects.base, decorator() that refreshes / expunges models
from the active session now opens a subtransaction for the whole span of
call / refresh / expunge, so that we can safely refresh model regardless
of whether caller opened another parent subtransaction (it was not the
case for create_subnetpool in base db plugin code).
- in neutron.db.l3_fip_qos, removed code that updates obj.db_model
relationship directly after corresponding insertions for child policy
binding model. This code is not needed because the only caller to the
_process_extra_fip_qos_update method refetches latest state of floating
ip OVO object anyway, and this code triggers several unit test failures.
- unit tests checking that a single commit happens for get_object and
get_objects are no longer valid for new facade objects that use reader
decorator that doesn't commit but close. This change is as intended, so
unit tests were tweaked to check close for new facade objects.
Change-Id: I15ec238c18a464f977f7d1079605b82965052311
Related-Bug: #1746996
In order to support LIKE statement filter in OVO objects,
StringMatchingFilterObj class is introduced. However, this class
does not provide initial attributes of "starts", "contains" and
"ends".
In real usage, if these attributes are not initialized, we will
hit failures in method apply_filters() in db/_model_query.py.
This patch adds the initialization back.
Secondly, a typo of "obj_utils.StringMatchingContains" is revised
in this patch too.
Change-Id: If828068d8d08ff09dff6c63d53320bc397d32448
Closes-Bug: #1724446
Function _kwargs_to_filters() in objects/db/api.py creates a filter
dictionary where all the values are made iterables except in the case
of StringMatchingFilterObj objects. As a consequence, when processing
such a filter, function apply_filters() in db/_model_query.py must
check if a value is a StringMatchingFilterObj before assuming it is
an iterable.
Change-Id: I1b43a06f6e35eaa61c6e6770076a167e6ee83987
Closes-Bug: #1724177
In some cases we may need to pass key=None filters to get_object()
and get_objects() methods, however we lack of NULL check before these
filters reach in_(), which will not return any matching queries in db
layer.
We need to do manual equals matches if NULL element exists in filters,
instead of pass them to in_() operator.
Change-Id: I7980b82e2627b7b097cae0a714d22e680cddd340
Closes-Bug: #1724177
Now that CommonDbMixin is just a shim we can stop using it.
Instead, use the model_query and resource_extend functions directly.
Related-Blueprint: neutron-lib
Change-Id: If1252c42c49cd59dba7ec7c02c9b887fdc169f51
Otherwise we risk a clash between multiple standard attribute records
created during a test run.
Change-Id: I965c6f9117a82aa8da20118c7b36a6dec32ac44b
Closes-Bug: #1683369
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for security groups and ports.
The object embeds segmentation information in addition to what's
available through the model. Specifically, binding_levels field exposes
all ml2 binding levels, that from their side load corresponding network
segment object. The order for level objects in binding_levels list field
is guaranteed to be in the order of level. So the consumers can eg.
access the bottom binding info with:
port_obj.binding_levels[-1].segment
For PortBindingLevel object, we want to expose segmentation info. This
is achieved through a 'segment' ObjectField. The database model itself
contains segment_id too. There is no reason though to expose it for
Level object in two places (one as a model field, another one through
the ObjectField), so we avoid adding ID field. The base class that
handles loading for ObjectField based synthetic fields was assuming that
objects always have a field per model attribute, so it needed a slight
adjustment to support this case, where we extract foreign_keys
attributes from the model itself if the field is not present on the
object.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Change-Id: I25de14e42e345d9235dbf4097c298ef5d606de51
Co-Authored-By: Martin Hickey <martin.hickey@ie.ibm.com>
Co-Authored-By: Rossella Sblendido <rsblendido@suse.com>
Co-Authored-By: Manjeet Singh Bhatia <manjeet.s.bhatia@intel.com>
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Those are needed to accommodate to API request needs without handling
sorting or pagination in Python.
Instead of adding four new arguments to get_objects interface, they are
consolidated using a single Pager object that is passed through the
_pager argument. The name uses underscore to avoid breaking objects that
want to pass 'pager' filter into SQLAlchemy. Hopefully, no objects will
ever have '_pager' attribute in their models.
Related-Bug: #1541928
Change-Id: I7dafc4dbd80f0ac35dbc2c2f30e56e441f5b1fc0
Slawek Kaplonski