The functionality within neutron.db.common_db_mixin is available via
neutron-lib APIs. This patch removes common_db_mixin and updates any
uses of it to use neutron-lib instead.
Depends-On: https://review.openstack.org/#/c/636159/
NeutronLibImpact
Change-Id: I2388f90b37abb09408809dda8c21da551bcd94bb
The following methods are no longer used in CommonDbMixin:
- register_dict_extend_funcs
- _apply_filters_to_query
- _filter_non_model_columns
This patch removes them from neutron.
NeutronLibImpact
Change-Id: Ic7042cdcb29e95cc3a13292819d77abc3971fe8a
The functionality from model_query is already in neutron-lib and
consumers are using it. This patch removes the _model_query module
from neutron and updates all imports to use neutron-lib's version of
it instead.
NeutronLibImpact
Change-Id: Ib2eae9edb009a93e60b3b0d63ca365056138566b
The _resource_extend module is already rehomed into neutron-lib and is
shimmed in neutron. This patch removes the module as no active
consumers are using it.
NeutronLibImpact
Change-Id: I1550075fa5fa2aa2f1a88ee7189d311a1fe78391
The register_model_query_hook and _apply_dict_extend_functions methods
from CommonDbMixin is not being used by consumers today. This patch
removes them and updates any doc references as well.
NeutronLibImpact
Change-Id: I3e72d7f33f5a7b0c9c023295302929410f94eefb
The APIs our consumers are using from neutron.db_utils were rehomed into
neutron-lib with https://review.openstack.org/#/c/540161/
This patch consumes them by removing the rehomed APIs and using lib's
implementation where applicable.
NeutronLibImpact
Change-Id: I7ee53bce917feae8e37bf278eb3121a5af47131c
The model_query_scope method is not used from common db mixin [1]. This
patch removes it on the path to eventually removing common db mixin
all together.
NeutronLibImpact
[1] http://codesearch.openstack.org/?q=%5C.model_query_scope%5C(
Change-Id: Ib8ce6e9f6000e9c01d28c1471b885b0f3c8a041b
The safe_reference property in common db mixin is only used by
vmware-nsx and they can use weakref directly (see depends on patch).
This patch removes safe_reference as we're working towards removing
common db mixin all together.
Depends-On: https://review.openstack.org/#/c/571221/
[1] http://codesearch.openstack.org/?q=%5C.safe_reference
Change-Id: I77f9561139fd88eff51638f6f41cf79f0b876dc1
A handful of the functions implemented in neutron.db._utils are exposed
in common_db_mixin as aliases. These are unneeded as they're already
defined elsewhere.
This patch removes the following aliases in common_db_mixin:
- safe_creation
- model_query_scope
- model_query
- resource_fields
It also renames the UT for common db mixin to
test__utils to reflect the fact that its testing functions
from the _utils db module.
NeutronLibImpact
Change-Id: I53cd533f107d950f163ad5b19b47097546a4691d
This refactoring is a step towards the goal of removing the
CommonDbMixin mixin class.
Related-Blueprint: neutron-lib
Change-Id: I1e2da0687310cc2da767dc2a6d13500307bba1ee
Move the CommonDbMixin methods to model_query.py and
resource_extend.py
This leaves CommonDbMixin as just a shim, and it should be easy
to remove once subprojects have stopped depending on it.
Related-Blueprint: neutron-lib
Change-Id: I5b804e09e630d88d551271d9731cc1f65c065259
By registering functions directly we cut off the dependency of the
"resource extend" functions on the plugin. This is a step towards
the goal of removing the CommonDbMixin mixin class.
Also, we register all "resource extend" functions at plugin create
(in __new__) instead of in the class definition (which caused the
hooks to be registered on import). This ensures the "resource
extend" functions are only registered for the plugins/mixins that
are actually used.
Note that decorators are used to register "resource extend" methods,
similar to the callback receiver decorators.
Related-Blueprint: neutron-lib
Change-Id: I128cfda773d5f9597df9cd61261fdc05f2a174aa
according to https://wiki.openstack.org/wiki/Python3, now we should avoid
using six.iteritems and replace it with dict.items.
Change-Id: I58a399baa2275f280acc0e6d649f81838648ce5c
Closes-Bug: #1680761
paginate_query was exploding when sort_keys had a
boolean column, since it was fixed in 4.18.0 of
oslo.db. This workaround is no longer needed.
Change-Id: Ie8df9c91c2828bb8886f9b175f06966fe9dd962c
Move the model query hook registration and resource extend funcs
registration methods out of the CommonDbMixin class and make them
regular utility functions.
This is a step in refactoring the CommonDbMixin class.
Change-Id: Iec1bb7f7098c83640ae695fd7cf2f4736f414ad2
This will prevent the common_db_mixin dictionary
extension functions and query hooks from stopping
the GC of plugins in tests and causing resource leaks.
Change-Id: I7576851a44abd14cbc337a3d3e28690c7316ec81
This avoids the logs being filled with warnings from
the sort utils about unstable sorting order.
"Unique keys not in sort_keys. The sorting order may be unstable."
Change-Id: I5cba69eb87abf1bec15fcc675369a725d67e23ce
This reverts commit 175bfe0482.
The change causes the following error if you're
unlucky enough to have such a marker column.
ArgumentError: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False
I saw test_floatingip_list_with_pagination failing,
where fixed_port_id of the marker was None.
Closes-Bug: #1656262
Change-Id: I6c32949d789f25d877d329ef0ae9d8650cb81acf
Extract all the common utils from common_db_mixin.py in preparation
for moving them to neutron-lib.
This is a preliminary step in preparation for refactoring the
CommonDbMixin class and moving it to neutron-lib also.
Partial Blueprint: neutron-lib
Change-Id: I3cba375a8162cb68e8f988f22f5c8b1ce7915180
apply_filters_to_query was performing an outerjoin to rbac_entries
unconditionally when model_query could have already performed an
outerjoin (if the request was from an unprivileged user) and/or when
the join wasn't even necessary (the '?shared=False' query that uses
a subquery and not a join). This resulted in terrible performance
because of cartesian products of rbac entries with themselves.
This fixes the issue by ensuring there is only an outerjoin to the
rbac table if it's going to be used for a filter condition and it's
not already joined because of a query scope imposed due to the user
not being privileged.
Unfortunately this doesn't include tests to prevent regressions because
we don't have any methods for testing the performance of individual
queries.
Closes-Bug: #1630939
Change-Id: I4364f4a97a29041e86b2fbd8aa895578153f4cf9
Update the API to accept project_id in requests and return
project_id in responses.
For now, the API treats tenant_id and project_id equivalently.
It accepts either or both in requests.
It returns both in responses, depending on filters.
We include an extension to indicate that support for project_id
is enabled in the API.
Completes: blueprint keystone-v3
APIImpact: Describe how the Networking API supports Keystone V3.
Co-Authored-By: Henry Gessau <HenryG@gessau.net>
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Change-Id: I8775aa8a477191ef21e7c3c6da31d098befefc3c
The paginate_query method was copied from nova which was copied
from glance. Now it is available in oslo_db.
Check and convert the sort keys and sort directions for
consumption by the oslo_db version of the method, and fix up
some grammar in the exception messages.
This work is related to the neutron-lib effort. The lib should
not propagate neutron's copy of paginate_query().
Related-Blueprint: neutron-lib
Change-Id: Ie7da16b94fa2023c9c3d84d96d55f33d0f76903f
This is unsafe when calling ML2 because ML2 assumes that its
functions will not be called inside of a transaction. This is
not only an issue for drivers that try to do DB lookups using
a different session in the post commit operation, but it's a
big issue for the delete methods.
The delete subnet and network methods in ML2 have 'while True'
loops that catch concurrency errors and retry the operation after
looking up info. If these are called inside a transaction, the
lookups will contain stale information and it can lead to the
while True loop never terminating!
Closes-Bug: #1551958
Change-Id: I33dc084ed15e5491fdda19da712a746ca87fbc8c
In order to give users and operators more flexibility in
annotating the purpose of various Neutron resources, this patch
adds a description field limited to 255 chars to all of the
Neutron resources that reference the standard attribute table.
The resource that reference the table are the following:
security_group_rules, security_groups, ports, subnets,
networks, routers, floatingips, subnetpools
This patch adds a description table related to standard attributes
and migrates over the existing security group description to the new
table as well.
Co-Authored-By: James Dempsey <jamesd@catalyst.net.nz>
APIImpact
DocImpact: Adds a description field to all resources outline in
commit message.
Closes-Bug: #1483480
Change-Id: I6e1ef53d7aae7d04a5485810cc1db0a8eb125953
The union model approach was completely broken because it
didn't keep track of which model each result actually was.
This patch just strips it out and replaces get_rbac_policies
with queries to each model. This will mean pagination is broken
once multiple rbac types are in place, but everything else should
work fine.
Co-Authored-By: Haim Daniel <hdaniel@redhat.com>
Closes-Bug: #1542815
Change-Id: I1e91aa22d093d50e5a9d318f24d09bb65e072246
Methods _create_ha_network, add_ha_port don't have wrapping
transaction in them, so they are prone to race conditions.
This commit adds a transaction to them. To avoid problem with
rolling back outmost transaction during exception handling,
internal methods have been wrapped in nested transaction.
Nested transaction is required in places like this:
def create():
create_something()
try:
_do_other_thing()
except Exception:
with excutils.save_and_reraise_exception():
delete_something()
def _do_other_thing():
with context.session.begin(subtransactions=True):
....
When exception is raised in _do_other_thing it
is caught in except block, but the object cannot be deleted in
except section because internal transaction has been rolled back.
A new method safe_creation and has been added
that provides a common way of handling such situations.
Closes-bug: #1501686
Change-Id: I952f6f7f8684743aa7f829bd92b1dc41b2c6aecf
Commit 5d53dfb8d6 removed the
method _get_tenant_id_for_create. This is used by various plugins
and the *aaS libaries.
Change-Id: I6d5e2555d6c198102a3d5400609f1d671e0d388d
The check of the tenant done in the method _get_tenant_id_for_create()
is already did by the Neutron Controller in prepare_request_body(),
with a call to attributes.populate_tenant_id().
Moreover, when the Controller processes a "create" requests, it
will add the 'tenant_id' to the resource dict.
Thus, _get_tenant_id_for_create() can be deleted.
Calls to this method are replaced by the res['tenant_id'].
Changes have to be done in UT to explicitly add the tenant_id while
creating resources, since the UT framework is bypassing the controller code
that automatically adds the tenant_id to the resource.
Co-Authored-By: Hong Hui Xiao <xiaohhui@cn.ibm.com>
Closes-Bug: #1513825
Change-Id: Icea06dc81344e1120bdf986a97a6b1094bbb765e
Depends-On: I31022e9230fc5404c6a94edabbb08d2b079c3a09
Depends-On: Iea3f014ef17a1e1b755cd2efe99afd1a36ebbc6a
Depends-On: I604602d023e0cbf7f6591149f914d73217d7a574
The _apply_filters_to_query method did not handle UnionModels
so objects leveraging it (i.e. RBAC policies) did not have
queries applied to them.
This patch corrects it by iterating through the component models
of the UnionModel and applying the filters to each component model.
It also adds an API test on RBAC that exercises the filtering.
Change-Id: I449acf359dd61189bbdacd200d7c41a4a88d3de8
Closes-Bug: #1517818
The subnet table was joined to the rbac table via an association proxy
which required a bunch of hacking in the model_query setup to get it
to work right.
This patch simplifies it quiet a bit by just using a direct relationship
between the subnets and networkrbacs tables joined via the subnet's
networkid.
It also unrolls an API test that was really difficult to debug because
it was hard to tell which iteration it was on.
Change-Id: I4da85218158aae624835b97053da9fbb6fb154ef
The query to find networks that aren't shared to the querier was
broken. It was querying for the inverse of RBAC entries that shared
to the querier, so it would return the network for each other tenant
it was shared to. This meant that if a network had multiple RBAC
entries, a shared=False filter wouldn't work in the API.
This patch corrects the behavior by adjusting the query that looks
for objects not shared to the caller to make sure the object ID doesn't
appear in the shared subquery.
This patch also adds a test that reliably reproduces the original issue.
The sporadically failing filter test that revealed this issue depended
on a race to have a network be shared to another tenant and to the wildcard
at the same time.
Change-Id: I9dcd869c1640b223221ba12e97284bbfcabbeb2b
Closes-Bug: #1495040
This adds the new API endpoint to create, update, and delete
role-based access control entries. These entries enable tenants
to grant access to other tenants to perform an action on an object
they do not own.
This was previously done using a single 'shared' flag; however, this
was too coarse because an object would either be private to a tenant
or it would be shared with every tenant.
In addition to introducing the API, this patch also adds support to
for the new entries in Neutron networks. This means tenants can now
share their networks with specific tenants as long as they know the
tenant ID.
This feature is backwards-compatible with the previous 'shared'
attribute in the API. So if a deployer doesn't want this new feature
enabled, all of the RBAC operations can be blocked in policy.json and
networks can still be globally shared in the legacy manner.
Even though this feature is referred to as role-based access control,
this first version only supports sharing networks with specific
tenant IDs because Neutron currently doesn't have integration with
Keystone to handle changes in a tenant's roles/groups/etc.
DocImpact
APIImpact
Change-Id: Ib90e2a931df068f417faf26e9c3780dc3c468867
Partially-Implements: blueprint rbac-networks
This patch implements the database model required for the network
RBAC work. In addition it migrates the current network and subnet
'shared' attributes to leverage the new table.
'shared' is no longer a property of the DB model because its status
is based on the tenant ID of the API caller. From an API perspective
this is the same (tenants will see networks as 'shared=True' if the
network is shared with them). However, internal callers (e.g. plugins,
drivers, services) will not be able to check for the 'shared' attribute
on network and subnet db objects any more.
This patch just achieves parity with the current shared behavior so it
doesn't add the ability to manipulate the RBAC entries directly. The
RBAC API is in the following patch.
Partially-Implements: blueprint rbac-networks
Change-Id: I3426b13eede8bfa29729cf3efea3419fb91175c4
This also adds a check to neutron/hacking/checks.py that should catch this
error in the future.
Blueprint: neutron-python3
Change-Id: Ie7b833ffa173772d39b85ee3ecaddace18e1274f
This patch simply adds a version of model_query in
neutron.db.common_db_mixin which can be invoked without
having to declare a class which inherits the mixin.
To this aim, model_query_scope has been refactored as well.
As the model query function being introduced in this patch
cannot use model query hooks (and does not need to), the
method was re-implemented rather than bringing out of the
mixin as it has been done for model_query_scope.
This change will allow for developing DB APIs without
having to use the baseDB/mixin classes models used so far.
Related-Blueprint: better-quotas
Change-Id: I7a79980f626e9eaf2775711c8a25f508067e5716
In Python 3, there is no "basestring". In Python 3, "six.string_types" is
"basestring", and "str" in Python 3.
Change-Id: Ic22e932cbf3c4b75cd424f4b41428da869f197cf
Blueprint: neutron-python3
All the Neutron code was scanned for places where in_ is being used
and added checks to ensure that the input is not an empty sequence.
Change-Id: I1e27f94ea350ce1dfabdd7eb14e4397ca29e8eb7
Closes-Bug:1264579
Simplfy register_model_query_hook() and register_dict_extend_funcs().
Move register_dict_extend_funcs() into CommonDbMixin attribute
because the related class attribute, _dict_extend_functions, is
defined in CommonDbMixin. They should be defined in same class.
Change-Id: Ib7b6df0c236f1d0804941147cc7cd7902a611311
Add in a default "advsvc" user and the logic in the Neutron policy
infrastructure which will allow this user to create/get/update/delete
ports on other tenants networks, as well as view other tenants
networks. This is for the use case of letting advanced services have
a user to put ports on other tenants networks. By default, we do not
define any roles for the policy "context_is_advsvc", but rely on
operators to specify the likely value of "role advsvc".
DocImpact
Closes-Bug: #1331836
Change-Id: I94cb3383eb1fed793934719603f888dbbdbbd85a
Co-Authored-By: Susanne Balle <sleipnir012@gmail.com>
I8badc249ad021fdbdb2367b5416c72435ed58994 causes anything importing
neutron/tests/unit/services/vpn/device_drivers/_test_cisco_csr_rest.py
to exit since httmock isn't a dependency.
Fix all hacking issues in addition to the revert, as this patch fixes 'tox -epep8'.
And just a reverting the patch will cause pep8 to fail.
Fixes-Bug: #1340881
This reverts commit 7f0a8f09ab.
Change-Id: I373a8c8ab16eb387be6a451b8146642389081afa
db_base_plugin_v2 imports too much modules that are not necessary
usually, so extract CommonDBMixin in different file.
Plus using db_base_plugin_v2 for some types of modules can lead to
cycles in imports, this refactoring should resolve the issue.
Closes-Bug: #1340145
Change-Id: Idb027d7c5cee2d5bc7598f805c56c55fd4aca048
Boden R