Remove the QoS constants from Neutron code. QoS constants are now
located in ``neutron_lib.services.qos.constants``.
This patch also reverts [1]. This patch was merged in order to
allow a newer neutron-lib release in "requirements". This test
was failing because the element order of the "VALID_RULE_TYPES"
list was different between Neutron and neutron-lib. That was
modifying the "QosRuleType" OVO hash.
[1]https://review.opendev.org/c/openstack/neutron/+/817940
Closes-Bug: #1950977
Related-Bug: #1922237
Change-Id: I31edea3cc0f4a284a773a35302997ca6069efc95
This patch implements support for CRUD operations for QoS minimum
packet rate, for example:
DELETE /qos/policies/$POLICY_ID/minimum_packet_rate_rules/$RULE_ID
Placement or dataplane enforcement is not implemented yet.
Partial-Bug: #1922237
See-Also: https://review.opendev.org/785236
Change-Id: Ie994bdab62bab33737f25287e568519c782dea9a
This patch adds new API extension to QoS service plugin
to allow CURD actions for packet rate limit (packet per
second) rule in Neutron server side.
NOTE: This patch will NOT implement the real functionality
in L2/L3 backend to limit the pps.
Co-Authored-By: NANALI <lin203@chinaunicom.cn>
Closes-bug: #1912460
Change-Id: Icc88accb88d9cec40c960c56f032c3c27317b42e
This reverts commit 062336e59b.
Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.
Related-Bug: #1920001
Conflicts:
neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
neutron/common/utils.py
neutron/db/address_group_db.py
neutron/services/segments/db.py
Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
In case when enforce_new_defaults is set to True and new policy rules
are used, context.is_admin flag isn't really working as it was with old
rules.
But in case when elevated context is needed, it means that we need
context which has full rights to the system. So we should also set
"system_scope" parameter to "all" to be sure that system scope queries
can be done with such elevated context always.
It is needed e.g. when elevated context is used to get some data from
db. In such case we need to have db query which will not be scoped to
the single project_id and with new defaults to achieve that system_scope
has to be set to "all".
Proper fix for that should be done in neutron-lib and it is proposed
in [1] already but as we are have frozen neutron-lib version for
stable/wallaby already this patch for neutron is temporary fix for that
issue.
We can revert that patch as soon as we will be in Xena development cycle
and [1] will be merged and released.
[1] https://review.opendev.org/c/openstack/neutron-lib/+/781625
Related-Bug: #1920001
Change-Id: I0068c1de09f5c6fae5bb5cd0d6f26f451e701939
assertItemsEqual was removed from Python's unittest.TestCase in
Python 3.3 [1][2]. We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.
NOTE(dmllr): added hacking check
[1] - https://bugs.python.org/issue17866
[2] - https://hg.python.org/cpython/rev/d9921cb6e3cd
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277
Change-Id: I7c20fec08e5dc9f67b34100c925ea6724bbd25f0
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
After change [1], if the system has a high number of security groups
with no associated RBAC entries, a non admin user owning only one
security group will experience unacceptable response times when
listing her security groups.
Change [1] added methods get_object and get_objects to class
RbacNeutronDbObjectMixin in neutron.objects.rbac_db, which retrieve with
and admin context all the objects (networks, subnets or security groups)
in the DB and then decide in memory whether the project that made the
query has access to them or not, based on their associated RBAC
policies. This change proposes to remove those methods and revert to
their counterparts in NeutronDbObject (neutron.objects.base), which use
a DB query scoped to the project to retrieve the objects based on their
associated RBAC policies by calling [2]. In this way, the potential
number of objects that are retrieved from the DB and that have to be
converted to OVOs is greatly reduced, improving significantly the
response time to the user.
[1] https://review.opendev.org/#/c/635311
[2] 7a58374fde/neutron_lib/db/model_query.py (L100)
Change-Id: Idd303778d83089da8fbeff40e3dda2bd19008d8e
Closes-Bug: #1830679
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
This patch moves the implementation of the function
_get_policy_obj to the policy object code scope.
Change-Id: I7057558a5ec32a55a37a6b93dabc997d69abfb98
Fixed all pep8 E265 errors and changed tox.ini to no longer
ignore them. Also removed an N536 comment missed from a
previous change.
Change-Id: Ie6db8406c3b884c95b2a54a7598ea83476b8dba1
New facade is enabled by setting new_facade = True for the object of
interest. With new_facade on, all OVO actions will use the new reader /
writer decorator to activate sessions.
There are two new facade decorators added to OVO: db_context_reader and
db_context_write that should be used instead of explicit
autonested_transaction / reader.using / writer.using in OVO context.
All neutron.objects.db.api helpers now receive OVO classes / objects
instead of model classes, since they need to know which type of engine
facade to use for which object. While it means we change signatures for
those helper functions, they are not used anywhere outside neutron tree
except vmware-nsx unit tests, and the latter pass anyway because the
tests completely mock out them disregarding their signatures.
This patch also adds several new OVO objects to be able to continue
using neutron.objects.db.api helpers to persist models that previously
didn't have corresponding OVO classes.
Finally, the patch adds registration for missing options in
neutron/tests/unit/extensions/test_qos_fip.py to be able to debug
failures in those unit tests. Strictly speaking, this change doesn't
belong to the patch, but I include it nevertheless to speed up merge in
time close to release.
There are several non-obvious changes included, specifically:
- in neutron.objects.base, decorator() that refreshes / expunges models
from the active session now opens a subtransaction for the whole span of
call / refresh / expunge, so that we can safely refresh model regardless
of whether caller opened another parent subtransaction (it was not the
case for create_subnetpool in base db plugin code).
- in neutron.db.l3_fip_qos, removed code that updates obj.db_model
relationship directly after corresponding insertions for child policy
binding model. This code is not needed because the only caller to the
_process_extra_fip_qos_update method refetches latest state of floating
ip OVO object anyway, and this code triggers several unit test failures.
- unit tests checking that a single commit happens for get_object and
get_objects are no longer valid for new facade objects that use reader
decorator that doesn't commit but close. This change is as intended, so
unit tests were tweaked to check close for new facade objects.
Change-Id: I15ec238c18a464f977f7d1079605b82965052311
Related-Bug: #1746996
neutron-lib contains the QoS constants. This patch removes them from
neutron and replaces all such uses to lib's QoS constants.
NeutronLibImpact
Change-Id: I6ed379b178a2b79fd14385a1d1e87d87eb04bfb9
The neutron-lib commit I360545b6ee4291547e0c5c8e668ad03d3efa4725 moved
the externally consumed globals from neutron.common.constants into lib.
With the exception of PROVISIONAL_IPV6_PD_PREFIX all other constants
in neutron.common.constants should only be used in neutron, and will
hopefully remain that way. External consumers needing access to other
common constants should move them into lib first.
NeutronLibImpact
Change-Id: Ie4bcffccf626a6e1de84af01f3487feb825f8b65
Added QoS policy network binding OVO. Direct access to
'qos_network_policy_bindings' DB table is removed.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I2a2037dae5a071bd220fa04b0988bc8e7172eb59
Added QoS policy port binding OVO. Direct access to 'qos_port_policy_bindings'
DB table is removed.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I40268f84bef808dcebd5c123fafc7a50a4d036f4
This patch implements the "default" behaviour for QoS policies.
If this flag is enabled for a QoS policy in a project, all
new networks created will have this QoS policy assigned by default.
If a new QoS policy is created or updated with this flag and another
QoS policy in the same project is set as the default policy, the new
one won't be created or updated. To set another QoS policy as default,
the current one must be unset.
DocImpact: A "default" flag is introduced for QoS policies. If this flag
is enabled in a QoS policy (attached to a project), then all
networks created in this project would have this QoS policy
assigned, unless an explicit policy is specified.
APIImpact
Closes-Bug: #1639220
Change-Id: If5ff2b00fa828f93aa089e275ddbd1ff542b79d4
This patch introduces the new parameter "direction" to
the QoS bandwidth limit rule. It will allow the creation
of bandwidth limit rules for either ingress or egress
traffic. For backwards compatibility the default direction
will be egress.
DocImpact: Ingress bandwidth limit available for QoS
APIImpact: New type of parameter for QoS rule in neutron API
Change-Id: Ia13568879c2b6f80fb190ccafe7e19ca05b0c6a8
Partial-Bug: #1560961
For version 1.2 or higher of QoS policy object it can contain
QoSMinumumBandwidtLimit rules and appending of such rule type
was missing in make_obj_compatible function.
Now such rules are appended to QoS policy.
Change-Id: I40d699db58c34e83272432376d1d59679a680db2
Closes-Bug: #1681440
Most of the _create_test methods create a property and only uses
its id during their execution. This change pretends to simplify
the returned data and reduce the number of lines used for the
creation of test data used during the execution of Unit Tests for
Oslo-Versioned Objects.
Change-Id: Icfc251a7d460246e91e260a4ac503d10c4031170
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Objects must use project_id and not tenant_id. The object framework
ensures that tenant_id is added as an extra field for backward
compatibility.
This patch reverts the workaround implemented in change
I4ec9340094bc51cd8aa6e5112bf8114aa26c2982 and implements a proper fix
by explicitly updating the objects.
Co-Authored-By: Artur Korzeniewski <artur.korzeniewski@intel.com>
Co-Authored-By: Darek Smigiel <smigiel.dariusz@gmail.com>
Closes-Bug: #1630748
Change-Id: Iab90bcab41655b2e210aea0e7581eb00b94ce5e5
Set standard_attr_id as property in DeclarativeObject class
to get access to standard_attr_id used during integration of
port and tag OVO.
Change-Id: I172e88be290e76f26bdd9044cc9486b3b9624ded
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
This patch is to introduce a modified get_random_fields() method to
set the attributes of an object as currently get_random_fields() returns
the modified fields which are suitable to be stored in the database,
however when fields of the objects are assigned with these values
the type won't match. For example, if an object has a custom type field
like DictOfStringField, get_random_fields() generates dict of string
values, however while returning the values it converts the dict to
a string but the object still expects a DictOfStringField type.
Also, everytime get_random_fields() is used for initializing the fields
of an object, modify_fields_to_db() gets called twice, once in the
get_random_fields() method and once while creating the object.
Change-Id: I8a5fc6603e625b0b98d330f6abd8c6588b1a8c31
In object base test framework _check_equal method was having
opposite argument order as assert* statements, this patch will swap
them to reflect the order followed in assert* statements.
Change-Id: I546027899985af3fadb09928af7b5b4ca4a0a3bc
This patch will replace the creation of network with
Network object, previously object_db_api was directly
used given now object is introduced we can use it.
Change-Id: I11a73f033171151e97c868108eda39fe178722f4
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for networks.
Segments are part of the network object. Since we need to reduce the
number of SQL queries executed per resource, and we want to include
segmentation information for networks, Network model was extended with
segments relationship that makes the information available on every
network resource fetch from the database. This change required some
change in test_dhcp_agent_scheduler to expire a session used in the
tests to avoid obsolete segment state to be reused to validate
scheduling.
This implementation of the object is not complete for the job of
updating the resource in database. For example, tags are not yet exposed
on the object; also attributes like availability_zones, or external
network attributes, are not covered. Those attributes are hopefully
needed on server side only, so until we adopt the object for server
side, it should be ok to live without them.
Another database related thing still missing in this patch is lack of
RBAC support for the object. To complete this support,
a real get_bound_tenant_ids should be put onto the object, instead of
the current no-op stub.
This patch also includes some rearrangements that simplified the work.
Specifically, all network related objects are consolidated in the
neutron.objects.network module, instead of being scattered through the
code base. Also, some setup code from test_policy relevant to RBAC was
moved into the base test class so that it can be utilized by other RBAC
enabled objects, like network.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Change-Id: I5160d0ab9e8042c356229420739db0ce42842368
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for security groups and ports.
The object embeds segmentation information in addition to what's
available through the model. Specifically, binding_levels field exposes
all ml2 binding levels, that from their side load corresponding network
segment object. The order for level objects in binding_levels list field
is guaranteed to be in the order of level. So the consumers can eg.
access the bottom binding info with:
port_obj.binding_levels[-1].segment
For PortBindingLevel object, we want to expose segmentation info. This
is achieved through a 'segment' ObjectField. The database model itself
contains segment_id too. There is no reason though to expose it for
Level object in two places (one as a model field, another one through
the ObjectField), so we avoid adding ID field. The base class that
handles loading for ObjectField based synthetic fields was assuming that
objects always have a field per model attribute, so it needed a slight
adjustment to support this case, where we extract foreign_keys
attributes from the model itself if the field is not present on the
object.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Change-Id: I25de14e42e345d9235dbf4097c298ef5d606de51
Co-Authored-By: Martin Hickey <martin.hickey@ie.ibm.com>
Co-Authored-By: Rossella Sblendido <rsblendido@suse.com>
Co-Authored-By: Manjeet Singh Bhatia <manjeet.s.bhatia@intel.com>
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Co-Authored-By: Victor Morales <victor.morales@intel.com>
This attribute duplicates self.db_objs[0], no need to store it
separately. Keeping it as is just makes it harder to write code that
need to modify the db_objs values used in some tests.
Change-Id: Ib552ef9844a3bbd4966a7b097e6ed346720e2f6b
to_dict() is used for conversions of OVO objects into
regular dictionaries to be used as plugin return values
to the API layer, etc. It provides the equivalent of
make_<resource>_dict that we use now (without the extension
processing). The values in these dictionaries should be
ready for representation in the API.
The issue was that the OVO to_dict() implementation was
placing complex types right into the dictionary which
would mean that the API would serialize them just by
calling str() on them (as part of json encoding). This
ignored the 'to_primitive' method defined on the OVO
type that defines how a field should be converted.
Therefore, when it came to timestamps to_dict() was
placing native datetime objects into the dictionary
which would convert to microsecond resolution, violating
the expected format of the OVO DateTime type.
This patch fixes the issue by calling 'to_primitive'
on each non-synthetic field in the to_dict() method
to ensure we match the expected format of the type before
we send it out the API.
Closes-Bug: #1620254
Change-Id: Ic0be54b1d4b23119e1458d4532e2f70bff0ff9f6
This should reflect the code under test better, and is needed for one of
patches in the review queue (I130609194f15b89df89e5606fb8193849edd14d8)
to pass some of those test cases.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: Id1ca4ce7b134d9729e68661cedb2f5556e58d6ff
This adds the standard attributes relationship to QoSPolicy.
This has the advantage of providing created_at and updated_at,
however, the primary reason is to ensure it has a revision number
so QoSPolicy objects can be safely used in push notifications.
Partially-Implements: blueprint push-notifications
Change-Id: I8c0c33eef5d53c704b609e5bc503f46f5caad1bb
This patch introduces the front end implementation for QoS
minimum bandwidth rule.
APIImpact: New type of parameter for QoS rule in neutron API
DocImpact
Change-Id: I6b619a96a2bfde164646c71409b671352bc6ce7d
Partial-Bug: #1560963
Limit the SQL query number when loading the synthetic fields for Neutron
DB objects. If synthetic fields is defined in SQL model as orm.relationship,
the objects are loaded with main object. No need to query for those resources
once more.
It has impact on performance in scale, to have constant number of SQL queries
when number of resources stored in DB is increasing.
Change-Id: Ie54d9f51642edec1f4533062eb8306b78a1b6f20
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
- Changes RULE_TYPE_DSCP_MARK to RULE_TYPE_DSCP_MARKING to conform
with the rules name.
- Added object versioning to qos related objects.
- qos/rule: Throws a QosRuleVersionUnavailable exception when
the QosDscpMarkingRule version is < '1.1'.
- removed test object version incrementation TODO from test_policy.py
- Object versioning can not be used to increment the object version.
Change-Id: I4f10ef3c1cbaa2a868de2b8e3abc4c39eb1f44c7
Partial-Bug: #1468353
The filter was working in Liberty, but was broken in Mitaka when
the field became synthetic as part of RBAC enablement for the resource.
_get_collection already knows how to handle the filter on database
level, so we just need to register the filter to pass it deeper into
db_api.
Change-Id: I40657bd15d9bf1e2bc6b0224254c916c35391cd1
Closes-Bug: #1591222
Those are needed to accommodate to API request needs without handling
sorting or pagination in Python.
Instead of adding four new arguments to get_objects interface, they are
consolidated using a single Pager object that is passed through the
_pager argument. The name uses underscore to avoid breaking objects that
want to pass 'pager' filter into SQLAlchemy. Hopefully, no objects will
ever have '_pager' attribute in their models.
Related-Bug: #1541928
Change-Id: I7dafc4dbd80f0ac35dbc2c2f30e56e441f5b1fc0
These methods were missing in the qos api.
These have been added to provide the ability to read the policy
and network/port binding, and will be accessed through the new
methods in the QosPolicy class.
Closes-bug: #1556812
Change-Id: I34e4851c0fec2cd037206868846644b99dfd7def
This patch adds the front end and back end implementation of QoS DSCP.
Associated patches that are dependent on this one:
* python-neutronclient: https://review.openstack.org/#/c/254280
* openstack-manuals: https://review.openstack.org/#/c/273638
* API Guide: https://review.openstack.org/#/c/275253
* Heat:
* Spec: https://review.openstack.org/#/c/272173
* QoSDscpMarkingRule resource: https://review.openstack.org/#/c/277567
* Fullstack tests: https://review.openstack.org/#/c/288392/
APIImpact - The API now supports marking traffic egressing from a VM's
dscp field with a valid dscp value.
Co-Authored-By: Nate Johnston <nate_johnston@cable.comcast.com>
Co-Authored-By: Victor Howard <victor.r.howard@gmail.com>
Co-Authored-By: Margaret Frances <margaret_frances@cable.comcast.com>
Co-Authored-By: James Reeves <james.reeves5546@gmail.com>
Co-Authored-By: John Schwarz <jschwarz@redhat.com>
Needed-By: I25ad60c1b9a66e568276a772b8c496987d9f8299
Needed-By: I881b8f5bc9024c20275bc56062de72a1c70c8321
Needed-By: I48ead4b459183db795337ab729830a1b3c0022da
Needed-By: Ib92b172dce48276b90ec75ee5880ddd69040d7c8
Needed-By: I4eb21495e84feea46880caf3360759263e1e8f95
Needed-By: I0ab6a1a0d1430c5791fea1d5b54106c6cc93b937
Partial-Bug: #1468353
Change-Id: Ic3baefe176df05f049a2e06529c58fd65fe6b419
Rodolfo Alonso Hernandez