The method ``set_resources_quota_usage_dirty`` can be used now to set
the dirty bit for a single resource or multiple ones.
Trivial-Fix
Change-Id: I13ef43b71fe7a080d55a84119784433ad84380b6
This fixes two queries still being executed without the context
decorator, which is causing warning messages.
Closes-Bug: #1995738
Change-Id: I61692e09119751ac7e19cecf6bf0b4b3a9fb38c7
This patch implements a new method specific for each quota driver
class. This method, "get_resource_count", returns the current number
of resources created in a project of a tracked resource. A tracked
resource is an instance of ``neutron.quota.resource.TrackedResource``.
This method does not count the current reservations, just the actual
resources created.
This new method, "get_resource_count", will be added to the abstract
class ``neutron_lib.db.quota_api.QuotaDriverAPI``.
This patch also fixes ``TestDbQuotaDriverNoLock``, that was using a
plugin inheriting from ``DbQuotaDriver`` instead of
``DbQuotaNoLockDriver``.
Closes-Bug: #1982962
Change-Id: I2707506468cb60d93a4459ea364f1e79faa83838
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
some of them, about 10%.
Feel free to reject if we think it will cause too much
trouble with cherry-picks, else I'll slowly work my way
through the rest of the tree.
Trivialfix
Change-Id: I3d484d11e273cb8ee617f9445a069887e7b2b89f
The goal of this patch is to make the Neutron code compliant
with SQLAlchemy 2.0.
All SQL transactions must be executed inside an explicit
writer/reader context. SQLAlchemy no longer will create an
implicit transaction if the session has no active transaction.
A warning message, only available in debug mode, is added. When
an ORM session calls "do_orm_execute", if there is no active
transaction, a warning message with a traceback will be logged
to help to debug the regression introduced.
Related-Bug: #1964575
Change-Id: I3da37fee205b8d67d10673075b9130147d9eab5f
Enabled ``DbQuotaDriverNull`` as a productio quota database
quota driver. This driver does not enforce any quota nor have access
to the database. When using this quota driver, the API will return
the default empty values expected from the ``QuotaDriverAPI`` class.
Closes-bug: #1960032
Change-Id: Iafa24753e657746a8b8165b5a63c17de9a9ba791
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Co-Authored-By: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
The goal of [1] is to, in case of failing when removing the quota
reservation, continue the operation. Any expired reservation will
be removed automatically in any driver.
If the DB transaction fails, it should affect only to the reservation
trying to be deleted. This is why this patch isolates the
"remove_reservation" method and guarantees it is called outside an
active DB session. That guarantees, in case of failure, no other DB
operation will be affected.
This patch also partially reverts [2] but still checks the security
group rule quota when a new security group is created. Instead of
creating and releasing a quota reservation for the security group
rules created, now only the available quota limit is checked before
creating them. That won't prevent another operation to create security
group rules in parallel, exceeding the available quota. However, this
is not even guaranteed with the current quota driver.
[1]https://review.opendev.org/c/openstack/neutron/+/805031
[2]https://review.opendev.org/c/openstack/neutron/+/701565
Closes-Bug: #1943714
Change-Id: Id73368576a948f78a043d7cf0be16661a65626a9
The quota driver ``ConfDriver`` was deprecated in Liberty release.
``NullQuotaDriver`` is created for testing although it could be used
in production if no quota enforcement is needed. However, because
the Quota engine is not plugable (is an extension always loaded), it
could be interesting to make it plugable as any other plugin.
This patch also creates a Quota engine driver API class that should be
used in any Quota engine driver. Currently it is used in the three
in-tree drivers implemented: ``NullQuotaDriver``, ``DbQuotaDriver``
and ``DbQuotaNoLockDriver``.
Change-Id: Ib4af80e18fac52b9f68f26c84a215415e63c2822
Closes-Bug: #1928211
This reverts commit 062336e59b.
Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.
Related-Bug: #1920001
Conflicts:
neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
neutron/common/utils.py
neutron/db/address_group_db.py
neutron/services/segments/db.py
Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
per (resource, project_id) but retrieves the instant (resource,
project_id) usage and the current (resource, project_id) reservations.
If the requested number of resources fit the available quota, a new
``Reservation`` register is created with the amount of units requested.
All those operations are done inside a DB transaction context. That
means the amount of resources and reservations is guaranteed inside
this transaction (depending on the DB backend isolation level defined)
and the new reservation created will not clash with other DB transation.
That will guarantee the number of resources and instant reservations
never exceed the quota limits defined for this (resource, project_id).
NOTES:
- This change tries to be as unobtrusive as possible. The new driver
uses the same ``DbQuotaDriver`` dabatase tables (except for
``QuotaUsage``) and the same Quota engine API, located in
``neutron.quota``. However, the Quota engine resources implements some
particular API actions like "dirty", that are not used in the new
driver.
- The Pecan Quota enforcement hooks,
``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
"resync", "mark_resources_dirty" or "set_resources_dirty", that has
no meaning in the new driver.
- The isolation between the Quota engine and the Pecan hook, and the
driver itself is not clearly defined. A refactor of the Quota engine,
Quota service, Quota drivers and a common API between the driver and
the engine is needed.
- If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
``TrackedResource`` will be joined in a single class. This resource
class will have a count method (countable) or a hard dependency on a
database table (tracked resource). The only difference will be the
"count" method implementation.
Closes-Bug: #1926787
Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1
With python 3.x, six.text_type and six.string_type
are just str.
Also removed a six.integer_type since it was the only
one left in a file.
Another step in removing all of six usage from neutron.
Change-Id: I5208dc41bff1983ecd323286f427296b722da62a
The hook starts a DB transaction and should be covered with
DB retry decorator.
Closes-Bug: #1777965
Closes-Bug: #1771293
Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.
NeutronLibImpact
Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
The retry_if_session_inactive decorator was rehomed into neutron-lib
[1]. This patch consumes it by removing the function from neutron and
using neutron-libs version where appropriate.
NeutronLibImpact
[1] https://review.openstack.org/#/c/557040/
Change-Id: I3e3289f33e62d45933d0fbf165bb4b25078f22d5
Michael Bayer while analysing neutron process function call trace,
suggested to run queries against specific columns rather than full
ORM entities as it can help reduce load both at the DB level and
in the Python level since they are much faster to fetch as
non-ORM entities. In this patch we are trying that on simpler
queries to improve neutron performance.
Co-Authored-By: Joe Talerico <jtaleric@redhat.com>
Change-Id: I6a41e9487a4427f876442bbeeae61974e892225e
The public db apis are available in neutron-lib. This patch consumes
the sqla_listen, sqla_remove and sqla_remove_all functions from lib
by removing them from neutron and using lib's version.
There's also a TODO added here so we can eventually rehome the
check_no_sqlalchemy_event_import hacking check to neutron-lib.
NeutronLibImpact
Change-Id: I3b3862016125303fb227b4ff41420a89bb33c917
It's of no guarantee that core plugin implements counter/getter function
for a CountableResource. Instead of just trying core plugin, try every
plugin registered in the directory.
To retain backwards compatibility, we also make sure that core plugin is
checked first.
Change-Id: I5245e217e1f44281f85febbdfaf873321253dc5d
Closes-Bug: #1714769
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
In a gate run the neutron.quota.resource module logged 5235
lines from this module, making it one of the top offenders.
This single line accounted for 3297 of those entries and it
offers limited information since the quota engine is quite
stable at this point and debugging tenants' dirty status isn't
an issue that comes up.
We can just get rid of this particular log line.
Change-Id: I9cb0fcfbad51a0863d04fb89fbb8b09e3e21bdf0
Partial-Bug: #1707307
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.
DocImpact:Documentation describing the new API as well as the new
information that it exposes.
APIImpact
Co-Authored-By: Prince Boateng<prince.a.owusu.boateng@intel.com>
Change-Id: Ief2a6a4d2d7085e2a9dcd901123bc4fe6ac7ca22
Related-bug: #1599488
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I8fb91bcb8ae4bfbfa68ddf7c38a4f1f5cf548b36
As one of query to fetch quota usage by resource and tenant
was done in lock mode earlier. This was done to protect the
dirty bit in the usage which tracks for update. For moving to
OVO carrying lock mode can be avoided by making sure that dirty
attribute of quotausages is still protected.
Change-Id: I0bb9f6fb7be51be590afb527a56e0569e922e98f
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Register all sqlalchemy events through a new function in
neutron.db.api so we can keep track of active events and
ensure all are removed at the end of each test run.
Without this, an instance of a plugin may be left around
with the only reference to it existing in SQLAlchemy, where
it will receive events for tests unrelated to it and potentially
interfere.
Change-Id: I8e93eb4e8ef5a13f015db9cd20e44941cdcb72ef
The devref was suggesting to do exactly the wrong thing for cases where
we have SQLAlchemy event listeners waiting for deleted objects.
DELETE statements fail to trigger any kind of SQLAlchemy event
listeners and the majority of our delete operations are small
(<10 items at once) so the performance difference is negligible
(especially since we do it so infrequently).
Change-Id: Id142179418a7b94d8d9695871b3fcd5dcc64730c
This decorates the quota system operations with
the retry decorators. This will help significantly
with the bug this marks as closed since operations
in the quota engine after commit should no longer
trigger retries of the full API operation.
The logic to find the args in the decorator had
to be adjusted to deal with functions already decorated.
This just uses the getargspec function from pecan that
deals with decorated functions.
Partial-Bug: #1612798
Closes-Bug: #1596075
Change-Id: Ib786117dcea08af75551770ea4c30d460382b829
If a resource was registered in the quota engine before
being set as a tracked resource, it would have been registered
as a CountableResource instead of a TrackedResource.
This would cause a failure later when the resources were
being unregistered because the quota engine thought it
should have been a TrackedResource and tried to call
unregister_events on it.
This patch just prevents resources from being registered
as both types. It also callse unregister_all_resources as
part of the standard test case cleanup since the vast majority
of tests are using a plugin that may register resources.
Closes-Bug: #1612222
Change-Id: I8a40f38d7c0e5aeca257ba62115fa9b02ad5aa93
The quota engine was still using oslo_db wrap_db_retry
which does not automatically take into account deadlocks
that occur inside of nested savepoint transactions.
In one case it didn't matter because it passed in the correct
exception checker but in the one that protected 'set_quota_usage'
it did not use is_retriable so a deadlock inside of the savepoint
would have resulted in a much more expensive retry all of the way
up at the API layer.
This patch just adjusts them to both use the standard neutron
retry_db_errors decorator.
Change-Id: I1e45eb15f14bf35881e5b1dce77733e831e9c6b1
Related-Bug: #1596075
It was deprecated at [1], and quota of resource will be registered
at initialization of APIRouter. So, no need to use the config now.
[1] https://review.openstack.org/#/c/181593
DocImpact: All references of 'quota_items' configuration option
and its description should be removed from the docs.
UpgradeImpact: Remove 'quota_items' configuration option from
neutron.conf file.
Change-Id: I0698772a49f51d7c65f1f4cf1ea7660cd07113a0
Closes-Bug: #1593772
Commit 3c584084 introduce a routine where the return value of a
function is assigned to a local variable named 'meh' and then
immediately returned.
This is a poor programming style and an even poorer variable name
choice.
Change-Id: I4da66c99689a999a30f83390dfe3c79db646e431
Python 3 deprecated the logger.warn method, see:
https://docs.python.org/3/library/logging.html#logging.warning
so we prefer to use warning to avoid DeprecationWarning.
Closes-Bugs: #1529913
Change-Id: Icc01ce5fbd10880440cf75a2e0833394783464a0
Co-Authored-By: Gary Kotton <gkotton@vmware.com>
Ensure that event handlers are invoked upon completion of
ML2 operations which add or remove tracked resources.
Also validate that the event handlers are called for the
appropriate resources and that quota usage's dirty bit
is set and unset as expected.
These are not unit tests, but added in the unit test tree
as they leverage code both from the DB unit test and the ML2
unit test framework. This module has indeed been added to
the 'exclusion list' in check_unit_test_structure.sh, and
should be moved to the functional test tree together with
the other modules.
Closes-Bug: #1499358
Change-Id: I78c432c35f3f3339607cd533019ae6d0fa2a5cd6
One of params, described in dosctring is not consequent with
param, sent to function.
This patch fixes this param.
Change-Id: Ie8fa630facd11d2985d3e53d6355efc973ea2166
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d