The method ``set_resources_quota_usage_dirty`` can be used now to set
the dirty bit for a single resource or multiple ones.
Trivial-Fix
Change-Id: I13ef43b71fe7a080d55a84119784433ad84380b6
This fixes two queries still being executed without the context
decorator, which is causing warning messages.
Closes-Bug: #1995738
Change-Id: I61692e09119751ac7e19cecf6bf0b4b3a9fb38c7
This patch implements a new method specific for each quota driver
class. This method, "get_resource_count", returns the current number
of resources created in a project of a tracked resource. A tracked
resource is an instance of ``neutron.quota.resource.TrackedResource``.
This method does not count the current reservations, just the actual
resources created.
This new method, "get_resource_count", will be added to the abstract
class ``neutron_lib.db.quota_api.QuotaDriverAPI``.
This patch also fixes ``TestDbQuotaDriverNoLock``, that was using a
plugin inheriting from ``DbQuotaDriver`` instead of
``DbQuotaNoLockDriver``.
Closes-Bug: #1982962
Change-Id: I2707506468cb60d93a4459ea364f1e79faa83838
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
some of them, about 10%.
Feel free to reject if we think it will cause too much
trouble with cherry-picks, else I'll slowly work my way
through the rest of the tree.
Trivialfix
Change-Id: I3d484d11e273cb8ee617f9445a069887e7b2b89f
The goal of this patch is to make the Neutron code compliant
with SQLAlchemy 2.0.
All SQL transactions must be executed inside an explicit
writer/reader context. SQLAlchemy no longer will create an
implicit transaction if the session has no active transaction.
A warning message, only available in debug mode, is added. When
an ORM session calls "do_orm_execute", if there is no active
transaction, a warning message with a traceback will be logged
to help to debug the regression introduced.
Related-Bug: #1964575
Change-Id: I3da37fee205b8d67d10673075b9130147d9eab5f
The goal of [1] is to, in case of failing when removing the quota
reservation, continue the operation. Any expired reservation will
be removed automatically in any driver.
If the DB transaction fails, it should affect only to the reservation
trying to be deleted. This is why this patch isolates the
"remove_reservation" method and guarantees it is called outside an
active DB session. That guarantees, in case of failure, no other DB
operation will be affected.
This patch also partially reverts [2] but still checks the security
group rule quota when a new security group is created. Instead of
creating and releasing a quota reservation for the security group
rules created, now only the available quota limit is checked before
creating them. That won't prevent another operation to create security
group rules in parallel, exceeding the available quota. However, this
is not even guaranteed with the current quota driver.
[1]https://review.opendev.org/c/openstack/neutron/+/805031
[2]https://review.opendev.org/c/openstack/neutron/+/701565
Closes-Bug: #1943714
Change-Id: Id73368576a948f78a043d7cf0be16661a65626a9
This reverts commit 062336e59b.
Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.
Related-Bug: #1920001
Conflicts:
neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
neutron/common/utils.py
neutron/db/address_group_db.py
neutron/services/segments/db.py
Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
per (resource, project_id) but retrieves the instant (resource,
project_id) usage and the current (resource, project_id) reservations.
If the requested number of resources fit the available quota, a new
``Reservation`` register is created with the amount of units requested.
All those operations are done inside a DB transaction context. That
means the amount of resources and reservations is guaranteed inside
this transaction (depending on the DB backend isolation level defined)
and the new reservation created will not clash with other DB transation.
That will guarantee the number of resources and instant reservations
never exceed the quota limits defined for this (resource, project_id).
NOTES:
- This change tries to be as unobtrusive as possible. The new driver
uses the same ``DbQuotaDriver`` dabatase tables (except for
``QuotaUsage``) and the same Quota engine API, located in
``neutron.quota``. However, the Quota engine resources implements some
particular API actions like "dirty", that are not used in the new
driver.
- The Pecan Quota enforcement hooks,
``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
"resync", "mark_resources_dirty" or "set_resources_dirty", that has
no meaning in the new driver.
- The isolation between the Quota engine and the Pecan hook, and the
driver itself is not clearly defined. A refactor of the Quota engine,
Quota service, Quota drivers and a common API between the driver and
the engine is needed.
- If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
``TrackedResource`` will be joined in a single class. This resource
class will have a count method (countable) or a hard dependency on a
database table (tracked resource). The only difference will be the
"count" method implementation.
Closes-Bug: #1926787
Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.
NeutronLibImpact
Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
The retry_if_session_inactive decorator was rehomed into neutron-lib
[1]. This patch consumes it by removing the function from neutron and
using neutron-libs version where appropriate.
NeutronLibImpact
[1] https://review.openstack.org/#/c/557040/
Change-Id: I3e3289f33e62d45933d0fbf165bb4b25078f22d5
Michael Bayer while analysing neutron process function call trace,
suggested to run queries against specific columns rather than full
ORM entities as it can help reduce load both at the DB level and
in the Python level since they are much faster to fetch as
non-ORM entities. In this patch we are trying that on simpler
queries to improve neutron performance.
Co-Authored-By: Joe Talerico <jtaleric@redhat.com>
Change-Id: I6a41e9487a4427f876442bbeeae61974e892225e
The public db apis are available in neutron-lib. This patch consumes
the sqla_listen, sqla_remove and sqla_remove_all functions from lib
by removing them from neutron and using lib's version.
There's also a TODO added here so we can eventually rehome the
check_no_sqlalchemy_event_import hacking check to neutron-lib.
NeutronLibImpact
Change-Id: I3b3862016125303fb227b4ff41420a89bb33c917
It's of no guarantee that core plugin implements counter/getter function
for a CountableResource. Instead of just trying core plugin, try every
plugin registered in the directory.
To retain backwards compatibility, we also make sure that core plugin is
checked first.
Change-Id: I5245e217e1f44281f85febbdfaf873321253dc5d
Closes-Bug: #1714769
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
In a gate run the neutron.quota.resource module logged 5235
lines from this module, making it one of the top offenders.
This single line accounted for 3297 of those entries and it
offers limited information since the quota engine is quite
stable at this point and debugging tenants' dirty status isn't
an issue that comes up.
We can just get rid of this particular log line.
Change-Id: I9cb0fcfbad51a0863d04fb89fbb8b09e3e21bdf0
Partial-Bug: #1707307
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.
DocImpact:Documentation describing the new API as well as the new
information that it exposes.
APIImpact
Co-Authored-By: Prince Boateng<prince.a.owusu.boateng@intel.com>
Change-Id: Ief2a6a4d2d7085e2a9dcd901123bc4fe6ac7ca22
Related-bug: #1599488
As one of query to fetch quota usage by resource and tenant
was done in lock mode earlier. This was done to protect the
dirty bit in the usage which tracks for update. For moving to
OVO carrying lock mode can be avoided by making sure that dirty
attribute of quotausages is still protected.
Change-Id: I0bb9f6fb7be51be590afb527a56e0569e922e98f
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Register all sqlalchemy events through a new function in
neutron.db.api so we can keep track of active events and
ensure all are removed at the end of each test run.
Without this, an instance of a plugin may be left around
with the only reference to it existing in SQLAlchemy, where
it will receive events for tests unrelated to it and potentially
interfere.
Change-Id: I8e93eb4e8ef5a13f015db9cd20e44941cdcb72ef
The devref was suggesting to do exactly the wrong thing for cases where
we have SQLAlchemy event listeners waiting for deleted objects.
DELETE statements fail to trigger any kind of SQLAlchemy event
listeners and the majority of our delete operations are small
(<10 items at once) so the performance difference is negligible
(especially since we do it so infrequently).
Change-Id: Id142179418a7b94d8d9695871b3fcd5dcc64730c
This decorates the quota system operations with
the retry decorators. This will help significantly
with the bug this marks as closed since operations
in the quota engine after commit should no longer
trigger retries of the full API operation.
The logic to find the args in the decorator had
to be adjusted to deal with functions already decorated.
This just uses the getargspec function from pecan that
deals with decorated functions.
Partial-Bug: #1612798
Closes-Bug: #1596075
Change-Id: Ib786117dcea08af75551770ea4c30d460382b829
The quota engine was still using oslo_db wrap_db_retry
which does not automatically take into account deadlocks
that occur inside of nested savepoint transactions.
In one case it didn't matter because it passed in the correct
exception checker but in the one that protected 'set_quota_usage'
it did not use is_retriable so a deadlock inside of the savepoint
would have resulted in a much more expensive retry all of the way
up at the API layer.
This patch just adjusts them to both use the standard neutron
retry_db_errors decorator.
Change-Id: I1e45eb15f14bf35881e5b1dce77733e831e9c6b1
Related-Bug: #1596075
Commit 3c584084 introduce a routine where the return value of a
function is assigned to a local variable named 'meh' and then
immediately returned.
This is a poor programming style and an even poorer variable name
choice.
Change-Id: I4da66c99689a999a30f83390dfe3c79db646e431
Ensure that event handlers are invoked upon completion of
ML2 operations which add or remove tracked resources.
Also validate that the event handlers are called for the
appropriate resources and that quota usage's dirty bit
is set and unset as expected.
These are not unit tests, but added in the unit test tree
as they leverage code both from the DB unit test and the ML2
unit test framework. This module has indeed been added to
the 'exclusion list' in check_unit_test_structure.sh, and
should be moved to the functional test tree together with
the other modules.
Closes-Bug: #1499358
Change-Id: I78c432c35f3f3339607cd533019ae6d0fa2a5cd6
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
This patch deals with the lock wait timeout and the deadlock errors
observed under high concurrency (api_workers >= 4) with the pymysql
driver. It includes the following changes:
- Stop setting dirty status for resource usage when creating
reservation, as usage of reserved resources is not tracked anymore;
- Add a variable, increasing delay when retrying make_reservation
upon a DBDeadlock error in order to reduce the chances of further
collisions;
- Enable transaction retry upon DBDeadlock errors for set_quota_usage;
- Do not resync quota usage while making reservation. This puts a lot
of stress on the database and is also wasteful since resource usage
is very likely to change again once the transaction is committed;
- Use autonested_transaction to simplify logic around when the
nested flag should be used.
Change-Id: I7a335f9ebea3c0d6fee6e6b757554e045a66075c
Closes-Bug: #1486134
Related-Blueprint: better-quotas
The oslo db retry decorator logs a traceback everytime a deadlock
is encountered even though it is being retried. With multiple workers
and a Galera cluster, deadlocks are common occurences due to our use
of with_lockmode update so we should not be polluting the logs.
This patch adjusts our usage of the retry decorator to catch deadlocks
with the exception checker which does not log them until the retries
are exhausted.
Change-Id: I433fbbad61070e20ebe934b9247e36fc190fa3e0
Reservations have a transient nature: a reservation lifespan
typically begins and ends with a single request.
Therefore tracking reserved amounts for each tenant and resource
is not nearly as efficient as tracking resource usage.
Indeed it is fairly easy to verify that the overhead for tracking
reserved amounts is much greater than the one needed for counting
active reservations for each tenant and resource.
This patch removes the logic for tracking reservations, and
replaces it with an explicit count of active reservations.
Please note that this patch does not adjust accordingly the
ResourceUsage DB model. This will be done in a separate patch with
an expand migration; this should avoid most merge conflicts before
the final patch for restoring reservation logic merges.
Related-Blueprint: better-quotas
Change-Id: Ib5e3bd61c1bc0fc8a5d612dae5c1740a8834a980
This lock was used to avoid errors due to list contents changing
during iteration, but is causing issues with pymysql. This patch
proposes an alternative approach which makes the use of a lock
unnecessary.
With this change a copy of the dirty_tenants set is made before
setting the dirty bit on resources, and then the mark_dirty routine
operates on this copy. This still guaranteses operations
correctness, as all the tenants that should be marked dirty are
marked dirty before the completion of the relevant API request.
Related-Blueprint: better-quotas
Change-Id: Ib39e7089889d3f906bdc025c843128a1fa3e8797
Reservations are temporarily disabled, and therefore querying them
is pointless, and potentially harmful.
Change-Id: Iab1d0ffdc54cb5bd06a0d4fbd4eb095ac4b754b8
Related-Bug: #1486134
We should synchronize every access or modification of
self._dirty_tenants or self._out_of_sync_tenants.
Closes-Bug: #1485969
Change-Id: If17f57e8905fd8d13438d0421f73468e77f723d9
Add the concept of resource reservation in neutron.
Usage tracking logic is also updated to support reservations.
Reservations are not however available with the now deprecated
configuration-based quota driver.
The base API controller will now use reservations to perform
quota checks rather than counting resource usage and then
invoking the limit_check routine.
The limit_check routine however has not been removed and
depreacated as a part of this patch. In order to ensure all
quota drivers expose a consistent interface, a
make_reservation method has been added to the configuration
based driver as well. This method simply performs "old-style"
limit checks by counting resource usage and then invoking
limit_check.
DocImpact
Implements blueprint better-quotas.
Change-Id: Ifea07f461def564884af5b291c8a56655a4d818b
Specify which resources should be tracked when initializing
the ML2 and l3_router plugins. This will enable usage tracking
for the following resources:
- Networks
- Ports
- Subnets
- Subnet pools
- Security groups
- Security group rules
- Routers
- Floating IPs
Partially implements blueprint: bp/better-quotas
Change-Id: I57287b15ffdadc30297651a01e9f05110e9ce6b6
Introduce a 'plural_name' attribute in the BaseResource
class in the neutron.quota.resource module. This will
enable to specify the plural name of a resource when it's
entered in the quota registry.
This will come handy for managing reservations and also
removes the need for passing the collection name when
counting a resource.
Also, the name of the 'resources' parameter for the
_count_resource routine has ben changed to collection_name,
as the previous one was misleading, since it led to believe
it should be used for a collection, whereas it is simply
the name of a collection.
Change-Id: I0b61ba1856e9552cc14574be28fec6c77fb8783c
Related-Blueprint: better-quotas
This patch introduces application logic support for tracking
Neutron resource usage data, thus introducing a different
way of enforcing quota limits, which now relies on records
tracking resource usage for each tenant.
When these records go "out-of-sync" with actual resource usage,
the quota usage table entry is marked "dirty".
And it will be resynchronized the next time a resource count is
requested. Changes in resource utilization are detected using
SQLAlchemy events.
This patch however does not automatically enable resource usage
tracking for any plugin. Plugins must explicitly declare for which
resources they wish to enable tracking. To this aim, this patch
provides the @tracked_resources decorator.
As some operators might wish to not use resource usage tracking, this
patch adds the track_quota_usage configuration option. If set to
False, DB-level usage tracking will not be performed, and
resources will be counted as before, ie: invoking a plugin method.
Usage tracking is performed using a new resource type,
TrackedResource, which can work side by side with CountableResource,
the one Neutron used so far. To this aim a ResourceRegistry class
has been introduced for registering and managing resources. Most
of the resource management code previously embedded in the
QuotaEngine class is being moved to ResourceRegistry as a part of
this patch.
Partially implements blueprint better-quota
Change-Id: If461399900973a38d7b82e0b3ac54fc052f09529
Rodolfo Alonso Hernandez