Commit Graph

72 Commits

Author SHA1 Message Date
Rodolfo Alonso Hernandez 2c0e9cfa71 Create a single method to set the quota usage dirty bit
The method ``set_resources_quota_usage_dirty`` can be used now to set
the dirty bit for a single resource or multiple ones.

Trivial-Fix

Change-Id: I13ef43b71fe7a080d55a84119784433ad84380b6
2023-07-06 06:06:23 +00:00
Rodolfo Alonso Hernandez 234c5a376b [sqlalchemy-20] Add missing DB context decorator
This fixes two queries still being executed without the context
decorator, which is causing warning messages.

Closes-Bug: #1995738
Change-Id: I61692e09119751ac7e19cecf6bf0b4b3a9fb38c7
2022-12-02 02:38:36 +01:00
Rodolfo Alonso Hernandez bd60f0833b Implement specific tracked resource count method per quota driver
This patch implements a new method specific for each quota driver
class. This method, "get_resource_count", returns the current number
of resources created in a project of a tracked resource. A tracked
resource is an instance of ``neutron.quota.resource.TrackedResource``.
This method does not count the current reservations, just the actual
resources created.

This new method, "get_resource_count", will be added to the abstract
class ``neutron_lib.db.quota_api.QuotaDriverAPI``.

This patch also fixes ``TestDbQuotaDriverNoLock``, that was using a
plugin inheriting from ``DbQuotaDriver`` instead of
``DbQuotaNoLockDriver``.

Closes-Bug: #1982962

Change-Id: I2707506468cb60d93a4459ea364f1e79faa83838
2022-07-28 06:01:18 +02:00
Brian Haley 58b1df699d Fix some pylint indentation warnings
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
some of them, about 10%.

Feel free to reject if we think it will cause too much
trouble with cherry-picks, else I'll slowly work my way
through the rest of the tree.

Trivialfix

Change-Id: I3d484d11e273cb8ee617f9445a069887e7b2b89f
2022-07-01 17:52:59 -04:00
Zuul a5bcc34bf4 Merge "[quota] Enable ``DbQuotaDriverNull`` as a production driver" 2022-04-12 14:55:22 +00:00
Rodolfo Alonso Hernandez eeb918e1b9 Add the corresponding DB context to all SQL transactions
The goal of this patch is to make the Neutron code compliant
with SQLAlchemy 2.0.

All SQL transactions must be executed inside an explicit
writer/reader context. SQLAlchemy no longer will create an
implicit transaction if the session has no active transaction.

A warning message, only available in debug mode, is added. When
an ORM session calls "do_orm_execute", if there is no active
transaction, a warning message with a traceback will be logged
to help to debug the regression introduced.

Related-Bug: #1964575

Change-Id: I3da37fee205b8d67d10673075b9130147d9eab5f
2022-04-08 09:09:54 +00:00
Jakub Libosvar 8ccbbb2292 [quota] Enable ``DbQuotaDriverNull`` as a production driver
Enabled ``DbQuotaDriverNull`` as a productio quota database
quota driver. This driver does not enforce any quota nor have access
to the database. When using this quota driver, the API will return
the default empty values expected from the ``QuotaDriverAPI`` class.

Closes-bug: #1960032

Change-Id: Iafa24753e657746a8b8165b5a63c17de9a9ba791
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Co-Authored-By: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
2022-04-05 10:10:46 +00:00
Rodolfo Alonso Hernandez 603abeb977 Execute the quota reservation removal in an isolated DB txn
The goal of [1] is to, in case of failing when removing the quota
reservation, continue the operation. Any expired reservation will
be removed automatically in any driver.

If the DB transaction fails, it should affect only to the reservation
trying to be deleted. This is why this patch isolates the
"remove_reservation" method and guarantees it is called outside an
active DB session. That guarantees, in case of failure, no other DB
operation will be affected.

This patch also partially reverts [2] but still checks the security
group rule quota when a new security group is created. Instead of
creating and releasing a quota reservation for the security group
rules created, now only the available quota limit is checked before
creating them. That won't prevent another operation to create security
group rules in parallel, exceeding the available quota. However, this
is not even guaranteed with the current quota driver.

[1]https://review.opendev.org/c/openstack/neutron/+/805031
[2]https://review.opendev.org/c/openstack/neutron/+/701565

Closes-Bug: #1943714

Change-Id: Id73368576a948f78a043d7cf0be16661a65626a9
2021-09-30 13:53:23 +00:00
Rodolfo Alonso Hernandez 7dcddeb0bd Replace "tenant_id" with "project_id" in Quota engine
This is part of the remaining technical debt of the specs
https://specs.openstack.org/openstack/neutron-specs/specs/newton/moving-to-keystone-v3.html

Blueprint: https://blueprints.launchpad.net/neutron/+spec/keystone-v3

Change-Id: I1faf520d3cdafe2de873525c8ebe1fa2114bdcd7
2021-09-22 08:27:10 +00:00
Zuul 96f1ea140e Merge "Remove ``ConfDriver`` code" 2021-08-16 17:36:13 +00:00
Rodolfo Alonso Hernandez ad31c58d60 Remove ``ConfDriver`` code
The quota driver ``ConfDriver`` was deprecated in Liberty release.

``NullQuotaDriver`` is created for testing although it could be used
in production if no quota enforcement is needed. However, because
the Quota engine is not plugable (is an extension always loaded), it
could be interesting to make it plugable as any other plugin.

This patch also creates a Quota engine driver API class that should be
used in any Quota engine driver. Currently it is used in the three
in-tree drivers implemented: ``NullQuotaDriver``, ``DbQuotaDriver``
and ``DbQuotaNoLockDriver``.

Change-Id: Ib4af80e18fac52b9f68f26c84a215415e63c2822
Closes-Bug: #1928211
2021-07-26 15:00:32 +00:00
Slawek Kaplonski d7371e13e4 Revert "Set system_scope='all' in elevated context"
This reverts commit 062336e59b.

Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.

Related-Bug: #1920001

Conflicts:
    neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
    neutron/common/utils.py
    neutron/db/address_group_db.py
    neutron/services/segments/db.py

Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
2021-06-15 10:29:20 +02:00
Rodolfo Alonso Hernandez e135a8221d New Quota driver ``DbQuotaNoLockDriver``
This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
per (resource, project_id) but retrieves the instant (resource,
project_id) usage and the current (resource, project_id) reservations.
If the requested number of resources fit the available quota, a new
``Reservation`` register is created with the amount of units requested.

All those operations are done inside a DB transaction context. That
means the amount of resources and reservations is guaranteed inside
this transaction (depending on the DB backend isolation level defined)
and the new reservation created will not clash with other DB transation.
That will guarantee the number of resources and instant reservations
never exceed the quota limits defined for this (resource, project_id).

NOTES:
- This change tries to be as unobtrusive as possible. The new driver
  uses the same ``DbQuotaDriver`` dabatase tables (except for
  ``QuotaUsage``) and the same Quota engine API, located in
  ``neutron.quota``. However, the Quota engine resources implements some
  particular API actions like "dirty", that are not used in the new
  driver.
- The Pecan Quota enforcement hooks,
  ``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
  "resync", "mark_resources_dirty" or "set_resources_dirty", that has
  no meaning in the new driver.
- The isolation between the Quota engine and the Pecan hook, and the
  driver itself is not clearly defined. A refactor of the Quota engine,
  Quota service, Quota drivers and a common API between the driver and
  the engine is needed.
- If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
  ``TrackedResource`` will be joined in a single class. This resource
  class will have a count method (countable) or a hard dependency on a
  database table (tracked resource). The only difference will be the
  "count" method implementation.

Closes-Bug: #1926787

Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1
2021-05-20 07:55:59 +00:00
Rodolfo Alonso Hernandez ddd5832323 Remove "six" library
Last step to remove "six" library usage in Neutron.

Change-Id: Idd42e0c51c8c3bd598c9cf91602596be238bccae
2020-07-28 16:55:52 +00:00
Brian Haley 4f10c3bd3f Remove usage of six.text_type and six.string_type
With python 3.x, six.text_type and six.string_type
are just str.

Also removed a six.integer_type since it was the only
one left in a file.

Another step in removing all of six usage from neutron.

Change-Id: I5208dc41bff1983ecd323286f427296b722da62a
2020-05-22 14:02:55 -04:00
Oleg Bondarev ab286bcdac Set DB retry for quota_enforcement pecan_wsgi hook
The hook starts a DB transaction and should be covered with
DB retry decorator.

Closes-Bug: #1777965
Closes-Bug: #1771293
Change-Id: I044980a98845edc7b0a02e3323a1e62eb54c10c7
2019-11-08 15:34:17 +04:00
Brian Haley eaf990b2bc Fix pep8 E128 warnings in non-test code
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.

Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
2019-03-12 21:22:33 +00:00
Boden R 68fd13af40 remove neutron.common.exceptions
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.

NeutronLibImpact

Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
2019-02-01 14:35:00 -07:00
Boden R e4aa5902f7 use context manager from neutron-lib
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.

NeutronLibImpact

Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
2018-10-24 07:18:46 -06:00
Brian Haley fc95db987d Fix flake8 N534 untranslated exception message
Fix N534 untranslated exception message warnings and
enable enforcement.

Trivialfix

Change-Id: I9e2b51c768cbb6fcf5588070d1b9e9835775b374
2018-10-19 15:46:04 -04:00
Boden R 6d9f1c662f use retry_if_session_inactive from neutron-lib
The retry_if_session_inactive decorator was rehomed into neutron-lib
[1]. This patch consumes it by removing the function from neutron and
using neutron-libs version where appropriate.

NeutronLibImpact

[1] https://review.openstack.org/#/c/557040/

Change-Id: I3e3289f33e62d45933d0fbf165bb4b25078f22d5
2018-10-12 14:47:35 -06:00
venkata anil 72ef0e7814 Fetch specific columns rather than full ORM entities
Michael Bayer while analysing neutron process function call trace,
suggested to run queries against specific columns rather than full
ORM entities as it can help reduce load both at the DB level and
in the Python level since they are much faster to fetch as
non-ORM entities. In this patch we are trying that on simpler
queries to improve neutron performance.

Co-Authored-By: Joe Talerico <jtaleric@redhat.com>
Change-Id: I6a41e9487a4427f876442bbeeae61974e892225e
2018-08-22 10:14:09 +00:00
Boden R c774930aed use sqla functions from neutron-lib
The public db apis are available in neutron-lib. This patch consumes
the sqla_listen, sqla_remove and sqla_remove_all functions from lib
by removing them from neutron and using lib's version.

There's also a TODO added here so we can eventually rehome the
check_no_sqlalchemy_event_import hacking check to neutron-lib.

NeutronLibImpact

Change-Id: I3b3862016125303fb227b4ff41420a89bb33c917
2018-07-25 21:04:20 +00:00
Brian Haley 7cfdf4aa81 Fix all pep8 E129 errors
Fixed all pep8 E129 errors and changed tox.ini to no longer
ignore them.

Change-Id: I0b06d99ce1d473b79a4cfdd173baa4f02e653847
2018-05-03 13:44:04 +09:00
Ihar Hrachyshka 07bfe6adb9 CountableResource: try count/get functions for all plugins
It's of no guarantee that core plugin implements counter/getter function
for a CountableResource. Instead of just trying core plugin, try every
plugin registered in the directory.

To retain backwards compatibility, we also make sure that core plugin is
checked first.

Change-Id: I5245e217e1f44281f85febbdfaf873321253dc5d
Closes-Bug: #1714769
2017-09-08 10:50:12 -07:00
Jenkins 7c1e21a3f3 Merge "Make code follow log translation guideline" 2017-08-14 17:42:09 +00:00
Inessa Vasilevskaya 7322bd6efb Make code follow log translation guideline
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.

Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
2017-08-14 02:01:48 +00:00
Kevin Benton c36676877e Remove 'persisted dirty' log message
In a gate run the neutron.quota.resource module logged 5235
lines from this module, making it one of the top offenders.
This single line accounted for 3297 of those entries and it
offers limited information since the quota engine is quite
stable at this point and debugging tenants' dirty status isn't
an issue that comes up.

We can just get rid of this particular log line.

Change-Id: I9cb0fcfbad51a0863d04fb89fbb8b09e3e21bdf0
Partial-Bug: #1707307
2017-08-10 07:54:34 +00:00
Sergey Belous a8109af65f Extend Quota API to report usage statistics
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.

DocImpact:Documentation describing the new API as well as the new
information that it exposes.
APIImpact

Co-Authored-By: Prince Boateng<prince.a.owusu.boateng@intel.com>
Change-Id: Ief2a6a4d2d7085e2a9dcd901123bc4fe6ac7ca22
Related-bug: #1599488
2017-07-17 20:51:48 +00:00
Ann Kamyshnikova 9195c66cbf Use new enginefacade for quota and provisioning blocks
Use reader and writer for db operations.

Partially-Implements blueprint: enginefacade-switch

Change-Id: I3adaec4cae814c1feb88aa646b99823de9c0eb9e
2017-03-29 14:31:03 +00:00
Duan Jiong c5f5bd0c4f Fix some reST field lists in docstrings
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.

[1] http://sphinx-doc.org/domains.html#info-field-lists

Change-Id: I8fb91bcb8ae4bfbfa68ddf7c38a4f1f5cf548b36
2017-03-23 08:50:01 +08:00
Manjeet Singh Bhatia 85b70a9c18 Make query in quota api lockless
As one of query to fetch quota usage by resource and tenant
was done in lock mode earlier. This was done to protect the
dirty bit in the usage which tracks for update. For moving to
OVO carrying lock mode can be avoided by making sure that dirty
attribute of quotausages is still protected.

Change-Id: I0bb9f6fb7be51be590afb527a56e0569e922e98f
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
2017-03-07 20:49:55 +00:00
Kevin Benton 553ab6d86e Register sqlalchemy events through hook for UT cleanup
Register all sqlalchemy events through a new function in
neutron.db.api so we can keep track of active events and
ensure all are removed at the end of each test run.

Without this, an instance of a plugin may be left around
with the only reference to it existing in SQLAlchemy, where
it will receive events for tests unrelated to it and potentially
interfere.

Change-Id: I8e93eb4e8ef5a13f015db9cd20e44941cdcb72ef
2017-01-20 04:15:01 -08:00
Kevin Benton b87715d764 Protect against '.delete()' for quota and revisions
The devref was suggesting to do exactly the wrong thing for cases where
we have SQLAlchemy event listeners waiting for deleted objects.
DELETE statements fail to trigger any kind of SQLAlchemy event
listeners and the majority of our delete operations are small
(<10 items at once) so the performance difference is negligible
(especially since we do it so infrequently).

Change-Id: Id142179418a7b94d8d9695871b3fcd5dcc64730c
2017-01-09 09:09:18 -08:00
Jenkins 660f3d58d0 Merge "Mark quota operations as retriable" 2016-09-13 20:34:31 +00:00
Kevin Benton 12420c1585 Mark quota operations as retriable
This decorates the quota system operations with
the retry decorators. This will help significantly
with the bug this marks as closed since operations
in the quota engine after commit should no longer
trigger retries of the full API operation.

The logic to find the args in the decorator had
to be adjusted to deal with functions already decorated.
This just uses the getargspec function from pecan that
deals with decorated functions.

Partial-Bug: #1612798
Closes-Bug: #1596075
Change-Id: Ib786117dcea08af75551770ea4c30d460382b829
2016-09-13 10:33:10 +00:00
Gary Kotton 4984c7c24e Quotas: add in missing translation
Commit 5d597367cf raised an
exception without a translation.

TrivialFix

Change-Id: Iaa4b4ca2b79505ded35221ab25d439aecf8d6d8a
2016-09-08 05:58:19 -07:00
Jenkins fcc15554c5 Merge "Except if tracked resource registered as countable" 2016-09-08 07:04:25 +00:00
Kevin Benton 5d597367cf Except if tracked resource registered as countable
If a resource was registered in the quota engine before
being set as a tracked resource, it would have been registered
as a CountableResource instead of a TrackedResource.

This would cause a failure later when the resources were
being unregistered because the quota engine thought it
should have been a TrackedResource and tried to call
unregister_events on it.

This patch just prevents resources from being registered
as both types. It also callse unregister_all_resources as
part of the standard test case cleanup since the vast majority
of tests are using a plugin that may register resources.

Closes-Bug: #1612222
Change-Id: I8a40f38d7c0e5aeca257ba62115fa9b02ad5aa93
2016-09-07 05:22:40 -07:00
Kevin Benton c46edbc7d6 Use db_api.retry_db_errors in quota engine
The quota engine was still using oslo_db wrap_db_retry
which does not automatically take into account deadlocks
that occur inside of nested savepoint transactions.

In one case it didn't matter because it passed in the correct
exception checker but in the one that protected 'set_quota_usage'
it did not use is_retriable so a deadlock inside of the savepoint
would have resulted in a much more expensive retry all of the way
up at the API layer.

This patch just adjusts them to both use the standard neutron
retry_db_errors decorator.

Change-Id: I1e45eb15f14bf35881e5b1dce77733e831e9c6b1
Related-Bug: #1596075
2016-07-18 22:48:19 -06:00
Hong Hui Xiao ff73054e86 Remove the deprecated config "quota_items"
It was deprecated at [1], and quota of resource will be registered
at initialization of APIRouter. So, no need to use the config now.

[1] https://review.openstack.org/#/c/181593

DocImpact: All references of 'quota_items' configuration option
and its description should be removed from the docs.

UpgradeImpact: Remove 'quota_items' configuration option from
neutron.conf file.

Change-Id: I0698772a49f51d7c65f1f4cf1ea7660cd07113a0
Closes-Bug: #1593772
2016-06-24 14:45:39 +00:00
Aradhana Singh 97372a1431 Refactoring config options for quota
Refactoring Neutron configuration options for quota to be in
neutron/conf. This will allow centralization of all configuration
options and provide an easy way to import.
It is similar to the Nova implementation found here:
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/centralize-config-options.html

Partial-Bug: #1563069

Change-Id: I7723ba9b24dd5cf74b321bd0fc54df0666594bbc
2016-05-11 09:15:38 -05:00
Henry Gessau ae5bad49cc Use exceptions from neutron-lib
Related-Blueprint: neutron-lib

Change-Id: Ia014468bd621c4ee6aea95bf19328c61070174c4
2016-04-21 21:29:44 -04:00
Gary Kotton 215cf3cd8b QUOTAS: add in missing space to help text
Commit 3c584084c3 added the text
and it was missing a space.

TrivialFix

Change-Id: Ibbc7b3c5b0d08f688d312a878d4dabbc175cb50a
2016-04-18 02:18:26 -07:00
Salvatore Orlando d090087df2 Remove local variable named 'meh'
Commit 3c584084 introduce a routine where the return value of a
function is assigned to a local variable named 'meh' and then
immediately returned.
This is a poor programming style and an even poorer variable name
choice.

Change-Id: I4da66c99689a999a30f83390dfe3c79db646e431
2016-03-18 10:10:24 -07:00
LiuNanke 83ef6b5677 Using LOG.warning replace LOG.warn
Python 3 deprecated the logger.warn method, see:
https://docs.python.org/3/library/logging.html#logging.warning
so we prefer to use warning to avoid DeprecationWarning.

Closes-Bugs: #1529913

Change-Id: Icc01ce5fbd10880440cf75a2e0833394783464a0
Co-Authored-By: Gary Kotton <gkotton@vmware.com>
2016-03-10 11:02:45 -08:00
Salvatore Orlando cdd049e4c4 ML2: Add tests to validate quota usage tracking
Ensure that event handlers are invoked upon completion of
ML2 operations which add or remove tracked resources.
Also validate that the event handlers are called for the
appropriate resources and that quota usage's dirty bit
is set and unset as expected.

These are not unit tests, but added in the unit test tree
as they leverage code both from the DB unit test and the ML2
unit test framework. This module has indeed been added to
the 'exclusion list' in check_unit_test_structure.sh, and
should be moved to the functional test tree together with
the other modules.

Closes-Bug: #1499358

Change-Id: I78c432c35f3f3339607cd533019ae6d0fa2a5cd6
2015-12-09 15:17:23 -08:00
Akihiro Motoki 2d8632e412 Use _ from neutron._i18n
Partial-Bug: #1520094
Change-Id: I874a4aa1d71d1f7034a1ff0b7450b419ef5c6864
2015-12-06 19:39:04 +09:00
Dariusz Smigiel (dasm) 1f8e58f538 Fix typo in Docstring
One of params, described in dosctring is not consequent with
param, sent to function.

This patch fixes this param.

Change-Id: Ie8fa630facd11d2985d3e53d6355efc973ea2166
2015-12-03 10:57:00 +01:00
Doug Wiegley dd726ed494 Move i18n to _i18n, as per oslo_i18n guidelines
- This does NOT break other projects that rely on neutron.i18n,
  as this change includes a debtcollector shim to maintain those
  older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
  http://docs.openstack.org/developer/oslo.i18n/usage.html

Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
2015-12-01 19:29:10 -07:00