The method ``set_resources_quota_usage_dirty`` can be used now to set
the dirty bit for a single resource or multiple ones.
Trivial-Fix
Change-Id: I13ef43b71fe7a080d55a84119784433ad84380b6
The following tables and columns were defined as primary key and key
(index). A column defined as primary key creates an index in the table.
The is no need to create a second one.
Tables and columns affected:
* portdataplanestatuses, port_id
* portdnses, port_id
* portuplinkstatuspropagation, port_id
* qos_policies_default, project_id
* quotausages, resource
* quotausages, project_id
* subnet_dns_publish_fixed_ips, subnet_id
* segmenthostmappings, segment_id
* segmenthostmappings, host
* networkdnsdomains, network_id
* floatingipdnses, floatingip_id
Closes-Bug: #2024044
Change-Id: I271c109a597eb0aa088a7a9c785e8631bfaa01d7
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/db.
Trivialfix
Change-Id: I9311cfe5efc51552008072d84aa238e5d0c9de60
This patch implements a new method specific for each quota driver
class. This method, "get_resource_count", returns the current number
of resources created in a project of a tracked resource. A tracked
resource is an instance of ``neutron.quota.resource.TrackedResource``.
This method does not count the current reservations, just the actual
resources created.
This new method, "get_resource_count", will be added to the abstract
class ``neutron_lib.db.quota_api.QuotaDriverAPI``.
This patch also fixes ``TestDbQuotaDriverNoLock``, that was using a
plugin inheriting from ``DbQuotaDriver`` instead of
``DbQuotaNoLockDriver``.
Closes-Bug: #1982962
Change-Id: I2707506468cb60d93a4459ea364f1e79faa83838
The goal of this patch is to make the Neutron code compliant
with SQLAlchemy 2.0.
All SQL transactions must be executed inside an explicit
writer/reader context. SQLAlchemy no longer will create an
implicit transaction if the session has no active transaction.
A warning message, only available in debug mode, is added. When
an ORM session calls "do_orm_execute", if there is no active
transaction, a warning message with a traceback will be logged
to help to debug the regression introduced.
Related-Bug: #1964575
Change-Id: I3da37fee205b8d67d10673075b9130147d9eab5f
Enabled ``DbQuotaDriverNull`` as a productio quota database
quota driver. This driver does not enforce any quota nor have access
to the database. When using this quota driver, the API will return
the default empty values expected from the ``QuotaDriverAPI`` class.
Closes-bug: #1960032
Change-Id: Iafa24753e657746a8b8165b5a63c17de9a9ba791
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Co-Authored-By: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
The "DbQuotaNoLockDriver" quota driver "Reservation" registers clean up
is done now in a "PeriodicWorker" spawned by ML2Plugin during the
initialization. The "Reservation" registers are no longer deleted
synchronously during the API calls.
That will prevent from possible database deadlocks when concurrent
delete operations clash (as seen in very busy systems, with more
then 500 parallel requests). Although those database deadlocks were
recoverable, this new implementation will avoid this by allowing
onle single thread to execute this command periodically.
Related-Bug: #1954662
Change-Id: I50bab57830ce4c1d123b2cbd9d9832690bd4c8f9
In "DbQuotaNoLockDriver", when a new reservation is being made,
first the expired reservations are removed. That guarantees the
freshness of the existing reservations.
In systems with high concurrency of operations, the
"DbQuotaNoLockDriver.make_reservation" method will be called in
parallel. The expired reservations removal implies a deletion
on the "reservation" table that could be executed by several
workers at the same time (in the same controller or not). That
could lead to a "DBDeadlock" exception if multiple workers want
to delete the same registers.
In case an API worker receives this exception, it should continue
as the expired reservations have been deleted by other worker. It
should not retry this operation.
If the reservations are not deleted, the quota engine will filter
out those expired reservations when counting the current number of
reservations [1][2][3]. That means even if in a particular request
the expired reservations are not deleted, these won't count in the
resource quota calculation.
The default reservation expiration timeout is set to 120 seconds
(as it should have been initially set) that is the default
expiration delta for a reservation since 2015.
[1]e99d9a9d06/neutron/quota/resource.py (L340)
[2]e99d9a9d06/neutron/db/quota/api.py (L226)
[3]e99d9a9d06/neutron/objects/quota.py (L100-L101)
Closes-Bug: #1954662
Change-Id: I8af6565d2537db7f0df2e8e567ea046a0a6e003a
The goal of [1] is to, in case of failing when removing the quota
reservation, continue the operation. Any expired reservation will
be removed automatically in any driver.
If the DB transaction fails, it should affect only to the reservation
trying to be deleted. This is why this patch isolates the
"remove_reservation" method and guarantees it is called outside an
active DB session. That guarantees, in case of failure, no other DB
operation will be affected.
This patch also partially reverts [2] but still checks the security
group rule quota when a new security group is created. Instead of
creating and releasing a quota reservation for the security group
rules created, now only the available quota limit is checked before
creating them. That won't prevent another operation to create security
group rules in parallel, exceeding the available quota. However, this
is not even guaranteed with the current quota driver.
[1]https://review.opendev.org/c/openstack/neutron/+/805031
[2]https://review.opendev.org/c/openstack/neutron/+/701565
Closes-Bug: #1943714
Change-Id: Id73368576a948f78a043d7cf0be16661a65626a9
Patch [1] changed to not fail if DBError will happend when releasing
quota reservation. That may lead to the errors while commiting db
transaction in the neutron/api/v2/base.py module when in same
transaction Neutron commits reservation (which removes reservation from
db) and then set resources dirty. In case if DB error happens in the
commit_reservation() and we will simply pass this error and move on,
transaction can't be commited without rollback.
This patch adds handle of such DBErrors in the remove_reservation
function so transaction can be rolled back in case of DB error happens.
[1] https://review.opendev.org/c/openstack/neutron/+/805031
Closes-Bug: #1943714
Change-Id: I295a4f0eb1eaf0286f0e34b96db29c8f08340b84
The quota reservation release (deletion) is done at the end of a
server call to create a new resource (or resources). The quota
drivers implemented have mechanisms to clean up the expired
reservations when calculating the resource quota.
If a reservation deletion fails, the API call should not be retried.
Instead of this, the reservation should be left and collected by
the quota driver clean up mechanisms.
This patch also adds a timeout delay for expired reservations in
``DbQuotaNoLockDriver``. Now this driver deletes any existing
reservation created ``RESERVATION_EXPIRATION_TIMEOUT = 20`` (seconds)
before. It is assumed that any reservation should be released before
this time (regardless if the resource is created or not).
Closes-Bug: #1940311
Change-Id: I155c401ec5e2fe6e3af6390855852764ee983cf5
The quota driver ``ConfDriver`` was deprecated in Liberty release.
``NullQuotaDriver`` is created for testing although it could be used
in production if no quota enforcement is needed. However, because
the Quota engine is not plugable (is an extension always loaded), it
could be interesting to make it plugable as any other plugin.
This patch also creates a Quota engine driver API class that should be
used in any Quota engine driver. Currently it is used in the three
in-tree drivers implemented: ``NullQuotaDriver``, ``DbQuotaDriver``
and ``DbQuotaNoLockDriver``.
Change-Id: Ib4af80e18fac52b9f68f26c84a215415e63c2822
Closes-Bug: #1928211
The method executes DB operations outside a DB context and with an
inactive session.
Change-Id: Ifd1c7b99421768dfa9462237e2b1b14af0e68f41
Closes-Bug: #1930876
This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
per (resource, project_id) but retrieves the instant (resource,
project_id) usage and the current (resource, project_id) reservations.
If the requested number of resources fit the available quota, a new
``Reservation`` register is created with the amount of units requested.
All those operations are done inside a DB transaction context. That
means the amount of resources and reservations is guaranteed inside
this transaction (depending on the DB backend isolation level defined)
and the new reservation created will not clash with other DB transation.
That will guarantee the number of resources and instant reservations
never exceed the quota limits defined for this (resource, project_id).
NOTES:
- This change tries to be as unobtrusive as possible. The new driver
uses the same ``DbQuotaDriver`` dabatase tables (except for
``QuotaUsage``) and the same Quota engine API, located in
``neutron.quota``. However, the Quota engine resources implements some
particular API actions like "dirty", that are not used in the new
driver.
- The Pecan Quota enforcement hooks,
``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
"resync", "mark_resources_dirty" or "set_resources_dirty", that has
no meaning in the new driver.
- The isolation between the Quota engine and the Pecan hook, and the
driver itself is not clearly defined. A refactor of the Quota engine,
Quota service, Quota drivers and a common API between the driver and
the engine is needed.
- If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
``TrackedResource`` will be joined in a single class. This resource
class will have a count method (countable) or a hard dependency on a
database table (tracked resource). The only difference will be the
"count" method implementation.
Closes-Bug: #1926787
Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1
now table quotas not set unique index in project_id and resource,
when set quota concurrently, there may be multiple quota limit
records for the same resource.
Closes-bug: #1893314
Change-Id: I78d2a5accd81a58cf4d6f3be26942a4780f5d43f
Removed E125 (continuation line does not distinguish itself
from next logical line) from the ignore list and fixed all
the indentation issues. Didn't think it was going to be
close to 100 files when I started.
Change-Id: I0a6f5efec4b7d8d3632dd9dbb43e0ab58af9dff3
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.
NeutronLibImpact
Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
The retry_if_session_inactive decorator was rehomed into neutron-lib
[1]. This patch consumes it by removing the function from neutron and
using neutron-libs version where appropriate.
NeutronLibImpact
[1] https://review.openstack.org/#/c/557040/
Change-Id: I3e3289f33e62d45933d0fbf165bb4b25078f22d5
Fixed all pep8 E265 errors and changed tox.ini to no longer
ignore them. Also removed an N536 comment missed from a
previous change.
Change-Id: Ie6db8406c3b884c95b2a54a7598ea83476b8dba1
It's of no guarantee that core plugin implements counter/getter function
for a CountableResource. Instead of just trying core plugin, try every
plugin registered in the directory.
To retain backwards compatibility, we also make sure that core plugin is
checked first.
Change-Id: I5245e217e1f44281f85febbdfaf873321253dc5d
Closes-Bug: #1714769
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.
DocImpact:Documentation describing the new API as well as the new
information that it exposes.
APIImpact
Co-Authored-By: Prince Boateng<prince.a.owusu.boateng@intel.com>
Change-Id: Ief2a6a4d2d7085e2a9dcd901123bc4fe6ac7ca22
Related-bug: #1599488
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I783202f2896acd3c2aee5a7408664b3008fa8854
Quota list API returns tenant_id with the project's resource quota
but it doesn't return project_id. When neutron supported keystone v3
feature, it was missed[1]. This patch also removes an useless check
from UT.
[1]: https://review.openstack.org/#/c/357977/
APIImpact
Closes-Bug: #1667827
Change-Id: I78f4aa38a0d775e7600afafdd6941ef485f62ade
This patch introduces and integrates Oslo-Versioned Object for
ResourceDelta, Reservation, Quota and QuotaUsage model classes.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Change-Id: Ic058e66e6780e1f6ff25a5b2bbe7294959905765
As one of query to fetch quota usage by resource and tenant
was done in lock mode earlier. This was done to protect the
dirty bit in the usage which tracks for update. For moving to
OVO carrying lock mode can be avoided by making sure that dirty
attribute of quotausages is still protected.
Change-Id: I0bb9f6fb7be51be590afb527a56e0569e922e98f
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Extract all the common utils from common_db_mixin.py in preparation
for moving them to neutron-lib.
This is a preliminary step in preparation for refactoring the
CommonDbMixin class and moving it to neutron-lib also.
Partial Blueprint: neutron-lib
Change-Id: I3cba375a8162cb68e8f988f22f5c8b1ce7915180
This decorates the quota system operations with
the retry decorators. This will help significantly
with the bug this marks as closed since operations
in the quota engine after commit should no longer
trigger retries of the full API operation.
The logic to find the args in the decorator had
to be adjusted to deal with functions already decorated.
This just uses the getargspec function from pecan that
deals with decorated functions.
Partial-Bug: #1612798
Closes-Bug: #1596075
Change-Id: Ib786117dcea08af75551770ea4c30d460382b829
If the previous action that let to the quota reservation
cancelling was the result of a database connection getting
interrupted, attempting to query without calling session.begin()
will result in a sqlalchemy.exc.ResourceClosedError.
This alters the quota methods that mutate DB state to use a
transaction with the new oslo DB enginefacade decorators that
start a transaction for us.
Partial-Bug: #1596075
Partially-Implements: blueprint enginefacade-switch
Change-Id: I3d0539b11795cbcf97e70e1ec39013221a00d6d5
All occurences of ``tenant_id`` across the database are renamed
to ``project_id``. Both options are equally valid, but ``project_id``
is preferred.
To inform external users about the change, HasTenant class was
deprecated.
UpgradeImpact
Partially-Implements: blueprint keystone-v3
Change-Id: I87a8ef342ccea004731ba0192b23a8e79bc382dc
The quota engine was still using oslo_db wrap_db_retry
which does not automatically take into account deadlocks
that occur inside of nested savepoint transactions.
In one case it didn't matter because it passed in the correct
exception checker but in the one that protected 'set_quota_usage'
it did not use is_retriable so a deadlock inside of the savepoint
would have resulted in a much more expensive retry all of the way
up at the API layer.
This patch just adjusts them to both use the standard neutron
retry_db_errors decorator.
Change-Id: I1e45eb15f14bf35881e5b1dce77733e831e9c6b1
Related-Bug: #1596075
String interpolation should be delayed to be handled
by the logging code, rather than being done
at the point of the logging call.
So add a hacking rule for it.
See the oslo i18n guideline.
* http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Change-Id: I91e8d59d508c594256d5f74514e62f8f928d1df5
Closes-Bug: #1596829
There is a bug in pep8, when 'select' used, it omits all default checks
and runs only those specified by 'select'. We got hit by this issue
since I2d26534230ffe5d01aa0aab6ec902f81cfba774d was merged which lead to
almost no static checks in pep8 job.
Also note that off_by_default decorator has no effect for now because
factory in hacking is triggered after ignored checks are collected.
There will be a follow-up patch for that in order to make pep8 doing
its job quickly.
[1] https://github.com/PyCQA/pycodestyle/issues/390
Related-Bug: 1594756
Change-Id: I8e27f40908e1bb4307cc7c893169a9d99f3433c4
If an object is updated in a compare and swap fashion, it
can fail when the change is flushed and raise a
StaleDataError if another process has updated the SQL record
at the same time. We need to catch these with the retry
decorator to restart the update process in a new transaction
with a fresh read of the latest state.
Partially-Implements: bp/push-notifications
Change-Id: I151ffcf47926f5ac66e452974f87e8bc2a906151
Currently there is no support to retrieve default quotas set
for all projects. This patch adds a new API function to get
default quotas.
GET /v2.0/quotas/<tenant-id>/default
DocImpact: Document new API to used to retrieve default quotas
APIImpact: New Read-only API to retrieve default quotas
Change-Id: If40a44348e305da444acd6196d2e0c04202b8f7a
Closes-Bug: #1204956
Thomas Goirand