Commit Graph

576 Commits

Author SHA1 Message Date
Stephen Finucane d638b011bb db: Wrap raw SQL query in sqlalchemy.text
Another change in SQLAlchemy 2.x.

Change-Id: I31123c70c9664844181eeaaf9d0846a319af7c2c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-09-27 15:01:53 +00:00
Zuul a85a8be323 Merge "Modify the comment that is confused" 2023-08-31 21:00:31 +00:00
Zuul 07b2a3a249 Merge "tests: Warn on *any* SAWarning warning" 2023-06-15 23:45:35 +00:00
Zuul 6fe6b3713c Merge "tests: Use base class for all functional tests" 2023-06-15 11:49:16 +00:00
Zuul b62cbabc18 Merge "db: Replace use of deprecated API" 2023-06-14 22:48:12 +00:00
Takashi Natsume e62c3a6fea Fix a wrong assertion method
Replace 'called_with' with 'assert_has_calls'.

Change-Id: Ie1c825750d892db53fcf7d6b04e91bc6fd5663e8
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
2023-05-07 01:11:45 +00:00
Stephen Finucane b2f4fbd5f1 tests: Warn on *any* SAWarning warning
We were explicitly warning on one type of warning. We should warn on all
of them, since they are issues that need prompt addressing.

Change-Id: Ice70d5f0eb93537c4580964d0c70d439182977c0
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-04-17 13:23:05 +01:00
Stephen Finucane d00a9fedd5 tests: Use base class for all functional tests
We were not using our own base class for some of the functional tests,
which resulted in us missing SQLAlchemy 2.0 issues. Correct this.

Change-Id: I921646053c9c68b06df54d206c3d51838c4cafa6
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-04-17 13:22:59 +01:00
Stephen Finucane f71603d695 db: Replace use of deprecated API
We were missing a number of users of SQLALchemy APIs that were either
changed or removed in 2.0. These were yielding the following errors in
2.0.

  TypeError: MetaData.__init__() got an unexpected keyword argument
  'bind'

  TypeError: TableClause.insert() got an unexpected keyword argument
  'values'

  TypeError: TableClause.update() got an unexpected keyword argument
  'values'

  AttributeError: 'Engine' object has no attribute 'execute'

Resolve these issues. We will fix the test class issue in a follow-up.

Change-Id: I8724b0738f097739f55cd8566bb30b7ddbe154d9
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2023-04-17 11:47:55 +01:00
Zuul d7e9a330f0 Merge "Db: Drop redundant indexes for columns already having unique constraint" 2023-04-12 09:47:12 +00:00
Vasyl Saienko 9668281a4e Do not use coalesce for consumers.uuid
The consumers.uuid  and allocations.consumer_id are not Nullable.

Change-Id: I37065f7eb0809921a885f62ad6f30e6a2fbfd174
2023-03-18 10:37:46 +02:00
Christian Rohmann 0a8c7c5a9e Db: Drop redundant indexes for columns already having unique constraint
* inventories.provider_id
 * inventories.resource_class
 * placement_aggregates.uuid
 * resource_providers.name
 * resource_providers.uuid

Story: 2010251
Task: 46118

Change-Id: Ia834a991edf8f21ee9bea7f97399cd6c003b54e5
2023-02-24 10:17:21 +01:00
Ghanshyam Mann 636d65e3ef Modify the placement API policies defaults and scope_type
Placement API policies have been modified to drop the system
scope (every policy is now project scoped) and also modified
the defaults. Most of the policies are default to admin_or_service
role except reshape which is service role only and project resource
usage which is allowed for project reader and admin-or-service role.

Implement: policy-defaults-improvement
Change-Id: I806753e5b36a18be191a839256aaa84b511778f4
2023-01-26 11:31:16 -06:00
Ghanshyam Mann ff8bee1fbc Avoid rbac defaults conflict in functional tests
RBAC defaults for enforce_scope and enforce_new_defaults for
placement service is False. Nova is setting them for Nova service
to True in https://review.opendev.org/c/openstack/nova/+/866218

Placement tests fixture are used in Nova functional tests and
setting the rbac defaults causing placement to run with defaults
of what Nova is setting. To avoid this conflict, in placement
tests fixture, we need to set enforce_scope and enforce_new_defaults
to the same value it is for placement service.

Needed-By: https://review.opendev.org/c/openstack/nova/+/866218
Change-Id: I7e94d107962efdd7779733375327f5c8248c7087
2023-01-09 18:09:14 +00:00
Zuul b9c1167fac Merge "Fix typos" 2022-09-19 13:42:15 +00:00
Balazs Gibizer 5ec38f6d3b Make us compatible with oslo.db 12.1.0
With oslo.db 12.1.0 the following sqlalchemy warning become an error:

  sqlalchemy.exc.RemovedIn20Warning: Retrieving row members using strings
  or other non-integers is deprecated; use row._mapping for a dictionary
  interface to the row (Background on SQLAlchemy 2.0 at:
  https://sqlalche.me/e/b8d9)

We tried to fix this before but missed a test_case that still used dict
access to get the fields of a Row instead of attribute access. We fixed
that test here. Also while fixed it I noticed that the generic
_AttributeCache object states that it stores dicts but actually it
sometimes stores dict but sometimes it stores Row objects. So the doc is
updated and the dict path converted to store namedtuple objects in the
cache instead. Note that Row is also acts like a namedtuple except that
._mapping does not exists in namedtuple but exists in Row. The trait
object assumed it gets a Row object with ._mapping from the cache so
that is adjusted to only assume a namedtuple and use _asdict() to covert
it to dict which is available both in Row and namedtuple.

Change-Id: I23ac1d85290a2dec307f8e76aafb02096259b605
2022-09-05 11:44:53 +02:00
Zuul 723da65faf Merge "Fix typo in schema" 2022-09-02 17:34:54 +00:00
Zuul d0e05e89ac Merge "Remove unicode literal strings" 2022-08-05 14:36:10 +00:00
anguoming 5fbb22901b Remove unicode literal strings
Python 2 has been deprecated. This patch removes all unicode strings.

Change-Id: Ie323d459923b36de544c15c45d61eb89866fc550
2022-08-05 02:15:19 +00:00
Balazs Gibizer 9c7f869b84 Func test for os-traits and os-resource-classes lib sync
Placement synchronizes the content of the os-traits and
os-resource-classes lib to its internal DB at service startup. We had
gabbi tests to test that by asserting a hard coded number of traits and
RCs. These test were always blocking the global version bump of such os-
libs as the placement test needed to be updated. This created a deadlock
between the version bump in the requirements repo and the test update in
the placement repo. To avoid the need to always do three steps for a lib
version bump (turn of test, bump global req, update an re-enable the test)
the test logic is changed. Now these tests compares the content of the
lib with the DB automatically instead of hard coding a number of
expected traits and RCs. To be able to this these tests are move from
gabbi to python.

Change-Id: I31431a6eb4f144135da0dc9c6d5e4c6b75d1af5d
2022-08-03 10:31:38 +02:00
Tony Breeds c4e89253a5 Update placement for os-traits 2.8.0 release
This change re enables checking the standard
trait count and updates placement to require
os-traits 2.8.0

Depends-on: https://review.opendev.org/c/openstack/requirements/+/851423
Change-Id: If5c07703d7c236c9fe0eb70e31ef1afb1d3c7a3d
2022-08-02 17:52:46 +10:00
Tony Breeds cbd0198b3e disable traits count check to allow os-traits 2.8.0
This is a workaound to unblock the requirements repo.

Longer term, we probably need to work out a way for this test to pass
with multiple versions of os-traits.

This change follows the same format as:
    I5aed5674975529b32390d3e216423cba8d501bc5 ; and
    I3685c3caac3be4754d1f9e8479d8a047eaf8f3e8

Change-Id: I406afae77def90d6f8b20dc3a72310a81fee7910
2022-08-02 17:51:44 +10:00
Stephen Finucane 7d4d49f41c Fix typo in schema
We should have been specifying a minimum number of properties for the
'mappings' field when creating allocations [1]. We were not, thanks to a
typo. Correct this typo.

[1] https://docs.openstack.org/api-ref/placement/#allocations

Change-Id: I79bd5bf36a57983c38abc9235a8420931f373ca4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-07-11 13:09:34 +01:00
Rajesh Tailor 4fff36d52b Fix typos
This change fixes typos in comments, log messages and tests.

Change-Id: Ic035853facfad866be5f99ac28247b2864ca5f8b
2022-07-08 21:46:57 +05:30
Stephen Finucane 20fdc44bfd tox: Enable SQLAlchemy 2.0 warnings
As described in [1]. Unfortunately it doesn't seem to be possible to
silence these for other modules (cough...oslo.db...cough), presumably
due to how SQLAlchemy has configured warnings. We'll have to ignore
these for now.

[1] https://docs.sqlalchemy.org/en/14/changelog/migration_20.html

Change-Id: I28bc91d515440553828976dd1d3a3aef36a6d6a9
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:39:27 +01:00
Stephen Finucane 13bbdba06d db: Use Row, not LegacyRow
We added this check in change I8b845a29fa21f4ca8340d91bf5c6394094c6ab97
("db: Remove use of non-integer/slice indices") because we were using
the same code to handle two different types of object in the same
method. However, while the functions being called to generate this
function return a LegacyRow in recent versions of SQLAlchemy, it seems
older versions return a Row. The former is a subclass of the latter so
we can use the latter for both. This is also a better idea long-term,
since SQLAlchemy 2.0 removes LegacyRow.

Change-Id: Ia30c03b3878d80550cefd39271e315065f1a199d
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:39:27 +01:00
Stephen Finucane 5a0f79827a tests: Restore - don't reset - warning filters
There are more various warning filters pre-configured in a typical
Python environment, including a few from third-party libraries such as
requests [1][2] and urllib3 [3] as well as stdlib [4]. Our fixture to
configure warnings, 'WarningsFixture', called 'warnings.resetwarnings'
which *reset* all the warning filters [5]. This is clearly not something
we want to do, and resulted in tests puking warnings after the initial
test run.

Resolve this by backing up the existing warning filters before applying
the filter, and then *restoring* this original list of warning filters
after the test run.

[1] https://github.com/psf/requests/blob/v2.26.0/requests/__init__.py#L127
[2] https://github.com/psf/requests/blob/v2.26.0/requests/__init__.py#L152
[3] https://github.com/urllib3/urllib3/blob/1.26.7/src/urllib3/__init__.py#L68-L78
[4] https://docs.python.org/3.8/library/warnings.html#default-warning-filter
[5] https://docs.python.org/3.8/library/warnings.html#warnings.resetwarnings

Change-Id: Ic5b0c723a6f02fce5bddbf0e4ac179dcfe62523a
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:39:27 +01:00
Stephen Finucane c68d472dca db: Remove unnecessary use of '_mapping'
Now that we've addressed the SQLAlchemy 2.0 issues, we can focus on
cleaning things up. Replace uses of '_mapping' with explicit references
to the fields.

Change-Id: I3da80bebac4a2013a3c7cbf128615c19eb836735
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:39:27 +01:00
Stephen Finucane e5b89581c5 db: Use explicit transactions
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:

  The current statement is being autocommitted using implicit
  autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
  method of Engine or Connection in order to use an explicit transaction
  for DML and DDL statements.

For more information, refer to http://sqlalche.me/e/b8d9.

Change-Id: I6fc825906a63da8b0dbe51b7b859c2eb7fa1af82
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:39:27 +01:00
Stephen Finucane 3aca87e4fc db: Replace deprecated 'FromClause.select().whereclause' parameter
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:

  The FromClause.select().whereclause parameter is deprecated and will
  be removed in version 2.0.  Please make use of the Select.where()
  method to add WHERE criteria to the SELECT statement.

For more information, refer to http://sqlalche.me/e/b8d9.

Change-Id: Ie93f52777da62fa98e8b33d4d1ca0ae6da4de9e0
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:39:27 +01:00
Stephen Finucane b3fe04f081 db: Remove use of non-integer/slice indices
Resolve the following sqlalchemy.exc.RemovedIn20Warning:

  Using non-integer/slice indices on Row is deprecated and will be
  removed in version 2.0; please use row._mapping[<key>], or the
  mappings() accessor on the Result object.

For more information, refer to http://sqlalche.me/e/b8d9.

Change-Id: I8b845a29fa21f4ca8340d91bf5c6394094c6ab97
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:39:27 +01:00
Stephen Finucane cb6bfc56a6 db: Update 'select()' calls
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:

  The legacy calling style of select() is deprecated and will be removed
  in SQLAlchemy 2.0.  Please use the new calling style described at
  select().

For more information, refer to http://sqlalche.me/e/b8d9.

Change-Id: Iec3f239c4df1083a8fca5113636b4faf1f36a548
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:37:28 +01:00
Stephen Finucane 932988e8bf db: Replace 'as_scalar()' with 'scalar_subquery()'
Resolve another deprecation warning for a feature being removed in
SQLAlchemy 2.0.

  SADeprecationWarning: The SelectBase.as_scalar() method is deprecated
  and will be removed in a future release.  Please refer to
  SelectBase.scalar_subquery(). (deprecated since: 1.4)

Change-Id: Ia62493017758db218a57c3e48a7b4dbe9c5494ee
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:31:53 +01:00
Stephen Finucane 5a97e855f5 db: Replace implicit conversion of SELECT into FROM
Resolve the following warnings issued in recent versions of SQLAlchemy:

  SADeprecationWarning: Implicit coercion of SELECT and textual SELECT
  constructs into FROM clauses is deprecated; please call .subquery() on
  any Core select or ORM Query object in order to produce a subquery
  object.

The resolution is simple: instead of using 'FromClause.alias', use
'Query.subquery'. A warning filter is included to prevent us
reintroducing this issue in the future. It can be dropped once we start
using SQLAlchemy 2.0 where this behavior will be removed.

This change requires bumping SQLAlchemy to 1.4.0, which is the first
version to introduce the selectable '.subquery()' method. We must also
bump oslo.db to 8.6.0, which is the first version to support SQLAlchemy
1.4.x. The alternative would be to introduce a switch based on
SQLAlchemy versions, but that's tech debt that'll have to be cleaned up.

Change-Id: I6300be962dc3ce391f70d7c7e8af7cb4d6ce9baf
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-04-07 15:31:53 +01:00
Zuul 11757852a9 Merge "tests: Silence noisy tests" 2022-03-23 15:57:13 +00:00
Zuul ff0f2dda77 Merge "Add microversion 1.39 to support any-trait queries" 2022-02-22 20:23:52 +00:00
Zuul 35f61658eb Merge "Remove unused compatibility code" 2022-02-22 20:20:04 +00:00
Zuul 75a4af0409 Merge "Add any-traits support for allocation candidates" 2022-02-22 19:40:27 +00:00
Zuul 4b43b80f11 Merge "Add any-traits support for listing resource providers" 2022-02-22 16:27:17 +00:00
Zuul 26bd968add Merge "Extend the RP tree DB query to support any-traits" 2022-02-22 16:27:15 +00:00
Zuul da8bb3d04d Merge "Enhance doc of _get_trees_with_traits" 2022-02-22 16:27:12 +00:00
Zuul 88bb5e33b3 Merge "DB layer should only depend on trait id not names" 2022-02-22 16:27:09 +00:00
Zuul 9f75ce599e Merge "Extend the RP db query to support any-traits" 2022-02-22 16:27:07 +00:00
Balazs Gibizer b2afade159 Add microversion 1.39 to support any-trait queries
The new microversion adds support for the ``in:`` syntax in the ``required``
query parameter in the ``GET /resource_providers`` API as well as to the
``required`` and ``requiredN`` query params of the
``GET /allocation_candidates`` API. Also adds support for repeating the
``required`` and ``requiredN`` parameters in the respective APIs. So

  required=in:T3,T4&required=T1,!T2

is supported and it means T1 and not T2 and (T3 or T4).

Story: 2005345
Story: 2005346
Change-Id: I66543c0c5509739d1461af2fb2c327a003202d74
2022-02-10 11:09:20 +01:00
Balazs Gibizer 8245f9e5ec Remove unused compatibility code
During the development of the any-traits feature the signature of the DB
methods are changed and to allow gradual change in the caller sites
fallback logic was added to accept the old signature as well.
Now that all the call sites uses a new signature such code is removed.

Story: 2005345
Story: 2005346
Change-Id: Icdbdbb8e68bba37a2a5c1ffb81df231581c4ae02
2022-02-10 11:09:20 +01:00
Balazs Gibizer 87e56f749b Add any-traits support for allocation candidates
The patch I8704fe7350f74e0567e574eb00fc40b330817381 added support for
any-traits in the DB query when considering RP trees. This patch extends
the allocation candidates code path to be able to parse the extended
'in:' syntax of the 'required' query parameter as well as to handle the
nested required trait structure both in the unnamed and in the name
request groups.

This patch refers to microversion 1.39 which has not been added yet so
the changes in this patch cannot be triggered from the REST API. A later
patch will add the microversion bump.

Note that supporting the 'in:' syntax in root_required query param was
not planned and it is not implemented as part of this patch series and
API microversion.

Story: 2005345
Story: 2005346
Change-Id: I760be29201bce8dfc6d4e571bc8a842404eaff09
2022-02-10 11:09:20 +01:00
Balazs Gibizer faa1ad516f Add any-traits support for listing resource providers
The patch Id908822e8e03b872b204016345fba30b05ff5b1f added support for
any-traits in the DB layer. This patch extends the resource provider
object and API layer to be able to parse the extended 'in:' syntax of
the 'required' query parameter as well as to handle the nested required
trait structure.

This patch refers to microversion 1.39 which has not been added yet so
the changes in this patch cannot be triggered from the REST API. A later
patch will add the microversion bump after the allocation_candidates
code path also gained support for the same query structure.

Story: 2005345
Story: 2005346
Change-Id: I1ef8e31c73ffbc84ecdfed806098ca860c60a396
2022-02-10 11:09:09 +01:00
Zuul 91b85ba7ab Merge "Refactor trait normalization" 2022-02-09 21:32:24 +00:00
Balazs Gibizer c19481a5f3 Extend the RP tree DB query to support any-traits
This extends the RP tree query at the DB layer to support any-traits with a
nested required_traits syntax [{A, B}, {C}] meaning ((A or B) and C).
The object and API layers do not support such queries yet.

Story: 2005345
Story: 2005346
Change-Id: I8704fe7350f74e0567e574eb00fc40b330817381
2022-02-08 14:51:50 +01:00
Balazs Gibizer 849c9afd2e Enhance doc of _get_trees_with_traits
The documentation of the research_context._get_trees_with_traits
function was not complete. So this patch adds characterisation tests for
this function and documents the behavior of the call.
The behavior might seem to be broken at first as it returns RPs that
are actually forbidden by forbidden traits. Still a later call in the
call chain does a proper trait filtering with a comment that this
later filtering should be moved to _get_trees_with_traits. So this is
probably not a bug just a limitation of the current
_get_trees_with_traits call.

Change-Id: I3216b951ac755a09326357809748823426f20acd
2022-02-08 14:42:29 +01:00