We were explicitly warning on one type of warning. We should warn on all
of them, since they are issues that need prompt addressing.
Change-Id: Ice70d5f0eb93537c4580964d0c70d439182977c0
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We were not using our own base class for some of the functional tests,
which resulted in us missing SQLAlchemy 2.0 issues. Correct this.
Change-Id: I921646053c9c68b06df54d206c3d51838c4cafa6
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We were missing a number of users of SQLALchemy APIs that were either
changed or removed in 2.0. These were yielding the following errors in
2.0.
TypeError: MetaData.__init__() got an unexpected keyword argument
'bind'
TypeError: TableClause.insert() got an unexpected keyword argument
'values'
TypeError: TableClause.update() got an unexpected keyword argument
'values'
AttributeError: 'Engine' object has no attribute 'execute'
Resolve these issues. We will fix the test class issue in a follow-up.
Change-Id: I8724b0738f097739f55cd8566bb30b7ddbe154d9
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Placement API policies have been modified to drop the system
scope (every policy is now project scoped) and also modified
the defaults. Most of the policies are default to admin_or_service
role except reshape which is service role only and project resource
usage which is allowed for project reader and admin-or-service role.
Implement: policy-defaults-improvement
Change-Id: I806753e5b36a18be191a839256aaa84b511778f4
RBAC defaults for enforce_scope and enforce_new_defaults for
placement service is False. Nova is setting them for Nova service
to True in https://review.opendev.org/c/openstack/nova/+/866218
Placement tests fixture are used in Nova functional tests and
setting the rbac defaults causing placement to run with defaults
of what Nova is setting. To avoid this conflict, in placement
tests fixture, we need to set enforce_scope and enforce_new_defaults
to the same value it is for placement service.
Needed-By: https://review.opendev.org/c/openstack/nova/+/866218
Change-Id: I7e94d107962efdd7779733375327f5c8248c7087
With oslo.db 12.1.0 the following sqlalchemy warning become an error:
sqlalchemy.exc.RemovedIn20Warning: Retrieving row members using strings
or other non-integers is deprecated; use row._mapping for a dictionary
interface to the row (Background on SQLAlchemy 2.0 at:
https://sqlalche.me/e/b8d9)
We tried to fix this before but missed a test_case that still used dict
access to get the fields of a Row instead of attribute access. We fixed
that test here. Also while fixed it I noticed that the generic
_AttributeCache object states that it stores dicts but actually it
sometimes stores dict but sometimes it stores Row objects. So the doc is
updated and the dict path converted to store namedtuple objects in the
cache instead. Note that Row is also acts like a namedtuple except that
._mapping does not exists in namedtuple but exists in Row. The trait
object assumed it gets a Row object with ._mapping from the cache so
that is adjusted to only assume a namedtuple and use _asdict() to covert
it to dict which is available both in Row and namedtuple.
Change-Id: I23ac1d85290a2dec307f8e76aafb02096259b605
Placement synchronizes the content of the os-traits and
os-resource-classes lib to its internal DB at service startup. We had
gabbi tests to test that by asserting a hard coded number of traits and
RCs. These test were always blocking the global version bump of such os-
libs as the placement test needed to be updated. This created a deadlock
between the version bump in the requirements repo and the test update in
the placement repo. To avoid the need to always do three steps for a lib
version bump (turn of test, bump global req, update an re-enable the test)
the test logic is changed. Now these tests compares the content of the
lib with the DB automatically instead of hard coding a number of
expected traits and RCs. To be able to this these tests are move from
gabbi to python.
Change-Id: I31431a6eb4f144135da0dc9c6d5e4c6b75d1af5d
This is a workaound to unblock the requirements repo.
Longer term, we probably need to work out a way for this test to pass
with multiple versions of os-traits.
This change follows the same format as:
I5aed5674975529b32390d3e216423cba8d501bc5 ; and
I3685c3caac3be4754d1f9e8479d8a047eaf8f3e8
Change-Id: I406afae77def90d6f8b20dc3a72310a81fee7910
We should have been specifying a minimum number of properties for the
'mappings' field when creating allocations [1]. We were not, thanks to a
typo. Correct this typo.
[1] https://docs.openstack.org/api-ref/placement/#allocations
Change-Id: I79bd5bf36a57983c38abc9235a8420931f373ca4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
As described in [1]. Unfortunately it doesn't seem to be possible to
silence these for other modules (cough...oslo.db...cough), presumably
due to how SQLAlchemy has configured warnings. We'll have to ignore
these for now.
[1] https://docs.sqlalchemy.org/en/14/changelog/migration_20.html
Change-Id: I28bc91d515440553828976dd1d3a3aef36a6d6a9
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We added this check in change I8b845a29fa21f4ca8340d91bf5c6394094c6ab97
("db: Remove use of non-integer/slice indices") because we were using
the same code to handle two different types of object in the same
method. However, while the functions being called to generate this
function return a LegacyRow in recent versions of SQLAlchemy, it seems
older versions return a Row. The former is a subclass of the latter so
we can use the latter for both. This is also a better idea long-term,
since SQLAlchemy 2.0 removes LegacyRow.
Change-Id: Ia30c03b3878d80550cefd39271e315065f1a199d
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Now that we've addressed the SQLAlchemy 2.0 issues, we can focus on
cleaning things up. Replace uses of '_mapping' with explicit references
to the fields.
Change-Id: I3da80bebac4a2013a3c7cbf128615c19eb836735
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:
The current statement is being autocommitted using implicit
autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
method of Engine or Connection in order to use an explicit transaction
for DML and DDL statements.
For more information, refer to http://sqlalche.me/e/b8d9.
Change-Id: I6fc825906a63da8b0dbe51b7b859c2eb7fa1af82
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:
The FromClause.select().whereclause parameter is deprecated and will
be removed in version 2.0. Please make use of the Select.where()
method to add WHERE criteria to the SELECT statement.
For more information, refer to http://sqlalche.me/e/b8d9.
Change-Id: Ie93f52777da62fa98e8b33d4d1ca0ae6da4de9e0
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following sqlalchemy.exc.RemovedIn20Warning:
Using non-integer/slice indices on Row is deprecated and will be
removed in version 2.0; please use row._mapping[<key>], or the
mappings() accessor on the Result object.
For more information, refer to http://sqlalche.me/e/b8d9.
Change-Id: I8b845a29fa21f4ca8340d91bf5c6394094c6ab97
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following sqlalchemy.exc.RemovedIn20Warning warning:
The legacy calling style of select() is deprecated and will be removed
in SQLAlchemy 2.0. Please use the new calling style described at
select().
For more information, refer to http://sqlalche.me/e/b8d9.
Change-Id: Iec3f239c4df1083a8fca5113636b4faf1f36a548
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve another deprecation warning for a feature being removed in
SQLAlchemy 2.0.
SADeprecationWarning: The SelectBase.as_scalar() method is deprecated
and will be removed in a future release. Please refer to
SelectBase.scalar_subquery(). (deprecated since: 1.4)
Change-Id: Ia62493017758db218a57c3e48a7b4dbe9c5494ee
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Resolve the following warnings issued in recent versions of SQLAlchemy:
SADeprecationWarning: Implicit coercion of SELECT and textual SELECT
constructs into FROM clauses is deprecated; please call .subquery() on
any Core select or ORM Query object in order to produce a subquery
object.
The resolution is simple: instead of using 'FromClause.alias', use
'Query.subquery'. A warning filter is included to prevent us
reintroducing this issue in the future. It can be dropped once we start
using SQLAlchemy 2.0 where this behavior will be removed.
This change requires bumping SQLAlchemy to 1.4.0, which is the first
version to introduce the selectable '.subquery()' method. We must also
bump oslo.db to 8.6.0, which is the first version to support SQLAlchemy
1.4.x. The alternative would be to introduce a switch based on
SQLAlchemy versions, but that's tech debt that'll have to be cleaned up.
Change-Id: I6300be962dc3ce391f70d7c7e8af7cb4d6ce9baf
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The new microversion adds support for the ``in:`` syntax in the ``required``
query parameter in the ``GET /resource_providers`` API as well as to the
``required`` and ``requiredN`` query params of the
``GET /allocation_candidates`` API. Also adds support for repeating the
``required`` and ``requiredN`` parameters in the respective APIs. So
required=in:T3,T4&required=T1,!T2
is supported and it means T1 and not T2 and (T3 or T4).
Story: 2005345
Story: 2005346
Change-Id: I66543c0c5509739d1461af2fb2c327a003202d74
During the development of the any-traits feature the signature of the DB
methods are changed and to allow gradual change in the caller sites
fallback logic was added to accept the old signature as well.
Now that all the call sites uses a new signature such code is removed.
Story: 2005345
Story: 2005346
Change-Id: Icdbdbb8e68bba37a2a5c1ffb81df231581c4ae02
The patch I8704fe7350f74e0567e574eb00fc40b330817381 added support for
any-traits in the DB query when considering RP trees. This patch extends
the allocation candidates code path to be able to parse the extended
'in:' syntax of the 'required' query parameter as well as to handle the
nested required trait structure both in the unnamed and in the name
request groups.
This patch refers to microversion 1.39 which has not been added yet so
the changes in this patch cannot be triggered from the REST API. A later
patch will add the microversion bump.
Note that supporting the 'in:' syntax in root_required query param was
not planned and it is not implemented as part of this patch series and
API microversion.
Story: 2005345
Story: 2005346
Change-Id: I760be29201bce8dfc6d4e571bc8a842404eaff09
The patch Id908822e8e03b872b204016345fba30b05ff5b1f added support for
any-traits in the DB layer. This patch extends the resource provider
object and API layer to be able to parse the extended 'in:' syntax of
the 'required' query parameter as well as to handle the nested required
trait structure.
This patch refers to microversion 1.39 which has not been added yet so
the changes in this patch cannot be triggered from the REST API. A later
patch will add the microversion bump after the allocation_candidates
code path also gained support for the same query structure.
Story: 2005345
Story: 2005346
Change-Id: I1ef8e31c73ffbc84ecdfed806098ca860c60a396
This extends the RP tree query at the DB layer to support any-traits with a
nested required_traits syntax [{A, B}, {C}] meaning ((A or B) and C).
The object and API layers do not support such queries yet.
Story: 2005345
Story: 2005346
Change-Id: I8704fe7350f74e0567e574eb00fc40b330817381
The documentation of the research_context._get_trees_with_traits
function was not complete. So this patch adds characterisation tests for
this function and documents the behavior of the call.
The behavior might seem to be broken at first as it returns RPs that
are actually forbidden by forbidden traits. Still a later call in the
call chain does a proper trait filtering with a comment that this
later filtering should be moved to _get_trees_with_traits. So this is
probably not a bug just a limitation of the current
_get_trees_with_traits call.
Change-Id: I3216b951ac755a09326357809748823426f20acd