Placement API policies have been modified to drop the system
scope (every policy is now project scoped) and also modified
the defaults. Most of the policies are default to admin_or_service
role except reshape which is service role only and project resource
usage which is allowed for project reader and admin-or-service role.
Implement: policy-defaults-improvement
Change-Id: I806753e5b36a18be191a839256aaa84b511778f4
Placement synchronizes the content of the os-traits and
os-resource-classes lib to its internal DB at service startup. We had
gabbi tests to test that by asserting a hard coded number of traits and
RCs. These test were always blocking the global version bump of such os-
libs as the placement test needed to be updated. This created a deadlock
between the version bump in the requirements repo and the test update in
the placement repo. To avoid the need to always do three steps for a lib
version bump (turn of test, bump global req, update an re-enable the test)
the test logic is changed. Now these tests compares the content of the
lib with the DB automatically instead of hard coding a number of
expected traits and RCs. To be able to this these tests are move from
gabbi to python.
Change-Id: I31431a6eb4f144135da0dc9c6d5e4c6b75d1af5d
This is a workaound to unblock the requirements repo.
Longer term, we probably need to work out a way for this test to pass
with multiple versions of os-traits.
This change follows the same format as:
I5aed5674975529b32390d3e216423cba8d501bc5 ; and
I3685c3caac3be4754d1f9e8479d8a047eaf8f3e8
Change-Id: I406afae77def90d6f8b20dc3a72310a81fee7910
We should have been specifying a minimum number of properties for the
'mappings' field when creating allocations [1]. We were not, thanks to a
typo. Correct this typo.
[1] https://docs.openstack.org/api-ref/placement/#allocations
Change-Id: I79bd5bf36a57983c38abc9235a8420931f373ca4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The new microversion adds support for the ``in:`` syntax in the ``required``
query parameter in the ``GET /resource_providers`` API as well as to the
``required`` and ``requiredN`` query params of the
``GET /allocation_candidates`` API. Also adds support for repeating the
``required`` and ``requiredN`` parameters in the respective APIs. So
required=in:T3,T4&required=T1,!T2
is supported and it means T1 and not T2 and (T3 or T4).
Story: 2005345
Story: 2005346
Change-Id: I66543c0c5509739d1461af2fb2c327a003202d74
The patch Id908822e8e03b872b204016345fba30b05ff5b1f added support for
any-traits in the DB layer. This patch extends the resource provider
object and API layer to be able to parse the extended 'in:' syntax of
the 'required' query parameter as well as to handle the nested required
trait structure.
This patch refers to microversion 1.39 which has not been added yet so
the changes in this patch cannot be triggered from the REST API. A later
patch will add the microversion bump after the allocation_candidates
code path also gained support for the same query structure.
Story: 2005345
Story: 2005346
Change-Id: I1ef8e31c73ffbc84ecdfed806098ca860c60a396
Some of our gabbit tests have the 'verbose' flag set, which means they
are rather noisy, dumping the (mostly) raw HTTP requests to stdout.
Resolve this by dropping the 'verbose' flag.
Tests for the 'placement-status' command don't capture stdout which
means we see their output. Resolve this by simply capturing stdout.
Change-Id: If6c1f88fbb94cb13459c9506c04cfbeee7ac9619
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This patch adds some tests to prove the following for both GET
/resource_providers and /allocation_candidates API:
* in:trait1,trait2 is not yet supported, this will makes sure we don't
leak the support to older microversions
* repeated `required` query params are not supported. Interestingly the
current behavior is that one `required` query param will overwrite the
rest.
Story: 2005345
Story: 2005346
Change-Id: I67d1af5b8b6e945c744180358d588e95e13c33f6
The attribute cache added for consumer types in
I32499b2616e253a06f8a003fd3eef91006b58f42. It added the
`unknown` value to the cache as that is not persisted. But it
mixed up which internal dict represents which direction of
the cache and only populated one direction of the cache.
This is causes lookup errors for allocation with
consumer_type_id = None.
The cache is fixed now.
Story: 2009167
Task: 43183
Change-Id: I232acaad8b350b3cf924dd7a59d97eb8ad5d0fcc
If an allocation is created with older than 1.38 then it is without
consumer_type then querying that allocation with 1.38 results in
HTTP 404 "No such consumer type: None." error.
This patch reproduces the error in the functional test environment.
Story: 2009167
Task: 43184
Change-Id: Ib65b79485780789463954aa7cc19ea7d1ed395a7
The latest os-traits release introduced 3 new standard traits
While bumping the os-traits requirements it turned out that the
lower-constraints become extra slow causing a timeout. This result is a
catch-22 situation. The lower-constraint job cannot be fixed alone as
the global requirement bump already happened for os-traits 2.6.0
requiring fixing the placement tests. But fixing the placement tests
alone is not possible as the lower-constraints job will time out.
So this patch squashes in Icd0a304fc989c9535db125cd483ecac92deb011d:
Pip is slow to resolve version constraints with many unrestricted
dependencies. The recent slowness seems to be due to amqp is being
unrestricted and pip starts with amqp 5.0.3 but the real minimum is
somewhere around amqp 2.5.0. This patch adds amqp 2.5.0 to
lower-constraints to speed up the pip version resolution.
Change-Id: I89b330d0647b0883eecaf40b988a1a4779bcb1dd
Update allocations, reshaper and usage APIs to accept and present
consumer_type in microversion 1.38.
ensure_consumer in placement/handlers/util.py is updated to be consumer
type aware.
allocation, usage and reshaper schema and handlers are updated
gabbits/consumer-types-1.38.yaml adds tests across the various URIs
A TODO is left in placement/handlers/allocation.py where the database
is being accessed in a way that is not ideal. This will be cleared
up in a followup patch (to add use of an AttributeCache).
Co-Authored-By: Surya Seetharaman <suryaseetharaman.9@gmail.com>
Co-Authored-By: melanie witt <melwittt@gmail.com>
Story: 2005473
Task: 36421
Change-Id: I24c2315093e07dbf25c4fb53152e6a4de7477a51
os-resource-classes 1.1.0 was recently released and upper
constraints were bumped. This brings 3 new standard resource classes so
the tests are adjusted.
blueprint: qos-minimum-guaranteed-packet-rate
Change-Id: I6d1c84ddc4094635bebe001ea046cd01f1c98bbb
This patch adds the new microversion 1.37 in which changing the
parent_provider_uuid of a resource provider to any other provider
(except in the same subtree to avoid loops) or to null (un-parenting /
orphaning) is supported.
Story: 2008764
Task: 42131
Change-Id: I9d16ab7944d7e6ce725a892827b9508b7ba52d7c
In I0438e39a3c61253f781761a5aae895993f897adb,
I333147dcd47c6d0b926338a5a0c545f5adc63961 , and
Ia4d1871b719037174506fb784b77572e63303762 six new
traits were added, and a new os-traits was released
triggering the canary test for updating constraints.
Change-Id: I4fbab60f04f85d70c703385d7b43457045e3661d
This commit updates the policies for the resource classes in placement to
support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
Change-Id: Ifeb5ae29d9d637708cd5c0bc62a2abfcbac3ca6e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This commit updates the policies for the usage resource in
placement to support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
This commit also allows project readers to see project-specific usages,
resolve a long-standing policy TODO.
Co-Authored-By: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I80afaae965d3c5f862320ac11a7d05db2f6b6553
This commit updates the policies for traits in placement to support read-only
roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
Change-Id: I5aec5f0ad24acd2c14c85a0a878c50790c2ab415
This commit updates the policies for the resource classes in placement to
support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
Change-Id: Ie56fac6e0714f418b26c875735caf8867a1a1d9f
This commit updates the policies for the inventories resource in
placement to support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
Co-Authored-By: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I9add612f1533d0810cbfd0eadc358759e285ab6a
This commit updates the policies for the allocation candidates resource in
placement to support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
Change-Id: I5197484395fd1f1e665e1882bd5884cba88ad40d
This commit updates the policies for the allocations resource in
placement to support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
Change-Id: I1f47f1a96e32d3bc9526c14e5541af7992fd1f72
This commit updates the policies for the aggregates resource in
placement to support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
Change-Id: I1242287418361ee48940f5bd2ee1bdc72cf1543e
This commit updates the policies for the resource providers in placement to
support read-only roles.
This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.
This commit also introduces some testing infrastructure and plumbing
that is useful for implementing protection tests with gabbi. Including
testing for deprecated policies.
Change-Id: Icc904bf325eaa60377171a22a86932135c384e6a
The upper constraint for os-traits was bumped to 2.4.0 in change
Idf9ae2ea576f4ed882489f03e4dbc94193243c98 and additional traits
are available.
Update the tests and requirements accordingly.
Change-Id: I2d67ea2025bab811b00d98759ac4b97ad0737377
os-traits has a new release 2.2.0 with an additional trait; update tests
and requirements accordingly.
Change-Id: I549a1ae99daa524ec881796f4442d781d4cf50bf
os-traits has a new release 2.1.0 with an additional trait; update tests
and requirements accordingly.
Change-Id: I7de7d2bc32b46a505a1837ff3af5861e2ccbd553
os-traits has a new release 2.0.0 with an additional trait; update tests
and requirements accordingly.
Change-Id: If9a26b4d85bd902457228bafc28abb75980b168d
There was a note in the API reference for "List allocations" indicating
the response payload when a nonexistent consumer is queried, but the
parameter table entry for `consumer_generation` was duplicated from the
"Manage" and "Update" operations, which was confusing.
This commit clarifies that the `consumer_generation` will be entirely
absent from the response payload in this scenario. It also adds the same
clarification for the `project_id` and `user_id` fields. And tightens up
the gabbi coverage proving same.
Change-Id: I70f22b246d0c0acd1ae87a75bc438a9258b704ad
Story: #2006726
Task: #37146
Modify the NUMANetworkFixture so we make a subclass
that pre-creates cn1 and cn2 and puts them under distinct parents and a
shared grandparent. This helps to model a deeply nested but sparse
topology where providers that contribute to the solution are low down
in the tree.
This provides a deeper topology to test the same_subtree and
_merge_candidates changes that come in the following patches.
The added gabbit intentionally uses yaml anchors to indicate that the
expected response on each test is the same.
Understanding the structure of the available resource providers and
the UUIDs in use can be super painful, but osc_placement_tree [1] can
help. In my own tetsing I added the following to the end of
make_entities to trigger it to dump:
from osc_placement_tree import utils as placement_visual
from placement import direct
import time
with direct.PlacementDirect(
self.conf_fixture.conf, latest_microversion=True) as client:
placement_visual.dump_placement_db_to_dot(
placement_visual.PlacementDirectAsClientWrapper(client),
'/tmp/dump.%s.dot' % time.time(),
hidden_fields=['inventories', 'generation', 'aggregates', 'resource_provider_generation'])
[1] https://pypi.org/project/osc-placement-tree/
Change-Id: I2aedac0ce3a4f5a40de796bb9f74824541a95a65
Based on the analysis at [1], we need to continue to use fully-populated
ancestry data when calculating same_subtree-ness. This commit introduces
a gabbi test that covers a particular case that can be broken if we
don't do that: where a provider involved in the request has a broken
ancestry path to another provider involved in the request.
[1] https://review.opendev.org/#/c/675606/2/placement/objects/allocation_candidate.py@908
Change-Id: I5062e725d6f29cd997b974a0dad7dc4d0ed2900f
While doing something else, I noticed a bug in the NUMANetworkFixture:
we were trying to consume two VCPU from numa0 and three from cn2; but
our second call to tb.set_allocation() was using the same consumer as
the first, so numa0 was losing its allocation.
This commit uses a new consumer for the cn2 allocation, and tweaks an
existing gabbi test to prove the fix.
Change-Id: I893b27b7a07d20ef245c6056cde0e2c5bc5f4daa
The Trait and ResourceClass db objects have the same essential
structure and are used throughout the code in similar ways:
* turn a name into an id
* turn an id into a name
* get an unfiltered list of names
* make a mapping of ids to names
In I409a5e819a72d64e66ee390e4528da0c503d8d05 we made the resource
class cache request specific. Doing that work made it pretty clear
we could have a similar cache for traits and as a result visit the
traits db fewer times per request.
The implementation is straightforward: make an _AttributeCache super
class that is a parent to ResourceClassCache. Create TraitCache as
a sibling. The sole difference is the table used for the data authority
and the exception raised when an attribute is not found.
The new super class has been refactored to use private attributes and
methods.
A 'get_all' method is added to the cache to list the full collection
of dicts it contains. That can be be directly transformed into Trait
and ResourceClass objects. The order of the results of this method
are not predictable, and sorting them would add cost for no benefit,
so gabbi tests which had previously relied on the ordered of returned
resource classes have been removed.
From the API, listing traits and resource classes (without filters) now
uses the cache instead of going to the db. Where filters (in traits) are
required, the db is accessed.
The research_context turns lists of trait names into id, name maps for
required and forbidden traits.
Further, anywhere the traits table was joined to create a name of an id,
the cache is used instead. This allows to drop some joins and operate
fully in-process and in-RAM. No additional loops are added to make this
happen: the translation is done in existing loops.
The upshot of these changes is that unless there is a write operation on
a trait or resource class, both tables are scanned at most once in any
request. And when they are scanned it is to list their entire contents.
As noted in the _AttributeCache docstring there are restrictions
on what kinds of entities can use the cache and some necessary
precautions.
Change-Id: Ia19ea2b4ecdde25323579edf60ad6269d05e75a2