Commit Graph

91 Commits

Author SHA1 Message Date
Ghanshyam Mann 636d65e3ef Modify the placement API policies defaults and scope_type
Placement API policies have been modified to drop the system
scope (every policy is now project scoped) and also modified
the defaults. Most of the policies are default to admin_or_service
role except reshape which is service role only and project resource
usage which is allowed for project reader and admin-or-service role.

Implement: policy-defaults-improvement
Change-Id: I806753e5b36a18be191a839256aaa84b511778f4
2023-01-26 11:31:16 -06:00
Zuul b9c1167fac Merge "Fix typos" 2022-09-19 13:42:15 +00:00
Zuul 723da65faf Merge "Fix typo in schema" 2022-09-02 17:34:54 +00:00
Balazs Gibizer 9c7f869b84 Func test for os-traits and os-resource-classes lib sync
Placement synchronizes the content of the os-traits and
os-resource-classes lib to its internal DB at service startup. We had
gabbi tests to test that by asserting a hard coded number of traits and
RCs. These test were always blocking the global version bump of such os-
libs as the placement test needed to be updated. This created a deadlock
between the version bump in the requirements repo and the test update in
the placement repo. To avoid the need to always do three steps for a lib
version bump (turn of test, bump global req, update an re-enable the test)
the test logic is changed. Now these tests compares the content of the
lib with the DB automatically instead of hard coding a number of
expected traits and RCs. To be able to this these tests are move from
gabbi to python.

Change-Id: I31431a6eb4f144135da0dc9c6d5e4c6b75d1af5d
2022-08-03 10:31:38 +02:00
Tony Breeds c4e89253a5 Update placement for os-traits 2.8.0 release
This change re enables checking the standard
trait count and updates placement to require
os-traits 2.8.0

Depends-on: https://review.opendev.org/c/openstack/requirements/+/851423
Change-Id: If5c07703d7c236c9fe0eb70e31ef1afb1d3c7a3d
2022-08-02 17:52:46 +10:00
Tony Breeds cbd0198b3e disable traits count check to allow os-traits 2.8.0
This is a workaound to unblock the requirements repo.

Longer term, we probably need to work out a way for this test to pass
with multiple versions of os-traits.

This change follows the same format as:
    I5aed5674975529b32390d3e216423cba8d501bc5 ; and
    I3685c3caac3be4754d1f9e8479d8a047eaf8f3e8

Change-Id: I406afae77def90d6f8b20dc3a72310a81fee7910
2022-08-02 17:51:44 +10:00
Stephen Finucane 7d4d49f41c Fix typo in schema
We should have been specifying a minimum number of properties for the
'mappings' field when creating allocations [1]. We were not, thanks to a
typo. Correct this typo.

[1] https://docs.openstack.org/api-ref/placement/#allocations

Change-Id: I79bd5bf36a57983c38abc9235a8420931f373ca4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-07-11 13:09:34 +01:00
Rajesh Tailor 4fff36d52b Fix typos
This change fixes typos in comments, log messages and tests.

Change-Id: Ic035853facfad866be5f99ac28247b2864ca5f8b
2022-07-08 21:46:57 +05:30
Zuul 11757852a9 Merge "tests: Silence noisy tests" 2022-03-23 15:57:13 +00:00
Zuul ff0f2dda77 Merge "Add microversion 1.39 to support any-trait queries" 2022-02-22 20:23:52 +00:00
Zuul 4b43b80f11 Merge "Add any-traits support for listing resource providers" 2022-02-22 16:27:17 +00:00
Balazs Gibizer b2afade159 Add microversion 1.39 to support any-trait queries
The new microversion adds support for the ``in:`` syntax in the ``required``
query parameter in the ``GET /resource_providers`` API as well as to the
``required`` and ``requiredN`` query params of the
``GET /allocation_candidates`` API. Also adds support for repeating the
``required`` and ``requiredN`` parameters in the respective APIs. So

  required=in:T3,T4&required=T1,!T2

is supported and it means T1 and not T2 and (T3 or T4).

Story: 2005345
Story: 2005346
Change-Id: I66543c0c5509739d1461af2fb2c327a003202d74
2022-02-10 11:09:20 +01:00
Balazs Gibizer faa1ad516f Add any-traits support for listing resource providers
The patch Id908822e8e03b872b204016345fba30b05ff5b1f added support for
any-traits in the DB layer. This patch extends the resource provider
object and API layer to be able to parse the extended 'in:' syntax of
the 'required' query parameter as well as to handle the nested required
trait structure.

This patch refers to microversion 1.39 which has not been added yet so
the changes in this patch cannot be triggered from the REST API. A later
patch will add the microversion bump after the allocation_candidates
code path also gained support for the same query structure.

Story: 2005345
Story: 2005346
Change-Id: I1ef8e31c73ffbc84ecdfed806098ca860c60a396
2022-02-10 11:09:09 +01:00
Stephen Finucane 50bacea31c tests: Silence noisy tests
Some of our gabbit tests have the 'verbose' flag set, which means they
are rather noisy, dumping the (mostly) raw HTTP requests to stdout.
Resolve this by dropping the 'verbose' flag.

Tests for the 'placement-status' command don't capture stdout which
means we see their output. Resolve this by simply capturing stdout.

Change-Id: If6c1f88fbb94cb13459c9506c04cfbeee7ac9619
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-02-06 02:00:06 +09:00
Zuul 7e0ad4956f Merge "Extra tests around required traits" 2022-02-03 18:27:56 +00:00
Balazs Gibizer 396634e07c Extra tests around required traits
This patch adds some tests to prove the following for both GET
/resource_providers and /allocation_candidates API:

* in:trait1,trait2 is not yet supported, this will makes sure we don't
  leak the support to older microversions

* repeated `required` query params are not supported. Interestingly the
  current behavior is that one `required` query param will overwrite the
  rest.

Story: 2005345
Story: 2005346

Change-Id: I67d1af5b8b6e945c744180358d588e95e13c33f6
2022-01-31 18:28:31 +01:00
Sean Mooney 89cc7332f5 update placement for os-traits 2.7.0 release
This change re enables checking the standard
trait count and updates placement to require
os-traits 2.7.0

Depends-on: https://review.opendev.org/c/openstack/requirements/+/826447
Change-Id: I3685c3caac3be4754d1f9e8479d8a047eaf8f3e8
2022-01-26 15:16:16 +00:00
Sean Mooney affb87af23 disable traits count check to allow os-traits 2.7.0
This is a workaound to unblock the requirements repo.

Change-Id: I5aed5674975529b32390d3e216423cba8d501bc5
2022-01-26 15:13:11 +00:00
Balazs Gibizer e9b6a2ec25 Fix adding 'unknown' to the ConsumerTypeCache
The attribute cache added for consumer types in
I32499b2616e253a06f8a003fd3eef91006b58f42. It added the
`unknown` value to the cache as that is not persisted. But it
mixed up which internal dict represents which direction of
the cache and only populated one direction of the cache.
This is causes lookup errors for allocation with
consumer_type_id = None.

The cache is fixed now.

Story: 2009167
Task: 43183
Change-Id: I232acaad8b350b3cf924dd7a59d97eb8ad5d0fcc
2021-09-02 15:59:56 +02:00
Balazs Gibizer 40a756c7b3 Reproduce 404 when allocation queried with 1.38
If an allocation is created with older than 1.38 then it is without
consumer_type then querying that allocation with 1.38 results in
HTTP 404 "No such consumer type: None." error.

This patch reproduces the error in the functional test environment.

Story: 2009167
Task: 43184
Change-Id: Ib65b79485780789463954aa7cc19ea7d1ed395a7
2021-09-02 15:59:28 +02:00
Zuul 068c6d3380 Merge "Microversion 1.38: API support for consumer types" 2021-08-25 22:18:29 +00:00
Balazs Gibizer cd1202ea46 Bump os-traits to latest 2.6.0
The latest os-traits release introduced 3 new standard traits

While bumping the os-traits requirements it turned out that the
lower-constraints become extra slow causing a timeout. This result is a
catch-22 situation. The lower-constraint job cannot be fixed alone as
the global requirement bump already happened for os-traits 2.6.0
requiring fixing the placement tests. But fixing the placement tests
alone is not possible as the lower-constraints job will time out.

So this patch squashes in Icd0a304fc989c9535db125cd483ecac92deb011d:

Pip is slow to resolve version constraints with many unrestricted
dependencies. The recent slowness seems to be due to amqp is being
unrestricted and pip starts with amqp 5.0.3 but the real minimum is
somewhere around amqp 2.5.0. This patch adds amqp 2.5.0 to
lower-constraints to speed up the pip version resolution.

Change-Id: I89b330d0647b0883eecaf40b988a1a4779bcb1dd
2021-08-25 14:20:14 +02:00
Chris Dent 3772132579 Microversion 1.38: API support for consumer types
Update allocations, reshaper and usage APIs to accept and present
consumer_type in microversion 1.38.

ensure_consumer in placement/handlers/util.py is updated to be consumer
type aware.

allocation, usage and reshaper schema and handlers are updated

gabbits/consumer-types-1.38.yaml adds tests across the various URIs

A TODO is left in placement/handlers/allocation.py where the database
is being accessed in a way that is not ideal. This will be cleared
up in a followup patch (to add use of an AttributeCache).

Co-Authored-By: Surya Seetharaman <suryaseetharaman.9@gmail.com>
Co-Authored-By: melanie witt <melwittt@gmail.com>

Story: 2005473
Task: 36421

Change-Id: I24c2315093e07dbf25c4fb53152e6a4de7477a51
2021-08-05 23:23:47 +00:00
Zuul c4d785cc35 Merge "Add support for RP re-parenting and orphaning" 2021-07-23 20:22:36 +00:00
Balazs Gibizer 366167009d Bump os-resource-classes requirements
os-resource-classes 1.1.0 was recently released and upper
constraints were bumped. This brings 3 new standard resource classes so
the tests are adjusted.

blueprint: qos-minimum-guaranteed-packet-rate
Change-Id: I6d1c84ddc4094635bebe001ea046cd01f1c98bbb
2021-07-16 10:58:33 +00:00
Balazs Gibizer 00795f31b8 Add support for RP re-parenting and orphaning
This patch adds the new microversion 1.37 in which changing the
parent_provider_uuid of a resource provider to any other provider
(except in the same subtree to avoid loops) or to null (un-parenting /
orphaning) is supported.

Story: 2008764
Task: 42131
Change-Id: I9d16ab7944d7e6ce725a892827b9508b7ba52d7c
2021-07-14 17:55:41 +02:00
Chris Dent f41599e1ad Update traits in tests and requirements
In I0438e39a3c61253f781761a5aae895993f897adb,
I333147dcd47c6d0b926338a5a0c545f5adc63961 , and
Ia4d1871b719037174506fb784b77572e63303762 six new
traits were added, and a new os-traits was released
triggering the canary test for updating constraints.

Change-Id: I4fbab60f04f85d70c703385d7b43457045e3661d
2021-02-23 17:41:12 +00:00
Stephen Finucane 94279af1e1 Implement secure RBAC for reshaper
This commit updates the policies for the resource classes in placement to
support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

Change-Id: Ifeb5ae29d9d637708cd5c0bc62a2abfcbac3ca6e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2021-02-10 11:00:40 +00:00
Lance Bragstad 7a7365c7e0 Implement secure RBAC for usage
This commit updates the policies for the usage resource in
placement to support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

This commit also allows project readers to see project-specific usages,
resolve a long-standing policy TODO.

Co-Authored-By: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I80afaae965d3c5f862320ac11a7d05db2f6b6553
2021-01-27 17:29:47 +00:00
Lance Bragstad 1f9529f8a0 Implement secure RBAC for traits
This commit updates the policies for traits in placement to support read-only
roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

Change-Id: I5aec5f0ad24acd2c14c85a0a878c50790c2ab415
2021-01-27 17:23:16 +00:00
Lance Bragstad e0c9a5a94d Implement secure RBAC for resource classes
This commit updates the policies for the resource classes in placement to
support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

Change-Id: Ie56fac6e0714f418b26c875735caf8867a1a1d9f
2021-01-27 17:22:10 +00:00
Lance Bragstad d0e49b794b Implement secure RBAC for inventories
This commit updates the policies for the inventories resource in
placement to support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

Co-Authored-By: Stephen Finucane <stephenfin@redhat.com>

Change-Id: I9add612f1533d0810cbfd0eadc358759e285ab6a
2021-01-27 17:18:39 +00:00
Lance Bragstad 8959e3f80c Implement secure RBAC for allocation candidates
This commit updates the policies for the allocation candidates resource in
placement to support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

Change-Id: I5197484395fd1f1e665e1882bd5884cba88ad40d
2021-01-27 17:16:30 +00:00
Lance Bragstad b8e648f13a Implement secure RBAC for allocations
This commit updates the policies for the allocations resource in
placement to support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

Change-Id: I1f47f1a96e32d3bc9526c14e5541af7992fd1f72
2021-01-27 17:15:56 +00:00
Lance Bragstad 468c1b7385 Implement secure RBAC for aggregates
This commit updates the policies for the aggregates resource in
placement to support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

Change-Id: I1242287418361ee48940f5bd2ee1bdc72cf1543e
2021-01-27 17:08:47 +00:00
Lance Bragstad 73203a91b6 Implement secure RBAC for resource providers
This commit updates the policies for the resource providers in placement to
support read-only roles.

This is part of a broader community effort to support read-only roles
and implement secure, consistent default policies.

This commit also introduces some testing infrastructure and plumbing
that is useful for implementing protection tests with gabbi. Including
testing for deprecated policies.

Change-Id: Icc904bf325eaa60377171a22a86932135c384e6a
2021-01-27 17:05:06 +00:00
melanie witt ffb9d5fdee Update for os-traits 2.4.0
The upper constraint for os-traits was bumped to 2.4.0 in change
Idf9ae2ea576f4ed882489f03e4dbc94193243c98 and additional traits
are available.

Update the tests and requirements accordingly.

Change-Id: I2d67ea2025bab811b00d98759ac4b97ad0737377
2020-08-04 20:59:31 +00:00
Tetsuro Nakamura 113da59345 Update for os-traits 2.2.0
os-traits has a new release 2.2.0 with an additional trait; update tests
and requirements accordingly.

Change-Id: I549a1ae99daa524ec881796f4442d781d4cf50bf
2020-01-11 12:49:04 +00:00
Eric Fried 3eab6acb65 Update for os-traits 2.1.0
os-traits has a new release 2.1.0 with an additional trait; update tests
and requirements accordingly.

Change-Id: I7de7d2bc32b46a505a1837ff3af5861e2ccbd553
2019-12-12 13:46:01 -06:00
Eric Fried eff1092233 Update for os-traits 2.0.0
os-traits has a new release 2.0.0 with an additional trait; update tests
and requirements accordingly.

Change-Id: If9a26b4d85bd902457228bafc28abb75980b168d
2019-12-05 10:37:50 -06:00
Eric Fried f3558482ab Clarify GET /allocations/$c for nonexistent $c
There was a note in the API reference for "List allocations" indicating
the response payload when a nonexistent consumer is queried, but the
parameter table entry for `consumer_generation` was duplicated from the
"Manage" and "Update" operations, which was confusing.

This commit clarifies that the `consumer_generation` will be entirely
absent from the response payload in this scenario. It also adds the same
clarification for the `project_id` and `user_id` fields. And tightens up
the gabbi coverage proving same.

Change-Id: I70f22b246d0c0acd1ae87a75bc438a9258b704ad
Story: #2006726
Task: #37146
2019-10-18 01:09:45 +00:00
Eric Fried d958cfa095 Update for os-traits 1.1.0
We've had three traits added since 0.16.0. Update accordingly.

Change-Id: I8c7f99b64434eed513d7462b68778b1f0da54d80
2019-10-17 09:45:34 -05:00
Chris Dent 978408b900 Add gabbits using a DeepNUMANetworkFixture
Modify the NUMANetworkFixture so we make a subclass
that pre-creates cn1 and cn2 and puts them under distinct parents and a
shared grandparent. This helps to model a deeply nested but sparse
topology where providers that contribute to the solution are low down
in the tree.

This provides a deeper topology to test the same_subtree and
_merge_candidates changes that come in the following patches.

The added gabbit intentionally uses yaml anchors to indicate that the
expected response on each test is the same.

Understanding the structure of the available resource providers and
the UUIDs in use can be super painful, but osc_placement_tree [1] can
help. In my own tetsing I added the following to the end of
make_entities to trigger it to dump:

        from osc_placement_tree import utils as placement_visual
        from placement import direct
        import time

        with direct.PlacementDirect(
              self.conf_fixture.conf, latest_microversion=True) as client:
            placement_visual.dump_placement_db_to_dot(
                    placement_visual.PlacementDirectAsClientWrapper(client),
                    '/tmp/dump.%s.dot' % time.time(),
                    hidden_fields=['inventories', 'generation', 'aggregates', 'resource_provider_generation'])

[1] https://pypi.org/project/osc-placement-tree/

Change-Id: I2aedac0ce3a4f5a40de796bb9f74824541a95a65
2019-08-16 10:44:37 +01:00
Eric Fried 9d256fa6c7 gabbi test for same_subtree with an ancestry hole
Based on the analysis at [1], we need to continue to use fully-populated
ancestry data when calculating same_subtree-ness. This commit introduces
a gabbi test that covers a particular case that can be broken if we
don't do that: where a provider involved in the request has a broken
ancestry path to another provider involved in the request.

[1] https://review.opendev.org/#/c/675606/2/placement/objects/allocation_candidate.py@908

Change-Id: I5062e725d6f29cd997b974a0dad7dc4d0ed2900f
2019-08-16 10:44:37 +01:00
Eric Fried a6e1a19a85 Add tests demonstrating overlapping same_subtreeZ
Pursuant to discussions at [1], this commit adds a pair of gabbi tests
demonstrating that same_subtree=A,B&same_subtree=B,C does not
necessarily act the same as same_subtree=A,B,C.

Foundation of the tests shamelessly purloined from [1].

[1] https://review.opendev.org/#/c/676204/4/placement/tests/functional/gabbits/same-subtree-deep.yaml@29

Change-Id: I73efe96c24581ac5c399968440581bc6918ad6c1
2019-08-16 08:00:38 +00:00
Eric Fried 0e199901df Fix allocation bug in NUMANetworkFixture
While doing something else, I noticed a bug in the NUMANetworkFixture:
we were trying to consume two VCPU from numa0 and three from cn2; but
our second call to tb.set_allocation() was using the same consumer as
the first, so numa0 was losing its allocation.

This commit uses a new consumer for the cn2 allocation, and tweaks an
existing gabbi test to prove the fix.

Change-Id: I893b27b7a07d20ef245c6056cde0e2c5bc5f4daa
2019-08-14 13:35:38 -05:00
Chris Dent d04168088c Bump os-traits minimum to 0.16.0
os-traits 0.16.0 was just released, adding 3 standard traits.
Require it.

Change-Id: I51f2a77b0f21c6145457bb33fe4db931edca4c99
2019-08-01 10:05:15 +01:00
Zuul 541052ad5e Merge "Extra gabbi tests for same_subtree" 2019-07-31 04:42:42 +00:00
Eric Fried 3b484c80d5 Extra gabbi tests for same_subtree
This is a followon for test gaps in the same_subtree patch [1].

[1] https://review.opendev.org/#/c/668376/8/placement/tests/functional/gabbits/granular-same-subtree.yaml

Change-Id: I82cf1aec44e251b30295b6119949a5995c0b6e3f
2019-07-30 09:43:48 -05:00
Chris Dent b09f2f917f Make a TraitCache similar to ResourceClassCache
The Trait and ResourceClass db objects have the same essential
structure and are used throughout the code in similar ways:

* turn a name into an id
* turn an id into a name
* get an unfiltered list of names
* make a mapping of ids to names

In I409a5e819a72d64e66ee390e4528da0c503d8d05 we made the resource
class cache request specific. Doing that work made it pretty clear
we could have a similar cache for traits and as a result visit the
traits db fewer times per request.

The implementation is straightforward: make an _AttributeCache super
class that is a parent to ResourceClassCache. Create TraitCache as
a sibling. The sole difference is the table used for the data authority
and the exception raised when an attribute is not found.

The new super class has been refactored to use private attributes and
methods.

A 'get_all' method is added to the cache to list the full collection
of dicts it contains. That can be be directly transformed into Trait
and ResourceClass objects. The order of the results of this method
are not predictable, and sorting them would add cost for no benefit,
so gabbi tests which had previously relied on the ordered of returned
resource classes have been removed.

From the API, listing traits and resource classes (without filters) now
uses the cache instead of going to the db. Where filters (in traits) are
required, the db is accessed.

The research_context turns lists of trait names into id, name maps for
required and forbidden traits.

Further, anywhere the traits table was joined to create a name of an id,
the cache is used instead. This allows to drop some joins and operate
fully in-process and in-RAM. No additional loops are added to make this
happen: the translation is done in existing loops.

The upshot of these changes is that unless there is a write operation on
a trait or resource class, both tables are scanned at most once in any
request. And when they are scanned it is to list their entire contents.

As noted in the _AttributeCache docstring there are restrictions
on what kinds of entities can use the cache and some necessary
precautions.

Change-Id: Ia19ea2b4ecdde25323579edf60ad6269d05e75a2
2019-07-24 11:23:50 +01:00