Commit Graph

228 Commits

Author SHA1 Message Date
Chris Dent 787bb33606 Use external placement in functional tests
Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).

openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.

The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.

tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.

The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.

Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,

TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.

Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
2018-12-12 18:46:49 +00:00
Balazs Gibizer b93b40c6c0 Add bandwidth related standard resource classes
Introduce NET_BW_EGR_KILOBIT_PER_SEC and
NET_BW_IGR_KILOBIT_PER_SEC resource classes.

blueprint bandwidth-resource-provider

Change-Id: I996bf705b14b564106426a2e57299638fb178750
2018-11-05 17:47:13 -05:00
Zuul 18e801d4de Merge "reshaper gabbit: Nix comments re doubled max_unit" 2018-08-30 17:21:13 +00:00
Zuul 06e5a22bc6 Merge "reshaper: Look up provider if not in inventories" 2018-08-30 01:12:50 +00:00
Zuul 7116d8daf8 Merge "[placement] Add /reshaper handler for POST" 2018-08-29 09:22:10 +00:00
Eric Fried 2a50107606 reshaper gabbit: Nix comments re doubled max_unit
As noted in a review [1] on the reshaper series, the comments about
doubling max_unit were obsolete (fixed via [2]), and are hereby removed.

[1] https://review.openstack.org/#/c/576927/35/nova/tests/functional/api/openstack/placement/gabbits/reshaper.yaml@222
[2] I8ba378ff5eeaf6c9cca11c5874708a17d4640097

Change-Id: Iaf3df651a76705d0f64182c46c665f037ea51b68
2018-08-28 14:46:00 -05:00
Eric Fried d13e5a5011 reshaper: Look up provider if not in inventories
Per the referenced bug, we weren't accounting for the scenario where a
reshape operation was removing *all* inventories for a provider (which
could be fairly common). With this fix, we do a three-stage lookup of
the provider object: If it's not in the inventories, we look in the
allocations; if it's not in the allocations, we look it up in the
database.

Change-Id: I594bb64f87c61b7ffd39c19e0fd42c4c087a3a11
Closes-Bug: #1783130
2018-08-23 08:45:34 -05:00
rajat29 d27d8e44a3 Normalize dashless 'resource provider create' uuid
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.

This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.

Co-Authored-By: Chen <dstbtgagt@foxmail.com>

Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
2018-08-23 11:20:42 +08:00
Chris Dent 4d525b4ec1 [placement] Add /reshaper handler for POST
/reshaper provides a way to atomically modify some allocations and
inventory in a single transaction, allowing operations like migrating
some inventory from a parent provider to a new child.

A fair amount of code is reused from handler/inventory.py, some
refactoring is in order before things get too far with that.

In handler/allocation.py some code is extracted to its own methods
so it can be reused from reshaper.py.

This is done as microversion 1.30.

A suite of gabbi tests is provided which attempt to cover various
failures including schema violations, generation conflicts, and
data conflicts.

api-ref, release notes and rest history are updated

Change-Id: I5b33ac3572bc3789878174ffc86ca42ae8035cfa
Partially-Implements: blueprint reshape-provider-tree
2018-08-23 00:36:17 +00:00
Tetsuro Nakamura e13b765e55 Not use project table for user table
`GET /resource_provider/{uuid}/allocations` API didn't
return all the allocations made by multiple users.

This was because the placement wrongly used project table
for user table. This patch fixes it with the test case.

Change-Id: I7c808dec5de1204ced0d1f1b31d8398de8c51679
Closes-Bug: #1785382
2018-08-04 19:34:04 +09:00
Tetsuro Nakamura 9cfc598acf Adds a test for getting allocations API
`GET /resource_provider/{uuid}/allocations` API currently doesn't
return all the allocations made by multiple users.

This patch adds a test to describe this bug. The fix for this
is coming in a follow up.

Change-Id: I2b01e27922f11bef2defcb01fe415692de1578ea
Partial-Bug: #1785382
2018-08-04 19:20:30 +09:00
Tetsuro Nakamura e4923abaeb Increase max_unit in placement test fixture
In the AllocationFixture setup, to avoid the limitation of the
max_unit that is set to the inventory, we were using a bit hacky
way creating multiple allocation objects that have the same resource
class, the same consumer, and the same resource provider.

Since this is not how it works in real cases, and this prevents us
from refactoring, this patch fixes it.

Change-Id: I8ba378ff5eeaf6c9cca11c5874708a17d4640097
2018-08-02 14:00:45 +09:00
Zuul 18dcfbb0b6 Merge "Add placement.concurrent_udpate to generation pre-checks" 2018-08-01 12:19:49 +00:00
Chris Dent 0fc4f95914 [placement] disallow additional fields in allocations
Back in microversion 1.12, when the allocations structure was extended
to allow project_id and user_id on PUT /allocations/{uuid},
"additionalProperties" was not set in the JSON schema, so it has been
possible since then to include unused fields in the input. The schema
was then reused in the creation of subsequent schema for new
microversions and for new URIs, such as POST /allocations and the
forthcoming /reshaper.

This change fixes it by fixing the old microversion. This is the "just
fix it" option from the discussion on the associated bug. The other
option is to create a new microversion that corrects the behavior. This
is more complex than it might initially sound because of the way in
which the original schema is used to compose new ones.

Change-Id: Ied464744803864e61a45e03c559760a8a2e2581f
Closes-Bug: #1782340
2018-07-19 10:46:13 +01:00
Chris Dent 6006ccd83b [placement] cover bad content-length header
The main dispatch handler in placement checks to see if the
content-length header is a valid int and returns a 400 status
code if not. There had been no gabbi coverage for this behavior.
This patch adds that coverage.

Change-Id: If12fc7e374c7933bd2f10beb4be3e0a8d8e622e9
2018-07-18 16:00:13 +01:00
Chris Dent 7b0168b736 [placement] Add gabbi coverage for inv of missing rp
There was no gabbi coverage for the case where a GET request was being
made for single class of inventory on a non-existent resource provider.
This adds it.

Note that it would have been possible to add an error code/comment for
the raised exception, but this was not done in this change because there
is another change in progress which is adding the appropriate code.

Change-Id: I1858f4eb617cc81b41610195bb9a33a9059850f7
2018-07-18 15:53:55 +01:00
Chris Dent 32034a93d3 [placement] Add gabbi coverage for an inventory change
There's a special case of changing inventories that we didn't have
gabbit coverage for: For an existing resource provider, we allow its
inventory to be changed to have a lower total than existing allocations
(See commit 09627f2a0b). There was no gabbi coverage for that case.
This adds it.

Change-Id: I34e104a766d4578f83a455b4b900ea50c9a6ec0f
2018-07-18 15:46:03 +01:00
Zuul fdd4253f14 Merge "Add UUID validation for consumer_uuid" 2018-07-16 20:03:37 +00:00
Zuul 6ca30fd832 Merge "Resource_provider API handler does not return specific error codes" 2018-07-16 14:52:50 +00:00
Balazs Gibizer 928746a7bb Add UUID validation for consumer_uuid
When an allocation is created with PUT /allocations/{consumer_uuid} the
consumer_uuid is not validated as UUID in the API and the DB schema only
checks that is fits into String(36). However the Consumer object stores
this value in a UUIDField oslo only raises a warning today.

This patch adds a UUID validation for that URL path.

Change-Id: Idedd85ca9266f10ce09231c34cab7ca43029a56c
Closes-Bug: #1780238
2018-07-16 15:23:05 +02:00
Chris Dent dd13c2784b Add placement.concurrent_udpate to generation pre-checks
We have two kinds of generation conflicts that happen (for both
resource providers and consumers). In
I5a164a0ee298846231113ad7b62b82a3e8f1f409 error codes are added for
conflicts that happen when incrementing the generation in the database.s

In this change the codes are added for those situations where in code we
are checking an incoming generation against the stored generation.

Since there are incoming genearations in this case, we can add tests
so this does, via gabbi.

Change-Id: I5ed005068c1c19194ed67556376604ae90727412
2018-07-11 15:56:12 +01:00
Eric Fried a644b22056 Test for unsanitized consumer UUID
Add a gabbi test case demonstrating that the consumer with UUID
{8}-{4}-{4}-{4}-{12} is not the same as the consumer with UUID
{8}{4}{4}{4}{12}.

Change-Id: I1852ac6004fedc5dfa9dd3de187c937ab385d1b5
Related-Bug: #1758057
2018-07-11 13:24:28 +00:00
Zuul 08bca9a48e Merge "update project/user for consumer in allocation" 2018-07-11 00:53:41 +00:00
Jay Pipes 4f9a7da581 update project/user for consumer in allocation
Adds code to the util.ensure_consumer() method that updates the
consumer's project and user foreign key identifiers when a user supplies
a different external project or user identifier in the paylod of PUT
/allocations/{consumer_uuid} or POST /allocations.

Change-Id: I9368ad2280551ffa9a011d3a30c4b45861305455
Closes-bug: #1779717
Related-Bug: #1781008
2018-07-10 16:05:38 +00:00
Tetsuro Nakamura bc8027515d Update root providers in same tree
When updating a parent provider of a resource provider, placement
didn't update a root provider of another resource provider in the
same tree.

This patch fixes it to update the root provider field of all the
resource providers in the same tree.

Change-Id: Icdedc10cdd5ebfda672ca2d65a75bf0143aa769c
Closes-Bug: #1779818
2018-07-10 14:39:44 +00:00
Zuul 8469fa70da Merge "make incomplete_consumer_project_id a valid UUID" 2018-07-10 14:39:17 +00:00
Zuul 4c5b439830 Merge "Regression test for bug 1779818" 2018-07-10 10:03:28 +00:00
Jay Pipes c641981826 placement: delete auto-created consumers on fail
When we fail to create allocations for new consumers (either when
issuing a PUT /allocations/{new_consumer_uuid} or a POST /allocations
where the payload includes a new consumer UUID), we need to ensure that
we delete the Consumer object and underlying record in the consumers
table that gets auto-created before calling AllocationList.create_all().

This auto-created consumer record is what is used to compare things like
consumer generation in later calls to PUT|POST /allocations, and this
phantom consumer record was causing confusion when normal retries (for
things like 409 Conflict due to concurrent provider or inventory
updates) would be rejected stating that the expected consumer generation
was 0 and not null (null being the sentinel that indicates the caller is
expecting the consumer is a new consumer).

Change-Id: If37ef318bd5482a2d19928002c6f1fa24932946f
Closes-bug: #1779725
Closes-bug: #1778576
2018-07-09 19:05:50 -04:00
Jay Pipes 4b62b6bfc9 make incomplete_consumer_project_id a valid UUID
Unfortunately, when I added the CONF.incomplete_consumer_project_id and
CONF.incomplete_consumer_user_id options, the default value for those
options was an invalid UUID (it had 13 zeroes for the last part of the
UUID instead of 12). This is preventing some work that tries to disable
warnings about invalid UUIDs being stored in ovo UUIDField objects.

The work to "heal allocations" will be able to fix up any deployments
that deployed the invalid UUID default CONF value because it looks for
any instances in Nova where the placement allocation records don't have
a matching user/project and replaces the mismatched user/project IDs on
the consumer appropriately.

Change-Id: I67b3b771f20e0b83225fce4839d378696fe1ab24
Closes-bug: #1780107
2018-07-09 19:15:46 +01:00
deepak_mourya a6411afbfa Resource_provider API handler does not return specific error codes
There are a few places in the resource_provider.py placement
API handler module that need to be updated to return custom
error codes.

Change-Id: I7efdf7f82d581cf5dc9a3decc063b69c8cef22ff
Closes-Bug: #1778929
2018-07-06 15:38:44 +05:30
Zuul ef35e47e40 Merge "Use valid UUID in the placement gabbits" 2018-07-06 00:54:59 +00:00
Balazs Gibizer 9012a09328 Use valid UUID in the placement gabbits
It seems gabbi does not resolve yaml refs in the URL part of a REST
call. This causes that the tests in ensure-consumer.yaml uses invalid
consumer id '*consumer_id'

This patches replace the *consumer_id with a fixture ref that works.

Change-Id: I0576f282b1801340db6827fb4c9520a0d48ad7f0
Related-Bug: #1780238
2018-07-05 14:09:04 +02:00
Balazs Gibizer 00b13a2e2f Prevent updating an RP's parent to form a loop
Placement had RP loop detection for RP creation but if an RP is created
without a parent (e.g. root RP) then the parent can be set later with a
PUT /resource_providers/{uuid} request by providing the UUID of the
parent. In this code path the loop detection was missing from the
validation. Moreover there are different loop cases for create than for
set. For create the only possible loop is when the RP being created is
points to itself as a parent. However when the parent is provided later
in a PUT the RP being updated can have descendant RPs. Setting a parent
to a descendant also creates a loop.

This patch adds the missing check and returns HTTP 400 if loop is detected.

Closes-Bug: #1779635
Change-Id: I42c91f5f752f0a4fba8b1d95489fc3f87a1c5b6e
2018-07-03 10:08:04 +02:00
Balazs Gibizer 16812e8979 Regression test for bug 1779635
This patch adds a test that shows that placement allows setting the
parent of an RP to itself.

Change-Id: Ie81de7bdd43bfdb2ba48996f79d41efc2d70c1cd
Related-Bug: #1779635
2018-07-02 12:13:56 +02:00
Tetsuro Nakamura f87ed5109b Regression test for bug 1779818
This patch adds a test that shows that when updating a parent provider
of a resource provider, placement doesn't update a root provider of
another resource provider in the same tree.

Change-Id: I4f58752444cf6dff6b19dabd7a5a829cbf636678
Related-Bug: #1779818
2018-07-02 15:26:55 +09:00
Zuul 162e55d07e Merge "Add microversion for nested allocation candidate" 2018-06-29 15:03:02 +00:00
Zuul 3f8b30e261 Merge "[placement] Fix capacity tracking in POST /allocations" 2018-06-29 13:16:19 +00:00
Tetsuro Nakamura 5b4aa78459 Add microversion for nested allocation candidate
This patch adds a microversion with a release note for allocation
candidates with nested resource provider trees.

From now on we support allocation candidates with nested resource
providers with the following features.

1) ``GET /allocation_candidates`` is aware of nested providers.
   Namely, when provider trees are present, ``allocation_requests``
   in the response of ``GET /allocation_candidates`` can include
   allocations on combinations of multiple resource providers
   in the same tree.
2) ``root_provider_uuid`` and ``parent_provider_uuid`` fields are
    added to ``provider_summaries`` in the response of
   ``GET /allocation_candidates``.

Change-Id: I6cecb25c6c16cecc23d4008474d150b1f15f7d8a
Blueprint: nested-resource-providers-allocation-candidates
2018-06-29 17:38:10 +09:00
Zuul 46ef5738ad Merge "placement: s/None/null/ in consumer conflict msg" 2018-06-29 07:23:21 +00:00
Zuul be655a3f40 Merge "[placement] demonstrate part of bug 1778591 with a gabbi test" 2018-06-28 02:33:31 +00:00
Zuul db016a8ba3 Merge "Handle CannotDeleteParentResourceProvider to 409 Conflict" 2018-06-28 01:02:21 +00:00
Chris Dent 623cd722e0 [placement] demonstrate part of bug 1778591 with a gabbi test
The test creates a resource provider with a small amount of VCPU
inventory. It then tries to allocate a large number VCPU via a new
consumer (and thus using consumer_generation of null).

This fails with a 409 because capacity is violated.

Then it queries to GET allocations on that consumer, to see if consumer
generation is available. It is not.

The it tries to allocate a smaller amount and though it might be
expected that the generation should still be null, it takes a 0
for the allocation to be accepted.

Once we decide what the right behavior is here, we can fix it.

Change-Id: I196834efc2cb6ece4204abbf8544c8f2621aaa36
Related-Bug: #1778591
2018-06-27 14:57:37 +01:00
deepak.mourya 1ea3d5ec74 Handle CannotDeleteParentResourceProvider to 409 Conflict
Error handling to raise the exception 409 while deleting
the parent if it has children.

Closes-Bug: #1770636
Change-Id: I87df68992e4e635f009974f5205ca4919a4f2576
2018-06-27 13:28:46 +00:00
Chris Dent 8ad33cf855 [placement] Fix capacity tracking in POST /allocations
Add a change to _check_capacity_exceeded to also compare the amount
needed by a given allocation to a running total of amounts needed
against this class of resource on this resource provider.

Change-Id: Id8dde9a1f4b62112925616dfa54e77704109481c
Closes-Bug: #1778743
2018-06-27 11:10:52 +01:00
Zuul 3feaa1671c Merge "[placement] Demonstrate bug in consumer generation handling" 2018-06-27 04:59:36 +00:00
Chris Dent 4f14614052 [placement] Add test demonstrating bug 1778743
Add a suite of gabbit tests which demonstrate that it is possible to
violate inventory constraints in a POST to /allocations. Subsequent
patch will have a fix with the test adjusted to reflect the proper
behavior.

Change-Id: Ie6510deac262e040a56caedb5f21e414e897ed70
Related-Bug: #1778743
2018-06-26 17:01:17 +01:00
Chris Dent 6b56bba8ed [placement] Demonstrate bug in consumer generation handling
When PUTting allocations to a non-existent consumer, the code
expects the consumer generation to be 'null'. When there is only
one resource provider in the set of allocations sent, this works
well.

Unfortunately when there is more than one resource provider the
loop which calls _new_allocations still uses the 'null' generation
but now the code expects the generation to be 0, because the
consumer now exists.

This patch provides a gabbi test that demonstrates the problem
by creating a shared disk resource provider and associating it
as an aggregate with the existing RP_UUID. A GET
/allocation_candidates is made and a single allocation_request is
returned including both resource providers. A test with a new
consumer is marked as an xfail.

In the next patch the problem will be fixed.

Change-Id: I1238f82dfc46fd9dae126abfd16ce3dd1506e991
Related-Bug: #1778576
2018-06-25 20:25:16 +01:00
Eric Fried 07460610f8 placement: s/None/null/ in consumer conflict msg
The placement API had a message complaining that it was "expecting None"
which is python-speak when it should be JSON speak.  This change
corrects the message and adds a gabbi test for it.

Change-Id: Ibfb3d581588c8eb954a9faca19d9cb26d2c28861
Closes-Bug: #1778071
2018-06-21 09:16:17 -05:00
Matt Riedemann aca9b9243a Fix nits from change Id609789ef6b4a4c745550cde80dd49cabe03869a
Change-Id: Ia321b2e33c11595db83221698b2c98b9b991ba48
2018-06-20 12:11:09 +01:00
Jay Pipes 092820939d Add a microversion for consumer generation support
This patch adds new placement API microversion for handling consumer
generations.

Change-Id: I978fdea51f2d6c2572498ef80640c92ab38afe65
Co-Authored-By: Ed Leafe <ed@leafe.com>
Blueprint: add-consumer-generation
2018-06-20 12:11:09 +01:00