Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).
openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.
The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.
tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.
The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.
Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,
TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.
Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
Per the referenced bug, we weren't accounting for the scenario where a
reshape operation was removing *all* inventories for a provider (which
could be fairly common). With this fix, we do a three-stage lookup of
the provider object: If it's not in the inventories, we look in the
allocations; if it's not in the allocations, we look it up in the
database.
Change-Id: I594bb64f87c61b7ffd39c19e0fd42c4c087a3a11
Closes-Bug: #1783130
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.
This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.
Co-Authored-By: Chen <dstbtgagt@foxmail.com>
Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
/reshaper provides a way to atomically modify some allocations and
inventory in a single transaction, allowing operations like migrating
some inventory from a parent provider to a new child.
A fair amount of code is reused from handler/inventory.py, some
refactoring is in order before things get too far with that.
In handler/allocation.py some code is extracted to its own methods
so it can be reused from reshaper.py.
This is done as microversion 1.30.
A suite of gabbi tests is provided which attempt to cover various
failures including schema violations, generation conflicts, and
data conflicts.
api-ref, release notes and rest history are updated
Change-Id: I5b33ac3572bc3789878174ffc86ca42ae8035cfa
Partially-Implements: blueprint reshape-provider-tree
`GET /resource_provider/{uuid}/allocations` API didn't
return all the allocations made by multiple users.
This was because the placement wrongly used project table
for user table. This patch fixes it with the test case.
Change-Id: I7c808dec5de1204ced0d1f1b31d8398de8c51679
Closes-Bug: #1785382
`GET /resource_provider/{uuid}/allocations` API currently doesn't
return all the allocations made by multiple users.
This patch adds a test to describe this bug. The fix for this
is coming in a follow up.
Change-Id: I2b01e27922f11bef2defcb01fe415692de1578ea
Partial-Bug: #1785382
In the AllocationFixture setup, to avoid the limitation of the
max_unit that is set to the inventory, we were using a bit hacky
way creating multiple allocation objects that have the same resource
class, the same consumer, and the same resource provider.
Since this is not how it works in real cases, and this prevents us
from refactoring, this patch fixes it.
Change-Id: I8ba378ff5eeaf6c9cca11c5874708a17d4640097
Back in microversion 1.12, when the allocations structure was extended
to allow project_id and user_id on PUT /allocations/{uuid},
"additionalProperties" was not set in the JSON schema, so it has been
possible since then to include unused fields in the input. The schema
was then reused in the creation of subsequent schema for new
microversions and for new URIs, such as POST /allocations and the
forthcoming /reshaper.
This change fixes it by fixing the old microversion. This is the "just
fix it" option from the discussion on the associated bug. The other
option is to create a new microversion that corrects the behavior. This
is more complex than it might initially sound because of the way in
which the original schema is used to compose new ones.
Change-Id: Ied464744803864e61a45e03c559760a8a2e2581f
Closes-Bug: #1782340
The main dispatch handler in placement checks to see if the
content-length header is a valid int and returns a 400 status
code if not. There had been no gabbi coverage for this behavior.
This patch adds that coverage.
Change-Id: If12fc7e374c7933bd2f10beb4be3e0a8d8e622e9
There was no gabbi coverage for the case where a GET request was being
made for single class of inventory on a non-existent resource provider.
This adds it.
Note that it would have been possible to add an error code/comment for
the raised exception, but this was not done in this change because there
is another change in progress which is adding the appropriate code.
Change-Id: I1858f4eb617cc81b41610195bb9a33a9059850f7
There's a special case of changing inventories that we didn't have
gabbit coverage for: For an existing resource provider, we allow its
inventory to be changed to have a lower total than existing allocations
(See commit 09627f2a0b). There was no gabbi coverage for that case.
This adds it.
Change-Id: I34e104a766d4578f83a455b4b900ea50c9a6ec0f
When an allocation is created with PUT /allocations/{consumer_uuid} the
consumer_uuid is not validated as UUID in the API and the DB schema only
checks that is fits into String(36). However the Consumer object stores
this value in a UUIDField oslo only raises a warning today.
This patch adds a UUID validation for that URL path.
Change-Id: Idedd85ca9266f10ce09231c34cab7ca43029a56c
Closes-Bug: #1780238
We have two kinds of generation conflicts that happen (for both
resource providers and consumers). In
I5a164a0ee298846231113ad7b62b82a3e8f1f409 error codes are added for
conflicts that happen when incrementing the generation in the database.s
In this change the codes are added for those situations where in code we
are checking an incoming generation against the stored generation.
Since there are incoming genearations in this case, we can add tests
so this does, via gabbi.
Change-Id: I5ed005068c1c19194ed67556376604ae90727412
Add a gabbi test case demonstrating that the consumer with UUID
{8}-{4}-{4}-{4}-{12} is not the same as the consumer with UUID
{8}{4}{4}{4}{12}.
Change-Id: I1852ac6004fedc5dfa9dd3de187c937ab385d1b5
Related-Bug: #1758057
Adds code to the util.ensure_consumer() method that updates the
consumer's project and user foreign key identifiers when a user supplies
a different external project or user identifier in the paylod of PUT
/allocations/{consumer_uuid} or POST /allocations.
Change-Id: I9368ad2280551ffa9a011d3a30c4b45861305455
Closes-bug: #1779717
Related-Bug: #1781008
When updating a parent provider of a resource provider, placement
didn't update a root provider of another resource provider in the
same tree.
This patch fixes it to update the root provider field of all the
resource providers in the same tree.
Change-Id: Icdedc10cdd5ebfda672ca2d65a75bf0143aa769c
Closes-Bug: #1779818
When we fail to create allocations for new consumers (either when
issuing a PUT /allocations/{new_consumer_uuid} or a POST /allocations
where the payload includes a new consumer UUID), we need to ensure that
we delete the Consumer object and underlying record in the consumers
table that gets auto-created before calling AllocationList.create_all().
This auto-created consumer record is what is used to compare things like
consumer generation in later calls to PUT|POST /allocations, and this
phantom consumer record was causing confusion when normal retries (for
things like 409 Conflict due to concurrent provider or inventory
updates) would be rejected stating that the expected consumer generation
was 0 and not null (null being the sentinel that indicates the caller is
expecting the consumer is a new consumer).
Change-Id: If37ef318bd5482a2d19928002c6f1fa24932946f
Closes-bug: #1779725
Closes-bug: #1778576
Unfortunately, when I added the CONF.incomplete_consumer_project_id and
CONF.incomplete_consumer_user_id options, the default value for those
options was an invalid UUID (it had 13 zeroes for the last part of the
UUID instead of 12). This is preventing some work that tries to disable
warnings about invalid UUIDs being stored in ovo UUIDField objects.
The work to "heal allocations" will be able to fix up any deployments
that deployed the invalid UUID default CONF value because it looks for
any instances in Nova where the placement allocation records don't have
a matching user/project and replaces the mismatched user/project IDs on
the consumer appropriately.
Change-Id: I67b3b771f20e0b83225fce4839d378696fe1ab24
Closes-bug: #1780107
There are a few places in the resource_provider.py placement
API handler module that need to be updated to return custom
error codes.
Change-Id: I7efdf7f82d581cf5dc9a3decc063b69c8cef22ff
Closes-Bug: #1778929
It seems gabbi does not resolve yaml refs in the URL part of a REST
call. This causes that the tests in ensure-consumer.yaml uses invalid
consumer id '*consumer_id'
This patches replace the *consumer_id with a fixture ref that works.
Change-Id: I0576f282b1801340db6827fb4c9520a0d48ad7f0
Related-Bug: #1780238
Placement had RP loop detection for RP creation but if an RP is created
without a parent (e.g. root RP) then the parent can be set later with a
PUT /resource_providers/{uuid} request by providing the UUID of the
parent. In this code path the loop detection was missing from the
validation. Moreover there are different loop cases for create than for
set. For create the only possible loop is when the RP being created is
points to itself as a parent. However when the parent is provided later
in a PUT the RP being updated can have descendant RPs. Setting a parent
to a descendant also creates a loop.
This patch adds the missing check and returns HTTP 400 if loop is detected.
Closes-Bug: #1779635
Change-Id: I42c91f5f752f0a4fba8b1d95489fc3f87a1c5b6e
This patch adds a test that shows that placement allows setting the
parent of an RP to itself.
Change-Id: Ie81de7bdd43bfdb2ba48996f79d41efc2d70c1cd
Related-Bug: #1779635
This patch adds a test that shows that when updating a parent provider
of a resource provider, placement doesn't update a root provider of
another resource provider in the same tree.
Change-Id: I4f58752444cf6dff6b19dabd7a5a829cbf636678
Related-Bug: #1779818
This patch adds a microversion with a release note for allocation
candidates with nested resource provider trees.
From now on we support allocation candidates with nested resource
providers with the following features.
1) ``GET /allocation_candidates`` is aware of nested providers.
Namely, when provider trees are present, ``allocation_requests``
in the response of ``GET /allocation_candidates`` can include
allocations on combinations of multiple resource providers
in the same tree.
2) ``root_provider_uuid`` and ``parent_provider_uuid`` fields are
added to ``provider_summaries`` in the response of
``GET /allocation_candidates``.
Change-Id: I6cecb25c6c16cecc23d4008474d150b1f15f7d8a
Blueprint: nested-resource-providers-allocation-candidates
The test creates a resource provider with a small amount of VCPU
inventory. It then tries to allocate a large number VCPU via a new
consumer (and thus using consumer_generation of null).
This fails with a 409 because capacity is violated.
Then it queries to GET allocations on that consumer, to see if consumer
generation is available. It is not.
The it tries to allocate a smaller amount and though it might be
expected that the generation should still be null, it takes a 0
for the allocation to be accepted.
Once we decide what the right behavior is here, we can fix it.
Change-Id: I196834efc2cb6ece4204abbf8544c8f2621aaa36
Related-Bug: #1778591
Error handling to raise the exception 409 while deleting
the parent if it has children.
Closes-Bug: #1770636
Change-Id: I87df68992e4e635f009974f5205ca4919a4f2576
Add a change to _check_capacity_exceeded to also compare the amount
needed by a given allocation to a running total of amounts needed
against this class of resource on this resource provider.
Change-Id: Id8dde9a1f4b62112925616dfa54e77704109481c
Closes-Bug: #1778743
Add a suite of gabbit tests which demonstrate that it is possible to
violate inventory constraints in a POST to /allocations. Subsequent
patch will have a fix with the test adjusted to reflect the proper
behavior.
Change-Id: Ie6510deac262e040a56caedb5f21e414e897ed70
Related-Bug: #1778743
When PUTting allocations to a non-existent consumer, the code
expects the consumer generation to be 'null'. When there is only
one resource provider in the set of allocations sent, this works
well.
Unfortunately when there is more than one resource provider the
loop which calls _new_allocations still uses the 'null' generation
but now the code expects the generation to be 0, because the
consumer now exists.
This patch provides a gabbi test that demonstrates the problem
by creating a shared disk resource provider and associating it
as an aggregate with the existing RP_UUID. A GET
/allocation_candidates is made and a single allocation_request is
returned including both resource providers. A test with a new
consumer is marked as an xfail.
In the next patch the problem will be fixed.
Change-Id: I1238f82dfc46fd9dae126abfd16ce3dd1506e991
Related-Bug: #1778576
The placement API had a message complaining that it was "expecting None"
which is python-speak when it should be JSON speak. This change
corrects the message and adds a gabbi test for it.
Change-Id: Ibfb3d581588c8eb954a9faca19d9cb26d2c28861
Closes-Bug: #1778071
This patch adds new placement API microversion for handling consumer
generations.
Change-Id: I978fdea51f2d6c2572498ef80640c92ab38afe65
Co-Authored-By: Ed Leafe <ed@leafe.com>
Blueprint: add-consumer-generation