Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).
openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.
The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.
tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.
The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.
Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,
TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.
Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
There are cases where ``root_provider_id`` of a resource provider is
set to NULL just after it is upgraded to the Rocky release. In such
cases getting allocation candidates raises a Keyerror.
This patch fixes that bug for cases there is no sharing or nested
providers in play.
Change-Id: I9639d852078c95de506110f24d3f35e7cf5e361e
Closes-Bug:#1799892
Later patches will introduce a field in RequestSpec using this type as
the field type to store the resource requests coming from outside of
Nova like the bandwidth request coming from the Neutron ports.
This patch refactors the usage of placement.lib.RequestGroup. Until now
this class was used both by placement and nova services and they used
it only as a util class. However after this series the nova services
would like to use such a class via RPC which requires an OVO. This
patch makes sure that the new OVO is used by nova and the old plain
object is used by placement. This way placement is not forced to use
an OVO where no OVO functionality is required.
The minimum required version of oslo.versionedobjects is updated to
1.33.3 to include the fix for bug 1771804.
Change-Id: I46c97d2641d9685ef59771314665a17a5236097d
blueprint: bandwidth-resource-provider
There are cases where ``root_provider_id`` of a resource provider is
set to NULL just after it is upgraded to the Rocky release. In such
cases getting allocation candidates raises a Keyerror.
This patch recreate that bug by simulating the situation by
inserting the records to the database directly.
Change-Id: Iaed912314f3e8fef2f46453a6bf12011702ae1dd
Related-Bug:#1799892
If there are multiple consumers having allocations to the same
resource provider, with different classes, it will attempt
multiple INSERTs with the same consumer_id which is not allowed
because of the database constraints.
This patch adds a simple GROUP BY in order to ensure that the
database server only provides us with unique values to avoid
trying to INSERT duplicate values.
Change-Id: I1acba5e65cd562472f29e354c6077f82844fa87d
Closes-Bug: #1798163
Change Icae5038190ab8c7bbdb38d54ae909fcbf9048912 in Rocky
attempts to online migrate missing consumers table records
when listing allocations for a given resource provider. The
problem is when it's doing an insert-from-select, it's not
handling multiple allocations on the same provider for the
same consumer, like you'd have with a compute instance that
has VCPU, MEMORY_MB and DISK_GB allocations against a single
compute node resource provider. As a result, the insert
statement has duplicate consumer IDs in it which results in
a unique constraint violation.
The existing tests never caught this because they tested with
3 unique consumers with a single allocation each.
The functional test added here hits both online data migration
routines: via the API when listing allocations for a resource
provider and the direct online data migration CLI.
Change-Id: Iba56aa6b227b6455d2437e4fabcd296b1b0f06ee
Related-Bug: #1798163
Per the referenced bug, we weren't accounting for the scenario where a
reshape operation was removing *all* inventories for a provider (which
could be fairly common). With this fix, we do a three-stage lookup of
the provider object: If it's not in the inventories, we look in the
allocations; if it's not in the allocations, we look it up in the
database.
Change-Id: I594bb64f87c61b7ffd39c19e0fd42c4c087a3a11
Closes-Bug: #1783130
Add a test that traverses all available placement URLs at the latest
microversion and tries to access them as non-admin. If something other
than a 403 response is given a failed test with a message like
method POST on route /resource_providers/{uuid}/inventories
is open for user, status: 404
is produced.
This works because we do authZ handling early in each handler, before
data processing and path parameter handling.
The method is pretty straightforward: traverse ROUTE_DECLARATIONS, visit
every url with each the declared methods, except the root version document,
and confirm a 403 response when the provided auth token is non-admin.
This has been created to avoid situations where a route is added without
policy like happened on https://review.openstack.org/#/c/576927/ . Until
recently we had a failover where any route not defined to have policy
would default to admin. That went away so now we need some test
automation to catch our forgetful humanness.
Change-Id: Id582886ec4b621b97d7cc7237b4670ad7bb12295
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.
This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.
Co-Authored-By: Chen <dstbtgagt@foxmail.com>
Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
/reshaper provides a way to atomically modify some allocations and
inventory in a single transaction, allowing operations like migrating
some inventory from a parent provider to a new child.
A fair amount of code is reused from handler/inventory.py, some
refactoring is in order before things get too far with that.
In handler/allocation.py some code is extracted to its own methods
so it can be reused from reshaper.py.
This is done as microversion 1.30.
A suite of gabbi tests is provided which attempt to cover various
failures including schema violations, generation conflicts, and
data conflicts.
api-ref, release notes and rest history are updated
Change-Id: I5b33ac3572bc3789878174ffc86ca42ae8035cfa
Partially-Implements: blueprint reshape-provider-tree
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have other tests running prior to
gabbi tests. This change ensure the opts are registered and defaulted.
Change-Id: I1c7e347b6e788928bef96e32c3365d0fdc5ba00f
Related-Bug: #1786498
Closes-Bug: #1788176
This patch adds a test for _get_provider_ids_matching()
to verify it works correctly with required traits.
Related-Bug: #1786519
Change-Id: I2512e361f5eaa4e60701be7c8bf57b2e0a02a146
For code refactoring purpose in the placement granular fixture
setup, this patch substitutes common functions in test_base.py
for existing local functions of _create_providers(),
_add_inventory(), and _set_traits().
Change-Id: I76a3f7d7e446e0f3af379f83c9d8333279884c73
`GET /resource_provider/{uuid}/allocations` API didn't
return all the allocations made by multiple users.
This was because the placement wrongly used project table
for user table. This patch fixes it with the test case.
Change-Id: I7c808dec5de1204ced0d1f1b31d8398de8c51679
Closes-Bug: #1785382
`GET /resource_provider/{uuid}/allocations` API currently doesn't
return all the allocations made by multiple users.
This patch adds a test to describe this bug. The fix for this
is coming in a follow up.
Change-Id: I2b01e27922f11bef2defcb01fe415692de1578ea
Partial-Bug: #1785382
This patch refactors the allocation fixture used in some placement
api tests by substituting common functions in test_base.py for the
existing object management functions.
Change-Id: Ide6544d1cf9e1ed154b42075acdd7af986c3afe8
In the AllocationFixture setup, to avoid the limitation of the
max_unit that is set to the inventory, we were using a bit hacky
way creating multiple allocation objects that have the same resource
class, the same consumer, and the same resource provider.
Since this is not how it works in real cases, and this prevents us
from refactoring, this patch fixes it.
Change-Id: I8ba378ff5eeaf6c9cca11c5874708a17d4640097
For code refactoring purpose in the placement NonSharedStorageFixture
setup, this patch substitutes common functions in test_base.py for
existing native setup functions for creating providers and setting
inventories.
Change-Id: I312333ed8ecd51b9f3f6b818c33b3ef54703f997
1. Remove "GROUP by" line in _anchors_for_sharing_providers, because
this has been removed in Ib1738fb4a4664aa7b78398655fd23159a54f5f69.
2. Add reminder note when we are sure all root_provider_id values are
NOT NULL.
3. Fix note in test_anchors_for_sharing_providers, s1 get r3 only via
agg3.
trivialfix
Change-Id: Id8bfd83db58366047267ff0eeb2930a19bddbf4e
There is a redundant join when we want to get id from
_anchors_for_sharing_providers. The last outerjoin is used to get the
rp.UUID according rp.id, if we set get_id=True, we no longer need this
outer join.
So, we remove the redundant join in this patch.
Change-Id: Ib5fc6e4efae29dd88ce92df834700d2121ed8076
Closes-bug: #1784604
This change adds a fast retry loop around
AllocationList._set_allocations if a resource provider generation
conflict happens. It turns out that under high concurrency of allocation
claims being made on the same resource provider conflicts can be quite
common and client side retries are insufficient.
Because both consumer generation and resource provider generations had
raised the same exception there was no way to distinguish between the
two so a child of ConcurrentUpdateDetected has been created as
ResourceProviderConcurrentUpdateDetected. In the future this will allow
us to send different error codes to the client as well, but that change
is not done here.
When the conflict is detected, all the resource providers in the
AllocationList are reloaded and the list objects refreshed.
Logging is provided to indicate:
* at debug that a retry is going to happen
* at warning that all the retries failed and the client is going to
see the conflict
The tests for this are a bit funky: Some mocks are used to cause the
conflicts, then the real actions after a couple of iterations.
Change-Id: Id614d609fc8f3ed2d2ff29a2b52143f53b3b1b9a
Closes-Bug: #1719933
Use a WarningsFixture specific to placement that worries about fewer
warnings and is not dependent on nova.
blueprint: placement-extract
Change-Id: Idfcc6882d7fe5141dcc793f0409f75c51fd26234
We want to avoid using the nova StandardLogging fixture, to limit
imports from nova, but it has two useful features that we want:
* always capture
* if the chosen log level is more than DEBUG, format DEBUG message
anyway, but don't output
blueprint: placement-extract
Change-Id: Iadd32c731ebfb5a62308a4d5f907a69f93590935
Instead of the nova fixture OutputStreamCapture. They do effectively
the same thing and once placement is extracted we'd like to not
have duplication.
blueprint: placement-extract
Change-Id: I4636533b1262f819e34ea78cca33ad9f90a35702
There is now a placement.base.TestCase for placement functional tests
which assembles the necessary configuration and fixtures. This change
uses that base class in the db functional tests and extends the base
class as required to add all the necessary functionality.
In the process issues were exposed in the fixtures.gabbits use of
oslo_config (causing tests to fail based on fallout from changes
elsewhere in the functional tests) so this change also fixes that
and limits the gabbi tests to only caring about the placement database
connection, which is more correct and how it should have been all
along.
This change removes the ConfPatcher fixture in fixtures.placement
because it is no better than using the oslo_config provided
fixture and was in the way while diagnosing difficulties with
getting these changes to work correctly.
The root cause of those problems was that placement changes were
at cross purposes with how the nova.tests.fixtures.Database fixture
expects to work: The goal of these changes was to only configure
and establish those fixtures that were strictly necessary for
placement in the placements tests. However, when _any_ database
is requested from the Database fixture, the context managers for
all of them are configured.
This means, for example, that if a placement fixture, which
originally was not configuring a 'connection' string for the api
or main databases, ran before a later api db fixture, the api db
fixture would fail with no connection string available.
The quick and dirty fix is used here to fix the problem: we set
reasonable configuration for all three databases in the placement
tests that need the placement database fixture.
In the future when placement is extracted, these problems go away
so it does not seem worth the effort (at least not now) to
restructure the nova Database fixture.
blueprint: placement-extract
Change-Id: Ice89e9a25f74caaa53b7df079bd529d172354524
In subsequent patches this will be used for making the rest of the
placement functional tests either not-use or centralized their
dependence on nova test fixtures and base classes.
blueprint: placement-extract
Change-Id: I24d9fb2d06bf5c99bcea6c9f138ce31713bb8356