Commit Graph

414 Commits

Author SHA1 Message Date
Chris Dent 787bb33606 Use external placement in functional tests
Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).

openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.

The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.

tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.

The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.

Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,

TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.

Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
2018-12-12 18:46:49 +00:00
Tetsuro Nakamura cdbedac920 Consider root id is None in the database case
There are cases where ``root_provider_id`` of a resource provider is
set to NULL just after it is upgraded to the Rocky release. In such
cases getting allocation candidates raises a Keyerror.

This patch fixes that bug for cases there is no sharing or nested
providers in play.

Change-Id: I9639d852078c95de506110f24d3f35e7cf5e361e
Closes-Bug:#1799892
2018-11-20 14:53:59 +00:00
Zuul ed5e7fb2c2 Merge "Add recreate test for bug 1799892" 2018-11-09 23:20:34 +00:00
Balazs Gibizer b93b40c6c0 Add bandwidth related standard resource classes
Introduce NET_BW_EGR_KILOBIT_PER_SEC and
NET_BW_IGR_KILOBIT_PER_SEC resource classes.

blueprint bandwidth-resource-provider

Change-Id: I996bf705b14b564106426a2e57299638fb178750
2018-11-05 17:47:13 -05:00
Balazs Gibizer d4f8974f87 Add request_spec.RequestGroup versioned object
Later patches will introduce a field in RequestSpec using this type as
the field type to store the resource requests coming from outside of
Nova like the bandwidth request coming from the Neutron ports.

This patch refactors the usage of placement.lib.RequestGroup. Until now
this class was used both by placement and nova services and they used
it only as a util class. However after this series the nova services
would like to use such a class via RPC which requires an OVO. This
patch makes sure that the new OVO is used by nova and the old plain
object is used by placement. This way placement is not forced to use
an OVO where no OVO functionality is required.

The minimum required version of oslo.versionedobjects is updated to
1.33.3 to include the fix for bug 1771804.

Change-Id: I46c97d2641d9685ef59771314665a17a5236097d
blueprint: bandwidth-resource-provider
2018-11-05 17:47:06 -05:00
Tetsuro Nakamura 95d0ebc3d8 Add recreate test for bug 1799892
There are cases where ``root_provider_id`` of a resource provider is
set to NULL just after it is upgraded to the Rocky release. In such
cases getting allocation candidates raises a Keyerror.

This patch recreate that bug by simulating the situation by
inserting the records to the database directly.

Change-Id: Iaed912314f3e8fef2f46453a6bf12011702ae1dd
Related-Bug:#1799892
2018-10-26 07:27:28 +00:00
Mohammed Naser 730936e535 Use unique consumer_id when doing online data migration
If there are multiple consumers having allocations to the same
resource provider, with different classes, it will attempt
multiple INSERTs with the same consumer_id which is not allowed
because of the database constraints.

This patch adds a simple GROUP BY in order to ensure that the
database server only provides us with unique values to avoid
trying to INSERT duplicate values.

Change-Id: I1acba5e65cd562472f29e354c6077f82844fa87d
Closes-Bug: #1798163
2018-10-16 15:30:38 -04:00
Matt Riedemann 618b47627d Add recreate test for bug 1798163
Change Icae5038190ab8c7bbdb38d54ae909fcbf9048912 in Rocky
attempts to online migrate missing consumers table records
when listing allocations for a given resource provider. The
problem is when it's doing an insert-from-select, it's not
handling multiple allocations on the same provider for the
same consumer, like you'd have with a compute instance that
has VCPU, MEMORY_MB and DISK_GB allocations against a single
compute node resource provider. As a result, the insert
statement has duplicate consumer IDs in it which results in
a unique constraint violation.

The existing tests never caught this because they tested with
3 unique consumers with a single allocation each.

The functional test added here hits both online data migration
routines: via the API when listing allocations for a resource
provider and the direct online data migration CLI.

Change-Id: Iba56aa6b227b6455d2437e4fabcd296b1b0f06ee
Related-Bug: #1798163
2018-10-16 15:21:35 -04:00
Eric Fried 8e1ca5bf34 Use uuidsentinel from oslo.utils
oslo.utils release 3.37.0 [1] introduced uuidsentinel [2]. This change
rips out nova's uuidsentinel and replaces it with the one from
oslo.utils.

[1] https://review.openstack.org/#/c/599754/
[2] https://review.openstack.org/#/c/594179/

Change-Id: I7f5f08691ca3f73073c66c29dddb996fb2c2b266
Depends-On: https://review.openstack.org/600041
2018-09-05 09:08:54 -05:00
Zuul 3779211031 Merge "Fix race condition in reshaper handler" 2018-08-30 17:35:09 +00:00
Zuul 18e801d4de Merge "reshaper gabbit: Nix comments re doubled max_unit" 2018-08-30 17:21:13 +00:00
Zuul 06e5a22bc6 Merge "reshaper: Look up provider if not in inventories" 2018-08-30 01:12:50 +00:00
Zuul 7116d8daf8 Merge "[placement] Add /reshaper handler for POST" 2018-08-29 09:22:10 +00:00
Eric Fried 2a50107606 reshaper gabbit: Nix comments re doubled max_unit
As noted in a review [1] on the reshaper series, the comments about
doubling max_unit were obsolete (fixed via [2]), and are hereby removed.

[1] https://review.openstack.org/#/c/576927/35/nova/tests/functional/api/openstack/placement/gabbits/reshaper.yaml@222
[2] I8ba378ff5eeaf6c9cca11c5874708a17d4640097

Change-Id: Iaf3df651a76705d0f64182c46c665f037ea51b68
2018-08-28 14:46:00 -05:00
Eric Fried d98986415c Fix race condition in reshaper handler
Fix the race condition identified at [1] by indexing the `inventories`
argument to the resource_provider.reshape method by ResourceProvider
object rather than its UUID. That means we never have to do a fresh
lookup for the provider.

[1] https://review.openstack.org/#/c/585033/18/nova/api/openstack/placement/objects/resource_provider.py@4195

Change-Id: Ie21bddc186364db2b10af45546c35b9ad5a0208b
2018-08-25 10:45:49 -05:00
Zuul 7107eff2a2 Merge "[placement] Add functional test to verify presence of policy" 2018-08-23 21:30:55 +00:00
Eric Fried d13e5a5011 reshaper: Look up provider if not in inventories
Per the referenced bug, we weren't accounting for the scenario where a
reshape operation was removing *all* inventories for a provider (which
could be fairly common). With this fix, we do a three-stage lookup of
the provider object: If it's not in the inventories, we look in the
allocations; if it's not in the allocations, we look it up in the
database.

Change-Id: I594bb64f87c61b7ffd39c19e0fd42c4c087a3a11
Closes-Bug: #1783130
2018-08-23 08:45:34 -05:00
Chris Dent 3a24000514 [placement] Add functional test to verify presence of policy
Add a test that traverses all available placement URLs at the latest
microversion and tries to access them as non-admin. If something other
than a 403 response is given a failed test with a message like

    method POST on route /resource_providers/{uuid}/inventories
    is open for user, status: 404

is produced.

This works because we do authZ handling early in each handler, before
data processing and path parameter handling.

The method is pretty straightforward: traverse ROUTE_DECLARATIONS, visit
every url with each the declared methods, except the root version document,
and confirm a 403 response when the provided auth token is non-admin.

This has been created to avoid situations where a route is added without
policy like happened on https://review.openstack.org/#/c/576927/ . Until
recently we had a failover where any route not defined to have policy
would default to admin. That went away so now we need some test
automation to catch our forgetful humanness.

Change-Id: Id582886ec4b621b97d7cc7237b4670ad7bb12295
2018-08-23 10:27:53 +01:00
rajat29 d27d8e44a3 Normalize dashless 'resource provider create' uuid
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.

This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.

Co-Authored-By: Chen <dstbtgagt@foxmail.com>

Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
2018-08-23 11:20:42 +08:00
Chris Dent 4d525b4ec1 [placement] Add /reshaper handler for POST
/reshaper provides a way to atomically modify some allocations and
inventory in a single transaction, allowing operations like migrating
some inventory from a parent provider to a new child.

A fair amount of code is reused from handler/inventory.py, some
refactoring is in order before things get too far with that.

In handler/allocation.py some code is extracted to its own methods
so it can be reused from reshaper.py.

This is done as microversion 1.30.

A suite of gabbi tests is provided which attempt to cover various
failures including schema violations, generation conflicts, and
data conflicts.

api-ref, release notes and rest history are updated

Change-Id: I5b33ac3572bc3789878174ffc86ca42ae8035cfa
Partially-Implements: blueprint reshape-provider-tree
2018-08-23 00:36:17 +00:00
Chris Dent 2b46354d5a Set policy_opt defaults in placement gabbi fixture
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have other tests running prior to
gabbi tests. This change ensure the opts are registered and defaulted.

Change-Id: I1c7e347b6e788928bef96e32c3365d0fdc5ba00f
Related-Bug: #1786498
Closes-Bug: #1788176
2018-08-21 14:28:32 +01:00
Eric Fried 327448e7cb Test case for multiple forbidden traits
Test case to expose a bug at [1] where symmetric difference (^) is being
used instead of difference (-).

[1] https://review.openstack.org/#/c/590041/6/nova/api/openstack/placement/objects/resource_provider.py@2911

Change-Id: I2e80fe75da1d1104d0562f087c74117fbe6b457b
2018-08-10 16:19:41 -04:00
Tetsuro Nakamura dc26780ef8 Adds a test for _get_provider_ids_matching()
This patch adds a test for _get_provider_ids_matching()
to verify it works correctly with required traits.

Related-Bug: #1786519
Change-Id: I2512e361f5eaa4e60701be7c8bf57b2e0a02a146
2018-08-10 16:19:41 -04:00
Zuul 681cb7f21f Merge "Use common functions in granular fixture" 2018-08-07 23:16:15 +00:00
Zuul 41692fc5e9 Merge "Use common functions in NonSharedStorageFixture" 2018-08-07 05:01:48 +00:00
Tetsuro Nakamura 45e34808f5 Use common functions in granular fixture
For code refactoring purpose in the placement granular fixture
setup, this patch substitutes common functions in test_base.py
for existing local functions of _create_providers(),
_add_inventory(), and _set_traits().

Change-Id: I76a3f7d7e446e0f3af379f83c9d8333279884c73
2018-08-07 11:03:37 +09:00
Zuul 5f3b4aa44f Merge "Not use project table for user table" 2018-08-07 01:11:39 +00:00
Zuul 1d97b663d3 Merge "Adds a test for getting allocations API" 2018-08-07 00:50:17 +00:00
Zuul 954038d795 Merge "Refactor AllocationFixture in placement test" 2018-08-07 00:46:16 +00:00
Zuul ef50733478 Merge "Increase max_unit in placement test fixture" 2018-08-06 20:21:40 +00:00
Tetsuro Nakamura e13b765e55 Not use project table for user table
`GET /resource_provider/{uuid}/allocations` API didn't
return all the allocations made by multiple users.

This was because the placement wrongly used project table
for user table. This patch fixes it with the test case.

Change-Id: I7c808dec5de1204ced0d1f1b31d8398de8c51679
Closes-Bug: #1785382
2018-08-04 19:34:04 +09:00
Tetsuro Nakamura 9cfc598acf Adds a test for getting allocations API
`GET /resource_provider/{uuid}/allocations` API currently doesn't
return all the allocations made by multiple users.

This patch adds a test to describe this bug. The fix for this
is coming in a follow up.

Change-Id: I2b01e27922f11bef2defcb01fe415692de1578ea
Partial-Bug: #1785382
2018-08-04 19:20:30 +09:00
Chris Dent 08f23b5810 [placement] Move resource_class_cache into placement hierarchy
Since only placement uses the resource_class_cache, move it
into the placement hierarchy.

Change-Id: I837d4b691f94af0487a66dacf5ae563e4423f6a3
2018-08-03 14:55:49 +01:00
Tetsuro Nakamura 7d824e6d37 Refactor AllocationFixture in placement test
This patch refactors the allocation fixture used in some placement
api tests by substituting common functions in test_base.py for the
existing object management functions.

Change-Id: Ide6544d1cf9e1ed154b42075acdd7af986c3afe8
2018-08-02 14:46:10 +09:00
Tetsuro Nakamura e4923abaeb Increase max_unit in placement test fixture
In the AllocationFixture setup, to avoid the limitation of the
max_unit that is set to the inventory, we were using a bit hacky
way creating multiple allocation objects that have the same resource
class, the same consumer, and the same resource provider.

Since this is not how it works in real cases, and this prevents us
from refactoring, this patch fixes it.

Change-Id: I8ba378ff5eeaf6c9cca11c5874708a17d4640097
2018-08-02 14:00:45 +09:00
Tetsuro Nakamura e23237c375 Use common functions in NonSharedStorageFixture
For code refactoring purpose in the placement NonSharedStorageFixture
setup, this patch substitutes common functions in test_base.py for
existing native setup functions for creating providers and setting
inventories.

Change-Id: I312333ed8ecd51b9f3f6b818c33b3ef54703f997
2018-08-02 10:56:57 +09:00
Zuul 392b38b6f9 Merge "Fix comments in _anchors_for_sharing_providers and related test" 2018-08-01 13:45:28 +00:00
Zuul 5a76921712 Merge "[placement] Use a simplified WarningsFixture" 2018-08-01 13:32:37 +00:00
Zuul fbfe876423 Merge "[placement] Use a non-nova log capture fixture" 2018-08-01 13:32:31 +00:00
Zuul 72226f95d8 Merge "[placement] Use oslotest CaptureOutput fixture" 2018-08-01 13:31:38 +00:00
Zuul 18dcfbb0b6 Merge "Add placement.concurrent_udpate to generation pre-checks" 2018-08-01 12:19:49 +00:00
Yikun Jiang 04afe2aae9 Fix comments in _anchors_for_sharing_providers and related test
1. Remove "GROUP by" line in _anchors_for_sharing_providers, because
this has been removed in Ib1738fb4a4664aa7b78398655fd23159a54f5f69.
2. Add reminder note when we are sure all root_provider_id values are
NOT NULL.
3. Fix note in test_anchors_for_sharing_providers, s1 get r3 only via
agg3.

trivialfix

Change-Id: Id8bfd83db58366047267ff0eeb2930a19bddbf4e
2018-08-01 16:34:50 +08:00
Zuul c90d6101f4 Merge "Remove redundant join in _anchors_for_sharing_providers" 2018-07-31 18:28:39 +00:00
Yikun Jiang 1c66a4b029 Remove redundant join in _anchors_for_sharing_providers
There is a redundant join when we want to get id from
_anchors_for_sharing_providers. The last outerjoin is used to get the
rp.UUID according rp.id, if we set get_id=True, we no longer need this
outer join.

So, we remove the redundant join in this patch.

Change-Id: Ib5fc6e4efae29dd88ce92df834700d2121ed8076
Closes-bug: #1784604
2018-07-31 18:27:39 +08:00
Chris Dent 72e4c4c8d7 [placement] Retry allocation writes server side
This change adds a fast retry loop around
AllocationList._set_allocations if a resource provider generation
conflict happens. It turns out that under high concurrency of allocation
claims being made on the same resource provider conflicts can be quite
common and client side retries are insufficient.

Because both consumer generation and resource provider generations had
raised the same exception there was no way to distinguish between the
two so a child of ConcurrentUpdateDetected has been created as
ResourceProviderConcurrentUpdateDetected. In the future this will allow
us to send different error codes to the client as well, but that change
is not done here.

When the conflict is detected, all the resource providers in the
AllocationList are reloaded and the list objects refreshed.

Logging is provided to indicate:

* at debug that a retry is going to happen
* at warning that all the retries failed and the client is going to
  see the conflict

The tests for this are a bit funky: Some mocks are used to cause the
conflicts, then the real actions after a couple of iterations.

Change-Id: Id614d609fc8f3ed2d2ff29a2b52143f53b3b1b9a
Closes-Bug: #1719933
2018-07-31 10:57:16 +01:00
Chris Dent 4b1d38e88a [placement] Use a simplified WarningsFixture
Use a WarningsFixture specific to placement that worries about fewer
warnings and is not dependent on nova.

blueprint: placement-extract

Change-Id: Idfcc6882d7fe5141dcc793f0409f75c51fd26234
2018-07-30 19:38:35 +01:00
Chris Dent 1de7ac302a [placement] Use a non-nova log capture fixture
We want to avoid using the nova StandardLogging fixture, to limit
imports from nova, but it has two useful features that we want:

* always capture
* if the chosen log level is more than DEBUG, format DEBUG message
  anyway, but don't output

blueprint: placement-extract

Change-Id: Iadd32c731ebfb5a62308a4d5f907a69f93590935
2018-07-30 19:38:22 +01:00
Chris Dent bee0a133e5 [placement] Use oslotest CaptureOutput fixture
Instead of the nova fixture OutputStreamCapture. They do effectively
the same thing and once placement is extracted we'd like to not
have duplication.

blueprint: placement-extract

Change-Id: I4636533b1262f819e34ea78cca33ad9f90a35702
2018-07-30 19:12:19 +01:00
Chris Dent f5783d90bc [placement] Use base test in placement functional tests
There is now a placement.base.TestCase for placement functional tests
which assembles the necessary configuration and fixtures. This change
uses that base class in the db functional tests and extends the base
class as required to add all the necessary functionality.

In the process issues were exposed in the fixtures.gabbits use of
oslo_config (causing tests to fail based on fallout from changes
elsewhere in the functional tests) so this change also fixes that
and limits the gabbi tests to only caring about the placement database
connection, which is more correct and how it should have been all
along.

This change removes the ConfPatcher fixture in fixtures.placement
because it is no better than using the oslo_config provided
fixture and was in the way while diagnosing difficulties with
getting these changes to work correctly.

The root cause of those problems was that placement changes were
at cross purposes with how the nova.tests.fixtures.Database fixture
expects to work: The goal of these changes was to only configure
and establish those fixtures that were strictly necessary for
placement in the placements tests. However, when _any_ database
is requested from the Database fixture, the context managers for
all of them are configured.

This means, for example, that if a placement fixture, which
originally was not configuring a 'connection' string for the api
or main databases, ran before a later api db fixture, the api db
fixture would fail with no connection string available.

The quick and dirty fix is used here to fix the problem: we set
reasonable configuration for all three databases in the placement
tests that need the placement database fixture.

In the future when placement is extracted, these problems go away
so it does not seem worth the effort (at least not now) to
restructure the nova Database fixture.

blueprint: placement-extract
Change-Id: Ice89e9a25f74caaa53b7df079bd529d172354524
2018-07-26 17:52:59 +01:00
Chris Dent d1b087d59e [placement] Extract base functional test case from test_direct
In subsequent patches this will be used for making the rest of the
placement functional tests either not-use or centralized their
dependence on nova test fixtures and base classes.

blueprint: placement-extract
Change-Id: I24d9fb2d06bf5c99bcea6c9f138ce31713bb8356
2018-07-26 01:55:13 +01:00