Commit Graph

15 Commits

Author SHA1 Message Date
Chris Dent 787bb33606 Use external placement in functional tests
Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).

openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.

The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.

tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.

The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.

Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,

TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.

Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
2018-12-12 18:46:49 +00:00
Tetsuro Nakamura 5b4aa78459 Add microversion for nested allocation candidate
This patch adds a microversion with a release note for allocation
candidates with nested resource provider trees.

From now on we support allocation candidates with nested resource
providers with the following features.

1) ``GET /allocation_candidates`` is aware of nested providers.
   Namely, when provider trees are present, ``allocation_requests``
   in the response of ``GET /allocation_candidates`` can include
   allocations on combinations of multiple resource providers
   in the same tree.
2) ``root_provider_uuid`` and ``parent_provider_uuid`` fields are
    added to ``provider_summaries`` in the response of
   ``GET /allocation_candidates``.

Change-Id: I6cecb25c6c16cecc23d4008474d150b1f15f7d8a
Blueprint: nested-resource-providers-allocation-candidates
2018-06-29 17:38:10 +09:00
Tetsuro Nakamura 97530c2ca3 Return all resources in provider_summaries
The response of ``GET /allocation_candidates`` API provides two fields
of ``allocation_requests`` and ``provider_summaries``. The callers,
like the filter scheduler in nova, would use information in
``provider_summaries`` in sorting or filtering providers to allocate
consumers. However, currently ``provider_summaries`` doesn't contain
resource classes that aren't requested.

With this patch, ``GET /allocation_candidates`` API returns all
resource classes with a new microversion.

Change-Id: Ic491f190ebd97d94c18931a0e78d779a55ee47a1
Closes-Bug: #1760276
Blueprint: placement-return-all-resources
2018-05-29 03:16:13 +09:00
Chris Dent 4e07d81260 [placement] Support forbidden traits in API
In a new microversion (1.22) expose support for processing
forbidden traits in GET /resource_providers and GET
/allocation_candidates. A forbidden trait is expressed as
part of the required parameter with a "!" prefix:

    required=CUSTOM_FAST,!CUSTOM_SLOW

This change uses db and query processing code adjustments
already present in the code but guarded by a flag. If the
currently requested microversion matches 1.22 or beyond
that flag is True, otherwise False.

Reno, api-ref update and api history update are included.
Because this microversion changes the value of an existing
parameter it was unclear how to best express that in the
api-ref. In this case existing parameter references were
annotated.

Partially implements blueprint placement-forbidden-traits

Change-Id: I43e92bc5f97db7a2b09e64c6cb953c07d0561e63
2018-04-13 19:24:08 +01:00
Chris Dent 8bde4042da [placement] Parse forbidden traits in query strings
Add support to parse_qs_request_groups to optionally process the
"required" parameter for forbidden traits, expressed as the
normal trait form prefixed with "!". The parsing removes "!"
prefixed traits from the required_traits attribute on a
RequestGroup and puts them (without the "!") in a new
forbidden_traits attribute.

The optionality allows the caller to control the processing
based on the microversion. Later code will add that to
the list resource providers and list allocation canidates
handler code, in the same microversion.

This allows declaring forbidden handling only at the API layer.
Subsequent patches will turn on forbidden handling in the object/
data layer but no forbidden traits will reach that layer until
the microversion turns on allow_forbidden in the API.

Partially implements blueprint placement-forbidden-traits
Change-Id: Icc9dc2631a0eca9e998b9ce8706c7b6920e0cb69
2018-04-13 19:16:07 +01:00
Tetsuro Nakamura a6122a7a64 [placement] Add test for provider summaries
In ``GET /allocation_candidates`` API, ``provider_summaries``
should show all the inventories for all the resource classes
in all the resource providers.

However, ``provider_summaries`` currently doesn't contain
resources that aren't requested.

This patch adds a simple test case to describe the bug.

Change-Id: Ida9cddc5a2b43917e7b5900fe68a413b18c6cb2d
Partial-Bug: #1760276
2018-04-01 18:41:47 +09:00
He Jie Xu 8a307bbdf4 placement: support traits in allocation candidates API
This patch add new query parameter `required` to the
`GET /allocation_candidates` API, which is used to filter candidates
with required traits.  The candidate attached traits return in the
provider summary also. Those API changes are added by new microversion.

Also using specific exception TraitNotFound instead of the generic
exception ValueError when invalid traits in the request.

Change-Id: Id821b5b2768dcc698695ba6570c6201e1e9a8233
Implement blueprint add-trait-support-in-allocation-candidates
2018-01-22 22:10:10 +08:00
Chris Dent 4a97bbd824 [placement] Enable limiting GET /allocation_candidates
This adds a limit query parameter to GET
/allocation_candidates?limit=5&resource=VCPU:1

A 'limit' filter is added to the AllocationCandidates. If set, after
the database query has been run to create the allocation requests and
provider summaries, a slice or sample of the allocation requests is
taken to limit the results. The summaries are then filtered to only
include those in the allocation requests.

This method avoids needing to make changes to the generated SQL, the
creation of which is fairly complex, or the database tables. The amount
of data queried is still high in the extreme case, but the amount of
data sent over the wire (as JSON) is shrunk. This is a trade-off that
was discussed in the spec and the discussion surrounding its review.
If it turns out that memory use server-side is an issue we can
investigate changing the SQL.

A configuration setting, [placement]/randomize_allocation_candidates,
is added to allow deployers to declare whether they want the results
to be returned in whatever order the database chooses or a random
order. The default is "False" which is expected to preserve existing
behavior and impose a packing placement strategy.

When the config setting is combined with the limit parameter, if
"True" the limited results are a random sampling from the full
results. If "False", it is a slice from the front.

This is done as a new microversion, 1.16, with updates to docs, a reno
and adjustments to the api history doc.

Change-Id: I5f3d4f49c34fd3cd6b9d2e12b3c3c4cdcb409bec
Implements: bp allocation-candidates-limit
2017-12-20 20:08:39 +00:00
Chris Dent 83030804cc [placement] Add cache headers to placement api requests
In relevant requests to the placement API add last-modified
and cache-control headers.

According the HTTP 1.1 RFC headers last-modified headers SHOULD always
be sent and should have a tie to the real last modified time. If we do
send them, we need Cache-Control headers to prevent inadvertent caching
of resources.

This change adds a microversion 1.15 which adds the headers to GET
requests and some PUT or POST requests.

Despite what it says 'no-cache' means "check to see if the version you
have is still valid as far as the server is concerned". Since our server
doesn't currently validate conditional requests and will always return an
entity, it ends up meaning "don't cache" (which is what we want).

The main steps in the patch are:

* To both the get single entity and get collection handlers add
  response.cache_control = 'no-cache'
* For single entity add response.last_modified = obj.updated_at or
  obj.created_at
* For collections, discover the max modified time when traversing the
  list of objects to create the serialized JSON output. In most of
  those loops an optimization is done where we only check for
  last-modified information if we have a high enough microversion such
  that the information will be used. This is not done when listing
  inventories because the expectation is that no single resource
  provider will ever have a huge number of inventory records.
* Both of the prior steps are assisted by a new util method:
  pick_last_modfied.

Where a time cannot be determined the current time is used.

In typical placement framework fashion this has been done in a very
explicit way, as it makes what the handler is doing very visible, even
though it results in a bit of boilerplate.

For those requests that are created from multiple objects or by doing
calculations, such as usages and aggregate associations, the current time
is used.

The handler for PUT /traits is modified a bit more extensively than some
of the others: This is because the method can either create or validate
the existence of the trait. In the case where the trait already exists,
we need to get it from the DB to get its created_at time. We only do
this if the microversion is high enough (at least 1.15) to warrant
needing the info.

Because these changes add new headers (even though they don't do
anything) a new microversion, 1.15, is added.

Partial-Bug: #1632852
Partially-Implements: bp placement-cache-headers

Change-Id: I727d4c77aaa31f0ef31c8af22c2d46cad8ab8b8e
2017-12-12 15:51:58 +00:00
Chris Dent 808323e0c5 [placement] Symmetric GET and PUT /allocations/{consumer_uuid}
In a new microversion, 1.12, include project_id and user_id in the
output of GET /allocations/{consumer_uuid} and add JSON schema
to enable PUT to /allocations/{consumer_uuid} using the same dict-based
format for request body that is used in the GET response. In later
commits a similar format will be used in POST /allocations. This
symmetry is general good form and also will make client code a little
easier.

Since GET /allocation_candiates includes objects which are capable
of being PUT to /allocations/{consumer_uuid}, its response body has
been updated as well, to change the 'allocation_requests' object
to use the dict-based format.

Internally to handlers/allocation.py the same method (_set_allocations)
is used for every microversion. Any previous data structure is
transformed into the dict-ish form. This means that pre-existing tests
(like allocation-bad-class.yaml) continue to exercise the problems it
was made for, but needs to be pinned to an older microversion, rather than
being latest.

Info about these changes is added to placement-api-ref,
rest_api_version_history and a reno.

Change-Id: I49f5680c15413bce27f2abba68b699f3ea95dcdc
Implements: bp symmetric-allocations
Closes-Bug: #1708204
2017-11-21 19:39:59 +00:00
Chris Dent 643ed234da [placement] Confirm that empty resources query causes 400
When a request is made to /resource_providers or /allocation_candidates
with a 'resources' query parameter, and empty value should lead to a
400. This was indeed happening, but there was no gabbi test coverage for
it. This adds tests at both URLs.

Note that because the query parameter is processed by the same code for
both /resource_providers and /allocation_candidates the test duplicates
a code a path, however, that's an implementation detail that we don't
know at the HTTP API layer.

Change-Id: Ia6fc0ac09efc5407a899de4215a2ba0ba0b017fb
Closes-Bug: #1723122
2017-10-12 15:29:38 +01:00
Chris Dent 7792ba33a0 [placement] Update allocation-candidates.yaml for gabbi 1.35
Gabbi 1.35 provides template variable susbtitution on the left hand side
of a JSONPath expression. That's used here to remove the need for static
fake uuids in the test data for allocation-candidates and the
SharedStorageFixture.

A couple of other clean ups to the use of gabbi, unrelated to the 1.35
update:

* It's possible to test the entire value in one response:

    resources:
         VCPU: 4
         DISK_GB: 5

  instead of:

    resources.VCPU: 4
    resources.DISK_GB: 5

  This is handy when the root of the expression is calculated and
  repeating that calculation is noisy.

* [?foo.bar=baz] is the more canonical form of [?(@.foo.bar=baz)]. The
  former is built into jsonpath, the latter dips into the underlying
  implementation.

Change-Id: Ie6b5e0186533581815d04411ed7b321948464bb6
2017-07-17 13:04:48 +00:00
Chris Dent 5da190ff12 [placement] Improve allocation_candidates coverage
The case where a bad resource class in resources can cause a 400
response was not covered. This adds it; things work as desired.

Change-Id: Ieb93889f76070b9ae37c082301488a953106d914
2017-07-06 17:34:40 +00:00
Sylvain Bauza 07d82cf8ef Reset the traits sync flag in the placement fixtures
We had a problem with tests running that were unable to find traits because a
previous test cleaned up the DB without resetting the sync flag and we fixed
that in I5dc0220310986c5b9fe5b71013c573c3d98cb9aa

Unfortunately, fixtures we use in Placement functional tree don't inherit from
general test cases (for obvious reasons) so we need to also reset the flag
there.

Co-Authored-By: Sylvain Bauza <sbauza@redhat.com>

Change-Id: Ib7019db36ce90bf6d762a6d37697c3a215e091b4
Closes-Bug: #1702304
2017-07-04 18:40:39 +02:00
Jay Pipes 1d01a8811a placement: support GET /allocation_candidates
A new 1.10 API microversion is added to return information that the
scheduler can use to select a particular set of resource providers to
claim resources for an instance.

The GET /allocation_candidates endpoint takes a "resources" querystring
parameter similar to the GET /resource_providers endpoint and returns a
dict with two top-level elements:

"allocation_requests" is a list of JSON objects that contain a
serialized HTTP body that the scheduler may subsequently use in a call
to PUT /allocations/{consumer_uuid} to claim resources against a
related set of resource providers.

"provider_summaries" is a JSON object, keyed by resource provider UUID,
of JSON objects of inventory/capacity information that the scheduler
can use to sort/weigh the results of the call when making its
destination host decisions.

Change-Id: I8dadb364746553d9495aa8bcffd0346ebc0b4baa
blueprint: placement-allocation-requests
2017-07-03 10:57:59 -04:00