Commit Graph

28 Commits

Author SHA1 Message Date
Chris Dent 787bb33606 Use external placement in functional tests
Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).

openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.

The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.

tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.

The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.

Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,

TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.

Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
2018-12-12 18:46:49 +00:00
Zuul fdd4253f14 Merge "Add UUID validation for consumer_uuid" 2018-07-16 20:03:37 +00:00
Zuul 6ca30fd832 Merge "Resource_provider API handler does not return specific error codes" 2018-07-16 14:52:50 +00:00
Balazs Gibizer 928746a7bb Add UUID validation for consumer_uuid
When an allocation is created with PUT /allocations/{consumer_uuid} the
consumer_uuid is not validated as UUID in the API and the DB schema only
checks that is fits into String(36). However the Consumer object stores
this value in a UUIDField oslo only raises a warning today.

This patch adds a UUID validation for that URL path.

Change-Id: Idedd85ca9266f10ce09231c34cab7ca43029a56c
Closes-Bug: #1780238
2018-07-16 15:23:05 +02:00
Eric Fried a644b22056 Test for unsanitized consumer UUID
Add a gabbi test case demonstrating that the consumer with UUID
{8}-{4}-{4}-{4}-{12} is not the same as the consumer with UUID
{8}{4}{4}{4}{12}.

Change-Id: I1852ac6004fedc5dfa9dd3de187c937ab385d1b5
Related-Bug: #1758057
2018-07-11 13:24:28 +00:00
deepak_mourya a6411afbfa Resource_provider API handler does not return specific error codes
There are a few places in the resource_provider.py placement
API handler module that need to be updated to return custom
error codes.

Change-Id: I7efdf7f82d581cf5dc9a3decc063b69c8cef22ff
Closes-Bug: #1778929
2018-07-06 15:38:44 +05:30
Chris Dent 83030804cc [placement] Add cache headers to placement api requests
In relevant requests to the placement API add last-modified
and cache-control headers.

According the HTTP 1.1 RFC headers last-modified headers SHOULD always
be sent and should have a tie to the real last modified time. If we do
send them, we need Cache-Control headers to prevent inadvertent caching
of resources.

This change adds a microversion 1.15 which adds the headers to GET
requests and some PUT or POST requests.

Despite what it says 'no-cache' means "check to see if the version you
have is still valid as far as the server is concerned". Since our server
doesn't currently validate conditional requests and will always return an
entity, it ends up meaning "don't cache" (which is what we want).

The main steps in the patch are:

* To both the get single entity and get collection handlers add
  response.cache_control = 'no-cache'
* For single entity add response.last_modified = obj.updated_at or
  obj.created_at
* For collections, discover the max modified time when traversing the
  list of objects to create the serialized JSON output. In most of
  those loops an optimization is done where we only check for
  last-modified information if we have a high enough microversion such
  that the information will be used. This is not done when listing
  inventories because the expectation is that no single resource
  provider will ever have a huge number of inventory records.
* Both of the prior steps are assisted by a new util method:
  pick_last_modfied.

Where a time cannot be determined the current time is used.

In typical placement framework fashion this has been done in a very
explicit way, as it makes what the handler is doing very visible, even
though it results in a bit of boilerplate.

For those requests that are created from multiple objects or by doing
calculations, such as usages and aggregate associations, the current time
is used.

The handler for PUT /traits is modified a bit more extensively than some
of the others: This is because the method can either create or validate
the existence of the trait. In the case where the trait already exists,
we need to get it from the DB to get its created_at time. We only do
this if the microversion is high enough (at least 1.15) to warrant
needing the info.

Because these changes add new headers (even though they don't do
anything) a new microversion, 1.15, is added.

Partial-Bug: #1632852
Partially-Implements: bp placement-cache-headers

Change-Id: I727d4c77aaa31f0ef31c8af22c2d46cad8ab8b8e
2017-12-12 15:51:58 +00:00
Zuul 96fcf0c942 Merge "[placement] Clean up TODOs in allocations.yaml gabbit" 2017-11-28 19:09:07 +00:00
Chris Dent 8caf4f5148 [placement] POST /allocations to set allocations for >1 consumers
This provides microversion 1.13 of the placement API, giving the
ability to POST to /allocations to set (or clear) allocations for
more than one consumer uuid.

It builds on the recent work to support a dict-based JSON format
when doing a PUT to /allocations/{consumer_uuid}.

Being able to set allocations for multiple consumers in one request
helps to address race conditions when cleaning up allocations during
move operations in nova.

Clearing allocations is done by setting the 'allocations' key for a
specific consumer to an empty dict.

Updates to placement-api-ref, rest version history and a reno are
included.

Change-Id: I239f33841bb9fcd92b406f979674ae8c5f8d57e3
Implements: bp post-allocations
2017-11-28 12:15:53 +00:00
Chris Dent 2948ca86fe [placement] Clean up TODOs in allocations.yaml gabbit
With the avent of gabbi 1.35.0 (already in test-requirements),
subtitutions in the left hand side of a json path check are now allowed.
allocations.yaml had two TODOs related to this which this change fixes.

Also fixed:

* a TODO about the nature of consumer id's is changed to a NOTE since
  the behavior is as designed
* a whitespace nit is fixed, since I'm in here already may as well

Change-Id: I3bbeb2bcf0ae23467f795dc6fb4a1de46c08d3ef
2017-10-19 10:59:37 +00:00
Chris Dent ad2c7a3c80 [placement] correct error on bad resource class in allocation
A 400 is returned when making an allocation with a resource class that
does not exist, however the error message refers to whatever the most
recent resource provider uuid was rather than the one that was given
the bad resource class. Since the exception happens outside the loop
that creates the allocations we don't have the association to resource
provider uuid, but we do know which consumer uuid, so report that
instead.

Change-Id: I81dc6b566c6cca0b7d2ac267c3e632c8ae8cfb47
Closes-Bug: #1714402
2017-09-09 02:47:30 +01:00
Chris Dent 9155da345e [placement] Require at least one allocation when PUT
When doing a PUT to /allocations/{consumer_uuid} if the list of
allocations is empty there is a 500 error coming from an uncaught
IndexError.

Adjust the schema to require at least one allocation. If no allocations
are provided in the list, the response will be a 400.

This doesn't require a microversion because it is changing a 500 to
a useful response.

Change-Id: I361fa1e534d7b20c985558e272b672b84c6fe439
Closes-Bug: #1708260
2017-08-02 21:21:12 +01:00
EdLeafe 6a6d021f81 Add check for invalid allocation amounts
This patch adds a check for the amounts in allocations passed to the
placement API, and returns an error if any amount is not greater than
zero.

Partial-Bug: #1673227

Change-Id: I2a8dc038d6489e82ad51fdc27d22cf0ab2d66322
2017-03-21 16:45:36 -07:00
Pushkar Umaranikar 8cef55c53c Placement api: set custom json_error_formatter in allocations
Any time placement API (v1.0) code wants to raise a webob.exc of some kind it
needs to add a json_formatter attribute to the call. This is easy to forget.

This change adds wrapper for webob.dec.wsgify which will set default custom
json error formatter. Resource based actions from placement need to use this
wrapper as decorater instead of webob.dec.wsgify.

In this change, decorator updated for allocations related actions.
Change in other resource handlers can be addressed in subsequent patches.

Change-Id: I3b81c5bd00a013f1659b9e6e80c71b373d965862
Partial-Bug: #1635182
2017-02-09 16:19:18 +00:00
Chris Dent efb0df6465 [placement] increase gabbi coverage of handlers.allocation
A case where a non-existent resource provider in an allocation PUT
request could raise a 400 was not covered. Now it is.

This still leave a ConcurrentUpdateDetected exception (which can
lead to 409) not covered. There's no simple way to cause such a
thing from a gabbi test.

Change-Id: I98243b679ba8101663d459cb9a76f7015078aff7
2016-11-22 15:34:51 +00:00
Jenkins 933343aba7 Merge "Typo error allocations.yaml" 2016-11-22 07:47:09 +00:00
Jenkins 17fbe727be Merge "[placement] Enforce min_unit, max_unit and step_size" 2016-11-16 11:42:07 +00:00
Jenkins 625a539358 Merge "Placement api: 404 response do not indicate what was not found" 2016-11-16 11:41:06 +00:00
pangliye 0b053ca992 Typo error allocations.yaml
Change-Id: Id3ce395785963bf9ff015a629f61a296fa343fcd
2016-11-16 18:59:46 +08:00
Chris Dent b000989635 [placement] Enforce min_unit, max_unit and step_size
Add support for checking min_unit, max_unit and step_size when
making allocations to the placement API. When the constraints are
violated a new exception InvalidAllocationConstraintsViolated is
raised.

Change-Id: I18596a3c0f2b0049aaccd0f3e73aef90b684c4a8
Closes-Bug: #1623545
2016-11-14 20:04:02 +00:00
Pushkar Umaranikar f87af91d90 Placement api: 404 response do not indicate what was not found
Add informative messages to the 404 error response from
nova/objects/resource_provider.py

For the method _update_inventory_in_db() from resource_provider.py,
gabbi test validation is not added since it is not being used and
from the placement API you create an inventory but then call
set_inventory or add_inventory on a resource_provider.

Similar change, i.e. adding informative messages to 404 error can be
done for api tests from inventory.yaml and usage.yaml

It includes adding informative logging in code for 404 exceptions and
adding gabbi tests for those.
This can be done in follow up patch.

Change-Id: If53f84ac5f7521e9926b97bdcce3cf77ec5b4ffd
Closes-Bug: #1634115
2016-11-09 21:11:07 +00:00
Jay Pipes 69bac2f99c placement: raise exc when resource class not found
The ResourceProvider.set_inventory() method was not raising NotFound
when an Inventory object in the supplied InventoryList object was using
a resource class that did not exist in the resource class cache. This
patch fixes that by having the string_from_id() and id_from_string()
methods of the resource class cache raise ResourceClassNotFound instead
of returning None. We raise specific ResourceClassNotFound and
InventoryWithResourceClassNotFound exception subclasses to differentiate
in the inventory API handler between an unknown resource class and a
known resource class that doesn't exist in a given resource provider's
inventory.

We return a 400 Not Found when InventoryWithResourceClassNotFound is
raised on updating a single inventory, but return a 409 Conflict when
InventoryWithResourceClassNotFound is raised on updating a set of
inventory records since in the latter scenario, it represents a race
condition that frankly should not occur.

Change-Id: I350b02dcdbaa9d30d885cd085f60daa6b53a7745
2016-11-03 11:48:53 -04:00
Jay Pipes 5661f879b0 placement: change resource class to a StringField
This patch modifies the fields.ResourceClass field type from an
EnumField to a StringField. This keeps the over-the-wire format of the
field backwards-compatible while allowing us to add non-standardized,
custom resource classes to the new resource_classes database table.

We change all locations of fields.ResourceClass.index() and
fields.ResourceClass.from_index() to use the ResourceClassCache object
added in the previous patch in this series and add some check logic in
Inventory and Allocation object's obj_make_compatible() methods to
ensure backversioned objects requesting or sending with a resource class
string different than the set of strings in the ResourceClass field at
the time of this patch raises a ValueError.

Change-Id: I2c1d4ae277ba25791c426e1c638dca1b1cb207a4
blueprint: custom-resource-classes
2016-10-12 17:24:09 -04:00
Jenkins daf4cc6b24 Merge "[placement] cleanup some incorrect comments" 2016-09-14 12:54:40 +00:00
Jay Pipes 09db0ba7cc [placement] make PUT inventory consistent with GET
The format of accepted request payload for PUT
/resource-providers/{uuid}/inventories was different from the response
payload describing inventories. It was using an anonymous dict approach
instead of a dict keyed by resource class identifier. This patch changes
the PUT implementation to accept a dict keyed by resource class
identifier.

Change-Id: I8770b36b11e10975e595f70084c9f06ee5b085a0
2016-09-12 07:53:38 -07:00
Chris Dent 8e33d7a5a9 [placement] cleanup some incorrect comments
Some comments in allocations.yaml were not accurate and had some
typos. This fixes them for the sake fixing broken windows.

Change-Id: Ie679ca54de6b481d51cbda39fac8931c9d28ba7e
2016-09-09 18:06:15 +00:00
Chris Dent eab8100b42 [placement] add two ways to GET allocations
Add two resources on the placement api for retrieving allocations.

One, /resource_providers/{uuid}/allocations, returns all the
allocations currently against this resource provider in this format:

    {'resource_provider_generation': GENERATION,
     'allocations':
        CONSUMER_ID_1: {
            'resources': {
                'DISK_GB': 4,
                'VCPU': 2
            }
        },
        CONSUMER_ID_2: {
            'resources': {
                'DISK_GB': 6,
                'VCPU': 3
            }
        }
    }

Another, /allocations/{consumer_id}, returns all the allocations for
the current consumer in this format:

    {'allocations':
      RP_UUID_1: {
        'generation': GENERATION,
        'resources': {
            'DISK_GB': 4,
            'VCPU': 2
        }
      },
      RP_UUID_2: {
          'generation': GENERATION,
          'resources': {
              'DISK_GB': 6,
              'VCPU': 3
          }
      }
    }

The formats are eminently usable on the client side, by having a
comparable dict reachable by a predictable key, but not self
describing (we don't know that the keys are consumer ids or resource
uuids).

In the first format the descent into a resources key is to maintain
parity with the second format and provide room for expansion.

Change-Id: I69fbc4e9834ec6dc80dacf43f8fd9dc6ec139006
Closes-Bug: #1621059
2016-09-08 19:49:58 +00:00
Chris Dent f441ee55c6 Add support for allocations in placement API
A new endpoint supporting PUT and DELETE methods with a path
of /allocations/{consumer_id} calls set_allocations and
delete_allocations to set or delete allocations for one or more
resource classes on one or more resource providers.

An example JSON body in PUT request takes the following form:

    {"allocations": [
      {
        "resource_provider": {
            "uuid": "d4b87fe1-30ca-4a65-8947-dfa3d85f10be"
        },
        "resources": {
            "MEMORY_MB": 64,
            "VCPU": 2
        }
      },
      {
        "resource_provider": {
            "uuid": "3cbe1001-bba9-4ef6-ae9c-feac1c3f9626"
        },
        "resources": {
            "DISK_GB": 28
        }
      }
    ]}

The body is validated against the JSON schema defined in the
ALLOCATION_SCHEMA constant.

The API code calls create_all and delete_all on the AllocationList
object. create_all() allows an existing allocation to be replaced.

Change-Id: Ic1e90636bad44b7dc76b45c5f93c67f3fe3297f7
Partially-Implements: blueprint generic-resource-pools
2016-09-01 11:39:37 -07:00