Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).
openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.
The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.
tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.
The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.
Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,
TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.
Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
There was no gabbi coverage for the case where a GET request was being
made for single class of inventory on a non-existent resource provider.
This adds it.
Note that it would have been possible to add an error code/comment for
the raised exception, but this was not done in this change because there
is another change in progress which is adding the appropriate code.
Change-Id: I1858f4eb617cc81b41610195bb9a33a9059850f7
We have two kinds of generation conflicts that happen (for both
resource providers and consumers). In
I5a164a0ee298846231113ad7b62b82a3e8f1f409 error codes are added for
conflicts that happen when incrementing the generation in the database.s
In this change the codes are added for those situations where in code we
are checking an incoming generation against the stored generation.
Since there are incoming genearations in this case, we can add tests
so this does, via gabbi.
Change-Id: I5ed005068c1c19194ed67556376604ae90727412
This is needed for ironic use case, when during cleaning, resources
are reserved by ironic itself. Cyborg will also benefit from this
during FPGA programming.
blueprint: allow-reserved-equal-total-inventory
Change-Id: I037d9b8c1bb554c3437434cc9a57ddb630dd62f0
In relevant requests to the placement API add last-modified
and cache-control headers.
According the HTTP 1.1 RFC headers last-modified headers SHOULD always
be sent and should have a tie to the real last modified time. If we do
send them, we need Cache-Control headers to prevent inadvertent caching
of resources.
This change adds a microversion 1.15 which adds the headers to GET
requests and some PUT or POST requests.
Despite what it says 'no-cache' means "check to see if the version you
have is still valid as far as the server is concerned". Since our server
doesn't currently validate conditional requests and will always return an
entity, it ends up meaning "don't cache" (which is what we want).
The main steps in the patch are:
* To both the get single entity and get collection handlers add
response.cache_control = 'no-cache'
* For single entity add response.last_modified = obj.updated_at or
obj.created_at
* For collections, discover the max modified time when traversing the
list of objects to create the serialized JSON output. In most of
those loops an optimization is done where we only check for
last-modified information if we have a high enough microversion such
that the information will be used. This is not done when listing
inventories because the expectation is that no single resource
provider will ever have a huge number of inventory records.
* Both of the prior steps are assisted by a new util method:
pick_last_modfied.
Where a time cannot be determined the current time is used.
In typical placement framework fashion this has been done in a very
explicit way, as it makes what the handler is doing very visible, even
though it results in a bit of boilerplate.
For those requests that are created from multiple objects or by doing
calculations, such as usages and aggregate associations, the current time
is used.
The handler for PUT /traits is modified a bit more extensively than some
of the others: This is because the method can either create or validate
the existence of the trait. In the case where the trait already exists,
we need to get it from the DB to get its created_at time. We only do
this if the microversion is high enough (at least 1.15) to warrant
needing the info.
Because these changes add new headers (even though they don't do
anything) a new microversion, 1.15, is added.
Partial-Bug: #1632852
Partially-Implements: bp placement-cache-headers
Change-Id: I727d4c77aaa31f0ef31c8af22c2d46cad8ab8b8e
Increase test coverage for deleting all inventory for a
resource provider. These new tests check if the inventory
was deleted for both DELETE and PUT methods, and check if
the new resource generation has been correctly updated as
expected.
The DELETE method is implemented by the following change:
I1ecb12c888f873e8330367c8411d5a2ef0458495.
These tests have been originally proposed in this change:
I626ca4a821b5ab0ebcaa79937a31771fbae5b8d9, however, since
these tests do not have anything about the client side code,
I broke it down into a separate change.
Change-Id: I73aff37aaecf1f3755f0445ab9edb8f6454cf6c3
This patch adds sane minimum and maximums to the fields for an inventory
posting, which will quickly return a 400 error if invalid values are
passed instead of proceeding, only to fail at the DB layer.
Partial-Bug: #1673227
Change-Id: I6296cee6b8a4be1dd53c52f6290ebda926cf6465
This patch adds a new method for deleting all inventories for a
resource provider: DELETE /resource-providers/{uuid}/inventories
Return codes:
204 NoContent on success
404 NotFound if the resource provider does not exist
405 MethodNotAllowed if a microversion is specified that is before
this change (1.5)
409 Conflict if inventory in use or if some other request concurrently
updates this resource provider
Change-Id: I1ecb12c888f873e8330367c8411d5a2ef0458495
Implements: bp delete-inventories-placement-api
Change decorator for resource based actions in inventory
handler to wsgi_wrapper.PlacementWsgify
This is a newly introduced wrapper class around webob.dec.wsgify
to set json formatter in case of webob exceptions.
Change-Id: Idfdc4dbde125b6ead203ab25ff995e0cfbc0f396
Partial-Bug: #1635182
Create/update inventory APIs returns 500 error if you pass value
greater than db.MAX_INT (2147483647) for total, reserved, min_unit,
max_unit, step_size int type parameters and db.SQL_SP_FLOAT_MAX
(3.40282e+38) for allocation_ratio.
Added maximum limit check in schema to ensure value is not greater than
maximum limit permitted by db for all mentioned parameters.
Closes-Bug: #1642484
Change-Id: I5b8c3c175ca617ae7bb82b81b0fe608ae1755b72
Add informative messages to the 404 error response from
inventory.py and usage.py
This change also includes adding/altering existing Gabbi tests
for 404 response and add resonse_strings validation checks for
that.
Change-Id: Ica46a5479dec06b035f8b43d6738e9d124f6ae37
Related-Bug: #1634115
Test for correctly formed but invalid data when creating an Inventory.
This increases the coverage of the inventory handler from gabbi.
Race condition and ConcurrentUpdateDetected scenarios are not
covered as those are hard to cause from a gabbi test.
Change-Id: I603558c08d540aa53918252a0a9a050701456f5a
The ResourceProvider.set_inventory() method was not raising NotFound
when an Inventory object in the supplied InventoryList object was using
a resource class that did not exist in the resource class cache. This
patch fixes that by having the string_from_id() and id_from_string()
methods of the resource class cache raise ResourceClassNotFound instead
of returning None. We raise specific ResourceClassNotFound and
InventoryWithResourceClassNotFound exception subclasses to differentiate
in the inventory API handler between an unknown resource class and a
known resource class that doesn't exist in a given resource provider's
inventory.
We return a 400 Not Found when InventoryWithResourceClassNotFound is
raised on updating a single inventory, but return a 409 Conflict when
InventoryWithResourceClassNotFound is raised on updating a set of
inventory records since in the latter scenario, it represents a race
condition that frankly should not occur.
Change-Id: I350b02dcdbaa9d30d885cd085f60daa6b53a7745
This patch modifies the fields.ResourceClass field type from an
EnumField to a StringField. This keeps the over-the-wire format of the
field backwards-compatible while allowing us to add non-standardized,
custom resource classes to the new resource_classes database table.
We change all locations of fields.ResourceClass.index() and
fields.ResourceClass.from_index() to use the ResourceClassCache object
added in the previous patch in this series and add some check logic in
Inventory and Allocation object's obj_make_compatible() methods to
ensure backversioned objects requesting or sending with a resource class
string different than the set of strings in the ResourceClass field at
the time of this patch raises a ValueError.
Change-Id: I2c1d4ae277ba25791c426e1c638dca1b1cb207a4
blueprint: custom-resource-classes
The correct form is for resource_provider_generation to be its on
separate key, not repeated in each individual inventory. This was
probably caused by the refactoring that create _send_inventory and
_send_inventories and a lack of sufficient test coverage.
Fixing this involves changes in both the placement api service, and
in the (thus far) only existing client, the scheduler reporting
client used in the resource tracker. The reporting client can be a
bit simpler now because of the cleaner behavior in the api.
Tests have been updated accordingly.
Change-Id: I3af1c7686a45c1a0d70fe704d3c7938810eff6a3
Closes-Bug: #1619771
The format of accepted request payload for PUT
/resource-providers/{uuid}/inventories was different from the response
payload describing inventories. It was using an anonymous dict approach
instead of a dict keyed by resource class identifier. This patch changes
the PUT implementation to accept a dict keyed by resource class
identifier.
Change-Id: I8770b36b11e10975e595f70084c9f06ee5b085a0
If InventoryList.find() was provided with a string of a resource
class that does not exist it would cause an uncaught ValueError.
This would eventually raise up to the Placement API as a 500 in
response, for example, to
'GET /resource_providers/{uuid}/inventories/HOUSE'
NotFound makes sense at both the object and HTTP levels. A unit test
for the object level has been added and a gabbi test for the HTTP
level.
Note: this was discovered while developing some test client code
for the placement API. A simple, but wrong, argument was passed
through the stack and caused a 500.
Change-Id: I0d4066a3f213f726a92d673986b702e08bda4459
Partially-Implements: blueprint generic-resource-pools
GET to list /resource_providers/{uuid}/inventories
POST to create a single new inventory via
/resource_provider/{uuid}/inventories
PUT to (re-)set all inventories at
/resource_providers/{uuid}/inventories
GET, DELETE or UPDATE one inventory of a particular resource class
/resource_provider/{uuid}/inventories/{resource_class}
Inventories are presented with a view marker named
'resource_provider_generation' which is used to enable consistent
views of inventory across concurrent updates. They are also used
in fashion similar to ETags to avoid concurrent updates.
ETags were considered for this functionality but discussion
amongst interested parties led to the conclusion that the marker
was more explicit, especially in the case of taking an inventory
representation "seen" in a collection representation, updating it,
and sending it back to the server.
Change-Id: Ib49c7cddd2f15869f01e8b1e09d8378c26fb7ddc
Partially-Implements: blueprint generic-resource-pools
Chris Dent