Adjust the fixtures used by the functional tests so they
use placement database and web fixtures defined by placement
code. To avoid making redundant changes, the solely placement-
related unit and functional tests are removed, but the placement
code itself is not (yet).
openstack-placement is required by the functional tests. It is not
added to test-requirements as we do not want unit tests to depend
on placement in any way, and we enforce this by not having placement
in the test env.
The concept of tox-siblings is used to ensure that the
placement requirement will be satisfied correctly if there is a
depends-on. To make this happen, the functional jobs defined in
.zuul.yaml are updated to require openstack/placement.
tox.ini has to be updated to use a envdir that is the same
name as job. Otherwise the tox siblings role in ansible cannot work.
The handling of the placement fixtures is moved out of nova/test.py
into the functional tests that actually use it because we do not
want unit tests (which get the base test class out of test.py) to
have anything to do with placement. This requires adjusting some
test files to use absolute import.
Similarly, a test of the comparison function for the api samples tests
is moved into functional, because it depends on placement functionality,
TestUpgradeCheckResourceProviders in unit.cmd.test_status is moved into
a new test file: nova/tests/functional/test_nova_status.py. This is done
because it requires the PlacementFixture, which is only available to
functional tests. A MonkeyPatch is required in the test to make sure that
the right context managers are used at the right time in the command
itself (otherwise some tables do no exist). In the test itself, to avoid
speaking directly to the placement database, which would require
manipulating the RequestContext objects, resource providers are now
created over the API.
Co-Authored-By: Balazs Gibizer <balazs.gibizer@ericsson.com>
Change-Id: Idaed39629095f86d24a54334c699a26c218c6593
The main dispatch handler in placement checks to see if the
content-length header is a valid int and returns a 400 status
code if not. There had been no gabbi coverage for this behavior.
This patch adds that coverage.
Change-Id: If12fc7e374c7933bd2f10beb4be3e0a8d8e622e9
In change Ice27c7080fc2df097cb387f7438c0aaf32b4c63d the placement api
was changed to set the accept header to application/json if the accept
header was not set. This was done to ensure that error responses are
formatted as JSON in the face of automation that webob does when
formatting errors.
That change was incomplete. Apparently many clients, notably requests,
which backs keystoneauth1, sets a default Accept header of '*/*'. This
is a reasonable default but means our error formatting default can go
awry.
This fixes it in the simplest way possible. See the referenced change
above for discussion on why doing this with a microversion would be
pointless.
Tested with a simple gabbi confirmation.
Change-Id: I3df5b9fb8fe7c0e41f7e1f495758973423485e11
Closes-Bug: #1771384
The API-sig has a guideline[1] for including error codes in error
responses to help distinguish errors with the same status code
from one another. This change provides a simplest-thing-that-could-
possibly-work solution to make that go.
This solution comes about after a few different constraints and attempts:
* We would prefer to go on using the existing webob.exc exceptions, not
make subclasses.
* We already have a special wrapper around our wsgi apps to deal with
setting the json_error_formatter.
* Though webob allows custom Request and Response objects, it uses the
default Response object as the parent of the HTTP exceptions.
* The Response object accepts kwargs, but only if they can be associated
with known attributes on the class. Since we can't subclass...
* The json_error_formatter method is not passed the raw exception, but
it does get the current WSGI environ
* The webob.exc classes take a 'comment' kwarg that is not used, but
is also not passed to the json_error_formatter.
Therefore, when we raise an exception, we can set 'comment' to a code
and then assign that comment to a well known field in the environ and if
that environ is set in json_error_formatter, we can set 'code' in the
output.
This is done in a new microversion, 1.23. Every response gets a default
code 'placement.undefined_code' from 1.23 on. Future development will
add specific codes where required. This change adds a stub code for
inventory in use when doing a PUT to .../inventories but the name
may need improvement.
[1] http://specs.openstack.org/openstack/api-wg/guidelines/errors.html
Implements blueprint placement-api-error-handling
Change-Id: I9a833aa35d474caa35e640bbad6c436a3b16ac5e
In relevant requests to the placement API add last-modified
and cache-control headers.
According the HTTP 1.1 RFC headers last-modified headers SHOULD always
be sent and should have a tie to the real last modified time. If we do
send them, we need Cache-Control headers to prevent inadvertent caching
of resources.
This change adds a microversion 1.15 which adds the headers to GET
requests and some PUT or POST requests.
Despite what it says 'no-cache' means "check to see if the version you
have is still valid as far as the server is concerned". Since our server
doesn't currently validate conditional requests and will always return an
entity, it ends up meaning "don't cache" (which is what we want).
The main steps in the patch are:
* To both the get single entity and get collection handlers add
response.cache_control = 'no-cache'
* For single entity add response.last_modified = obj.updated_at or
obj.created_at
* For collections, discover the max modified time when traversing the
list of objects to create the serialized JSON output. In most of
those loops an optimization is done where we only check for
last-modified information if we have a high enough microversion such
that the information will be used. This is not done when listing
inventories because the expectation is that no single resource
provider will ever have a huge number of inventory records.
* Both of the prior steps are assisted by a new util method:
pick_last_modfied.
Where a time cannot be determined the current time is used.
In typical placement framework fashion this has been done in a very
explicit way, as it makes what the handler is doing very visible, even
though it results in a bit of boilerplate.
For those requests that are created from multiple objects or by doing
calculations, such as usages and aggregate associations, the current time
is used.
The handler for PUT /traits is modified a bit more extensively than some
of the others: This is because the method can either create or validate
the existence of the trait. In the case where the trait already exists,
we need to get it from the DB to get its created_at time. We only do
this if the microversion is high enough (at least 1.15) to warrant
needing the info.
Because these changes add new headers (even though they don't do
anything) a new microversion, 1.15, is added.
Partial-Bug: #1632852
Partially-Implements: bp placement-cache-headers
Change-Id: I727d4c77aaa31f0ef31c8af22c2d46cad8ab8b8e
We plan to allow PUT requests to create/update both custom traits
and custom resource classes, without bodies. Prior to this change,
the code would not all a PUT, POST or PATCH to not have a body. This was
added in I6e7dffb5dc5f0cdc78a57e8df3ae9952c55163ae which was fixing an
issue with how webob handles exceptions.
This change does two things:
* It address the problem from bug #1623517, fixed in the change id'd
above, in a more narrow fashion, making sure the data source that
causes the KeyError is non-empty right before it is used. This allows
simplifying the following change.
* If a request has a content-length (indicating the presence of a body),
verify that there is also a content-type. If not, raise a 400.
basic-http.yaml has been change to modify one gabbi test to check a
response body is correct and to add another test to confirm that the
code that is doing the content-length check is passed through.
Change-Id: Ifb7446fd02ba3e54bbe2676dfd38e5dfecd15f98
Closes-Bug: #1674392
Related-Bug: #1623517
If 'cors.allowed_origin' is set in the nova.conf, configure the
placement API to use oslo_middleware.CORS. Simple gabbi tests are
added which confirm the basic operation of the middleware, modeled
on the tests in nova/tests/functional/test_middleware.py, as well as
additional tests which confirm that when the middleware is not
configured it is not present in the system.
The cors config options are registered in deploy.py to ensure that
the group is allowed to exist in conf (even if it doesn't). Without
that, a deployment that tries to configure cors would not actually
cause the middleware to run.
Change-Id: I571bc675facaecb523dcf906f4bb44a51102b514
If a PUT, POST or PATCH is sent without a content-type header,
webob.dec.wsgify will raise a KeyError. Avoid this by checking for
the content-type header before reaching any wsgify calls. As noted
in the TODO within this is not the most elegant solution, but
prevents an inadvertent 500 and returns a reasonable 400.
Change-Id: I6e7dffb5dc5f0cdc78a57e8df3ae9952c55163ae
Closes-Bug: #1623517
More detailed tests of 404 and 405 responses to ensure they
include expected information. The 404 tests revealed a missing
json_formatter for a 404.
This was shortcoming revealed by further work on the already merged
placement API tests.
Change-Id: I8150615705bbd2e350d0639f3c2cdb1b14c52518
Partially-Implements: blueprint generic-resource-pools
GET a list of /resource_providers
POST to /resource_providers for a new one
GET, DELETE, PUT a single /resource_providers/{uuid}
Uses jsonschema to validate input when creating or updating.
A 'uuid' FormatChecker is added in the util module to verify
the format of uuids in request bodies. Note the lengthy comment
adjacent explaining its magic.
The olso_context RequestContext is subclassed to allow it to
be an engine_facade transaction_context_provider. A database
fixture is added to the gabbi tests to create and dispose the
placement database per each gabbi testsuite (i.e. each YAML
file).
A resource_provider_url is added to util.py: given a
resource_provider object, give the url, accounting for any
prefix in SCRIPT_NAME
With the addition of working endpoints, basic_http.yaml
is extended with more complete testing of basic http behaviors.
Tests for resource_providers are in resource-provider.yaml.
Change-Id: I799d839101af78b4d89aca175f647efc2b56c401
Partially-Implements: blueprint generic-resource-pools
The placement API will initially host resource provider
information and then will grow to provide a full placement
service. Eventually it will be fully extracted from Nova.
To facilitate that extraction, this service is being developed
with few ties to existing nova.wsgi structures. Instead it
uses relatively plain WSGI apps that are:
* uses the Routes library with declarative mapping defined in
ROUTE_DECLARATIONS
* basic wsgi apps, with webob and the request class, for each handler
define as functions
* does not use a paste-ini file to compose middleware, instead code
which minimally inspects the oslo config to know how to adjust
middleware (in the initial case choosing an auth_strategy)
Many of these "features" will be demonstrated in commits that
follow this one that enable specific behaviors for resource
providers and their inventories.
In order to ensure that this change can be merged in an atomic
fashion it includes microversion support that was previously in its
own commit.
The microversion_parse library is used in a WSGI middleware
to parse incoming microversion headers and place the
determined value into the WSGI environment at the
'placement.microversion' key. Response headers are adjusted to
add the required outgoing headers as described in:
http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html
If a client requests an invalid microversion, they will receive
a 400 response. If the microversion is of a valid form but not
available, they will received a 406 response. The format of that
response is dependent on the accept header of the request. If it
is JSON, the min and max available microversions are presented.
A request to '/' will return version discovery information.
Thus far nothing else is done with the microversion information.
It is there for when we need it in the future. For now everything
is version 1.0.
The groundwork for using gabbi to test the API is in place in
nova/tests/functional/api/openstack/placement. The gabbi tests
are included in the functional target. To run just the placement
tests it is possible to run 'tox -efunctional placement'.
Change-Id: Icb9c8f7a1fa8a9eac66c2d72f4b7e4efd4e1944f
Partially-Implements: blueprint generic-resource-pools
Chris Dent