Merge "Docs: extract testing info to own sub-page"
This commit is contained in:
Zuul
Gerrit Code Review
commit
d2d925041e
|
|
@ -15,6 +15,24 @@
|
|||
Placement API Developer Notes
|
||||
===============================
|
||||
|
||||
Subpages
|
||||
========
|
||||
|
||||
.. # TODO(cdent): What I really want here is to be able to have this list
|
||||
# in the info column that is to the left of the page on the docs theme.
|
||||
# Thus far there doesn't seem to be a way to do that, so this ugly way
|
||||
# provides a method for making sure the subpages are more discoverable.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
api-ref-guideline
|
||||
goals
|
||||
quick-dev
|
||||
testing
|
||||
vision-reflection
|
||||
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
|
|
@ -153,7 +171,7 @@ the code:
|
|||
``TOTAL_VERSIONED_METHODS`` in ``placement/tests/unit/test_microversion.py``.
|
||||
This provides a confirmatory check just to make sure you are paying attention
|
||||
and as a helpful reminder to do the other things in this list.
|
||||
* Include functional gabbi tests as appropriate (see `Using Gabbi`_). At the
|
||||
* Include functional gabbi tests as appropriate (see :doc:`testing`). At the
|
||||
least, update the ``latest microversion`` test in
|
||||
``placement/tests/functional/gabbits/microversion.yaml``.
|
||||
* Update the `API Reference`_ documentation as appropriate. The source is
|
||||
|
|
@ -271,110 +289,7 @@ Code that adds newly raised exceptions should include an error code. Find
|
|||
additional guidelines on use in the docs for ``placement.errors``. When a
|
||||
new error code is added, also document it in the `API Reference`_.
|
||||
|
||||
Testing of handler code is described in the next section.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Most of the handler code in the placement API is tested using `gabbi`_. Some
|
||||
utility code is tested with unit tests found in `placement/tests/unit`. The
|
||||
back-end objects are tested with a combination of unit and functional tests
|
||||
found in ``placement/tests/unit/objects/test_resource_provider.py`` and
|
||||
`placement/tests/functional/db`.
|
||||
|
||||
When writing tests for handler code (that is, the code found in
|
||||
``placement/handlers``) a good rule of thumb is that if you feel like there
|
||||
needs to be a unit test for some of the code in the handler, that is a good
|
||||
sign that the piece of code should be extracted to a separate method. That
|
||||
method should be independent of the handler method itself (the one decorated by
|
||||
the ``wsgify`` method) and testable as a unit, without mocks if possible. If
|
||||
the extracted method is useful for multiple resources consider putting it in
|
||||
the ``util`` package.
|
||||
|
||||
As a general guide, handler code should be relatively short and where there are
|
||||
conditionals and branching, they should be reachable via the gabbi functional
|
||||
tests. This is merely a design goal, not a strict constraint.
|
||||
|
||||
Using Gabbi
|
||||
-----------
|
||||
|
||||
Gabbi was developed in the `telemetry`_ project to provide a declarative way to
|
||||
test HTTP APIs that preserves visibility of both the request and response of
|
||||
the HTTP interaction. Tests are written in YAML files where each file is an
|
||||
ordered suite of tests. Fixtures (such as a database) are set up and torn down
|
||||
at the beginning and end of each file, not each test. JSON response bodies can
|
||||
be evaluated with `JSONPath`_. The placement WSGI application is run via
|
||||
`wsgi-intercept`_, meaning that real HTTP requests are being made over a file
|
||||
handle that appears to Python to be a socket.
|
||||
|
||||
In the placement API the YAML files (aka "gabbits") can be found in
|
||||
``placement/tests/functional/gabbits``. Fixture definitions are in
|
||||
``placement/tests/functional/fixtures/gabbits.py``. Tests are frequently
|
||||
grouped by handler name (e.g., ``resource-provider.yaml`` and
|
||||
``inventory.yaml``). This is not a requirement and as we increase the number of
|
||||
tests it makes sense to have more YAML files with fewer tests, divided up by
|
||||
the arc of API interaction that they test.
|
||||
|
||||
The gabbi tests are integrated into the functional tox target, loaded via
|
||||
``placement/tests/functional/test_api.py``. If you
|
||||
want to run just the gabbi tests one way to do so is::
|
||||
|
||||
tox -efunctional test_api
|
||||
|
||||
If you want to run just one yaml file (in this example ``inventory.yaml``)::
|
||||
|
||||
tox -efunctional api.inventory
|
||||
|
||||
It is also possible to run just one test from within one file. When you do this
|
||||
every test prior to the one you asked for will also be run. This is because
|
||||
the YAML represents a sequence of dependent requests. Select the test by using
|
||||
the name in the yaml file, replacing space with ``_``::
|
||||
|
||||
tox -efunctional api.inventory_post_new_ipv4_address_inventory
|
||||
|
||||
.. note:: ``tox.ini`` in the placement repository is configured by a
|
||||
``group_regex`` so that each gabbi YAML is considered a group. Thus,
|
||||
all tests in the file will be run in the same process when running
|
||||
stestr concurrently (the default).
|
||||
|
||||
Writing More Gabbi Tests
|
||||
------------------------
|
||||
|
||||
The docs for `gabbi`_ try to be complete and explain the `syntax`_ in some
|
||||
depth. Where something is missing or confusing, please log a `bug`_.
|
||||
|
||||
While it is possible to test all aspects of a response (all the response
|
||||
headers, the status code, every attribute in a JSON structure) in one single
|
||||
test, doing so will likely make the test harder to read and will certainly make
|
||||
debugging more challenging. If there are multiple things that need to be
|
||||
asserted, making multiple requests is reasonable. Since database set up is only
|
||||
happening once per file (instead of once per test) and since there is no TCP
|
||||
overhead, the tests run quickly.
|
||||
|
||||
While `fixtures`_ can be used to establish entities that are required for
|
||||
tests, creating those entities via the HTTP API results in tests which are more
|
||||
descriptive. For example the ``inventory.yaml`` file creates the resource
|
||||
provider to which it will then add inventory. This makes it easy to explore a
|
||||
sequence of interactions and a variety of responses with the tests:
|
||||
|
||||
* create a resource provider
|
||||
* confirm it has empty inventory
|
||||
* add inventory to the resource provider (in a few different ways)
|
||||
* confirm the resource provider now has inventory
|
||||
* modify the inventory
|
||||
* delete the inventory
|
||||
* confirm the resource provider now has empty inventory
|
||||
|
||||
Nothing special is required to add a new set of tests: create a YAML file with
|
||||
a unique name in the same directory as the others. The other files can provide
|
||||
examples. Gabbi can provide a useful way of doing test driven development of a
|
||||
new handler: create a YAML file that describes the desired URLs and behavior
|
||||
and write the code to make it pass.
|
||||
|
||||
It's also possible to use gabbi against a running placement service, for
|
||||
example in devstack. See `gabbi-run`_ to get started. If you don't want to
|
||||
go to the trouble of using devstack, but do want a live server see
|
||||
:doc:`quick-dev`.
|
||||
Testing of handler code is described in :doc:`testing`.
|
||||
|
||||
Database Schema Changes
|
||||
=======================
|
||||
|
|
@ -459,14 +374,6 @@ information.
|
|||
.. _Request: http://docs.webob.org/en/latest/reference.html#request
|
||||
.. _Response: http://docs.webob.org/en/latest/#response
|
||||
.. _microversions: http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html
|
||||
.. _gabbi: https://gabbi.readthedocs.io/
|
||||
.. _telemetry: http://specs.openstack.org/openstack/telemetry-specs/specs/kilo/declarative-http-tests.html
|
||||
.. _wsgi-intercept: http://wsgi-intercept.readthedocs.io/
|
||||
.. _syntax: https://gabbi.readthedocs.io/en/latest/format.html
|
||||
.. _bug: https://github.com/cdent/gabbi/issues
|
||||
.. _fixtures: http://gabbi.readthedocs.io/en/latest/fixtures.html
|
||||
.. _JSONPath: http://goessner.net/articles/JsonPath/
|
||||
.. _gabbi-run: http://gabbi.readthedocs.io/en/latest/runner.html
|
||||
.. _errors: http://specs.openstack.org/openstack/api-wg/guidelines/errors.html
|
||||
.. _API Reference: https://developer.openstack.org/api-ref/placement/
|
||||
.. _Placement API Error Handling: http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/placement-api-error-handling.html
|
||||
|
|
|
|||
|
|
@ -0,0 +1,125 @@
|
|||
..
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
not use this file except in compliance with the License. You may obtain
|
||||
a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
===================
|
||||
Testing Placement
|
||||
===================
|
||||
|
||||
Most of the handler code in the placement API is tested using `gabbi`_. Some
|
||||
utility code is tested with unit tests found in `placement/tests/unit`. The
|
||||
back-end objects are tested with a combination of unit and functional tests
|
||||
found in ``placement/tests/unit/objects/test_resource_provider.py`` and
|
||||
`placement/tests/functional/db`.
|
||||
|
||||
When writing tests for handler code (that is, the code found in
|
||||
``placement/handlers``) a good rule of thumb is that if you feel like there
|
||||
needs to be a unit test for some of the code in the handler, that is a good
|
||||
sign that the piece of code should be extracted to a separate method. That
|
||||
method should be independent of the handler method itself (the one decorated by
|
||||
the ``wsgify`` method) and testable as a unit, without mocks if possible. If
|
||||
the extracted method is useful for multiple resources consider putting it in
|
||||
the ``util`` package.
|
||||
|
||||
As a general guide, handler code should be relatively short and where there are
|
||||
conditionals and branching, they should be reachable via the gabbi functional
|
||||
tests. This is merely a design goal, not a strict constraint.
|
||||
|
||||
Using Gabbi
|
||||
-----------
|
||||
|
||||
Gabbi was developed in the `telemetry`_ project to provide a declarative way to
|
||||
test HTTP APIs that preserves visibility of both the request and response of
|
||||
the HTTP interaction. Tests are written in YAML files where each file is an
|
||||
ordered suite of tests. Fixtures (such as a database) are set up and torn down
|
||||
at the beginning and end of each file, not each test. JSON response bodies can
|
||||
be evaluated with `JSONPath`_. The placement WSGI application is run via
|
||||
`wsgi-intercept`_, meaning that real HTTP requests are being made over a file
|
||||
handle that appears to Python to be a socket.
|
||||
|
||||
In the placement API the YAML files (aka "gabbits") can be found in
|
||||
``placement/tests/functional/gabbits``. Fixture definitions are in
|
||||
``placement/tests/functional/fixtures/gabbits.py``. Tests are frequently
|
||||
grouped by handler name (e.g., ``resource-provider.yaml`` and
|
||||
``inventory.yaml``). This is not a requirement and as we increase the number of
|
||||
tests it makes sense to have more YAML files with fewer tests, divided up by
|
||||
the arc of API interaction that they test.
|
||||
|
||||
The gabbi tests are integrated into the functional tox target, loaded via
|
||||
``placement/tests/functional/test_api.py``. If you
|
||||
want to run just the gabbi tests one way to do so is::
|
||||
|
||||
tox -efunctional test_api
|
||||
|
||||
If you want to run just one yaml file (in this example ``inventory.yaml``)::
|
||||
|
||||
tox -efunctional api.inventory
|
||||
|
||||
It is also possible to run just one test from within one file. When you do this
|
||||
every test prior to the one you asked for will also be run. This is because
|
||||
the YAML represents a sequence of dependent requests. Select the test by using
|
||||
the name in the yaml file, replacing space with ``_``::
|
||||
|
||||
tox -efunctional api.inventory_post_new_ipv4_address_inventory
|
||||
|
||||
.. note:: ``tox.ini`` in the placement repository is configured by a
|
||||
``group_regex`` so that each gabbi YAML is considered a group. Thus,
|
||||
all tests in the file will be run in the same process when running
|
||||
stestr concurrently (the default).
|
||||
|
||||
Writing More Gabbi Tests
|
||||
------------------------
|
||||
|
||||
The docs for `gabbi`_ try to be complete and explain the `syntax`_ in some
|
||||
depth. Where something is missing or confusing, please log a `bug`_.
|
||||
|
||||
While it is possible to test all aspects of a response (all the response
|
||||
headers, the status code, every attribute in a JSON structure) in one single
|
||||
test, doing so will likely make the test harder to read and will certainly make
|
||||
debugging more challenging. If there are multiple things that need to be
|
||||
asserted, making multiple requests is reasonable. Since database set up is only
|
||||
happening once per file (instead of once per test) and since there is no TCP
|
||||
overhead, the tests run quickly.
|
||||
|
||||
While `fixtures`_ can be used to establish entities that are required for
|
||||
tests, creating those entities via the HTTP API results in tests which are more
|
||||
descriptive. For example the ``inventory.yaml`` file creates the resource
|
||||
provider to which it will then add inventory. This makes it easy to explore a
|
||||
sequence of interactions and a variety of responses with the tests:
|
||||
|
||||
* create a resource provider
|
||||
* confirm it has empty inventory
|
||||
* add inventory to the resource provider (in a few different ways)
|
||||
* confirm the resource provider now has inventory
|
||||
* modify the inventory
|
||||
* delete the inventory
|
||||
* confirm the resource provider now has empty inventory
|
||||
|
||||
Nothing special is required to add a new set of tests: create a YAML file with
|
||||
a unique name in the same directory as the others. The other files can provide
|
||||
examples. Gabbi can provide a useful way of doing test driven development of a
|
||||
new handler: create a YAML file that describes the desired URLs and behavior
|
||||
and write the code to make it pass.
|
||||
|
||||
It's also possible to use gabbi against a running placement service, for
|
||||
example in devstack. See `gabbi-run`_ to get started. If you don't want to
|
||||
go to the trouble of using devstack, but do want a live server see
|
||||
:doc:`quick-dev`.
|
||||
|
||||
.. _bug: https://github.com/cdent/gabbi/issues
|
||||
.. _fixtures: http://gabbi.readthedocs.io/en/latest/fixtures.html
|
||||
.. _gabbi: https://gabbi.readthedocs.io/
|
||||
.. _gabbi-run: http://gabbi.readthedocs.io/en/latest/runner.html
|
||||
.. _JSONPath: http://goessner.net/articles/JsonPath/
|
||||
.. _syntax: https://gabbi.readthedocs.io/en/latest/format.html
|
||||
.. _telemetry: http://specs.openstack.org/openstack/telemetry-specs/specs/kilo/declarative-http-tests.html
|
||||
.. _wsgi-intercept: http://wsgi-intercept.readthedocs.io/
|
||||
|
|
@ -276,10 +276,6 @@ This history of placement microversions may be found in
|
|||
cli/index
|
||||
configuration/index
|
||||
contributor/index
|
||||
contributor/api-ref-guideline
|
||||
contributor/goals
|
||||
contributor/quick-dev
|
||||
contributor/vision-reflection
|
||||
install/index
|
||||
install/from-pypi
|
||||
install/controller-install-obs
|
||||
|
|
|
|||
Loading…
Reference in New Issue