From 531b68fb6a44cf41d310a5e17f86e56630000c65 Mon Sep 17 00:00:00 2001 From: Chris Dent Date: Wed, 27 Feb 2019 10:55:58 +0000 Subject: [PATCH] Docs: extract testing info to own sub-page In preparation for extending the testing documentation to include information about more than just gabbi, including the tempest, grenade and perload jobs, pull the content out to its own page. As noted in a TODO within, making subpages show up is cumbersome for the moment, but we can iterate to something swell. Change-Id: I71d000804985aaf5f3a1c467974ac7f8d0250a98 --- doc/source/contributor/index.rst | 133 +++++------------------------ doc/source/contributor/testing.rst | 125 +++++++++++++++++++++++++++ doc/source/index.rst | 4 - 3 files changed, 145 insertions(+), 117 deletions(-) create mode 100644 doc/source/contributor/testing.rst diff --git a/doc/source/contributor/index.rst b/doc/source/contributor/index.rst index ae0e4f4ec..d6274ec30 100644 --- a/doc/source/contributor/index.rst +++ b/doc/source/contributor/index.rst @@ -15,6 +15,24 @@ Placement API Developer Notes =============================== +Subpages +======== + +.. # TODO(cdent): What I really want here is to be able to have this list + # in the info column that is to the left of the page on the docs theme. + # Thus far there doesn't seem to be a way to do that, so this ugly way + # provides a method for making sure the subpages are more discoverable. + +.. toctree:: + :maxdepth: 1 + + api-ref-guideline + goals + quick-dev + testing + vision-reflection + + Overview ======== @@ -153,7 +171,7 @@ the code: ``TOTAL_VERSIONED_METHODS`` in ``placement/tests/unit/test_microversion.py``. This provides a confirmatory check just to make sure you are paying attention and as a helpful reminder to do the other things in this list. -* Include functional gabbi tests as appropriate (see `Using Gabbi`_). At the +* Include functional gabbi tests as appropriate (see :doc:`testing`). At the least, update the ``latest microversion`` test in ``placement/tests/functional/gabbits/microversion.yaml``. * Update the `API Reference`_ documentation as appropriate. The source is @@ -271,110 +289,7 @@ Code that adds newly raised exceptions should include an error code. Find additional guidelines on use in the docs for ``placement.errors``. When a new error code is added, also document it in the `API Reference`_. -Testing of handler code is described in the next section. - -Testing -======= - -Most of the handler code in the placement API is tested using `gabbi`_. Some -utility code is tested with unit tests found in `placement/tests/unit`. The -back-end objects are tested with a combination of unit and functional tests -found in ``placement/tests/unit/objects/test_resource_provider.py`` and -`placement/tests/functional/db`. - -When writing tests for handler code (that is, the code found in -``placement/handlers``) a good rule of thumb is that if you feel like there -needs to be a unit test for some of the code in the handler, that is a good -sign that the piece of code should be extracted to a separate method. That -method should be independent of the handler method itself (the one decorated by -the ``wsgify`` method) and testable as a unit, without mocks if possible. If -the extracted method is useful for multiple resources consider putting it in -the ``util`` package. - -As a general guide, handler code should be relatively short and where there are -conditionals and branching, they should be reachable via the gabbi functional -tests. This is merely a design goal, not a strict constraint. - -Using Gabbi ------------ - -Gabbi was developed in the `telemetry`_ project to provide a declarative way to -test HTTP APIs that preserves visibility of both the request and response of -the HTTP interaction. Tests are written in YAML files where each file is an -ordered suite of tests. Fixtures (such as a database) are set up and torn down -at the beginning and end of each file, not each test. JSON response bodies can -be evaluated with `JSONPath`_. The placement WSGI application is run via -`wsgi-intercept`_, meaning that real HTTP requests are being made over a file -handle that appears to Python to be a socket. - -In the placement API the YAML files (aka "gabbits") can be found in -``placement/tests/functional/gabbits``. Fixture definitions are in -``placement/tests/functional/fixtures/gabbits.py``. Tests are frequently -grouped by handler name (e.g., ``resource-provider.yaml`` and -``inventory.yaml``). This is not a requirement and as we increase the number of -tests it makes sense to have more YAML files with fewer tests, divided up by -the arc of API interaction that they test. - -The gabbi tests are integrated into the functional tox target, loaded via -``placement/tests/functional/test_api.py``. If you -want to run just the gabbi tests one way to do so is:: - - tox -efunctional test_api - -If you want to run just one yaml file (in this example ``inventory.yaml``):: - - tox -efunctional api.inventory - -It is also possible to run just one test from within one file. When you do this -every test prior to the one you asked for will also be run. This is because -the YAML represents a sequence of dependent requests. Select the test by using -the name in the yaml file, replacing space with ``_``:: - - tox -efunctional api.inventory_post_new_ipv4_address_inventory - -.. note:: ``tox.ini`` in the placement repository is configured by a - ``group_regex`` so that each gabbi YAML is considered a group. Thus, - all tests in the file will be run in the same process when running - stestr concurrently (the default). - -Writing More Gabbi Tests ------------------------- - -The docs for `gabbi`_ try to be complete and explain the `syntax`_ in some -depth. Where something is missing or confusing, please log a `bug`_. - -While it is possible to test all aspects of a response (all the response -headers, the status code, every attribute in a JSON structure) in one single -test, doing so will likely make the test harder to read and will certainly make -debugging more challenging. If there are multiple things that need to be -asserted, making multiple requests is reasonable. Since database set up is only -happening once per file (instead of once per test) and since there is no TCP -overhead, the tests run quickly. - -While `fixtures`_ can be used to establish entities that are required for -tests, creating those entities via the HTTP API results in tests which are more -descriptive. For example the ``inventory.yaml`` file creates the resource -provider to which it will then add inventory. This makes it easy to explore a -sequence of interactions and a variety of responses with the tests: - -* create a resource provider -* confirm it has empty inventory -* add inventory to the resource provider (in a few different ways) -* confirm the resource provider now has inventory -* modify the inventory -* delete the inventory -* confirm the resource provider now has empty inventory - -Nothing special is required to add a new set of tests: create a YAML file with -a unique name in the same directory as the others. The other files can provide -examples. Gabbi can provide a useful way of doing test driven development of a -new handler: create a YAML file that describes the desired URLs and behavior -and write the code to make it pass. - -It's also possible to use gabbi against a running placement service, for -example in devstack. See `gabbi-run`_ to get started. If you don't want to -go to the trouble of using devstack, but do want a live server see -:doc:`quick-dev`. +Testing of handler code is described in :doc:`testing`. Database Schema Changes ======================= @@ -459,14 +374,6 @@ information. .. _Request: http://docs.webob.org/en/latest/reference.html#request .. _Response: http://docs.webob.org/en/latest/#response .. _microversions: http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html -.. _gabbi: https://gabbi.readthedocs.io/ -.. _telemetry: http://specs.openstack.org/openstack/telemetry-specs/specs/kilo/declarative-http-tests.html -.. _wsgi-intercept: http://wsgi-intercept.readthedocs.io/ -.. _syntax: https://gabbi.readthedocs.io/en/latest/format.html -.. _bug: https://github.com/cdent/gabbi/issues -.. _fixtures: http://gabbi.readthedocs.io/en/latest/fixtures.html -.. _JSONPath: http://goessner.net/articles/JsonPath/ -.. _gabbi-run: http://gabbi.readthedocs.io/en/latest/runner.html .. _errors: http://specs.openstack.org/openstack/api-wg/guidelines/errors.html .. _API Reference: https://developer.openstack.org/api-ref/placement/ .. _Placement API Error Handling: http://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/placement-api-error-handling.html diff --git a/doc/source/contributor/testing.rst b/doc/source/contributor/testing.rst new file mode 100644 index 000000000..04575cef0 --- /dev/null +++ b/doc/source/contributor/testing.rst @@ -0,0 +1,125 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +=================== + Testing Placement +=================== + +Most of the handler code in the placement API is tested using `gabbi`_. Some +utility code is tested with unit tests found in `placement/tests/unit`. The +back-end objects are tested with a combination of unit and functional tests +found in ``placement/tests/unit/objects/test_resource_provider.py`` and +`placement/tests/functional/db`. + +When writing tests for handler code (that is, the code found in +``placement/handlers``) a good rule of thumb is that if you feel like there +needs to be a unit test for some of the code in the handler, that is a good +sign that the piece of code should be extracted to a separate method. That +method should be independent of the handler method itself (the one decorated by +the ``wsgify`` method) and testable as a unit, without mocks if possible. If +the extracted method is useful for multiple resources consider putting it in +the ``util`` package. + +As a general guide, handler code should be relatively short and where there are +conditionals and branching, they should be reachable via the gabbi functional +tests. This is merely a design goal, not a strict constraint. + +Using Gabbi +----------- + +Gabbi was developed in the `telemetry`_ project to provide a declarative way to +test HTTP APIs that preserves visibility of both the request and response of +the HTTP interaction. Tests are written in YAML files where each file is an +ordered suite of tests. Fixtures (such as a database) are set up and torn down +at the beginning and end of each file, not each test. JSON response bodies can +be evaluated with `JSONPath`_. The placement WSGI application is run via +`wsgi-intercept`_, meaning that real HTTP requests are being made over a file +handle that appears to Python to be a socket. + +In the placement API the YAML files (aka "gabbits") can be found in +``placement/tests/functional/gabbits``. Fixture definitions are in +``placement/tests/functional/fixtures/gabbits.py``. Tests are frequently +grouped by handler name (e.g., ``resource-provider.yaml`` and +``inventory.yaml``). This is not a requirement and as we increase the number of +tests it makes sense to have more YAML files with fewer tests, divided up by +the arc of API interaction that they test. + +The gabbi tests are integrated into the functional tox target, loaded via +``placement/tests/functional/test_api.py``. If you +want to run just the gabbi tests one way to do so is:: + + tox -efunctional test_api + +If you want to run just one yaml file (in this example ``inventory.yaml``):: + + tox -efunctional api.inventory + +It is also possible to run just one test from within one file. When you do this +every test prior to the one you asked for will also be run. This is because +the YAML represents a sequence of dependent requests. Select the test by using +the name in the yaml file, replacing space with ``_``:: + + tox -efunctional api.inventory_post_new_ipv4_address_inventory + +.. note:: ``tox.ini`` in the placement repository is configured by a + ``group_regex`` so that each gabbi YAML is considered a group. Thus, + all tests in the file will be run in the same process when running + stestr concurrently (the default). + +Writing More Gabbi Tests +------------------------ + +The docs for `gabbi`_ try to be complete and explain the `syntax`_ in some +depth. Where something is missing or confusing, please log a `bug`_. + +While it is possible to test all aspects of a response (all the response +headers, the status code, every attribute in a JSON structure) in one single +test, doing so will likely make the test harder to read and will certainly make +debugging more challenging. If there are multiple things that need to be +asserted, making multiple requests is reasonable. Since database set up is only +happening once per file (instead of once per test) and since there is no TCP +overhead, the tests run quickly. + +While `fixtures`_ can be used to establish entities that are required for +tests, creating those entities via the HTTP API results in tests which are more +descriptive. For example the ``inventory.yaml`` file creates the resource +provider to which it will then add inventory. This makes it easy to explore a +sequence of interactions and a variety of responses with the tests: + +* create a resource provider +* confirm it has empty inventory +* add inventory to the resource provider (in a few different ways) +* confirm the resource provider now has inventory +* modify the inventory +* delete the inventory +* confirm the resource provider now has empty inventory + +Nothing special is required to add a new set of tests: create a YAML file with +a unique name in the same directory as the others. The other files can provide +examples. Gabbi can provide a useful way of doing test driven development of a +new handler: create a YAML file that describes the desired URLs and behavior +and write the code to make it pass. + +It's also possible to use gabbi against a running placement service, for +example in devstack. See `gabbi-run`_ to get started. If you don't want to +go to the trouble of using devstack, but do want a live server see +:doc:`quick-dev`. + +.. _bug: https://github.com/cdent/gabbi/issues +.. _fixtures: http://gabbi.readthedocs.io/en/latest/fixtures.html +.. _gabbi: https://gabbi.readthedocs.io/ +.. _gabbi-run: http://gabbi.readthedocs.io/en/latest/runner.html +.. _JSONPath: http://goessner.net/articles/JsonPath/ +.. _syntax: https://gabbi.readthedocs.io/en/latest/format.html +.. _telemetry: http://specs.openstack.org/openstack/telemetry-specs/specs/kilo/declarative-http-tests.html +.. _wsgi-intercept: http://wsgi-intercept.readthedocs.io/ diff --git a/doc/source/index.rst b/doc/source/index.rst index cf9fef446..1cb1a6763 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -248,10 +248,6 @@ This history of placement microversions may be found in cli/index configuration/index contributor/index - contributor/api-ref-guideline - contributor/goals - contributor/quick-dev - contributor/vision-reflection install/index install/from-pypi install/controller-install-obs