Merge "docs: Add information on test removal/interop to REVIEWING"

This commit is contained in:
Zuul 2018-07-18 08:52:23 +00:00 committed by Gerrit Code Review
commit ea2d907756
2 changed files with 43 additions and 3 deletions

View File

@ -99,6 +99,39 @@ level docstring linking to the API ref doc in the API tests and a docstring for
scenario tests this is up to the reviewers discretion whether a docstring is
required or not.
Test Removal and Refactoring
----------------------------
Make sure that any test that is renamed, relocated (e.g. moved to another
class), or removed does not belong to the `interop`_ testing suite -- which
includes a select suite of Tempest tests for the purposes of validating that
OpenStack vendor clouds are interoperable -- or a project's `whitelist`_ or
`blacklist`_ files.
It is of critical importance that no interop, whitelist or blacklist test
reference be broken by a patch set introduced to Tempest that renames,
relocates or removes a referenced test.
Please check the existence of code which references Tempest tests with:
http://codesearch.openstack.org/
Interop
^^^^^^^
Make sure that modifications to an `interop`_ test are backwards-compatible.
This means that code modifications to tests should not undermine the quality of
the validation currently performed by the test or significantly alter the
behavior of the test.
Removal
^^^^^^^
Reference the :ref:`test-removal` guidelines for understanding best practices
associated with test removal.
.. _interop: https://www.openstack.org/brand/interop
.. _whitelist: https://docs.openstack.org/tempest/latest/run.html#test-selection
.. _blacklist: https://docs.openstack.org/tempest/latest/run.html#test-selection
Release Notes
-------------
Release notes are how we indicate to users and other consumers of Tempest what
@ -113,16 +146,18 @@ something extra.
.. _reno: https://docs.openstack.org/reno/latest/
Deprecated Code
---------------
Sometimes we have some bugs in deprecated code. Basically, we leave it. Because
we don't need to maintain it. However, if the bug is critical, we might need to
fix it. When it will happen, we will deal with it on a case-by-case basis.
When to approve
---------------
* Every patch needs two +2s before being approved.
* Its ok to hold off on an approval until a subject matter expert reviews it
* It's ok to hold off on an approval until a subject matter expert reviews it
* If a patch has already been approved but requires a trivial rebase to merge,
you do not have to wait for a second +2, since the patch has already had
two +2s.

View File

@ -1,3 +1,5 @@
.. _test-removal:
Tempest Test Removal Procedure
==============================
@ -107,16 +109,19 @@ intent of this prong was mostly for refstack/defcore and also for things that
running on the stable branches. We don't want to remove any tests if that
would break our API consistency checking between releases, or something that
defcore/refstack is depending on being in Tempest. It's worth pointing out
that if a test is used in defcore as part of interop testing then it will
that if a test is used in `defcore`_ as part of `interop`_ testing then it will
probably have continuing value being in Tempest as part of the
integration/integrated tests in general. This is one area where some overlap
is expected between testing in projects and Tempest, which is not a bad thing.
.. _defcore: https://wiki.openstack.org/wiki/Governance/InteropWG
.. _interop: https://www.openstack.org/brand/interop
Discussing the 3rd prong
""""""""""""""""""""""""
There are 2 approaches to addressing the 3rd prong. Either it can be raised
during a qa meeting during the Tempest discussion. Please put it on the agenda
during a QA meeting during the Tempest discussion. Please put it on the agenda
well ahead of the scheduled meeting. Since the meeting time will be well known
ahead of time anyone who depends on the tests will have ample time beforehand
to outline any concerns on the before the meeting. To give ample time for