From fcde1d72f15da446eed64a575387fc754f5724b0 Mon Sep 17 00:00:00 2001 From: Chris Dent Date: Wed, 7 Mar 2018 19:00:38 +0000 Subject: [PATCH] Clarify testing for interop programs This provides an alternative to I91315c3c42b8a329d1763f9de80aa3d0270ac044 to provide simplified guidance on use of plugins for trademark tests. The coauthors below represents the people who were activce in the creating and refining these ideas. Change-Id: I2e68c8e855e7c6c1e4f985f38a0a38de451407a3 Co-Authored-By: Graham Hayes Co-Authored-By: Zane Bitter Co-Authored-By: Andrea Frittoli Co-Authored-By: Mark T. Voelker Co-Authored-By: Jim Rollenhagen Co-Authored-By: ricolin --- ...180307-trademark-program-test-location.rst | 73 +++++++++++++++++++ .../20160504-defcore-test-location.rst | 5 ++ 2 files changed, 78 insertions(+) create mode 100644 resolutions/20180307-trademark-program-test-location.rst rename resolutions/{ => superseded}/20160504-defcore-test-location.rst (96%) diff --git a/resolutions/20180307-trademark-program-test-location.rst b/resolutions/20180307-trademark-program-test-location.rst new file mode 100644 index 000000000..e9ea74036 --- /dev/null +++ b/resolutions/20180307-trademark-program-test-location.rst @@ -0,0 +1,73 @@ +.. _201180307_interop_test_location: + +======================================================================= + 2018-03-07 Location of tempest tests for OpenStack Trademark Programs +======================================================================= + +Introduction +============ + +When :ref:`20160504_defcore_test_location` was proposed, the trademark program +was simpler, involving only a small subset of the projects available in +OpenStack. + +With the proposed "add on" and "vertical" trademark programs, more projects are +involved. This growth introduces some scaling problems into the management of +tests for the trademark programs. These issues were discussed at the Rocky PTG. +The outcome of those discussions are reported here. + +This resolution supersedes :ref:`20160504_defcore_test_location`. + +Updated Resolution +================== + +Whereas the Trademark Program has made it clear that they have the tooling and +willingness to support trademark tests located in any of + +* The `Tempest repository`_ +* Project specific tempest plugins (e.g., `image-trademark-tempest-plugin`) +* Vertical specific tempest plugins (e.g., + `nfv-vertical-trademark-tempest-plugin`) +* A global Trademark tempest plugin (e.g., + `openstack-trademark-tempest-plugin`) + +as long as they are tests that can be run by tempest. + +And the QA Team, Trademark Program and individual projects have shared +responsibility for the management of trademark tests, with varying degrees of +bandwidth for review attention. + +And trademark tests are expected to be changed rarely. + +It is resolved that trademark tests (and candidates thereof) for individual, +projects should live in one, and only one, of four possible locations, each +with differing strategies for review and repository ownership (status in +:ref:`projects`): + +* The `Tempest repository`_, with approval reviews performed by the QA Team, + repository ownership via the QA Team (already true). +* A vertical specific tempest plugin, with approval reviews performed by the QA + Team and the Trademark Program, repository ownership via the Trademark + Program. +* A global Trademark tempest plugin, with approval reviews performed by the QA + Team and the Trademark Program, repository ownership via the Trademark + Program. +* A project specific tempest plugin, specifically limited to trademark-oriented + tests, with approval reviews performed by the project, the QA Team, and the + Trademark Program, repository ownership via the Trademark Program if they + accept, otherwise the project team. + +The choice is up to the individual projects, in consultation with the QA Team +and the Trademark Program. + +.. note:: No overarching direction is provided here on what tooling must be + used in the tests but it is generally expected that the tooling + should be aggreed by the people who are reviewing the code. If a + project wants to use tooling other than the core tempest libraries + (e.g., gabbi_) they should prefer a project specific plugin and be + aware that their choice of tooling may limit the ability for + non-project parties to review tests. Any tooling chosen must be able + to produce tests that are run by tempest. + +.. _gabbi: https://gabbi.readthedocs.io/ +.. _Tempest repository: http://git.openstack.org/cgit/openstack/tempest diff --git a/resolutions/20160504-defcore-test-location.rst b/resolutions/superseded/20160504-defcore-test-location.rst similarity index 96% rename from resolutions/20160504-defcore-test-location.rst rename to resolutions/superseded/20160504-defcore-test-location.rst index b77227ebf..590e7878f 100644 --- a/resolutions/20160504-defcore-test-location.rst +++ b/resolutions/superseded/20160504-defcore-test-location.rst @@ -1,7 +1,12 @@ +.. _20160504_defcore_test_location: + ============================================================ 2016-05-04 Recommendation on Location of Tests for DefCore ============================================================ +.. note:: This resolution has been superseded by + ref:`201180305_interop_test_location` + At the Newton summit discussion about `QA, DefCore, and interoperability testing`_ the DefCore committee asked for direction about where they should look for tests to consider for verifying