From f800a1c5ca672033cd9e283c82972b87d0911648 Mon Sep 17 00:00:00 2001 From: Felipe Monteiro Date: Mon, 9 Jul 2018 00:23:46 -0400 Subject: [PATCH] Add documentation explaining why Patrole tests aren't in Tempest There have been concerns raised as to why Patrole tests aren't contained in Tempest (the concerns are not found in any written discussion online). This documentation puts such concerns to rest. It was agreed upon in the RBAC testing spec [0] that: "rbac tests will live in separate tempest plugin." This documentation formalizes that reasoning. [0] comment in https://review.openstack.org/#/c/382672/ Change-Id: I31d956b42440a5448a5be0a7e2c5b3b7ddacfab5 --- doc/source/field_guide/rbac.rst | 42 +++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/doc/source/field_guide/rbac.rst b/doc/source/field_guide/rbac.rst index 2654d31e..d2f4af07 100644 --- a/doc/source/field_guide/rbac.rst +++ b/doc/source/field_guide/rbac.rst @@ -53,6 +53,48 @@ the projects themselves, but that approach has the following shortcomings: * Patrole is designed to work with any role via configuration settings, but on the other hand the projects handpick which roles to test. +Why not use Patrole framework on Tempest tests? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Patrole framework can't be applied to existing Tempest tests via +:ref:`rbac-validation`, because: + +* Tempest tests aren't factored the right way: They're not granular enough. + They call too many APIs and too many policies are enforced by each test. +* Tempest tests assume default policy rules: Tempest uses ``os_admin`` + credentials for admin APIs and ``os_primary`` for non-admin APIs. + This breaks for custom policy overrides. +* Tempest doesn't have tests that enforce all the policy actions, regardless. + Some RBAC tests require that tests be written a very precise way for the + server to authorize the expected policy actions. + +Why are these tests not in Tempest? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Patrole should be a separate project that specializes in RBAC tests. + +Philosophically speaking: + +* Tempest supports `API and scenario testing`_. RBAC testing is out of scope. +* The `OpenStack project structure reform`_ evolved OpenStack "to a more + decentralized model where [projects like QA] provide processes and tools to + empower projects to do the work themselves". This model resulted in the + creation of the `Tempest external plugin interface`_. +* Tempest supports `plugins`_. Why not use one for RBAC testing? + +Practically speaking: + +* The Tempest team should not be burdened with having to support Patrole, too. + Tempest is a big project and having to absorb RBAC testing is difficult. +* Tempest already has many in-tree Zuul checks/gates. If Patrole tests lived + in Tempest, then adding more Zuul checks/gates for Patrole would only make it + harder to get changes merged in Tempest. + +.. _API and scenario testing: https://docs.openstack.org/tempest/latest/overview.html#tempest-the-openstack-integration-test-suite +.. _OpenStack project structure reform: https://governance.openstack.org/tc/resolutions/20141202-project-structure-reform-spec.html#impact-for-horizontal-teams +.. _Tempest external plugin interface: https://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/tempest-external-plugin-interface.html +.. _plugins: https://docs.openstack.org/tempest/latest/plugin.html + Scope of these tests --------------------