diff --git a/README.rst b/README.rst index 0c786b9e..fb8976ff 100644 --- a/README.rst +++ b/README.rst @@ -33,10 +33,22 @@ testing scope is confined to policies. * *Atomicity*. Patrole tests should be atomic: they should test policies in isolation. Unlike Tempest, a Patrole test strives to only call a single endpoint at a time. -* *Holistic coverage*. Patrole strives for complete coverage of the OpenStack - API. Additionally, Patrole strives to test the API-to-policy mapping - contained in each project's policy in code documentation. -* *Self-contained*. Patrole should attempt to clean up after itself; whenever +* *Complete coverage*. Patrole should validate all policy in code defaults. For + testing, Patrole uses the API-to-policy mapping contained in each project's + `policy in code`_ documentation where applicable. + + For example, Nova's policy in code documentation is located in the + `Nova repository`_ under ``nova/policies``. Likewise, Keystone's policy in + code documentation is located in the `Keystone repository`_ under + ``keystone/common/policies``. The other OpenStack services follow the same + directory layout pattern with respect to policy in code. + + .. note:: + + Realistically this is not always possible because some services have + not yet moved to policy in code. + +* *Self-cleaning*. Patrole should attempt to clean up after itself; whenever possible we should tear down resources when done. .. note:: @@ -45,7 +57,11 @@ testing scope is confined to policies. pre-provisioned credentials. Work is currently underway to clean up modifications made to pre-provisioned credentials. -* *Self-tested*. Patrole should be self-tested. +* *Self-testing*. Patrole should be self-testing. + +.. _policy in code: https://specs.openstack.org/openstack/oslo-specs/specs/newton/policy-in-code.html +.. _Nova repository: https://github.com/openstack/nova/tree/master/nova/policies +.. _Keystone repository: https://github.com/openstack/keystone/tree/master/keystone/common/policies Features -------- diff --git a/doc/source/HACKING.rst b/doc/source/HACKING.rst index 1847447d..8777875b 100644 --- a/doc/source/HACKING.rst +++ b/doc/source/HACKING.rst @@ -1,4 +1,5 @@ -======= -Hacking -======= +==================== +Patrole Coding Guide +==================== + .. include:: ../../HACKING.rst diff --git a/doc/source/framework/rbac_utils.rst b/doc/source/framework/rbac_utils.rst index 69ba0454..71439288 100644 --- a/doc/source/framework/rbac_utils.rst +++ b/doc/source/framework/rbac_utils.rst @@ -23,8 +23,19 @@ credentials, rather than using distinct credentials for setup/teardown and test execution, respectively. This is especially true when considering custom policy rule definitions, which can be arbitrarily complex. -Patrole, therefore, implicitly splits up each test into 3 stages: set up, -test execution, and teardown. +.. _role-overriding: + +Role Overriding +^^^^^^^^^^^^^^^ + +Role overriding is the way Patrole is able to create resources and delete +resources -- including those that require admin credentials -- while still +being able to exercise the same set of Tempest credentials to perform the API +action that authorizes the policy under test, by manipulating the role of +the Tempest credentials. + +Patrole implicitly splits up each test into 3 stages: set up, test execution, +and teardown. The role workflow is as follows: @@ -43,7 +54,7 @@ The role workflow is as follows: Test Setup ---------- -Automatic role switch in background. +Automatic role override in background. Resources can be set up inside the ``resource_setup`` class method that Tempest provides. These resources are typically reserved for "expensive" resources @@ -59,7 +70,7 @@ resources one needs, without having to worry about permissions. Test Execution -------------- -Manual role switch required. +Manual role override required. "Test execution" here means calling the API endpoint that enforces the policy action expected by the ``rbac_rule_validation`` decorator. Test execution @@ -152,7 +163,7 @@ everything extraneous outside. Test Cleanup ------------ -Automatic role switch in background. +Automatic role override in background. After the test -- no matter whether it ended successfully or in failure -- the credentials are overridden with the admin role by the Patrole framework, diff --git a/doc/source/index.rst b/doc/source/index.rst index 83682628..255fd9a1 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -2,6 +2,14 @@ Patrole: Tempest Plugin for RBAC Testing ======================================== +Overview +======== + +.. toctree:: + :maxdepth: 2 + + overview + User's Guide ============ @@ -12,7 +20,6 @@ Patrole Configuration Guide :maxdepth: 2 configuration - usage Patrole Installation Guide -------------------------- diff --git a/doc/source/overview.rst b/doc/source/overview.rst new file mode 100644 index 00000000..795359e1 --- /dev/null +++ b/doc/source/overview.rst @@ -0,0 +1,178 @@ +======================== +Team and repository tags +======================== + +.. image:: https://governance.openstack.org/tc/badges/patrole.svg + :target: https://governance.openstack.org/tc/reference/tags/index.html + +Patrole - The OpenStack RBAC Validation Test Suite +================================================== + +The documentation for Patrole is officially hosted at: +https://docs.openstack.org/patrole/latest/ + +This is a set of integration tests to be run against a live OpenStack +cluster. Patrole has a battery of tests dedicated to validating the correctness +and security of the cloud's RBAC implementation. + +Design Principles +----------------- + +As a `Tempest plugin`_, Patrole borrows some `design principles`_ from Tempest, +but not all, as its testing scope is confined to policies. + +* Patrole uses OpenStack public interfaces. Tests in Patrole should only touch + public OpenStack APIs. +* Patrole tests should be atomic: they should test policies in isolation. + Unlike Tempest, a Patrole test strives to only call a single endpoint at a + time. This is because it is important to validate each policy is authorized + correctly and the best way to do that is to validate the policy alone. +* Patrole should validate all policy in code defaults. For testing, Patrole + uses the API-to-policy mapping contained in each project's `policy in code`_ + documentation where applicable. + + For example, Nova's policy in code documentation is located in the + `Nova repository`_ under ``nova/policies``. Likewise, Keystone's policy in + code documentation is located in the `Keystone repository`_ under + ``keystone/common/policies``. The other OpenStack services follow the same + directory layout pattern with respect to policy in code. + + .. note:: + + Realistically this is not always possible because some services have + not yet moved to policy in code. + +* Patrole should attempt to clean up after itself; whenever possible it should + tear down resources when done. + + .. note:: + + Patrole modifies roles dynamically in the background, which affects + pre-provisioned credentials. Work is currently underway to clean up + modifications made to pre-provisioned credentials. + +* Patrole should be self-testing. + +.. _Tempest plugin: https://docs.openstack.org/tempest/latest/plugin.html +.. _design principles: https://docs.openstack.org/tempest/latest/overview.html#design-principles +.. _policy in code: https://specs.openstack.org/openstack/oslo-specs/specs/newton/policy-in-code.html +.. _Nova repository: https://github.com/openstack/nova/tree/master/nova/policies +.. _Keystone repository: https://github.com/openstack/keystone/tree/master/keystone/common/policies + +Quickstart +---------- + +To run Patrole, you must first have `Tempest`_ installed and configured +properly. Please reference Tempest's `Quickstart`_ guide to do so. Follow all +the steps outlined therein. Afterward, proceed with the steps below. + +#. You first need to install Patrole. This is done with pip after you check out + the Patrole repo:: + + $ git clone https://git.openstack.org/openstack/patrole + $ pip install patrole/ + + This can be done within a venv. + + .. note:: + + You may also install Patrole from source code by running:: + + pip install -e patrole/ + +#. Next you must properly configure Patrole, which is relatively + straightforward. For details on configuring Patrole refer to the + :ref:`patrole-configuration`. + +#. Once the configuration is done you're now ready to run Patrole. This can + be done using the `tempest_run`_ command. This can be done by running:: + + $ tempest run --regex '^patrole_tempest_plugin\.tests\.api' + + There is also the option to use testr directly, or any `testr`_ based test + runner, like `ostestr`_. For example, from the workspace dir run:: + + $ stestr --regex '(?!.*\[.*\bslow\b.*\])(^patrole_tempest_plugin\.tests\.api))' + + will run the same set of tests as the default gate jobs. + + You can also run Patrole tests using `tox`_. To do so, ``cd`` into the + **Tempest** directory and run:: + + $ tox -eall-plugin -- patrole_tempest_plugin.tests.api + + .. note:: + + It is possible to run Patrole via ``tox -eall`` in order to run Patrole + isolated from other plugins. This can be accomplished by including the + installation of services that currently use policy in code -- for example, + Nova and Keystone. For example:: + + $ tox -evenv-tempest -- pip install /opt/stack/patrole /opt/stack/keystone /opt/stack/nova + $ tox -eall -- patrole_tempest_plugin.tests.api + +#. Log information from tests is captured in ``tempest.log`` under the Tempest + repository. Some Patrole debugging information is captured in that log + related to expected test results and :ref:`role-overriding`. + + More detailed RBAC testing log output is emitted to ``patrole.log`` under + the Patrole repository. To configure Patrole's logging, see the + :ref:`patrole-configuration` guide. + +.. _Tempest: https://github.com/openstack/tempest +.. _Quickstart: https://docs.openstack.org/tempest/latest/overview.html#quickstart +.. _tempest_run: https://docs.openstack.org/tempest/latest/run.html +.. _testr: https://testrepository.readthedocs.org/en/latest/MANUAL.html +.. _ostestr: https://docs.openstack.org/os-testr/latest/ +.. _tox: https://tox.readthedocs.io/en/latest/ + +RBAC Tests +---------- + +To change the role that the patrole tests are being run as, edit +``rbac_test_role`` in the ``patrole`` section of tempest.conf: :: + + [patrole] + rbac_test_role = member + ... + +.. note:: + + The ``rbac_test_role`` is service-specific. member, for example, + is an arbitrary role, but by convention is used to designate the default + non-admin role in the system. Most Patrole tests should be run with + **admin** and **member** roles. However, other services may use entirely + different roles. + +For more information about the member role and its nomenclature, +please see: ``__. + +Unit Tests +---------- + +Patrole also has a set of unit tests which test the Patrole code itself. These +tests can be run by specifying the test discovery path:: + + $ stestr --test-path ./patrole_tempest_plugin/tests/unit run + +By setting ``--test-path`` option to ``./patrole_tempest_plugin/tests/unit`` +it specifies that test discovery should only be run on the unit test directory. + +Alternatively, there are the py27 and py35 tox jobs which will run the unit +tests with the corresponding version of Python. + +One common activity is to just run a single test; you can do this with tox +simply by specifying to just run py27 or py35 tests against a single test:: + + $ tox -e py27 -- -n patrole_tempest_plugin.tests.unit.test_rbac_utils.RBACUtilsTest.test_override_role_with_missing_admin_role + +Or all tests in the test_rbac_utils.py file:: + + $ tox -e py27 -- -n patrole_tempest_plugin.tests.unit.test_rbac_utils + +You may also use regular expressions to run any matching tests:: + + $ tox -e py27 -- test_rbac_utils + +For more information on these options and details about stestr, please see the +`stestr documentation `_. diff --git a/doc/source/usage.rst b/doc/source/usage.rst deleted file mode 100644 index 14c2cc74..00000000 --- a/doc/source/usage.rst +++ /dev/null @@ -1,61 +0,0 @@ -.. _patrole-usage: - -======== -Usage -======== - -Patrole (API) Tests -=================== - -If Patrole is installed correctly, then the RBAC tests can be executed -from inside the tempest root directory as follows:: - - $ tox -eall-plugin -- patrole_tempest_plugin.tests.api - -To execute patrole tests for a specific module, run:: - - $ tox -eall-plugin -- patrole_tempest_plugin.tests.api.compute - -.. note:: - - It is possible to run Patrole via ``tox -eall`` in order to run Patrole - isolated from other plugins. This can be accomplished by including the - installation of services that currently use policy in code -- for example, - Nova and Keystone. For example:: - - $ tox -evenv-tempest -- pip install /opt/stack/patrole /opt/stack/keystone /opt/stack/nova - $ tox -eall -- patrole_tempest_plugin.tests.api -.. - -To change the role that the patrole tests are being run as, edit -``rbac_test_role`` in the ``patrole`` section of tempest.conf: :: - - [patrole] - rbac_test_role = Member - ... - -.. note:: - - The ``rbac_test_role`` is service-specific. Member, for example, - is an arbitrary role, but by convention is used to designate the default - non-admin role in the system. Most patrole tests should be run with - **admin** and **Member** roles. However, some services, like Heat, take - advantage of a role called **heat_stack_user**, as it appears frequently - in Heat's policy.json. - -For more information about the Member role, -please see: ``__. - -Unit Tests -========== - -Patrole includes unit tests for its RBAC framework. They can be run by -executing:: - - $ tox -e py27 - -or:: - - $ tox -e py35 - -against the Python 3.5 interpreter.