Docstrings for more Syntribos components

Adding a few more docstrings, populating code-docs page, modifying
Sphinx documentation generation.

Implements: blueprint docstring-add-to-framework
Change-Id: I9506c9fdeab19d9b4bf52ee3a51c27e38476e82e

doc/source/code-docs.rst

@@ -7,20 +7,20 @@ Configuration
 This section describes the configuration specified in your configuration file
 (second argument to the runner).
 
-.. autoclass:: syntribos.config.MainConfig
+.. automodule:: syntribos.config
     :members:
+    :undoc-members:
+    :show-inheritance:
 
 ..
-    Arguments
-    ---------
-    .. autoclass:: syntribos.arguments.SyntribosCLI
+    .. automodule:: syntribos.arguments
         :members:
-
-..
-    Runner
-    ------
-    .. autoclass:: syntribos.runners.Runner
+        :undoc-members:
+        :show-inheritance:
+    .. automodule:: syntribos.runner
         :members:
+        :undoc-members:
+        :show-inheritance:
 
 Tests
 -----
@@ -34,19 +34,26 @@ either directly, or through a subclass like
 
 All tests are aggregated in the `syntribos.tests.base.test_table` variable
 
-.. autoclass:: syntribos.tests.base.TestType
+.. automodule:: syntribos.tests.base
     :members:
-
-.. autoclass:: syntribos.tests.base.BaseTestCase
-    :members:
-
-.. autofunction:: syntribos.tests.base.replace_invalid_characters
-
+    :undoc-members:
+    :show-inheritance:
 
 ..
-    .. autoclass:: syntribos.tests.fuzz.base_fuzz.BaseFuzzTestCase
+    .. automodule:: syntribos.tests.fuzz.base_fuzz
         :members:
+        :undoc-members:
+        :show-inheritance:
 
+.. automodule:: syntribos.tests.fuzz.config
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+.. automodule:: syntribos.tests.fuzz.datagen
+    :members:
+    :undoc-members:
+    :show-inheritance:
 
 Issues
 ------
@@ -54,8 +61,10 @@ Issues
 This section describes the representation of issues that are uncovered by
 Syntribos.
 
-.. autoclass:: syntribos.issue.Issue
+.. automodule:: syntribos.issue
     :members:
+    :undoc-members:
+    :show-inheritance:
 
 Results
 -------
@@ -63,8 +72,10 @@ Results
 This section describes the representation of results (collections of issues)
 from a given Syntribos run.
 
-.. autoclass:: syntribos.result.IssueTestResult
+.. automodule:: syntribos.result
     :members:
+    :undoc-members:
+    :show-inheritance:
 
 HTTP Requests
 -------------
@@ -72,13 +83,17 @@ HTTP Requests
 This section describes the components related to generating, fuzzing, and making
 HTTP requests.
 
-.. autoclass:: syntribos.clients.http.parser.RequestCreator
+.. automodule:: syntribos.clients.http.client
     :members:
-    :private-members:
+    :undoc-members:
+    :show-inheritance:
 
-.. autoclass:: syntribos.clients.http.models.RequestObject
+.. automodule:: syntribos.clients.http.models
     :members:
+    :undoc-members:
+    :show-inheritance:
 
-.. autoclass:: syntribos.clients.http.models.RequestHelperMixin
+.. automodule:: syntribos.clients.http.parser
     :members:
-    :private-members:
+    :undoc-members:
+    :show-inheritance:

doc/source/conf.py

@@ -16,7 +16,6 @@ import os
 import sys
 
 sys.path.insert(0, os.path.abspath('../../'))
-sys.path.insert(0, os.path.abspath('../../syntribos'))
 
 # -- General configuration ----------------------------------------------------
 
@@ -76,9 +75,10 @@ latex_documents = [
 
 # Example configuration for intersphinx: refer to the Python standard library.
 # intersphinx_mapping = {'http://docs.python.org/': None}
+intersphinx_mapping = {'cafe': ('http://opencafe.readthedocs.org/en/latest', None)}
 
-autodoc_mock_imports = [
-    'cafe',
-    'cafe.engine.http.client',
-    'cafe.drivers.unittest.arguments'
-]
+# autodoc_mock_imports = [
+#     'cafe',
+#     'cafe.engine.http.client',
+#     'cafe.drivers.unittest.arguments'
+# ]

doc/source/configuration.rst

@@ -1,14 +1,17 @@
 Configuration
 =============
 
-Copy the data files from Syntribos data directory to .opencafe/data directory created during "cafe-config init". This directory contains the fuzz string files. Copy the example configuration file to .opencafe/configs directory created during "cafe-config init".
+Copy the data files from Syntribos data directory to ``.opencafe/data``
+directory created during ``cafe-config init``. This directory contains the fuzz
+string files. Next, copy the example configuration file to the
+``.opencafe/configs`` directory.
 
 ::
 
     $ cp data/* .opencafe/data/
-    $ cp examples/configs/keystone.config  .opencafe/configs/.
+    $ cp examples/configs/keystone.config  .opencafe/configs/
 
-Modify the configuration files to update your keystone URL, API endpoint
+Modify the configuration files to update your Keystone URL, API endpoint
 and user credentials.
 
 ::
@@ -18,18 +21,29 @@ and user credentials.
 Example configuration file:
 
 ::
-
     [syntribos]
-    endpoint=<yourapiendpoint>
+    # End point URLs and versions of the services to be tested.
+    endpoint=http://localhost:5000
+    # Optional, api version if required.
+    # Used for cross auth tests (-t AUTH_WITH_SOMEONE_ELSE_TOKEN)
+    # version=v2
 
     [user]
     username=<yourusername>
     password=<yourpassword>
-    user_id=<youruserid>
+    # Optional, if username is provided
+    # user_id=<youruserid>
 
+    [alt_user]
+    # Used for cross auth tests (-t AUTH_WITH_SOMEONE_ELSE_TOKEN)
+    username=<alt_username>
+    password=<alt_password>
+    user_id=<alt_userid>
 
     [auth]
-    endpoint=<yourkeystoneurl>
+    # Config for authorization enpoint, so that the service can
+    # obtain a valid token, enter your keystone auth endpoint.
+    endpoint=http://localhost:5000
 
 You can create a directory to store the request templates for the resources
 being tested. The templates under the `examples` directory can give you a quick

doc/source/index.rst

@@ -34,6 +34,7 @@ For Developers
 .. toctree::
     :maxdepth: 1
 
+    structure
     contributing
     code-docs
     unittests

doc/source/structure.rst

@@ -0,0 +1,20 @@
+Project Structure
+=================
+
+- ``data/`` (textfiles containing data for use by Syntribos tests)
+- ``doc/source/`` (Sphinx documentation files)
+- ``examples/`` (example Syntribos request templates, config files)
+    - ``configs/`` (examples of Syntribos configs; currently only Keystone)
+    - ``templates/`` (examples of request templates; currently only Keystone/Solum)
+- ``scripts/`` (?)
+- ``syntribos/`` (core Syntribos code)
+    - ``clients/`` (clients for making calls, e.g. HTTP)
+        - ``http/`` (clients for making HTTP requests)
+    - ``extensions/`` (extensions that can be called in request templates)
+        - ``identity/`` (extension for interacting with Keystone/identity)
+        - ``random_data/`` (extension for generating random test data)
+    - ``formatters/`` (output formatters, e.g. JSON, XML/XUnit)
+    - ``tests/`` (location of tests that Syntribos can run against a target)
+        - ``auth/`` (tests related to authentication/authorization)
+        - ``fuzz/`` (tests that "fuzz" API requests)
+- ``tests/unit/`` (unittests for testing Syntribos itself)

doc/source/test.anatomy.rst

@@ -1,4 +1,4 @@
-Basic syntribos test anatomy
+Basic Syntribos Test Anatomy
 ============================
 
 **Test Types**
@@ -33,7 +33,13 @@ For all tests against HTTP headers only, use:
 Syntribos template files can be supplemented with variable data, or data
 retrieved from external sources. This is handled using 'extensions.'
 
-Extensions are found in ``syntribos/syntribos/extensions/`` .
+Extensions are found in ``syntribos/extensions/`` .
+
+Calls to extensions are made in this form: 
+
+::
+
+    CALL_EXTERNAL|{extension dot path}:{function}:{arguments}
 
 One example packaged with Syntribos enables the tester to obtain an auth
 token from keystone/identity. The code is located in
@@ -54,7 +60,7 @@ usernames when fuzzing a create user call.
 
 ::
 
-    "username": "CALL_EXTERNAL|syntribos.extensions.random_data.client:get_uuid:[]|",
+    "username": "CALL_EXTERNAL|syntribos.extensions.random_data.client:get_uuid:[]|"
 
 The extension function can return one value or be used as a generator if
 you want it to change for each test.
@@ -68,6 +74,6 @@ follows:
 
 ::
 
-    "ACTION_FIELD:id": "1a16f348-c8d5-42ec-a474-b1cdf78cf40f",
+    "ACTION_FIELD:id": "1a16f348-c8d5-42ec-a474-b1cdf78cf40f"
 
 The ID provided will remain static for every test.

syntribos/config.py

@@ -27,5 +27,5 @@ class MainConfig(data_interfaces.ConfigSectionInterface):
 
     @property
     def version(self):
-        """???"""
+        """Used for base_auth test."""
         return self.get("version")

syntribos/issue.py

@@ -30,15 +30,15 @@ class Issue(object):
     :ivar response: The response object returned after sending the request
     :ivar target: A hostname/IP/etc. to be tested
     :ivar path: A specific REST API method, i.e. a URL path associated with a
-        Target
+        Target.
     :ivar test_type: The type of vulnerability that is being tested for. This
         is not necessarily the same as the Defect Type, which may be something
         like 500 error or DoS.
     :ivar impacted_parameter: For fuzz tests only, a
         :class:`syntribos.tests.fuzz.base_fuzz.ImpactedParameter` that holds
-            data about what part of the request was affected by the fuzz test
-
+        data about what part of the request was affected by the fuzz test.
     """
+
     def __init__(self, test, severity, text, confidence,
                  request=None, response=None, impacted_parameter=None):
         self.defect_type = test
@@ -86,7 +86,6 @@ class Issue(object):
         """Convert the request object to a dict of values for outputting.
 
         :param req: The request object
-        :type req: (TODO)
         :rtype: `dict`
         :returns: dictionary of HTTP request data
         """
@@ -102,7 +101,6 @@ class Issue(object):
         """Convert the response object to a dict of values for outputting.
 
         :param res: The result object
-        :type res: (TODO)
         :rtype: `dict`
         :returns: dictionary of HTTP response data
         """

syntribos/tests/base.py

@@ -79,10 +79,7 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
 
     @classmethod
     def get_test_cases(cls, filename, file_content):
-        """Not sure what the point of this is.
-
-        TODO: FIGURE THIS OUT
-        """
+        """Returns tests for given TestCase class (overwritten by children)."""
         yield cls
 
     @classmethod
@@ -93,6 +90,8 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
         read in by the test runner as the master list of tests to be run.
 
         :param str new_name: Name of new class to be created
+        :param str fuzz_string: Fuzz string to insert
+        :param str param_path: String tracing location of the ImpactedParameter
         :param dict kwargs: Keyword arguments to pass to the new class
         :rtype: class
         :returns: A TestCase class extending :class:`BaseTestCase`
@@ -108,11 +107,23 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
         return new_cls
 
     def run_test(self):
+        """This kicks off the test(s) for a given TestCase class
+
+        After running the tests, an `AssertionError` is raised if any tests
+        were added to self.failures.
+
+        :raises: :exc:`AssertionError`
+        """
         self.test_case()
         if self.failures:
             raise AssertionError
 
     def test_case(self):
+        """This method is overwritten by individual TestCase classes
+
+        It represents the actual test that is called in :func:`run_test`,
+        and handles populating `self.failures`
+        """
         pass
 
     def register_issue(self, issue):
@@ -121,9 +132,10 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
         Registers a :class:`syntribos.issue.Issue` object as a failure and
         associates the test's metadata to it.
 
-        :param Issue issue: issue object to update
+        :param issue: issue object to update
+        :type issue: :class:`syntribos.issue.Issue`
         :returns: new issue object with metadata associated
-        :rtype: Issue
+        :rtype: :class:`syntribos.issue.Issue`
         """
 
         # Still associating request and response objects with issue in event of
@@ -142,7 +154,7 @@ class BaseTestCase(cafe.drivers.unittest.fixtures.BaseTestFixture):
         return issue
 
     def test_issues(self):
-        """Run assertions for each test registered in test_case."""
+        """(DEPRECATED) Run assertions for each test in test_case."""
         for issue in self.issues:
             try:
                 issue.run_tests()

syntribos/tests/fuzz/base_fuzz.py

@@ -21,7 +21,7 @@ from syntribos.tests import base
 import syntribos.tests.fuzz.config
 import syntribos.tests.fuzz.datagen
 
-data_dir = os.environ.get("CAFE_DATA_DIR_PATH")
+data_dir = os.environ.get("CAFE_DATA_DIR_PATH", "")
 
 
 class BaseFuzzTestCase(base.BaseTestCase):

syntribos/tests/fuzz/datagen.py

@@ -20,11 +20,13 @@ from syntribos.clients.http import parser
 
 
 class FuzzMixin(object):
-    """FuzzMixin Class
 
-    FuzzBehavior provides the fuzz_data function which yields a test name
-    and all iterations of a given piece of data (currently supports dict,
-    ElementTree.Element, and basestring formats) with each string provided.
+    """Mixin for fuzz tests
+
+    This class provides the :meth:`~.FuzzMixin._fuzz_data` function which
+    yields a test name and all iterations of a given piece of data (currently
+    supports `dict`, :class:`xml.etree.ElementTree.Element`, and `basestring`
+    formats) with each string provided.
     """
     @classmethod
     def _fuzz_data(cls, strings, data, skip_var, name_prefix):
@@ -33,6 +35,11 @@ class FuzzMixin(object):
         For each attribute in the model object, call the _build_combinations
         method corresponding to the type of the data parameter, which replaces
         the value with the fuzz string.
+
+        :param strings:
+        :param data:
+        :param skip_var:
+        :param name_prefix:
         """
         param_path = ""
         for str_num, stri in enumerate(strings, 1):
@@ -50,14 +57,18 @@ class FuzzMixin(object):
                 yield (name, model, stri, param_path)
 
     @classmethod
-    def _build_str_combinations(cls, string, data):
-        """Places fuzz string in fuzz location for string data."""
+    def _build_str_combinations(cls, fuzz_string, data):
+        """Places `fuzz_string` in fuzz location for string data.
+
+        :param str fuzz_string: Value to place in fuzz location
+        :param str data: Lines from the request template
+        """
         for match in re.finditer(r"{([\w]*):?([^}]*)}", data):
             # Match either "{identifier:value}" or "{value}"
             start, stop = match.span()
             model = "{0}{1}{2}".format(
                     cls.remove_braces(data[:start]),
-                    string, cls.remove_braces(data[stop:]))
+                    fuzz_string, cls.remove_braces(data[stop:]))
             if match.group(1):
                 # The string is of the format "{identifier:value}", so we just
                 # want the identifier as the param_path
@@ -67,7 +78,12 @@ class FuzzMixin(object):
 
     @classmethod
     def _build_combinations(cls, stri, dic, skip_var):
-        """Places fuzz string in fuzz location for object data."""
+        """Places fuzz string in fuzz location for object data.
+
+        :param stri:
+        :param dic:
+        :param skip_var:
+        """
         for key, val in dic.iteritems():
             if skip_var in key:
                 continue
@@ -94,10 +110,15 @@ class FuzzMixin(object):
 
     @staticmethod
     def _merge_dictionaries(x, y):
-        """merge the dictionaries
+        """Merge `dicts` together
 
-        Uses the copy function to create a merged dictionary without squashing
-        the passed in objects
+        Create a copy of `x`, and update that with elements of `y`, to prevent
+        squashing of passed in dicts.
+
+        :param dict x: Dictionary 1
+        :param dict y: Dictionary 2
+        :returns: Merged dictionary
+        :rtype: `dict`
         """
 
         z = x.copy()
@@ -123,20 +144,28 @@ class FuzzMixin(object):
                            "{0}/{1}".format(ele.tag, param_path))
 
     @staticmethod
-    def _update_element(ele, stri):
-        """update element
+    def _update_element(ele, text):
+        """Copies an XML element, updates its text attribute with `text`
 
-        Returns a copy of the element with the element text replaced by stri
+        :param ele: XML element to be copied, modified
+        :type ele: :class:`xml.ElementTree.Element`
+        :param str text: Text to populate `ele`'s text attribute with
+        :returns: XML element with "text" attribute set to `text`
+        :rtype: :class:`xml.ElementTree.Element`
         """
         ret = ele.copy()
-        ret.text = stri
+        ret.text = text
         return ret
 
     @staticmethod
     def _update_attribs(ele, attribs):
-        """update attributes
+        """Copies an XML element, populates attributes from `attribs`
 
-        Returns a copy of the element with the attributes replaced by attribs
+        :param ele: XML element to be copied, modified
+        :type ele: :class:`xml.ElementTree.Element`
+        :param dict attribs: Source of new attribute values for `ele`
+        :returns: XML element with all attributes overwritten by `attribs`
+        :rtype: :class:`xml.ElementTree.Element`
         """
         ret = ele.copy()
         ret.attrib = attribs
@@ -144,9 +173,14 @@ class FuzzMixin(object):
 
     @staticmethod
     def _update_inner_element(ele, list_):
-        """Update inner element
+        """Copies an XML element, populates sub-elements from `list_`
 
         Returns a copy of the element with the subelements given via list_
+        :param ele: XML element to be copied, modified
+        :type ele: :class:`xml.ElementTree.Element`
+        :param list list_: List of subelements to append to `ele`
+        :returns: XML element with new subelements from `list_`
+        :rtype: :class:`xml.ElementTree.Element`
         """
         ret = ele.copy()
         for i, v in enumerate(list_):
@@ -155,6 +189,7 @@ class FuzzMixin(object):
 
     @staticmethod
     def remove_braces(string):
+        """Remove braces from strings (in request templates)."""
         return string.replace("}", "").replace("{", "")
 
     @staticmethod
@@ -175,6 +210,13 @@ class FuzzRequest(RequestObject, FuzzMixin, RequestHelperMixin):
 
         Gets the name and the fuzzed request model from _fuzz_data, and
         creates a request object from the parameters of the model.
+
+        :param strings:
+        :param fuzz_type:
+        :param name_prefix:
+        :returns: Generator of tuples:
+            (name, request, fuzzstring, ImpactedParameter name)
+        :rtype: `tuple`
         """
         for name, data, stri, param_path in self._fuzz_data(
             strings, getattr(self, fuzz_type), self.action_field,