Add documentation for cross-project unit testing

Change-Id: I4270af00b14d33a66439fa8d7141c192df1f0c3a

doc/source/cross-testing.rst

@@ -0,0 +1,102 @@
+============================
+ Cross-project Unit Testing
+============================
+
+Libraries in OpenStack have an unusual ability to introduce breaking
+changes. All of the projects are run together from source in one form
+or another during the integration tests, but they are not combined
+from source when unit tests are run. The server applications do not
+generally import code from other projects, so their unit tests are
+isolated. The libraries, however, are fundamentally intended to be
+used during unit tests as well as integration tests. Testing the full
+cross-product of libraries and consuming projects would consume all
+available test servers, and so we cannot run all of the tests on all
+patches to the libraries. As an alternative, we have a few scripts in
+``oslotest`` for running the unit tests in serial. The result takes
+far too long (usually overnight) to run in the OpenStack
+infrastructure. Instead, they are usually run by hand on a dedicated
+system. A cloud VM works well for this purpose, especially considering
+how much of it is now automated.
+
+Check Out OpenStack Source
+==========================
+
+The first step for all of the cross-project unit tests tools is to
+ensure that you have a full copy of the OpenStack source checked
+out. You can do this yourself through gerrit's ssh API, or you can use
+the ``clone_openstack.sh`` command in the tools directory of the
+``openstack/oslo-incubator`` repository.
+
+For example::
+
+  $ mkdir -p ~/repos/openstack
+  $ cd ~/repos/openstack
+  $ git clone git://git.openstack.org/openstack/oslo-incubator
+  $ cd ~/repos
+  $ ./openstack/oslo-incubator/tools/clone_openstack.sh
+
+The first time the script runs it will take quite some time, since it
+has to download the entire history of every OpenStack project.
+
+Testing One Project
+===================
+
+``oslo_run_cross_tests`` runs one set of unit tests for one library
+against all of the consuming projects. It should be run from the
+directory with the library to be tested, and passed arguments telling
+it about another project whose tests should be run.
+
+For example, to run the ``py27`` test suite from nova using the
+currently checked out sources for ``oslo.config``, run::
+
+  $ cd ~/repos/openstack/oslo.config
+  $ tox -e venv -- oslo_run_cross_tests ~/repos/openstack/nova py27
+
+Testing All Consumers
+=====================
+
+``oslo_run_pre_release_tests`` builds on ``oslo_run_cross_tests`` to
+find all of the consuming projects and run their tests
+automatically. The pre-release script needs to know where the source
+has been checked out, so the first step is to create its configuration
+file.
+
+Edit ``~/.oslo.conf`` to contain::
+
+  [DEFAULT]
+  repo_root = /path/to/repos
+
+Replace ``/path/to/repos`` with the full, expanded, absolute path to
+the location where the source code was checked out. For example, if
+you followed the instructions above using ``clone_openstack.sh`` in
+``~/repos`` and your user name is ``theuser`` the path would be
+``/home/theuser/repos``.
+
+Returning to the earlier example, to test ``oslo.config`` with all of
+the projects that use it, go to the ``oslo.config`` source directory
+and run ``oslo_run_pre_release_tests``.
+
+::
+
+  $ cd ~/repos/openstack/oslo.config
+  $ tox -e venv -- oslo_run_pre_release_tests
+
+The output for each test set is logged to a separate file in the
+current directory, to make them easy to examine.
+
+Use the ``--update`` or ``-u`` option to force a ``git pull`` for each
+consuming projects before running its tests (useful for maintaining a
+long-running system to host these tests).
+
+Use the ``--verbose`` or ``-v`` option to report more verbosely about
+what is happening, including the number of projects being tested.
+
+Use ``--env`` or ``-e`` to add a tox environment to test. By default
+the ``py27`` and ``pep8`` environments are used because those have
+been shown to provide a good balance between finding problems and
+running the tests quickly.
+
+Use the ``--ref`` option to test a specific commit reference of the
+library under test. The default is to leave the source directory
+untouched, but if this option is specified ``git checkout`` is used to
+force the source tree to the specified reference.

doc/source/index.rst

@@ -13,6 +13,7 @@ Contents
    api
    testing
    features
+   cross-testing
    resources
    contributing
    history