Improved docs for os-testr commands
This commit adds real docs for using the 3 currently packaged commands in the os-testr project. As part of this change it also updates the sphinx config to generate proper man pages for all the tooling. At a future stage we need to ensure that these get installed properly. Change-Id: I0055c7961203a094590f6cfeb136a2236a2f65cc
This commit is contained in:
Matthew Treinish
parent
16330c0526
commit
2f29ca51be
|
|
@ -21,6 +21,14 @@ Features
|
|||
Release Notes
|
||||
=============
|
||||
|
||||
0.2.0
|
||||
-----
|
||||
* Adds support for comments in a blacklist file and printing comments for each
|
||||
exclude
|
||||
* Several bugfixes for subunit-trace
|
||||
* Switched subunit-trace behavior to disable printing percent change
|
||||
in run time by default and make it optional
|
||||
|
||||
0.1.0
|
||||
-----
|
||||
* First release which includes: ostestr, subunit-trace, and subunit2html
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ sys.path.insert(0, os.path.abspath('../..'))
|
|||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
#'sphinx.ext.intersphinx',
|
||||
# 'sphinx.ext.intersphinx',
|
||||
'oslosphinx'
|
||||
]
|
||||
|
||||
|
|
@ -38,7 +38,7 @@ master_doc = 'index'
|
|||
|
||||
# General information about the project.
|
||||
project = u'os-testr'
|
||||
copyright = u'2013, OpenStack Foundation'
|
||||
copyright = u'2015, Matthew Treinish'
|
||||
|
||||
# If true, '()' will be appended to :func: etc. cross-reference text.
|
||||
add_function_parentheses = True
|
||||
|
|
@ -68,8 +68,15 @@ latex_documents = [
|
|||
('index',
|
||||
'%s.tex' % project,
|
||||
u'%s Documentation' % project,
|
||||
u'OpenStack Foundation', 'manual'),
|
||||
u'Matthew Treinish', 'manual'),
|
||||
]
|
||||
|
||||
man_pages = [('ostestr', 'ostestr', 'tooling to run OpenStack tests',
|
||||
['Matthew Treinish'], 1),
|
||||
('subunit_trace', 'subunit-trace', 'pretty output filter for '
|
||||
'subunit streams', ['Matthew Treinish'], 1),
|
||||
('subunit2html', 'subunit2html', 'generate a html results page '
|
||||
'from a subunit stream', ['Matthew Treinish'], 1)]
|
||||
|
||||
# Example configuration for intersphinx: refer to the Python standard library.
|
||||
#intersphinx_mapping = {'http://docs.python.org/': None}
|
||||
# intersphinx_mapping = {'http://docs.python.org/': None}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,217 @@
|
|||
.. _ostestr:
|
||||
|
||||
ostestr
|
||||
=======
|
||||
|
||||
The ostestr command provides a wrapper around the testr command included in
|
||||
the testrepository package. It's designed to build on the functionality
|
||||
included in testr and workaround several UI bugs in the short term. By default
|
||||
it also has output that is much more useful for OpenStack's test suites which
|
||||
are lengthy in both runtime and number of tests. Please note that the CLI
|
||||
semantics are still a work in progress as the project is quite young, so
|
||||
default behavior might change in future version.
|
||||
|
||||
Summary
|
||||
-------
|
||||
ostestr [-b|--blacklist_file <blacklist_file>] [-r|--regex REGEX]
|
||||
[-p|--pretty] [--no-pretty] [-s|--subunit] [-l|--list]
|
||||
[-n|--no-discover <test_id>] [--slowest] [--no-slowest]
|
||||
[--pdb <test_id>] [--parallel] [--serial]
|
||||
[-c|--concurrency <workers>] [--until-failure] [--print-exclude]
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
--blacklist_file BLACKLIST_FILE, -b BLACKLIST_FILE
|
||||
Path to a blacklist file, this file contains a
|
||||
separate regex exclude on each newline
|
||||
--regex REGEX, -r REGEX
|
||||
A normal testr selection regex. If a blacklist file is
|
||||
specified, the regex will be appended to the end of
|
||||
the generated regex from that file
|
||||
--pretty, -p
|
||||
Print pretty output from subunit-trace. This is
|
||||
mutually exclusive with --subunit
|
||||
--no-pretty
|
||||
Disable the pretty output with subunit-trace
|
||||
--subunit, -s
|
||||
output the raw subunit v2 from the test run this is
|
||||
mutuall exclusive with --pretty
|
||||
--list, -l
|
||||
List all the tests which will be run.
|
||||
--no-discover TEST_ID, -n TEST_ID
|
||||
Takes in a single test to bypasses test discover and
|
||||
just excute the test specified
|
||||
--slowest
|
||||
After the test run print the slowest tests
|
||||
--no-slowest
|
||||
After the test run don't print the slowest tests
|
||||
--pdb TEST_ID
|
||||
Run a single test that has pdb traces added
|
||||
--parallel
|
||||
Run tests in parallel (this is the default)
|
||||
--serial
|
||||
Run tests serially
|
||||
--concurrency WORKERS, -c WORKERS
|
||||
The number of workers to use when running in parallel.
|
||||
By default this is the number of cpus
|
||||
--until-failure
|
||||
Run the tests in a loop until a failure is
|
||||
encountered. Running with subunit or prettyoutput
|
||||
enable will force the loop to run testsserially
|
||||
--print-exclude
|
||||
If an exclude file is used this option will prints the
|
||||
comment from the same line and all skipped tests
|
||||
before the test run
|
||||
|
||||
Running Tests
|
||||
-------------
|
||||
|
||||
os-testr is primarily for running tests at it's basic level you just invoke
|
||||
ostestr to run a test suite for a project. (assuming it's setup to run tests
|
||||
using testr already) For example::
|
||||
|
||||
$ ostestr
|
||||
|
||||
This will run tests in parallel (with the number of workers matching the number
|
||||
of CPUs) and with subunit-trace output. If you need to run tests in serial you
|
||||
can use the serial option::
|
||||
|
||||
$ ostestr --serial
|
||||
|
||||
Or if you need to adjust the concurrency but still run in parallel you can use
|
||||
-c/--concurrency::
|
||||
|
||||
$ ostestr --concurrency 2
|
||||
|
||||
If you only want to run an individual test module or more specific (a single
|
||||
class, or test) and parallel execution doesn't matter, you can use the
|
||||
-n/--no-discover to skip test discovery and just directly calls subunit.run on
|
||||
the tests under the covers. Bypassing discovery is desirable when running a
|
||||
small subset of tests in a larger test suite because the discovery time can
|
||||
often far exceed the total run time of the tests.
|
||||
|
||||
For example::
|
||||
|
||||
$ ostestr --no-discover test.test_thing.TestThing.test_thing_method
|
||||
|
||||
Additionally, if you need to run a single test module, class, or single test
|
||||
with pdb enabled you can use --pdb to directly call testtools.run under the
|
||||
covers which works with pdb. For example::
|
||||
|
||||
$ ostestr --pdb tests.test_thing.TestThing.test_thing_method
|
||||
|
||||
|
||||
Test Selection
|
||||
--------------
|
||||
|
||||
ostestr is designed to build on top of the test selection in testr. testr only
|
||||
exposed a regex option to select tests. This equivalent is functionality is
|
||||
exposed via the --regex option. For example::
|
||||
|
||||
$ ostestr --regex 'magic\.regex'
|
||||
|
||||
This will do a straight passthrough of the provided regex to testr.
|
||||
Additionally, ostestr allows you to specify a a blacklist file to define a set
|
||||
of regexes to exclude. You can specify a blacklist file with the
|
||||
--blacklist-file/-b option, for example::
|
||||
|
||||
$ ostestr --blacklist_file $path_to_file
|
||||
|
||||
The format for the file is line separated regex, with '#' used to signify the
|
||||
start of a comment on a line. For example::
|
||||
|
||||
# Blacklist File
|
||||
^regex1 # Excludes these tests
|
||||
.*regex2 # exclude those tests
|
||||
|
||||
Will generate a regex to pass to testr which will exclude both any tests
|
||||
matching '^regex1' and '.*regex2'. If a blacklist file is used in conjunction
|
||||
with the --regex option the regex specified with --regex will be appended to
|
||||
the generated output from the --blacklist_file. Also it's worth noting that the
|
||||
regex test selection options can not be used in conjunction with the
|
||||
--no-discover or --pdb options described in the previous section. This is
|
||||
because the regex selection requires using testr under the covers to actually
|
||||
do the filtering, and those 2 options do not use testr.
|
||||
|
||||
It's also worth noting that you can use the test list option to dry run any
|
||||
selection arguments you are using. You just need to use --list/-l with your
|
||||
selection options to do this, for example::
|
||||
|
||||
$ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list
|
||||
|
||||
This will list all the tests which will be run by ostestr using that combination
|
||||
of arguments.
|
||||
|
||||
Please not that all of this selection functionality will be expanded on in the
|
||||
future and a default grammar for selecting multiple tests will be chosen in a
|
||||
future release. However as of right now all current arguments (which have
|
||||
guarantees on always remaining in place) are still required to perform any
|
||||
selection logic while this functionality is still under development.
|
||||
|
||||
|
||||
Output Options
|
||||
--------------
|
||||
|
||||
By default ostestr will use subunit-trace as the output filter on the test
|
||||
run. It will also print the slowest tests from the run after the run is
|
||||
concluded. You can disable the printing the slowest tests with the --no-slowest
|
||||
flag, for example::
|
||||
|
||||
$ ostestr --no-slowest
|
||||
|
||||
If you'd like to disable the subunit-trace output you can do this using
|
||||
--no-pretty::
|
||||
|
||||
$ ostestr --no-pretty
|
||||
|
||||
ostestr also provides the option to just output the raw subunit stream on
|
||||
STDOUT with --subunit/-s. Note if you want to use this you also have to
|
||||
specify --no-pretty as the subunit-trace output and the raw subunit output
|
||||
are mutually exclusive. For example, to get raw subunit output the arguments
|
||||
would be::
|
||||
|
||||
$ ostestr --no-pretty --subunit
|
||||
|
||||
An additional option on top of the blacklist file is --print-exclude option.
|
||||
When this option is specified when using a blacklist file before the tests are
|
||||
run ostestr will print all the tests it will be excluding from the blacklist
|
||||
file. If a line in the blacklist file has a comment that will be printed before
|
||||
listing the tests which will be excluded by that line's regex. If no comment is
|
||||
present on a line the regex from that line will be used instead. For example,
|
||||
if you were using the example blacklist file from the previous section the
|
||||
output before the regular test run output would be::
|
||||
|
||||
$ ostestr -b blacklist-file blacklist.txt --print-exclude
|
||||
Excludes these tests
|
||||
regex1_match
|
||||
regex1_exclude
|
||||
|
||||
exclude those tests
|
||||
regex2_match
|
||||
regex2_exclude
|
||||
|
||||
...
|
||||
|
||||
Notes for running with tox
|
||||
--------------------------
|
||||
|
||||
If you use `tox`_ for running your tests and call ostestr as the test command
|
||||
.. _tox: https://tox.readthedocs.org/en/latest/
|
||||
it's recommended that you set a posargs following ostestr on the commands
|
||||
stanza. For example::
|
||||
|
||||
[testenv]
|
||||
commands = ostestr {posargs}
|
||||
|
||||
this will enable end users to pass args to configure the output, use the
|
||||
selection logic, or any other options directly from the tox cli. This will let
|
||||
tox take care of the venv management and the environment separation but enable
|
||||
direct access to all of the ostestr options to easily customize your test run.
|
||||
For example, assuming the above posargs usage you would be to do::
|
||||
|
||||
$ tox -epy34 -- --regex ^regex1
|
||||
|
||||
or to skip discovery::
|
||||
|
||||
$ tox -epy34 -- -n test.test_thing.TestThing.test_thing_method
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
.. _subunit2html:
|
||||
|
||||
subunit2html
|
||||
============
|
||||
|
||||
subunit2html is a tool that takes in a subunit stream file and will output an
|
||||
html page
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
subunit2html subunit_stream [output]
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
subunit2html takes in 1 mandatory argument. This is used to specify the location
|
||||
of the subunit stream file. For example::
|
||||
|
||||
$ subunit2html subunit_stream
|
||||
|
||||
By default subunit2html will store the generated html results file at
|
||||
results.html file in the current working directory.
|
||||
|
||||
An optional second argument can be provided to set the output path of the html
|
||||
results file that is generated. If it is provided this will be the output path
|
||||
for saving the generated file, otherwise results.html in the current working
|
||||
directory will be used. For example::
|
||||
|
||||
$ subunit2html subunit_stream test_results.html
|
||||
|
||||
will write the generated html results file to test_results.html in the current
|
||||
working directory
|
||||
|
|
@ -0,0 +1,110 @@
|
|||
.. _subunit_trace:
|
||||
|
||||
subunit-trace
|
||||
=============
|
||||
|
||||
subunit-trace is an output filter for subunit streams. It is often used in
|
||||
conjunction with test runners that emit subunit to enable a consistent and
|
||||
useful realtime output from a test run.
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
subunit-trace [--fails|-f] [--failonly] [--perc-diff|-d] [--no-summary]
|
||||
[--diff-threshold|-t <threshold>]
|
||||
|
||||
Options
|
||||
-------
|
||||
|
||||
--no-failure-debug, -n
|
||||
Disable printing failure debug information in realtime
|
||||
--fails, -f
|
||||
Print failure debug information after the stream is
|
||||
proccesed
|
||||
--failonly
|
||||
Don't print success items
|
||||
--perc-diff, -d
|
||||
Print percent change in run time on each test
|
||||
--diff-threshold THRESHOLD, -t THRESHOLD
|
||||
Threshold to use for displaying percent change from the
|
||||
avg run time. If one is not specified the percent
|
||||
change will always be displayed.
|
||||
--no-summary
|
||||
Don't print the summary of the test run after completes
|
||||
|
||||
Usage
|
||||
-----
|
||||
subunit-trace will take a subunit stream in via STDIN. This is the only input
|
||||
into the tool. It will then print on STDOUT the formatted test result output
|
||||
for the test run information contained in the stream.
|
||||
|
||||
A subunit v2 stream must be passed into subunit-trace. If only a subunit v1
|
||||
stream is available you must use the subunit-1to2 utility to convert it before
|
||||
passing the stream into subunit-trace. For example this can be done by chaining
|
||||
pipes::
|
||||
|
||||
$ cat subunit_v1 | subunit-1to2 | subunit-trace
|
||||
|
||||
Adjusting per test output
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
subunit-trace provides several options to customize it's output. This allows
|
||||
users to customize the output from subunit-trace to suit their needs. The output
|
||||
from subunit-trace basically comes in 2 parts, the per test output, and the
|
||||
summary at the end. By default subunit-trace will print failure messages during
|
||||
the per test output, meaning when a test fails it will also print the message
|
||||
and any traceback and other attachments at that time. However this can be
|
||||
disabled by using --no-failure-debug, -n. For example::
|
||||
|
||||
$ testr run --subunit | subunit-trace --no-failure-debug
|
||||
|
||||
Rhere is also the option to print all failures together at the end of the test
|
||||
run before the summary view. This is done using the --fails/-f option. For
|
||||
example::
|
||||
|
||||
$ testr run --subunit | subunit-trace --fails
|
||||
|
||||
Often the --fails and --no-failure-debug options are used in conjunction to
|
||||
only print failures at the end of a test run. This is useful for large test
|
||||
suites where an error message might be lost in the noise. To do this ::
|
||||
|
||||
$ testr run --subunit | subunit-trace --fails --no-failure-debug
|
||||
|
||||
By default subunit-trace will print a line for each test after it completes with
|
||||
the test status. However, if you only want to see the run time output for
|
||||
failures and not any other test status you can use the --failonly option. For
|
||||
example::
|
||||
|
||||
$ testr run --subunit | subunit-trace --failonly
|
||||
|
||||
The last output option provided by subunit-trace is to diable the summary view
|
||||
of the test run which is normally displayed at the end of a run. You can do
|
||||
this using the --no-summary option. For example::
|
||||
|
||||
$ testr run --subunit | subunit-trace --no-summary
|
||||
|
||||
|
||||
Show per test run time percent change
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
subunit-trace provides an option to display the percent change in run time
|
||||
from the previous run. To do this subunit-trace leverages the testr internals
|
||||
a bit. It uses the times.dbm database which, the file repository type in
|
||||
testrepository will create, to get the previous run time for a test. If testr
|
||||
hasn't ever been used before or for whatever reason subunit-trace is unable to
|
||||
find the times.dbm file from testr no percentages will be displayed even if it's
|
||||
enabled. Additionally, if a test is run which does not have an entry in the
|
||||
times.dbm file will not have a percentage printed for it.
|
||||
|
||||
To enable this feature you use --perc-diff/-d, for example::
|
||||
|
||||
$ testr run --subunit | subunit-trace --perc-diff
|
||||
|
||||
There is also the option to set a threshold value for this option. If used it
|
||||
acts as an absolute value and only percentage changes that exceed it will be
|
||||
printed. Use the --diff-threshold/-t option to set a threshold, for example::
|
||||
|
||||
$ testr run --subunit | subunit-trace --perc-diff --threshold 45
|
||||
|
||||
This will only display percent differences when the change in run time is either
|
||||
>=45% faster or <=45% slower.
|
||||
|
|
@ -1,7 +1,12 @@
|
|||
========
|
||||
=====
|
||||
Usage
|
||||
========
|
||||
=====
|
||||
|
||||
To use os-testr in a project::
|
||||
This section contains the documentation for each of tools packaged in os-testr
|
||||
|
||||
import os_testr
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
ostestr
|
||||
subunit_trace
|
||||
subunit2html
|
||||
|
|
|
|||
Loading…
Reference in New Issue