From 759c42b10cb3728f5549b05f68e826b1c62a968c Mon Sep 17 00:00:00 2001 From: Matthew Treinish Date: Tue, 14 Nov 2017 15:31:17 -0500 Subject: [PATCH] Update Python PTI for tests to be specific and explicit This commit updates the language in the Python PTI to be much more specific about how projects should be running tests. This is done so we can have a consistent experience between projects when running tests. The current state of affairs is far from ideal in that there are about 7 different ways projects run python unit tests. (some don't even comply with the looser previous PTI) For example:: * PBR's testr entrpoint "python setup.py test" https://github.com/openstack-dev/pbr/blob/master/pbr/testr_command.py * testr's setup tools entrypoint "python setup.py testr" https://github.com/testing-cabal/testrepository/blob/master/testrepository/setuptools_command.py * pretty_tox.sh wrapper scripts and variants (this was cargo culted from tempest and diverged significantly between each projects' forks) * ostestr (which is part of the os-testr package, but it's not the only thing there) * nosetests (the horizon case for django testing will likely be a continued exception) * proboscis (which are just the examples I know of off the top of my head) For anyone trying to work in multiple projects navigating this can be quite tricky, especially when trying to debug a failure. To clean this up, this patch defines one way to run python tests. Besides making the wording more explicit and providing specifics this commit also tries to provide some additional background information on why we use stestr. It's not always obvious the advantages it provides us, so this attempts to document it at a high level for the ages. Change-Id: I2637dd714cbb6d38ef8b8dc1083e359207118284 --- reference/pti/python.rst | 58 +++++++++++++++++++++++++++++++++------- 1 file changed, 49 insertions(+), 9 deletions(-) diff --git a/reference/pti/python.rst b/reference/pti/python.rst index bb23e27e6..aed7ac59c 100644 --- a/reference/pti/python.rst +++ b/reference/pti/python.rst @@ -69,16 +69,56 @@ Virtual Environment Management To support sensible testing across multiple python versions, we use tox config files in the projects. -unittest running ----------------- +Python test running +------------------- -OpenStack uses testrepository and stestr as its test runner, which supports a -number of things, most importantly to the expanded project is the subunit output -stream collection. This is useful for aggregating and displaying test output. -In support of that, the oslotest library is built on top of testtools, -testscenarios and fixtures. The usage of the testrepository project is -deprecated and things are being migrated to stestr which is an active and -currently maintained fork of testrepository. +OpenStack uses stestr as its test runner. stestr should be used for running +all python tests, this includes unit tests, functional tests, and integration +tests. stestr is used because of its real time subunit output and its support +for parallel execution of tests. In addition, stestr only runs tests conforming +to the python stdlib unittest model (and extensions on it like testtools). This +enables people to use any test runner they prefer locally. Other popular test +runners often include a testing ecosystem which is tied directly to the runner. +Using these precludes the use of alternative runners for other users. + +To have a consistent interface via tox between projects' unit test +jobs the command for running stestr in tox should be set to:: + + stestr run '{posargs}' + +.. note:: + While the use of wrapper scripts can sometimes be useful as a short term + crutch to work around a specific temporary issues, it should be avoided + because it creates a divergent experience between projects, and can mask + real issues. + +If there are additional mandatory args needed for running a test suite they +can be added before the posargs. (this way the end user experience is the same) +For example:: + + stestr --test-path ./tests/unit run '{posargs}' + +However, these arguments should try to be minimized because it just adds to the +complexity that people will need to understand when running tests on a project. + +Coverage Jobs +------------- + +For coverage jobs you need to invoke the test runner in the same way as for the +normal unit test jobs, but to switch the python executable to be +``coverage run``. To do this you need to setup the tox ``cover`` job like:: + + [testenv:cover] + setenv = + PYTHON=coverage run --source $project --parallel-mode + commands = + stestr run '{posargs}' + coverage combine + coverage html -d cover + coverage xml -o cover/coverage.xml + +Specifically, the output html directory ``cover`` and the ``coverage.xml`` file +added to that directory are mandatory output artifacts. Project Configuration