diff --git a/HACKING.rst b/HACKING.rst index 81a7c2ca53..04b5eb6d56 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -312,3 +312,57 @@ example of this would be:: * Boot an additional instance from the new snapshot based volume * Check written content in the instance booted from snapshot """ + +Branchless Tempest Considerations +--------------------------------- + +Starting with the OpenStack Icehouse release Tempest no longer has any stable +branches. This is to better ensure API consistency between releases because +the API behavior should not change between releases. This means that the stable +branches are also gated by the Tempest master branch, which also means that +proposed commits to Tempest must work against both the master and all the +currently supported stable branches of the projects. As such there are a few +special considerations that have to be accounted for when pushing new changes +to tempest. + +1. New Tests for new features +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When adding tests for new features that were not in previous releases of the +projects the new test has to be properly skipped with a feature flag. Whether +this is just as simple as using the @test.requires_ext() decorator to check +if the required extension (or discoverable optional API) is enabled or adding +a new config option to the appropriate section. If there isn't a method of +selecting the new **feature** from the config file then there won't be a +mechanism to disable the test with older stable releases and the new test won't +be able to merge. + +2. Bug fix on core project needing Tempest changes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When trying to land a bug fix which changes a tested API you'll have to use the +following procedure:: + + - Propose change to the project, get a +2 on the change even with failing + - Propose skip on Tempest which will only be approved after the + corresponding change in the project has a +2 on change + - Land project change in master and all open stable branches (if required) + - Land changed test in Tempest + +Otherwise the bug fix won't be able to land in the project. + +3. New Tests for existing features +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If a test is being added for a feature that exists in all the current releases +of the projects then the only concern is that the API behavior is the same +across all the versions of the project being tested. If the behavior is not +consistent the test will not be able to merge. + +API Stability +------------- + +For new tests being added to Tempest the assumption is that the API being +tested is considered stable and adheres to the OpenStack API stability +guidelines. If an API is still considered experimental or in development then +it should not be tested by Tempest until it is considered stable. diff --git a/README.rst b/README.rst index 7af0025da3..9aaea240cc 100644 --- a/README.rst +++ b/README.rst @@ -59,50 +59,49 @@ and reference data to be used in testing. will have a configuration file already set up to work with your devstack installation. -Tempest is not tied to any single test runner, but testr is the most commonly -used tool. After setting up your configuration file, you can execute -the set of Tempest tests by using ``testr`` :: +Tempest is not tied to any single test runner, but `testr`_ is the most commonly +used tool. Also, the nosetests test runner is **not** recommended to run tempest. + +After setting up your configuration file, you can execute the set of Tempest +tests by using ``testr`` :: $> testr run --parallel -To run one single test :: +.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html - $> testr run --parallel tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server +To run one single test serially :: + + $> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server Alternatively, you can use the run_tempest.sh script which will create a venv -and run the tests or use tox to do the same. +and run the tests or use tox to do the same. Tox also contains several existing +job configurations. For example:: + + $> tox -efull + +which will run the same set of tests as the OpenStack gate. (it's exactly how +the gate invokes tempest) Or:: + + $> tox -esmoke + +to run the tests tagged as smoke. + Configuration ------------- Detailed configuration of tempest is beyond the scope of this -document. The etc/tempest.conf.sample attempts to be a self -documenting version of the configuration. +document see :ref:`tempest-configuration` for more details on configuring +tempest. The etc/tempest.conf.sample attempts to be a self documenting version +of the configuration. -To generate the sample tempest.conf file, run the following +You can generate a new sample tempest.conf file, run the following command from the top level of the tempest directory: tox -egenconfig The most important pieces that are needed are the user ids, openstack -endpoints, and basic flavors and images needed to run tests. - -Common Issues -------------- - -Tempest was originally designed to primarily run against a full OpenStack -deployment. Due to that focus, some issues may occur when running Tempest -against devstack. - -Running Tempest, especially in parallel, against a devstack instance may -cause requests to be rate limited, which will cause unexpected failures. -Given the number of requests Tempest can make against a cluster, rate limiting -should be disabled for all test accounts. - -Additionally, devstack only provides a single image which Nova can use. -For the moment, the best solution is to provide the same image uuid for -both image_ref and image_ref_alt. Tempest will skip tests as needed if it -detects that both images are the same. +endpoint, and basic flavors and images needed to run tests. Unit Tests ---------- @@ -132,57 +131,3 @@ present in python 2.6 will be used. If you're running you're OpenStack services on an earlier release with python 2.6 you can easily run tempest against it from a remote system running python 2.7. (or deploy a cloud guest in your cloud that has python 2.7) - -Branchless Tempest Considerations ---------------------------------- - -Starting with the OpenStack Icehouse release Tempest no longer has any stable -branches. This is to better ensure API consistency between releases because -the API behavior should not change between releases. This means that the stable -branches are also gated by the Tempest master branch, which also means that -proposed commits to Tempest must work against both the master and all the -currently supported stable branches of the projects. As such there are a few -special considerations that have to be accounted for when pushing new changes -to tempest. - -1. New Tests for new features -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When adding tests for new features that were not in previous releases of the -projects the new test has to be properly skipped with a feature flag. Whether -this is just as simple as using the @test.requires_ext() decorator to check -if the required extension (or discoverable optional API) is enabled or adding -a new config option to the appropriate section. If there isn't a method of -selecting the new **feature** from the config file then there won't be a -mechanism to disable the test with older stable releases and the new test won't -be able to merge. - -2. Bug fix on core project needing Tempest changes -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When trying to land a bug fix which changes a tested API you'll have to use the -following procedure:: - - - Propose change to the project, get a +2 on the change even with failing - - Propose skip on Tempest which will only be approved after the - corresponding change in the project has a +2 on change - - Land project change in master and all open stable branches (if required) - - Land changed test in Tempest - -Otherwise the bug fix won't be able to land in the project. - -3. New Tests for existing features -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If a test is being added for a feature that exists in all the current releases -of the projects then the only concern is that the API behavior is the same -across all the versions of the project being tested. If the behavior is not -consistent the test will not be able to merge. - -API Stability -------------- - -For new tests being added to Tempest the assumption is that the API being -tested is considered stable and adheres to the OpenStack API stability -guidelines. If an API is still considered experimental or in development then -it should not be tested by Tempest until it is considered stable. diff --git a/doc/source/configuration.rst b/doc/source/configuration.rst index f772aa3adc..a7c8fb7f04 100644 --- a/doc/source/configuration.rst +++ b/doc/source/configuration.rst @@ -1,3 +1,5 @@ +.. _tempest-configuration: + Tempest Configuration Guide ===========================