Cleanup the tempest readme
This commit makes several needed cleanups and improvements to the tempest readme to make things clearer and elaborate on some sections. Also as the readme is intended to be a tempest users guide, this moves sections which are more developer focused to the hacking guide. Change-Id: I4180ce18268443873fe7d8d5e7d0aaebc2cd1e08
This commit is contained in:
Matthew Treinish
parent
efa399bc32
commit
a970d65486
54
HACKING.rst
54
HACKING.rst
|
|
@ -312,3 +312,57 @@ example of this would be::
|
|||
* Boot an additional instance from the new snapshot based volume
|
||||
* Check written content in the instance booted from snapshot
|
||||
"""
|
||||
|
||||
Branchless Tempest Considerations
|
||||
---------------------------------
|
||||
|
||||
Starting with the OpenStack Icehouse release Tempest no longer has any stable
|
||||
branches. This is to better ensure API consistency between releases because
|
||||
the API behavior should not change between releases. This means that the stable
|
||||
branches are also gated by the Tempest master branch, which also means that
|
||||
proposed commits to Tempest must work against both the master and all the
|
||||
currently supported stable branches of the projects. As such there are a few
|
||||
special considerations that have to be accounted for when pushing new changes
|
||||
to tempest.
|
||||
|
||||
1. New Tests for new features
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When adding tests for new features that were not in previous releases of the
|
||||
projects the new test has to be properly skipped with a feature flag. Whether
|
||||
this is just as simple as using the @test.requires_ext() decorator to check
|
||||
if the required extension (or discoverable optional API) is enabled or adding
|
||||
a new config option to the appropriate section. If there isn't a method of
|
||||
selecting the new **feature** from the config file then there won't be a
|
||||
mechanism to disable the test with older stable releases and the new test won't
|
||||
be able to merge.
|
||||
|
||||
2. Bug fix on core project needing Tempest changes
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When trying to land a bug fix which changes a tested API you'll have to use the
|
||||
following procedure::
|
||||
|
||||
- Propose change to the project, get a +2 on the change even with failing
|
||||
- Propose skip on Tempest which will only be approved after the
|
||||
corresponding change in the project has a +2 on change
|
||||
- Land project change in master and all open stable branches (if required)
|
||||
- Land changed test in Tempest
|
||||
|
||||
Otherwise the bug fix won't be able to land in the project.
|
||||
|
||||
3. New Tests for existing features
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If a test is being added for a feature that exists in all the current releases
|
||||
of the projects then the only concern is that the API behavior is the same
|
||||
across all the versions of the project being tested. If the behavior is not
|
||||
consistent the test will not be able to merge.
|
||||
|
||||
API Stability
|
||||
-------------
|
||||
|
||||
For new tests being added to Tempest the assumption is that the API being
|
||||
tested is considered stable and adheres to the OpenStack API stability
|
||||
guidelines. If an API is still considered experimental or in development then
|
||||
it should not be tested by Tempest until it is considered stable.
|
||||
|
|
|
|||
107
README.rst
107
README.rst
|
|
@ -59,50 +59,49 @@ and reference data to be used in testing.
|
|||
will have a configuration file already set up to work with your
|
||||
devstack installation.
|
||||
|
||||
Tempest is not tied to any single test runner, but testr is the most commonly
|
||||
used tool. After setting up your configuration file, you can execute
|
||||
the set of Tempest tests by using ``testr`` ::
|
||||
Tempest is not tied to any single test runner, but `testr`_ is the most commonly
|
||||
used tool. Also, the nosetests test runner is **not** recommended to run tempest.
|
||||
|
||||
After setting up your configuration file, you can execute the set of Tempest
|
||||
tests by using ``testr`` ::
|
||||
|
||||
$> testr run --parallel
|
||||
|
||||
To run one single test ::
|
||||
.. _testr: http://testrepository.readthedocs.org/en/latest/MANUAL.html
|
||||
|
||||
$> testr run --parallel tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
|
||||
To run one single test serially ::
|
||||
|
||||
$> testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
|
||||
|
||||
Alternatively, you can use the run_tempest.sh script which will create a venv
|
||||
and run the tests or use tox to do the same.
|
||||
and run the tests or use tox to do the same. Tox also contains several existing
|
||||
job configurations. For example::
|
||||
|
||||
$> tox -efull
|
||||
|
||||
which will run the same set of tests as the OpenStack gate. (it's exactly how
|
||||
the gate invokes tempest) Or::
|
||||
|
||||
$> tox -esmoke
|
||||
|
||||
to run the tests tagged as smoke.
|
||||
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
Detailed configuration of tempest is beyond the scope of this
|
||||
document. The etc/tempest.conf.sample attempts to be a self
|
||||
documenting version of the configuration.
|
||||
document see :ref:`tempest-configuration` for more details on configuring
|
||||
tempest. The etc/tempest.conf.sample attempts to be a self documenting version
|
||||
of the configuration.
|
||||
|
||||
To generate the sample tempest.conf file, run the following
|
||||
You can generate a new sample tempest.conf file, run the following
|
||||
command from the top level of the tempest directory:
|
||||
|
||||
tox -egenconfig
|
||||
|
||||
The most important pieces that are needed are the user ids, openstack
|
||||
endpoints, and basic flavors and images needed to run tests.
|
||||
|
||||
Common Issues
|
||||
-------------
|
||||
|
||||
Tempest was originally designed to primarily run against a full OpenStack
|
||||
deployment. Due to that focus, some issues may occur when running Tempest
|
||||
against devstack.
|
||||
|
||||
Running Tempest, especially in parallel, against a devstack instance may
|
||||
cause requests to be rate limited, which will cause unexpected failures.
|
||||
Given the number of requests Tempest can make against a cluster, rate limiting
|
||||
should be disabled for all test accounts.
|
||||
|
||||
Additionally, devstack only provides a single image which Nova can use.
|
||||
For the moment, the best solution is to provide the same image uuid for
|
||||
both image_ref and image_ref_alt. Tempest will skip tests as needed if it
|
||||
detects that both images are the same.
|
||||
endpoint, and basic flavors and images needed to run tests.
|
||||
|
||||
Unit Tests
|
||||
----------
|
||||
|
|
@ -132,57 +131,3 @@ present in python 2.6 will be used. If you're running you're OpenStack services
|
|||
on an earlier release with python 2.6 you can easily run tempest against it
|
||||
from a remote system running python 2.7. (or deploy a cloud guest in your cloud
|
||||
that has python 2.7)
|
||||
|
||||
Branchless Tempest Considerations
|
||||
---------------------------------
|
||||
|
||||
Starting with the OpenStack Icehouse release Tempest no longer has any stable
|
||||
branches. This is to better ensure API consistency between releases because
|
||||
the API behavior should not change between releases. This means that the stable
|
||||
branches are also gated by the Tempest master branch, which also means that
|
||||
proposed commits to Tempest must work against both the master and all the
|
||||
currently supported stable branches of the projects. As such there are a few
|
||||
special considerations that have to be accounted for when pushing new changes
|
||||
to tempest.
|
||||
|
||||
1. New Tests for new features
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When adding tests for new features that were not in previous releases of the
|
||||
projects the new test has to be properly skipped with a feature flag. Whether
|
||||
this is just as simple as using the @test.requires_ext() decorator to check
|
||||
if the required extension (or discoverable optional API) is enabled or adding
|
||||
a new config option to the appropriate section. If there isn't a method of
|
||||
selecting the new **feature** from the config file then there won't be a
|
||||
mechanism to disable the test with older stable releases and the new test won't
|
||||
be able to merge.
|
||||
|
||||
2. Bug fix on core project needing Tempest changes
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
When trying to land a bug fix which changes a tested API you'll have to use the
|
||||
following procedure::
|
||||
|
||||
- Propose change to the project, get a +2 on the change even with failing
|
||||
- Propose skip on Tempest which will only be approved after the
|
||||
corresponding change in the project has a +2 on change
|
||||
- Land project change in master and all open stable branches (if required)
|
||||
- Land changed test in Tempest
|
||||
|
||||
Otherwise the bug fix won't be able to land in the project.
|
||||
|
||||
3. New Tests for existing features
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If a test is being added for a feature that exists in all the current releases
|
||||
of the projects then the only concern is that the API behavior is the same
|
||||
across all the versions of the project being tested. If the behavior is not
|
||||
consistent the test will not be able to merge.
|
||||
|
||||
API Stability
|
||||
-------------
|
||||
|
||||
For new tests being added to Tempest the assumption is that the API being
|
||||
tested is considered stable and adheres to the OpenStack API stability
|
||||
guidelines. If an API is still considered experimental or in development then
|
||||
it should not be tested by Tempest until it is considered stable.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
.. _tempest-configuration:
|
||||
|
||||
Tempest Configuration Guide
|
||||
===========================
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue