This change fixes typos in tempest code, doc, zuul yaml
and tox.ini file, which are found by codespell.
Change-Id: I8051d979e28f7dbf532181f339cbef9cd7220c76
To releas a new tag for Tempest 2024.1, we modified the
compatible constraint in tox.ini
- https://review.opendev.org/c/openstack/tempest/+/913950/
As new tag is released now so we can switch back to the master
constraint in tox.ini.
Change-Id: I1ec341903929f240ef3e796ee0db2acda9af9823
We are going to release Tempest new tag to declare the start of
support for stable/2024.1. So that new tag use stable/2024.1
constraint in the tox env.
For example: if anyone use Tempest new tag in future say
1 year later then tox env also should use the 2024.1 contstraint for
compatibility instead master.
Also, making ironic-tempest-bios-ipmi-direct-tinyipa as
non voting to merge his change.
These need to be reverted once release is done.
Change-Id: I5ee1ce6b3ffbbb1f47e448fc15d540f74b21684f
We are going to release Tempest new tag 37.0.0 to declare
the end of support for stable/yoga. stable/2023.2 constraint
is latest stable constraints tested with current tempest hash
which is going to be used for 37.0.0 release.
Further information:
https://docs.openstack.org/tempest/latest/requirement_upper_constraint_for_tempest.html
Change-Id: I0b4863c219b71b75f665e7a7599063124c34b2de
This adds the '--slowest' option to the 'tempest run' command line for
the [testenv:integrated-compute] to show the slowest tests after the
tests finish running.
Change-Id: I53b0fa6dcf077191565e2aaf3f08a24b3238a34b
This patch proposes adding several new unit tests in effort to increase
the total unit test coverage.
Additionally, the ./tempest/serial_tests/ directory is not included in
coverage calculation. The reason being we don't want to test tests with
unit tests, which becomes a bit of an overkill.
Finally, there is an addition of a new option for the coverage program
called --fail-under. It will result the program in failure if the
coverage is under set percentage. Reason for this implementation is to
encourage writing unit tests for new code.
Change-Id: I804116413cd7d73cd7e5ae71409a8855ef937b88
To releas a new tag for Tempest 2023.2, we modified the
compatible constraint in tox.ini
- https://review.opendev.org/c/openstack/tempest/+/874359/1
As new tag is released now so we can switch back to the master
constraint in tox.ini.
Change-Id: I431ec42d81c4d2a95d6f541c794607de1f9cadb8
We are going to release Tempest new tag to declare the start of
support for stable/2023.2. So that new tag use stable/2023.2
constraint in the tox env.
For example: if anyone use Tempest new tag in future say
1 year later then tox env also should use the 2023.2 contstraint for
compatibility instead master.
Also, making ironic-tempest-bios-ipmi-direct-tinyipa as
non voting to merge his change.
These need to be reverted once release is done.
Change-Id: Iea83eb6b467f40b9398262d6ddf6912d0ea98d1a
this is to avoid current problem due to
https://github.com/mtreinish/stestr/issues/347
but also to equalize dependencies with other test envs.
Change-Id: I867bec362d83ab820aa6b11dc4770d50133ecdbf
We are going to release Tempest new tag 35.0.0 to declare
the end of support for stable/xena. stable/2023.1 constraint
is latest stable constraints tested with current tempest hash
which is going to be used for 35.0.0 release.
Further information:
https://docs.openstack.org/tempest/latest/requirement_upper_constraint_for_tempest.html
Change-Id: I658c195385418fe84e6b9574a089fd1ede3e8217
To releas a new tag for Tempest, we modified the
compatible constraint in tox.ini
- https://review.opendev.org/c/openstack/tempest/+/882354/
As new tag is released now so we can switch back to the master
constraint in tox.ini.
Change-Id: Ic4f60c6a1b8b657edf06cffc9165c49da4395389
We are going to release Tempest new tag which work with
the stable/2023.1 constraint so use that in the tox env.
For example: if anyone use Tempest new tag in future say
1 year later then tox env also should use the 2023.1 contstraint for
compatibility instead master.
These need to be move back to master constraint once release is done.
Change-Id: I81c251c7f006b8792c006db88bd47fdb40df14a1
To releas a new tag for Tempest 2023.1, we modified the
compatible constraint in tox.ini
- https://review.opendev.org/c/openstack/tempest/+/877713
As new tag is released now so we can switch back to the master
constraint in tox.ini.
Change-Id: Ic91e443ae1948d71fa276be03dcef4576c64df33
We will release Tempest new tag to get cinder tests
fix[1] in stable/2023.1. Without the cinder test fix[1], tempest
released for 2023.1 will not work with cinder in 2023.1.
So that new tag use stable/2023.1 constraint in the tox env.
For example: if anyone uses Tempest new tag in future say
1 year later then tox env also should use the 2023.1 contstraint for
compatibility instead master.
These need to be move back to master constraint once release is done.
[1] https://review.opendev.org/c/openstack/tempest/+/875372
Change-Id: Ibb348e0642190b3a83d8b5f86c8b49295cf071e0
Recently we are seeing a lot of job timeout(bug#2004780)
and we see many tests taking time and also number of tests
increasing over time. This is to prepare the list of extra tests
(here extra tests means the tests which are covered by the other
API or scenario tests) which we do not need to run in every
integrated jobs. Instead, we can run them in a separete job(s).
Currently I am adding admin (except keystone) and negative tests
in the 'extra tests' list but we can add more tests here which
we think are covered in some other tests.
As negative tests are important for interop, adding those extra
tests coverage for stable branch job also but running them in
periodic run only.
Related-Bug: #2004780
Change-Id: Id02221df0d6180519751c63e890851bd59fdafa0
multinode job run all the tests including multinode and
non multinode tests. But we do not need to run all the
non multinode tests in this job instead smoke tests along
with multinode tests should be enough to run. This make
multinode jobs to run only smoke and multinode tests. For
that, we need to tag the multinode tests with 'multinode' attr.
Relavant-Bug: #2004780
Change-Id: I7e87d1db3ef3a00b3d27f0904d0af6a270e03837
tempest-slow-py3 job run all the slow test serially which
takes lot of time and end up job timeout. This preparing
tempest-slow-parallel job which will run slow tests parallelly
in periodic run. Based on the results, later we can make
tempest-slow-py3 job to run tests in parallel.
Also, run tempest-full-parallel in periodic and based on the
result we can run tempest-full-py3 job scenario tests in parallel.
Relavant-Bug: #2004780
Change-Id: I876dacb40daa384cddc8faae3200cd3d39506ddc
With the new version of tox, the interpretation of escaping in the
command section changed [1]. For example, the `tox -e smoke` runs the
following command to execute tempest tests when tox==4.3.5. is installed:
tempest run --regex '[.*\bsmoke\b.*]'
(missing '\' in front of '[', ']') ^^
The issue can be fixed either by adding another backslash ('\\[') or
moving the regular expressions to separate variables. The first solution
won't, however, be backwards compatible with the older version of tox
therefore this patch uses the second approach.
[1] https://tox.wiki/en/latest/config.html#substitutions
Change-Id: Ie5ea7f9305c7e7611f205d2e0703849bd6321a68
Tempest provides a LockFixture to avoid two potentially interfering
tests to run in parallel. However, this solution does not scale when
we want to separate a set of tests from many other test cases. For
example, host aggregate and availability zone testing needs compute
hosts without any nova servers to be able to test moving computes
between aggregates but a lot of other tests are creating nova
servers. To fully separate these aggregate tests from the rest of
the tempest test cases, this patch proposes a @serial class decorator
to mark a test class to be run totally independently of any other test
classes.
Under the hood, the @serial decorator is implemented with a tempest-wide
interprocess read-write lock. The serial test classes always take the
write lock, while the non-serial classes take the read lock. The lock
allows in many readers OR a single writer. So the serial tests are run
independently from the rest.
To minimize the time a serial test blocks other tempest tests run in
parallel, this patch also introduced a serial_tests test directory to
store the serial tests. The current test ordering in a fresh env
uses alphabetical order so the serial tests will run at the end of
the execution not randomly in the middle. The gate uses fresh VMs
for every run so we can rely on this optimization there. In local
envs where tests are re-run, the subsequent runs will be ordered at
runtime by stestr. Therfore, a longer runtime might be observed due to
locking, but the correctness of the test execution is still kept.
Related-Bug: #821732
Change-Id: I0181517edab75f586464a38c4811417f888783b1
tox.ini started failing with Tox4 which had some
incompatible changes. One is passenv where we need
to pass each value in newline otherwise, it fails with
error:
failed with pass_env values cannot contain whitespace,
use comma to have multiple values in a single line,
invalid values found.....
Also adding tempest in allowlist_externals as it is required
by unit test also.
Fixing tox.ini for tox4 changes.
Change-Id: I55cd9855d8232c007b012c19755c837e97823aea
We are going to release Tempest new tag 33.0.0 to declare
the end of support for stable/wallaby. stable/zed constraint
is latest stable constraints tested with current tempest hash
which is going to be used for 33.0.0 release.
Further information:
https://docs.openstack.org/tempest/latest/requirement_upper_constraint_for_tempest.html
Change-Id: Ia8536e783df945a273b8697b128a5115104d3c12
We remove install_command because the value set here is basically the
default but with one small difference. It doesn't run using `python -m
pip` which means we could potentially find a global pip install which
installs to the wrong python by default. Using python -m pip as the tox
default does should ensure it consistently finds the venv pip.
Additionally we stop allowlisting all external commands to make problems
like this more apparent.
Change-Id: I75a6bd7985fce66b3e8a4cb81320b22d71f6bc4a
To releas a new tag for Tempest Zed, we modified the
compatible constraint in tox.ini
- https://review.opendev.org/c/openstack/tempest/+/857602/2
As new tag is released now so we can switch back to the master
constraint in tox.ini.
Change-Id: I20f8d726d5b9adf9e5b7669c1b3ff8b28d46f0ac
We are going to release Tempest new tag to declare the start of
support for stable/zed. So that new tag use stable/zed
constraint in the tox env.
For example: if anyone use Tempest new tag (32.0.0) in future say
1 year later then tox env also should use the zed contstraint for
compatibility instead master.
These need to be move back to master constraint once release is done.
Change-Id: I188a23bf77855358535986256b17fcc4cc6405f9
We are going to release a new tempest tag 31.1.0 cut an
intermediate release during the Zed development cycle to
make bug fixes available for plugins and other consumers.
Change-Id: Iebd6d6412537959e3e59bf899c3670601733cef3
We are going to release Tempest new tag 31.0.0 to declare
the end of support for stable/ussuri. Pinning constraints
in tox.ini with stable/yoga constraint because Tempest is tested
with stable/yoga and master constraints. It did not use master constraints
as master dev cycle is in progress and so does their constraints also
is in progress and keep getting modified.
Further information:
https://docs.openstack.org/tempest/latest/requirement_upper_constraint_for_tempest.html
Change-Id: Ief33c3ebb527b81ae8fbd222237ad5605fad045a
To releas a new tag for Tempest Yoga, we modified the
compatible constraint in tox.ini
- https://review.opendev.org/c/openstack/tempest/+/832739/1
As new tag is released now so we can switch back to the master
constraint in tox.ini.
Change-Id: I4330b958970c28ed373e6385c8ac9b813c1257a8
We are going to release Tempest new tag to declare the start of
support for stable/yoga. So that new tag use stable/yoga
constraint in the tox env.
For example: if anyone use Tempest new tag (30.0.0) in future say
1 year later then tox env also should use the yoga contstraint for
compatibility instead master.
These need to be move back to master constraint once release is done.
By using master constraint in tox for all the old tag is problem and
we faced the issue of constraint incompatibility due to that and we
have to work around it all over the devstack, tempest role, grenade to
set the stable constraint cia env var.
Change-Id: Ie3159da1b9b73652f52cf22d87ab1dd0ae6411d9
As Tempest is branchless and needs to support older
supported stable branches, let's test it with all the
python version supported in those supported stable branches.
For that, we need to explicitly dd the py jobs instead of using
the template. This way, we will explicitly know that we are testing
all supported python versions when the new release template bumps the
min python version in the template.
In Yoga testing, we are making py3.9 job to run as voting, so
adding py3.9 job as voting.
Change-Id: I422da8da06509e529f4091e21cdc3d392d3cf910
This patch proposes to ignore hacking H405, "multi line docstring
summary not separated with an empty line". This arbitrarily enforces
"title" lines for docstring that are more than 1 line, meaning a
simple doctring that needs just 2 lines will either need to be
condensed into a single line, or expanded to include a "title" and a
longer explanation paragraph underneath. This does not appear to bring
much value.
Change-Id: I1978c73ab6eda1dce85c96b612c4c5bff286efc6
While releasing a new tag for Tempest, we modified the
constraint in tox.ini so that they are set to xena instead of
master - https://review.opendev.org/c/openstack/tempest/+/808031
Now the new tag is released and we can bring back the master
constraint in tox.ini so that Tempest master uses master constraint.
Change-Id: Iaf9eaaf60ea3c2d15b1225f713364ebede25768a
We are going to release Tempest a new tag to declare the start of
support for stable/xena so the new tag should use stable/xena
constraint in the tox env.
For example: if anyone uses Tempest 29.0.0 in the future, say 1 year
later, then tox env also should use the xena contstraint instead
master for the compatibility.
These changes need to be moved back to master constraint once the
release is done. Using master constraint in tox.ini for all the older
tags is a problem and we faced the issue of constraint
incompatibility. Therefore, we have to work around it all over the
devstack, tempest, grenade and set the stable constraints via the
env var.
Change-Id: I2bbf550294e7873eb07598b204ccc8d63ff6b5eb
API microversions are inhertied in nature from features points
of view, means higher microversion will have all the features/
changes done in lower microversion.
In Tempest we write the microversion tests by capping
the min and max microversion so that they can request
the API with correct microversion.
But for non microversion tests we do not test if they are
run-able for all the configured microversion in Tempest config
file. To test it at some extend this commit adds a experimental
job to run the API tests (compute and volume) with 'latest'
microversion to know if that run successfully or need modification.
This job is experimental as now as I expect lot of test failure
with the 'latest' microversion and as we keep fixing those and all
the tests pass then we can move it to voting job in check/gate pipeline.
Change-Id: Ie840dd0997519d7e11a661853f9376f43c6e7905
Rajesh Tailor
yatin