Commit Graph

44 Commits

Author SHA1 Message Date
Martin Kopec 450096d30a Add set-src-dest-host role
The role will find out the available hosts and set
compute.migration_source_host and compute.migration_dest_host
options in tempest.conf if tempest_set_src_dest_host is set to
True.

Change-Id: Ic01970b100a7bb30d9c78229204c7de67681108c
2024-02-24 22:35:29 +01:00
Ghanshyam Mann 671a931e54 Increase the default concurrency for tempest run
run-tempest role which run tempest test for all tempest
based job limit the concurrency (number of worker used to
run the tests) to number-of-cpu/2 (if number of cpu > 3)
which limit the jobs to use half of the available cpu to
run the tests. We are seeing a lot of timeout now a days
and increasing the test runner worker can possibily solve
the timeout issue.

This commit modifies the logic of default concurrency to
set as number-of-cpu -2 so that we will be utlizing the
available cpu in more efficient way.

Related-Bug: #2004780

Change-Id: I8beeedade3098ad346d3a307b94bc3b254cde90a
2023-07-21 11:59:42 -05:00
Lukáš Piwowarski bd6481d934 Fix default values for variables in run-tempest role
With the new version of tox the parsing of postargs value has changed.
Previously an empty string was discarded by tox if it appeared in the
postargs value in contrast with the latest version of tox which
leaves the empty string intact.

The behavior of the latest tox version makes some jobs fail as some
variables in the run-tempest role have an empty string as a default
value. This leads in some cases to the execution of an incorrect tempest
command, for example:

tempest run --regex '[.*\bsmoke\b.*]' '' --concurrency=2

The patch changes the default value for blacklist_option,
exclude_list_option and tempest_test_regex variables to None to prevent
an empty string from being passed to postargs variable in tox.

Change-Id: I006e33c5ea0753b68cf1feb8580cb029209af9a8
2023-01-31 14:14:45 +01:00
Ghanshyam Mann 51a5d4f234 Use UPPER_CONSTRAINTS_FILE for stable/wallaby testing
stable/wallaby is in EM state and Tempest is going to be
pinned for stable/wallaby which need to modify run-tempest
role to use the constraints set by the devstack.

Change-Id: I985c222a1a461dff10ff33716fd2263a1a89813a
2023-01-25 14:49:48 -06:00
Lukáš Piwowarski be19f21813 Remove references to 'all-plugin' tox environment
The 'all-plugin' tox environment was deprecated by this patch [1].
Instead of the 'all-plugin' it is recommended to use the 'all' tox
environment.

This patch removes any reference to 'all-plugin' tox environment and
updates the documentation so that the installation steps work with
the 'all' tox venv.

[1] https://review.opendev.org/c/openstack/tempest/+/543974

Related-Bug: 1862240
Change-Id: I0a86d9cf33bff6576a6337967d6b663ab7d9f4a6
2022-10-10 14:01:49 +02:00
Ghanshyam Mann d9b6d7ca1a Use UPPER_CONSTRAINTS_FILE for stable/victoria testing
stable/victoria is in EM state and Tempest is going to be
pinned for stable/victoria which need to modify run-tempest
role to use the constraints set by the devstack or any job.

Because of Tempest 26.1.0 and with stestr 3.0.1 (beacause of upper
constraints of stestr 3.0.1 in stable/ussuri ) which does not have
new args exclude-regex so let's fallback to old arg if new arg is
passed.

Change-Id: I9cad0ccb02ee7bef0f94a1ee40e0fa117d89b5eb
2022-05-30 10:12:12 +00:00
Ghanshyam Mann 7aa3b21e89 Use UPPER_CONSTRAINTS_FILE for stable/ussuri testing
Tempest is going to be pinned for stable/ussuri(depends on) which
need to modify run-tempest role to use the constraints set by
the devstack or any job.

Because of Tempest 26.1.0 and with stestr 2.5.1 or 3.0.1 (beacause of upper
constraints of stestr 2.5.1 in stable/train and 3.0.1 in stable/ussuri )
which does not have new args exclude-regex so let's fallback to old arg if
new arg is passed.

Change-Id: I89ae6bd5137e57cb6ec5bc3c9edecc29489bca94
2022-05-28 23:53:16 +00:00
Ghanshyam Mann 5ff5dabd6e Fix stable/train testing with tempest 26.1.0
Tis issue came up before also during we cap the Tempest
version for stable branch tests(the one moved to EM state).

Tempest use the tox constraints file while creating the tempest
virtual env in devstack/lib/tempest, installing the tempest
plugins and running the tests like in run-tempest ansible roles.
If anywhere tox constraints file is used differently then venv
will be recreated and installed tempest plugins will be gone from
that recreated venv so tempest will not be able to run the plugins
tests and fail job.

Current failure: https://aca43ed1a01d95dea0ee-dcdb6cbb330bdac08ffee1284f86c919.ssl.cf2.rackcdn.com/822225/3/check/octavia-v2-dsvm-scenario/f60e4d3/job-output.txt

Earlier I tried to fix it in a generic way by fetching the devstack
localrc - https://review.opendev.org/c/openstack/tempest/+/774913
but this solution does not seems working as we can see it is skipped
in current stable/train testing with tempest 26.1.0

Adding stable/train explicitly in list for using the stable constraint
in run-tempest role also which will be in sync with what devstack/lib/tempest
use. And later I will debug why generic way is not working.

Because of Tempest 26.1.0 and with stestr 2.5.1 (beacause of upper
constraints of stestr 2.5.1 in stable/train) which does not have
new args exclude-regex so let's fallback to old arg if new arg is passed.

Depends-On: https://review.opendev.org/c/openstack/devstack/+/822380
Depends-On: https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/825708

Closes-Bug: #1955418
Change-Id: I105c839c38adfba1f9e22123ec3a99ffc0b5b72e
2022-01-25 17:03:31 +00:00
Ghanshyam Mann 617c84c7e5 Fix run-tempest-26 role for stable/stein
stable/stein uses old Tempest 26.0.0 to run the test
and that is why we use stable/stein upper-constraints
in the testing. But when any jobs with tempest plugins
try to run plugins tests, they fail.

This is because a new role run-tempest-26 (used for testing
with tempest 26.0.0) does not include the stable/stein in the
list of branches for whom we use stable constraints instead of
master constraints. and due to that run-tempest-26 finds constraints
mismatch and recreates the tempest virtual env. On recreation, venv
remove the previously installed tempest pluings and so does not run
their test.

Closes-Bug: #1938227
Change-Id: I00ac642fa602980d14612adc1ce65068d67db8ad
2021-07-27 16:24:53 -05:00
Ghanshyam Mann 1e4cb1da36 Use older run-tempest for stable branches
run-tempest is changed recently to add the new variables but
keep supporting the old ones too, for example:
tempest_black_regex, tempest_exclude_regex. and if both
old and new var are used in job definition (parent and child) then
new variables are picked. Because Tempest is branchless, zuul pick
the Tempest master playbooks/roles. That is why job running on stable
branch gate will pick the base job definition from Tempest master.
This way if any stable jobs which were defining the old var and using old
Tempest are broken if any of their parent job define the new var.

This commit pin the older run-tempest role for such stable branches.

Change-Id: If49ab0c31aca5b7837636727096a9bc83f891b1b
2021-07-14 10:38:16 -05:00
Radosław Piliszek 151cd7b9ac Move verify-ipv6-only-deployments
from Tempest to DevStack as it tests DevStack side of things and
is useful for projects not using Tempest.

This is part 2 of 2.
The 1st part is DevStack-side, in Depends-On.

The script is left calling out to devstack because legacy (dsvm)
jobs rely on its presence.

Depends-On: Ie166730843f874b9c99e37244e460d7ad33b7eeb
Change-Id: I6fa17ae413f106453303c4882925573bd8e05029
2021-06-10 07:44:44 +00:00
Ghanshyam Mann 926f823fbd Fix TEMPEST_VENV_UPPER_CONSTRAINTS default value comparision
TEMPEST_VENV_UPPER_CONSTRAINTS is default to 'master' in devstack
not 'default', fixing this in tempest role.

Change-Id: I5ccc379cc03d8c4d6bd7ebb914bbbc8ffb8ba34d
2021-04-07 16:52:41 -05:00
Ghanshyam Mann 62d11d8479 Use configured upper-constraints for non-master Tempest run
If any job use older Tempest version then we can not need
to use the master upper-constraints as they can be
incompatible most of the time. We need to use the compatible
constraints of what version of Tempest is configured to run.

Below devstack patch adding the new way to configure
the upper-constraints via new variable TEMPEST_VENV_UPPER_CONSTRAINTS
when Tempest master is not used.
- https://review.opendev.org/c/openstack/devstack/+/774711

We need to use the same variable to use the configured constraints
in run-tempest roles also so that when run-tempest recreate the
tempest venv then it honor the same configured constraints instead
if using master and error For example:
- https://zuul.opendev.org/t/openstack/build/854450dc514041a78fd250dde0eb138e/log/job-output.txt#31341

Depends-On: https://review.opendev.org/c/openstack/devstack/+/774716

Change-Id: I32d539b02366c8ae7ea70195674a97ed6a8ae759
2021-02-22 09:49:55 -06:00
Zuul 76f63460b3 Merge "Allow use custom uppper constraints in stable/stein" 2021-02-21 03:52:13 +00:00
Martin Kopec 8a8c85ddfc Fix run-tempest black/exclude var setting
Before [1] we didn't have conditionals for tempest_black_regex
being set or not, therefore it had a default value of empty
string set in the role's defaults. However after [1] we do
have such conditionals which make setting the default value
unnecessary - especially when we're asking if the var is
defined - it will be always defined.
Therefore this patch removes the default setting of
tempest_black_regex and tempest_exclude_regex.

The patch also removes 'Set tempest_exclude_regex' task
introduced by [1] as it appears unnecessary as well. This
task simply copied tempest_black_regex value to
tempest_exclude_regex. However there are 2 tasks below which
do the same logic plus more - they build CLI arguments for
tempest where depending on which ansible var is used the
appropriate CLI arg is chosen - tempest_black_regex is set,
--black-regex will be passed to tempest (for backward
compatibility) and if tempest_exclude_regex is set,
--exclude-regex will passed to tempest.

The patch creates a new temporary ansible var which will be used
only internally. The role will store in it the CLI argument
chosen dynamically which is passed to tempest later. Before we
reused the tempest_exclude_regex variable for that, however that
causes problems whent the var is set on a role level as set_fact
won't override such var.

[1] https://review.opendev.org/c/openstack/tempest/+/768583

Change-Id: I3e45741057e7a95b282b03bab4695f46c62d5271
2021-02-11 17:18:25 +00:00
Riccardo Pittau bd2acbfbc7 Allow use custom uppper constraints in stable/stein
Openstack Stein still supports Python 2.7 which means it is necessary to
set different upper constraints for that specific release for tempest
testing using that Python version.
The current master upper constraints won't work as they exclusively
consider Python version >= 3.x

Change-Id: Ieafd0ebc242eba2f4cbaf22823dbe94735236189
2021-02-04 11:23:34 +01:00
Martin Kopec dc84423b72 Inclusive jargon
Following stestr's example where arguments such as --blacklist-file,
--black-regex and --whitelist-file are deprecated since its
3.1.0 release, let's do the change here as well in order to
get tempest consumers some time for the transition.

This change deprecates the following arguments and replaces them
by new ones which are functionally equivavelnt:
* --black-regex is replaced by --exclude-regex
* --blacklist-file is replaced by --exclude-list
* --whitelist-file is replaced by --include-list

For now, Tempest will accept both (new and old) arguments to make
the transition smoother for all consumers.

The patch also bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23

Change-Id: I3e09b31f63d2cd7ea41c48e62432bd3bc54fcf44
2021-01-20 08:40:57 +00:00
Martin Kopec 05c35eb793 Improve tempest-cleanup ansible role
The patch implements a new flag which will fail a job when any
resources were leaked - that can be used for verification that
tests are cleaning their resources after they are finished.

Change-Id: I212cdce9713c80491153b0bbdb313e75c1f96f1d
2020-11-03 21:47:14 +00:00
Ghanshyam Mann b9bdeb4f5c Move process-stackviz role from Tempest to Devstack
process-stackviz role currently defined in Tempest and used
in tempest jobs. Issue with having it in branchless Tempest is that any
fix in that role cannot be backported to older Tempest. Also stackviz is
not Tempest things it is only used by Tempest jobs.

Stackviz can be considered as a service similar to Tempest so inatlling it in
Devstack side make sense. Also that will give us advantage to handle the
branch specific fixes or backpoting the fixes.

This can solve the current issues we are facing on rocky branch
- https://zuul.opendev.org/t/openstack/build/c1e2da80537448bfb24331fb4198a931/console#4/0/29/controller

Depends-On: https://review.opendev.org/#/c/737037/
Depends-On: https://review.opendev.org/#/c/737075/
Depends-On: https://review.opendev.org/#/c/737076/
Depends-On: https://review.opendev.org/#/c/737077/
Change-Id: Ide74cdf7967975fba9a3230007b35798d23e8bd2
2020-06-29 16:23:09 +00:00
Martin Kopec a857880423 Improve logging in tempest cleanup
* add additional handler so that INFO messages are printed to stdout
  besided tempest.log
* replace print commands by LOG.info calls
* when --debug argument used print DEBUG messages to stdout
* add additional debug messages to improve transparency of the cleanup
  execution
* use --debug argument within tempest-cleanup ansible role to see more
  details directly in the output

Change-Id: Iee72dc411fdf5cda8a8eed8dd7f349e0866648ab
2020-05-07 05:54:48 +00:00
Zuul 1dbaa8dff0 Merge "Add tempest-cleanup ansible role" 2020-05-07 01:56:51 +00:00
Zuul ccc41e6010 Merge "Use ensure-pip for stackviz pip install" 2020-05-06 02:25:44 +00:00
Martin Kopec f62dbc1c22 Add tempest-cleanup ansible role
The patch adds a new ansible role called tempest-cleanup which will
allow us to test tempest cleanup in the gate jobs.

Change-Id: I2cef2da6fee13e622da07b890da88850fe420152
2020-05-05 12:07:51 +00:00
Ian Wienand 08a40ad129 Use ensure-pip for stackviz pip install
Infra is trying to move to nodes that do not come with pip/virtualenv
pre-installed on the image; for details see the spec at [1].

zuul-jobs include ensure-pip [2] to provide pip dependencies; use that
before installing stackviz.  Note that if pip is installed this is a
noop.

[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/cleanup-test-node-python.html
[2] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#role-ensure-pip

Needed-By: https://review.opendev.org/723762
Change-Id: I24925ab8e885404da64a3ecfca3848041b6dfc05
2020-05-05 14:12:12 +10:00
Zuul b714ae3895 Merge "run-tempest: save the result of tempest execution" 2020-05-01 00:12:07 +00:00
Andreas Jaeger 99b085bdb9 Set TOX_CONSTRAINTS_FILE
UPPER_CONSTRAINTS_FILE is deprecatedi[1], set the newer
TOX_CONSTRAINTS_FILE as well when interacting with other repos for the
transition.

Update internal use but not the tox.ini file, since users calling it
might use the old variable.

[1] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file

Change-Id: Iea09402cf918a6978120a341950643185c94ec85
2020-04-22 15:01:21 +02:00
Luigi Toscano b774666aae run-tempest: save the result of tempest execution
This is useful when a job needs to execute additional tests
after tempest run, so ignoring the tempest run failures,
and combine the results afterwards.

Change-Id: I61fc5ebf69db47a035ae8294cfda4047d9f7e21a
2020-04-17 13:31:00 +02:00
Martin Kopec 4af65407b1 Update README for run-tempest role
Updates README in order to clear confusion about defining
multi-line regexes.

Closes-Bug: 1827234
Change-Id: Ie55ef879b1d6c971b202553b3906d02c29b981d6
2020-02-10 07:25:09 +00:00
Ghanshyam 56149024a0 Use stable constraint in run-tempest for stable branch testing
Stable branches till stable/rocky is using python
version <py3.6. Tempest test those branch in venv
but Tempest tox use the master upper-constraint[1]
which block installation due to dependencies
require >=py3.6. For exmaple, oslo.concurrency 4.0.0
is not compatible for <py3.6.

As we pin Tempest for EM stable brach, we should be
able to use stable constraint for Tempest installation
as well as while running during run-tempest playbook.

tox.ini is hard coded to use master constraint[1] which force
run-tempest to recreate the tox env and use the master constraint.

Devstack can set stable u-c to use via env var but this is
not enough as Tempest role run-tempest recreate the tox
env with master u-c.
- https://review.opendev.org/#/c/705089/4/lib/tempest

Currently we are fixing run-tempest to use stable constraint
till stable/rocky which use distro having <py3.6.

Ideally, we can change compatible branch constraint in tox.ini
while doing release tag for Tempest. But that can be done
for future tags only.

[1] bc9fe8eca8/tox.ini (L14)

Change-Id: I4268049ff91e5527e4b95fd8a70fa53ba27d56ed
2020-02-06 13:12:07 +00:00
Dmitry Tantsur 5952f85343 Allow overridding OS_TEST_TIMEOUT via the role variables
Change-Id: I7e464ba27704a0b4b13d6e0787b81750deee965e
2019-09-02 14:56:07 +02:00
Ghanshyam a0b8ca43e4 Setup framework for IPv6-only deployment verification
Devstack IPv6 base job 'devstack-IPv6' setup the IPv6 setting
to deploy the services to listen on IPv6 address.

Tempest 'devstack-tempest-ipv6' job derived from 'devstack-IPv6'
job adds the playbooks to run the tests.

As part of Train community goal 'Support IPv6-Only Deployments',
we will expand the 'devstack-tempest-ipv6' job to do
IPv6-only deployments verification.

This commit define the new roles of 'ipv6-only-deployments-verification'
which will be invoked as part of run phase of 'devstack-tempest-ipv6' job.
This role will do IPv6-only setting and deployments verification via
tools/verify-ipv6-only-deployement.sh

IPv6 verification script can be extended further to perform more checks
and via project specific test case. Those tests will run as part of project
specific child jobs.

The 'devstack-tempest-ipv6' job will be used as parent for project specific
IPv6-only job. Those child job can extend the project specific IPv6 verification
by defining new playebook for post-run. That way the base verification
done in 'devstack-tempest-ipv6' will still run in addition to project specific
verificaiton and tests run.

Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- Child jobs derived from 'devstack-tempest-ipv6' will run the IPv6 related test case or
  any further IPv6 deployment verification.

This commit also adds the new job 'tempest-ipv6-only' which will run smoke
and ipv6 related tests present in Tempest. This job will be used to run
on 6 services (Nova, Neutron, Cinder, Keystone, Glance, Swift) deployed
by devstack.

Story: #2005477
Task: #35923

Change-Id: Ie1e5e5e02ee45ab7a4aae0f35bdb42afc2c3fe2e
2019-07-29 07:37:33 +00:00
Sergey Vilgelm 2979bff949
Pass tox_extra_args to Run tempest task
Patrole needs access to global packages, but the 'all-plugin' is
deprecated[0] and 'all' defines a sitepackages option as False[1].
Adding tox_extra_args option makes it possible to configure Zuul and
pass --sitepackages argument to tox.

[0] 9b4c50cf63/tox.ini (L64)
[1] 9b4c50cf63/tox.ini (L7)

Change-Id: Ib3264f54075216471720483c8dd816887c1f413c
2018-11-06 15:33:04 -06:00
Chandan Kumar a9445043df Added tempest_black_regex to skip tempest tests
* https://review.openstack.org/547106 adds a black_regex flag to
  tempest run. We can use the same in devstack-tempest base job
  to skip/blacklist tests.

Change-Id: I9ed91b01ddb7ec0a3d23305031afbac7c4b16c55
2018-08-08 06:17:37 +00:00
Jens Harbott aacf6e89ac Do not fail on stackviz issues
The process-stackviz role only helps with analyzing runs, when it fails
it should not cause the whole job to suffer a POST_FAILURE.

Change-Id: Icfd6b2fc7adb42afb7900ea4458fa60cefc76fdb
Related-Bug: 1758054
2018-03-22 14:09:46 +00:00
Zuul 1525e67d7b Merge "Add support for blacklist file for Zuul jobs" 2018-03-07 17:47:31 +00:00
Michel Peterson ac140feca0 Use fetch-subunit-output role
Currently there is a zuul-jobs role only used in this tempest job while
other jobs mostly use fetch-subunit-output role.

This changes tries to consolidate towards the most used role, to be able
to clean up zuul-jobs a bit.

Change-Id: I404760ba34767660f8816f48712e13dccad67440
Depends-On: https://review.openstack.org/548406
2018-03-01 15:46:26 +02:00
Michel Peterson ef1a16d4a8 Add support for blacklist file for Zuul jobs
Currently there is tempest_test_regex that can be used to select or
ignore tests, however when the list is complex it can become complex to
maintain and define. For such complex scenarios is that the blacklist
file exist.

This patch adds support for the blacklist file in Zuul jobs.

Change-Id: Ib63955446ea59d81d0d882704666dcce4a625526
2018-02-26 15:17:58 +02:00
Andrea Frittoli (andreaf) b295335798 Fix the default value of stage_dir
The current default value of stage_dir does not match the valid
default from zuul-jobs, which causes stackviz to be skipped when
stage_dir is not overwritten, which is the case in all CI jobs.

Setting default and docs to "{{ ansible_user_dir }}".

Change-Id: I9fd3f8541c8e3fbc406c5e5c0c5fe8427bf74112
2018-01-30 08:37:28 +00:00
Andreas Jaeger ba2f479dc2 Rename tox_venvlist
For consistency with the tox jobs, use tox_envlist instead of
tox_venvlist. See
https://docs.openstack.org/infra/zuul-jobs/jobs.html#job-tox for
definition of tox job.

All projects using tox_venvlist have been changed to use tox_envlist as
well (see dependencies). Now we can rename tox_venvlist to tox_envlist -
and then the projects can remove tox_venvlist and use only tox_envlist.

Change-Id: I67a20b1faa77ff0166ad96fd95432c11948abc74
Depends-On: I035584aaeaa71b62c1d0c7e0a62dbf053a3488b6
Depends-On: I2fb02ead3b45c2a05d28f59fd28d062a3b40e07e
Depends-On: Id03d4c1351d5a002dc9d73247d0315bef274f627
Depends-On: Iff3093e61c7d2cb684667d5bc5f2a75580569aab
2018-01-12 16:49:47 +01:00
Michel Peterson 515edd4035 Document how to write multiline regex with comments
It is possible to add multiline and docummented regex to the
tempest_test_regex but it might not be evident at first, although very
useful for some scenarios.

This patch documents how this should be achieved to make it easier for
people in the future.

Change-Id: Id54679debedb1667cc70004b9a28fba921e50c0b
2017-12-25 14:40:32 +02:00
Zuul 9bd78bcd65 Merge "Add role to build the stackviz report" 2017-11-21 10:57:21 +00:00
Andrea Frittoli (andreaf) e4fdd2fece Add role to build the stackviz report
Add a role to build the stackviz report and include it in the
tempest post play.

Change-Id: I085460fc7a4fe6baee2d85c8ab3ea3a2128ea54d
2017-11-18 10:16:12 +00:00
Luigi Toscano e8a1f53986 run-tempest role: new tempest_test_regex variable
Specify a regular expression to filter the tests.

Change-Id: I44911c6bd6ede68c3376c8a6b42700310dce7a06
2017-11-16 15:22:07 +01:00
Andrea Frittoli (andreaf) 7d5445dae2 Initial skeleton for devstack-tempest base job
Setup the initial folder and play to run tempest.
This simply runs tempest full for now, with not support for config
options.

Change-Id: I5a76dd23900a1b5fb1764fafd837d69baf9ed8b1
Depends-on: Iffe54fbccbccd68db08f79a1b51dd7f76dbff408
2017-11-08 18:04:39 +11:00