tap-as-a-service-tempest-plugin tests were moved to
neutron-tempest-plugin when tap-as-a-service was
revived from x/, so this tempest plugin can be
removed from generate-tempest-plugins-list.py
Change-Id: I0a796b9bd5206092d5aee583db89c4227c626ff8
Recently we are seeing a lot of job timeout(bug#2004780)
and we see many tests taking time and also number of tests
increasing over time. This is to prepare the list of extra tests
(here extra tests means the tests which are covered by the other
API or scenario tests) which we do not need to run in every
integrated jobs. Instead, we can run them in a separete job(s).
Currently I am adding admin (except keystone) and negative tests
in the 'extra tests' list but we can add more tests here which
we think are covered in some other tests.
As negative tests are important for interop, adding those extra
tests coverage for stable branch job also but running them in
periodic run only.
Related-Bug: #2004780
Change-Id: Id02221df0d6180519751c63e890851bd59fdafa0
We currently have cryptsetup and LUKS v1 tests, but we are missing LUKS
v2 tests.
Code paths for LUKS v1 and LUKS v2 are different, because Nova uses
libvirt to decode LUKS v1 volumes but os-brick to decode LUKS v2
volumes.
This means that not only the attach process is different, but also some
features such as extend volume, so one of them may work while the other
doesn't.
This patch adds LUKS v2 encryption tests for all backends except for RBD
volumes, because they are not connected to the host but directly through
libvirt, so they can only use LUKS v1.
Depends-On: Ia1411f11ec4bf44af6a42d5f96c8a0903846ed66
Depends-On: I351f1a7769c9f915e4cd280f05a8b8b87f40df84
Related-Bug: #1967157
Change-Id: I40c518c3a3e760867efd6b6338fa9c16945b28c4
*vmware-nsx-tempest-plugin,networking-l2gw-tempest-plugin
x/tap-as-a-service are removed from the list
*Updating comments for required projects
Change-Id: I8bec628a07a4e555a95d542c87fa76f6c464f1d9
from Tempest to DevStack as it tests DevStack side of things and
is useful for projects not using Tempest.
This is part 2 of 2.
The 1st part is DevStack-side, in Depends-On.
The script is left calling out to devstack because legacy (dsvm)
jobs rely on its presence.
Depends-On: Ie166730843f874b9c99e37244e460d7ad33b7eeb
Change-Id: I6fa17ae413f106453303c4882925573bd8e05029
If first image is not captured in saving state,
then we should skip the test
- test_create_second_image_when_first_image_is_being_saved
Test is for creating the another image is first one is in
saving state and if image creation operation is fast and it
gets created before second API request then this test will
fail on its assert.
To handle the test scope and this race condition, we need to
assert on second operation if image is in saving state.This
can be done via capturing the timeout exception from first image
creation and waiting for saving state.
Change-Id: Id59f7ebb223f968109917182e29267faa808a072
Closes-Bug: #1881592
tempest/manager.py has been deprecated for over 4 years already,
it's time to remove it.
Implements: blueprint tempest-scenario-manager-stable
Change-Id: Iea813d478a93a1638be597af48929cd50a5e6366
Following stestr's example where arguments such as --blacklist-file,
--black-regex and --whitelist-file are deprecated since its
3.1.0 release, let's do the change here as well in order to
get tempest consumers some time for the transition.
This change deprecates the following arguments and replaces them
by new ones which are functionally equivavelnt:
* --black-regex is replaced by --exclude-regex
* --blacklist-file is replaced by --exclude-list
* --whitelist-file is replaced by --include-list
For now, Tempest will accept both (new and old) arguments to make
the transition smoother for all consumers.
The patch also bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23
Change-Id: I3e09b31f63d2cd7ea41c48e62432bd3bc54fcf44
During migrating the testing to Ubutnu Focal,
we skipped few device detach tests due to bug#1882521
- https://review.opendev.org/#/c/734700/8
Now Nova bug#1882521 is fixed so let's unskip those tests.
Change-Id: I9e48d72fc1192a775671ca339e0339e8ec1afa17
Related-Bug: #1882521
Related-Bug: #1894804
As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).
Most of the Tempest jobs will be migrate automatically once devstack
base job start running on Focal(Depends-On).
Few job like tempest-full-multinode and plugin sanity job needs to override
the nodeset. multinode job needs to run on Bionic till stable/ussuri and
on focal from master(victoria) onwards.
stable jobs testing stable branch needs to keep running on their supported
distro version which is bionic from stein till ussuri.
Keep 'tempest-integrated-compute' to run on bionic until
https://bugs.launchpad.net/qemu/+bug/1894804 is fixed.
Depends-On: https://review.opendev.org/#/c/731207/
[1] https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.html
Change-Id: I51fbd32a60c931d404e59e975049f7b01ed900dc
tempest-tox-plugin-sanity-check job is failing due to missing virtualenv.
As the job is running on master only and we don't support python2 anymore,
this patch edits the tempest-plugin-sanity.sh script so that it uses
python3 venv module to create a venv.
Change-Id: I3f711ec97ad9006c715bb45db9d8dd9d951bb6a1
The test_create_second_image_when_first_image_is_being_saved
introduces race and might if the first image was created quickly.
Blacklisting this for now untli we decide what to do with this next.
Change-Id: I15df606fd163c75bf265d406771e4e946a1d38b9
Closes-Bug: #1881592
Now we have more documents for plugins on stable release,
support and their testing policy. Let's prepare a
index page for it will be easy for plugins maintainer.
Also renaming the stable support policy section to have a clear
name.
Adding redirect links for moved documents.
Change-Id: I17a9a6e0e88605a0ee5047678d6ed168c31396ec
UPPER_CONSTRAINTS_FILE is deprecatedi[1], set the newer
TOX_CONSTRAINTS_FILE as well when interacting with other repos for the
transition.
Update internal use but not the tox.ini file, since users calling it
might use the old variable.
[1] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
Change-Id: Iea09402cf918a6978120a341950643185c94ec85
Add all the scenario network tests which mainly involves network
and compute.
Also, fix an incorrect entry which most likely has no matches
(test_network_v6.py test_security_groups_basic_ops.py are different
files at the same level).
Change-Id: I936f263b4c1d96689d276ecfcf25ef1560d66ced
This commit adds x/glare and x/tap-as-a-service to the BLACKLIST to
avoid sanity-job failures like following.
Failed Plugins: x/glare, x/tap-as-a-service,
+ [[ -n x/glare, x/tap-as-a-service, ]]
+ exit 1
It looks the root cause is related to pg_config installation but not
sure. So, this commit just blacklists them as a workaround.
Error: pg_config executable not found.
Change-Id: I03c6f01593b8e414b5e111d0e0c971ae156da715
x/kingbird doens't seem to be compatible with tempest anymore
which is breaking our tempest-tox-plugin-sanity-check gate job.
Change-Id: Icdaff49bbc2ba0e59cd528d55cad6c35014b7ad0
Those tests are related to Nova-Cinder and Nova-Glance interactions
and there is no need to run them on e.g. Neutron gate.
So this patch adds both groups of API tests to be blacklisted in
integrated networking template.
Change-Id: I500b4514f64c89e55d946b5958098511318ea358
tempest-tox-plugin-sanity-check zuul gates are currently failing for
plugins that use psycopg2, since the release of psycopg2 2.8.4. Global
upper constraints have the version pinned to 2.8.3.
This change updates the sanity check to use the global constraints.
Change-Id: I8bacf2d23a005e43c79d93086559b6f8462fc1bf
This commit removes barbican-tempest-plugin from the blacklist. The
patch[1] got already merged. The plugin should work for the sanity now.
[1] https://review.opendev.org/#/c/634631/
Change-Id: I34d119c58afa6b3c728d1a540b4ea1df27d9c4d0
This commit adds x/group-based-policy plugin to the blacklist. It looks
this plugin doesn't support python3 yet.
Change-Id: I2f97f2245434d21b83c790efd37a7092ef1bfeb2
This commit retries http requests to generate a plugins list. We often
see internal server errors(500) in the gate. It makes sanity jobs fail.
So, it would be better to retry internally instead of 'recheck'. To
accomplish it, this uses urllib3 module.
NOTE: The default total number of retries[0] is 10. It should be enough
for us.
[0] https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry
Change-Id: Id3da3e2895fc2324d5a2db91f82b12ec5bb610f8
This commit makes two bashate rules E005 and E042 treat as error and
ignores E006. It is difficult to keep E006 rule because bash scripts
tend to be long naturally. Both E005 and E042 are good to be errors
since it should be safer.
These additions can be seen in devstack[0] too.
E005: file does not begin with #! or have a .sh prefix
E006: check for lines longer than 79 columns
E042: local declaration hides errors
[0] 0fc6b2c5a8/tox.ini (L35)
Change-Id: I56c017569aa8660648764502cce5adfe2e946a58
This commit removes x/ranger-tempest-plugin from BLACKLIST. The patch[0]
is trying to support the tempest-sanity job. So, when the patch is
merged, we can remove it from BLACKLIST.
[0] https://review.opendev.org/#/c/672585/
Change-Id: Ia7f0bdad9f854c0f6724537e1ca0234f2952aca8
Depends-On: https://review.opendev.org/672585/
This commit introduces bashate[0] which is a code style checker for bash
scripts. We already have some bash scripts in Tempest repo. So, it might
be good to check the style automatically.
Some scripts under the tools directory are also fixed due to the bashate
violation.
Change-Id: I07820f10387552f93a9d8891b58a8c7fcdb83046
Devstack IPv6 base job 'devstack-IPv6' setup the IPv6 setting
to deploy the services to listen on IPv6 address.
Tempest 'devstack-tempest-ipv6' job derived from 'devstack-IPv6'
job adds the playbooks to run the tests.
As part of Train community goal 'Support IPv6-Only Deployments',
we will expand the 'devstack-tempest-ipv6' job to do
IPv6-only deployments verification.
This commit define the new roles of 'ipv6-only-deployments-verification'
which will be invoked as part of run phase of 'devstack-tempest-ipv6' job.
This role will do IPv6-only setting and deployments verification via
tools/verify-ipv6-only-deployement.sh
IPv6 verification script can be extended further to perform more checks
and via project specific test case. Those tests will run as part of project
specific child jobs.
The 'devstack-tempest-ipv6' job will be used as parent for project specific
IPv6-only job. Those child job can extend the project specific IPv6 verification
by defining new playebook for post-run. That way the base verification
done in 'devstack-tempest-ipv6' will still run in addition to project specific
verificaiton and tests run.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- Child jobs derived from 'devstack-tempest-ipv6' will run the IPv6 related test case or
any further IPv6 deployment verification.
This commit also adds the new job 'tempest-ipv6-only' which will run smoke
and ipv6 related tests present in Tempest. This job will be used to run
on 6 services (Nova, Neutron, Cinder, Keystone, Glance, Swift) deployed
by devstack.
Story: #2005477
Task: #35923
Change-Id: Ie1e5e5e02ee45ab7a4aae0f35bdb42afc2c3fe2e
This commit adds ranger-tempest-plugin to the blacklist. The plugin
looks like very new and it looks it doesn't work correctly yet. So, it
might be better to skip to sanity for the plugin now.
Change-Id: I6d5d647d92916e5b62e8c67094d04f028e25ee67
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-object-storage'
gate template for swift gate which will run only Swift, Cinder and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-object-storage template on swift gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I70851d41174462b83cc8c525cecdb1b3d65e4ec7
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-compute'
gate template for nova gate which will run only nova, neutron, cinder
and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-compute template on nova gate.
It will also help to disable the non-nova related serviecs and tests which
cannot be disabled from tempest-full job.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I807d4cac6214395df292a6955686ea44677c3481
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-placement'
gate template for placement gate which will run only nova and neutron tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-placement template on placement gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I200197e2fb57879aaeb382266d6309f0d1137b0c
elajkat