Merge pull request #2 from cdent/gabbi-tempest-path

Use a GABBI_TEMPEST_PATH to locate gabbits
This commit is contained in:
Chris Dent 2018-01-09 17:20:58 +00:00 committed by GitHub
commit 1fe8486eec
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 138 additions and 175 deletions

23
Makefile Normal file
View File

@ -0,0 +1,23 @@
# simple Makefile for some common tasks
.PHONY: clean dist release pypi tagv
clean:
find . -name "*.pyc" |xargs rm || true
rm -r dist || true
rm -r build || true
rm -r .eggs || true
rm -r gabbi_tempest.egg-info || true
tagv:
git tag -s \
-m `python -c 'import gabbi_tempest; print gabbi_tempest.__version__'` \
`python -c 'import gabbi_tempest; print gabbi_tempest.__version__'`
git push origin master --tags
dist:
python setup.py sdist bdist_wheel
release: clean tagv pypi
pypi:
python setup.py sdist bdist_wheel upload --sign

View File

@ -2,41 +2,82 @@
Gabbi + Tempest
===============
This is an exploration of running gabbi_ as a tempest plugin. This
code is based entirely on the work of Mehdi Abaakouk who made a
tempest plugin for gnocchi_. This code models that but will try to
be more generic, eventually.
Gabbi-tempest is an experimental Tempest_ plugin_ that enables
testing the APIs of running OpenStack services, integrated with
tempest but without needing to write Python. Instead the YAML
format_ provided by gabbi_ is used to write and evaluate HTTP
requests and responses.
For the time being it works with Nova.
Tests are placed in YAML files in one or more directories. Those
directories are added to a ``GABBI_TEMPEST_PATH`` environment
variable. When that variable is passed into a tempest test
runner that is aware of the gabbi plugin, the files on that path
will be used to create tempests tests.
To experiment with it you need a working tempest installation and
configuration. I used devstack with::
The test harness sets a series of enviornment variables that can
be used in the YAML to reach the available services. The available
variables may be extended in two ways:
* Adding them to the environment that calls tempest if the values are
known.
* Setting them in a subclass of the plugin if the values need to
be calculated from what tempest knows.
For each service in the service catalog there are
``<SERVICE_TYPE>_SERVICE`` and ``<SERVICE_TYPE>_BASE`` variables
(e.g., ``PLACEMENT_SERVICE`` and ``PLACEMENT_BASE``). A useful
``SERVICE_TOKEN``, ``IMAGE_REF``, ``FLAVOR_REF`` and ``FLAVOR_REF_ALT``
are also available.
For the time being the ``SERVICE_TOKEN`` is ``admin``.
Trying It
---------
To experiment with this you need a working tempest installation and
configuration. One way to do that is to use devstack_ with the
following added to the local.conf::
enable_service tempest
INSTALL_TEMPEST=True
in local.conf.
Once tempest is confirmed to be working, make a clone of this repo,
cd into it and do the equivalent of::
Once tempest is confirmed to be working, gabbi-tempest must be
installed. Either install it from PyPI::
pip install gabbi-tempest
Or make a clone of this repo_, cd into it, and do the equivalent of::
pip install -e .
If you are using virtualenvs or need sudo, your form will be
different.
Create some gabbi_ tests that exercise the OpenStack services. There
are sample files in the ``samples`` directory in the repo_.
Go to the tempest directory (often ``/opt/stack/tempest``) and run
tempest limiting the test run to gabbi related tests::
tempest as follows. Adding the ``regex`` will limit the test run
to just gabbi related tests::
tempest run --regex gabbi
You can be more specific if you like::
tempest run --regex PlacementNovaGabbi
GABBI_TEMPEST_PATH=/path/one:/path/two tempest run --regex gabbi
This will run the tests described by the YAML files in
``gabbi_tempest/tests/scenario/gabbits/``. Edit those files and run
the tempest command again for fun and adventure.
``/path/one`` and ``/path/two``.
History
-------
This code is based on the work of Mehdi Abaakouk who made a tempest
plugin for gnocchi_ that worked with gabbi_. He figured out the
details of the plugin structure.
.. _devstack: https://docs.openstack.org/devstack/latest/
.. _Tempest: https://docs.openstack.org/tempest/latest/
.. _plugin: https://docs.openstack.org/tempest/latest/plugin.html
.. _gnocchi: https://review.openstack.org/#/c/301585/
.. _gabbi: https://gabbi.readthedocs.org/
.. _format: https://gabbi.readthedocs.io/en/latest/format.html
.. _repo: https://github.com/cdent/gabbi-tempest

View File

@ -0,0 +1,14 @@
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
__version__ = '0.1.0'

View File

@ -0,0 +1,11 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

View File

@ -22,27 +22,36 @@ import tempest.test
CONF = config.CONF
class GenericGabbiTest(tempest.test.BaseTestCase):
credentials = []
service_name = None
service_type = None
@classmethod
def skip_checks(cls):
service = cls.service_name
super(GenericGabbiTest, cls).skip_checks()
if not service:
# FIXME(cdent): Hack to work around discoverability
# weirdness
raise cls.skipException('skipping the base class fake test')
if not CONF.service_available.get(service):
raise cls.skipException('%s support is required' % service)
credentials = ['admin']
@classmethod
def resource_setup(cls):
super(GenericGabbiTest, cls).resource_setup()
endpoints, token = cls._get_service_auth()
cls._set_environ()
# We support all enabled services, so use base hosts.
host = 'stub'
url = None
fallback_dir = os.path.join(os.getcwd(), 'gabbits')
gabbi_paths = os.environ.get('GABBI_TEMPEST_PATH', fallback_dir)
top_suite = unittest.TestSuite()
for test_dir in gabbi_paths.split(':'):
dotted_path = test_dir.replace(os.path.sep, '.')
inner_suite = driver.build_tests(
test_dir, unittest.TestLoader(), host=host, url=url,
test_loader_name='tempest.scenario%s' % dotted_path)
top_suite.addTest(inner_suite)
cls.tests = top_suite
@classmethod
def _set_environ(cls):
# Set test ENVIRON substitutions.
endpoints, token = cls._get_service_auth()
for service_type, url in endpoints.items():
name = '%s_SERVICE' % service_type.upper()
os.environ[name] = url
@ -53,20 +62,6 @@ class GenericGabbiTest(tempest.test.BaseTestCase):
os.environ['FLAVOR_REF'] = CONF.compute.flavor_ref
os.environ['FLAVOR_REF_ALT'] = CONF.compute.flavor_ref_alt
if cls.service_type in endpoints:
host = None
url = endpoints[cls.service_type]
else:
host = 'stub'
url = None
test_dir = os.path.join(os.path.dirname(__file__), 'gabbits',
cls.service_type)
cls.tests = driver.build_tests(
test_dir, unittest.TestLoader(), host=host, url=url,
test_loader_name='tempest.scenario.%s.%s' % (
cls.__name__, cls.service_type))
@classmethod
def clear_credentials(cls):
# FIXME(sileht): We don't want the token to be invalided, but
@ -104,21 +99,3 @@ class GenericGabbiTest(tempest.test.BaseTestCase):
# NOTE(sileht): A fake test is needed to have the class loaded
# by the test runner
pass
class NovaGabbiTest(GenericGabbiTest):
credentials = ['admin']
service_name = 'nova'
service_type = 'compute'
class GlanceGabbiTest(GenericGabbiTest):
credentials = ['admin']
service_name = 'glance'
service_type = 'image'
class PlacementNovaGabbiTest(GenericGabbiTest):
credentials = ['admin']
service_name = 'nova'
service_type = 'multi'

View File

@ -1,76 +0,0 @@
defaults:
request_headers:
x-auth-token: $ENVIRON['SERVICE_TOKEN']
# NOTE(cdent): The use of complex JSONPath to determine URLs is bad
# for readability.
tests:
- name: retrieve root
GET: /
response_json_paths:
# $NETLOC contains the /v2.1 prefix
$.version.links[?rel = "self"].href: /$ENVIRON['COMPUTE_SERVICE']/
$.version.status: CURRENT
- name: retrieve empty servers
GET: /servers
response_json_paths:
$.servers: []
- name: try bad accept
desc: https://bugs.launchpad.net/nova/+bug/1567966
xfail: True
GET: /servers
request_headers:
accept: text/plain
status: 406
- name: try bad method
desc: https://bugs.launchpad.net/nova/+bug/1567970
xfail: True
DELETE: /servers
status: 405
- name: post bad content-type
desc: https://bugs.launchpad.net/nova/+bug/1567977
xfail: True
POST: /servers
request_headers:
content-type: text/plain
data: I want a server so badly
status: 415
- name: create server
POST: /servers
request_headers:
content-type: application/json
data:
server:
name: new-server-one
imageRef: $ENVIRON['IMAGE_REF']
flavorRef: $ENVIRON['FLAVOR_REF']
status: 202
response_headers:
location: //servers/[a-f0-9-]+/
- name: wait for the server to be done
GET: $LOCATION
request_headers:
content-type: application/json
poll:
count: 10
delay: .5
response_json_paths:
$.server.status: ACTIVE
# bookmark link, whatever it is, is busted. Goes to bad version
# - name: get bookmark
# GET: $RESPONSE['$.server.links[?rel = "bookmark"].href']
- name: get server
GET: $RESPONSE['$.server.links[?rel = "self"].href']
response_json_paths:
$.server.name: new-server-one

View File

@ -1,34 +0,0 @@
defaults:
request_headers:
x-auth-token: $ENVIRON['SERVICE_TOKEN']
# NOTE(cdent): The use of complex JSONPath to determine URLs is bad
# for readability.
tests:
- name: retrieve root
GET: /
status: 300
- name: choose current api
desc: apparently the /v2/ url is a 404, despite being right there in discovery
xfail: true
GET: $RESPONSE['$.versions[?status = "CURRENT"].links[?rel = "self"].href']
- name: get images
GET: /v2/images
- name: get one image
GET: $RESPONSE['$.images[0].self']
response_json_paths:
$.status: active
$.schema: /v2/schemas/image
- name: get the schema
GET: $RESPONSE['$.schema']
response_json_paths:
$.name: image
# TODO(cdent): add some images

View File

@ -1 +1,3 @@
pbr
gabbi
six

View File

@ -89,7 +89,9 @@ tests:
response_json_paths:
$.server.status: VERIFY_RESIZE
# expected to fail because we don't double this way anymore
- name: check for double allocations
xfail: True
GET: *s1_alloc
poll:
count: 10

View File

@ -15,8 +15,8 @@ classifier =
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
Topic :: Software Development :: Testing
[files]
@ -31,3 +31,6 @@ source-dir = docs/source
[entry_points]
tempest.test_plugins =
gabbi = gabbi_tempest.plugin:GabbiTempestPlugin
[bdist_wheel]
universal=1