Merge pull request #2 from cdent/gabbi-tempest-path
Use a GABBI_TEMPEST_PATH to locate gabbits
This commit is contained in:
commit
1fe8486eec
|
@ -0,0 +1,23 @@
|
|||
# simple Makefile for some common tasks
|
||||
.PHONY: clean dist release pypi tagv
|
||||
|
||||
clean:
|
||||
find . -name "*.pyc" |xargs rm || true
|
||||
rm -r dist || true
|
||||
rm -r build || true
|
||||
rm -r .eggs || true
|
||||
rm -r gabbi_tempest.egg-info || true
|
||||
|
||||
tagv:
|
||||
git tag -s \
|
||||
-m `python -c 'import gabbi_tempest; print gabbi_tempest.__version__'` \
|
||||
`python -c 'import gabbi_tempest; print gabbi_tempest.__version__'`
|
||||
git push origin master --tags
|
||||
|
||||
dist:
|
||||
python setup.py sdist bdist_wheel
|
||||
|
||||
release: clean tagv pypi
|
||||
|
||||
pypi:
|
||||
python setup.py sdist bdist_wheel upload --sign
|
75
README.rst
75
README.rst
|
@ -2,41 +2,82 @@
|
|||
Gabbi + Tempest
|
||||
===============
|
||||
|
||||
This is an exploration of running gabbi_ as a tempest plugin. This
|
||||
code is based entirely on the work of Mehdi Abaakouk who made a
|
||||
tempest plugin for gnocchi_. This code models that but will try to
|
||||
be more generic, eventually.
|
||||
Gabbi-tempest is an experimental Tempest_ plugin_ that enables
|
||||
testing the APIs of running OpenStack services, integrated with
|
||||
tempest but without needing to write Python. Instead the YAML
|
||||
format_ provided by gabbi_ is used to write and evaluate HTTP
|
||||
requests and responses.
|
||||
|
||||
For the time being it works with Nova.
|
||||
Tests are placed in YAML files in one or more directories. Those
|
||||
directories are added to a ``GABBI_TEMPEST_PATH`` environment
|
||||
variable. When that variable is passed into a tempest test
|
||||
runner that is aware of the gabbi plugin, the files on that path
|
||||
will be used to create tempests tests.
|
||||
|
||||
To experiment with it you need a working tempest installation and
|
||||
configuration. I used devstack with::
|
||||
The test harness sets a series of enviornment variables that can
|
||||
be used in the YAML to reach the available services. The available
|
||||
variables may be extended in two ways:
|
||||
|
||||
* Adding them to the environment that calls tempest if the values are
|
||||
known.
|
||||
* Setting them in a subclass of the plugin if the values need to
|
||||
be calculated from what tempest knows.
|
||||
|
||||
For each service in the service catalog there are
|
||||
``<SERVICE_TYPE>_SERVICE`` and ``<SERVICE_TYPE>_BASE`` variables
|
||||
(e.g., ``PLACEMENT_SERVICE`` and ``PLACEMENT_BASE``). A useful
|
||||
``SERVICE_TOKEN``, ``IMAGE_REF``, ``FLAVOR_REF`` and ``FLAVOR_REF_ALT``
|
||||
are also available.
|
||||
|
||||
For the time being the ``SERVICE_TOKEN`` is ``admin``.
|
||||
|
||||
Trying It
|
||||
---------
|
||||
|
||||
To experiment with this you need a working tempest installation and
|
||||
configuration. One way to do that is to use devstack_ with the
|
||||
following added to the local.conf::
|
||||
|
||||
enable_service tempest
|
||||
INSTALL_TEMPEST=True
|
||||
|
||||
in local.conf.
|
||||
|
||||
Once tempest is confirmed to be working, make a clone of this repo,
|
||||
cd into it and do the equivalent of::
|
||||
Once tempest is confirmed to be working, gabbi-tempest must be
|
||||
installed. Either install it from PyPI::
|
||||
|
||||
pip install gabbi-tempest
|
||||
|
||||
Or make a clone of this repo_, cd into it, and do the equivalent of::
|
||||
|
||||
pip install -e .
|
||||
|
||||
If you are using virtualenvs or need sudo, your form will be
|
||||
different.
|
||||
|
||||
Create some gabbi_ tests that exercise the OpenStack services. There
|
||||
are sample files in the ``samples`` directory in the repo_.
|
||||
|
||||
Go to the tempest directory (often ``/opt/stack/tempest``) and run
|
||||
tempest limiting the test run to gabbi related tests::
|
||||
tempest as follows. Adding the ``regex`` will limit the test run
|
||||
to just gabbi related tests::
|
||||
|
||||
tempest run --regex gabbi
|
||||
|
||||
You can be more specific if you like::
|
||||
|
||||
tempest run --regex PlacementNovaGabbi
|
||||
GABBI_TEMPEST_PATH=/path/one:/path/two tempest run --regex gabbi
|
||||
|
||||
This will run the tests described by the YAML files in
|
||||
``gabbi_tempest/tests/scenario/gabbits/``. Edit those files and run
|
||||
the tempest command again for fun and adventure.
|
||||
``/path/one`` and ``/path/two``.
|
||||
|
||||
History
|
||||
-------
|
||||
|
||||
This code is based on the work of Mehdi Abaakouk who made a tempest
|
||||
plugin for gnocchi_ that worked with gabbi_. He figured out the
|
||||
details of the plugin structure.
|
||||
|
||||
.. _devstack: https://docs.openstack.org/devstack/latest/
|
||||
.. _Tempest: https://docs.openstack.org/tempest/latest/
|
||||
.. _plugin: https://docs.openstack.org/tempest/latest/plugin.html
|
||||
.. _gnocchi: https://review.openstack.org/#/c/301585/
|
||||
.. _gabbi: https://gabbi.readthedocs.org/
|
||||
.. _format: https://gabbi.readthedocs.io/en/latest/format.html
|
||||
.. _repo: https://github.com/cdent/gabbi-tempest
|
||||
|
|
|
@ -0,0 +1,14 @@
|
|||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
__version__ = '0.1.0'
|
|
@ -0,0 +1,11 @@
|
|||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
|
@ -22,27 +22,36 @@ import tempest.test
|
|||
CONF = config.CONF
|
||||
|
||||
class GenericGabbiTest(tempest.test.BaseTestCase):
|
||||
credentials = []
|
||||
service_name = None
|
||||
service_type = None
|
||||
|
||||
@classmethod
|
||||
def skip_checks(cls):
|
||||
service = cls.service_name
|
||||
super(GenericGabbiTest, cls).skip_checks()
|
||||
if not service:
|
||||
# FIXME(cdent): Hack to work around discoverability
|
||||
# weirdness
|
||||
raise cls.skipException('skipping the base class fake test')
|
||||
if not CONF.service_available.get(service):
|
||||
raise cls.skipException('%s support is required' % service)
|
||||
credentials = ['admin']
|
||||
|
||||
@classmethod
|
||||
def resource_setup(cls):
|
||||
super(GenericGabbiTest, cls).resource_setup()
|
||||
endpoints, token = cls._get_service_auth()
|
||||
|
||||
cls._set_environ()
|
||||
|
||||
# We support all enabled services, so use base hosts.
|
||||
host = 'stub'
|
||||
url = None
|
||||
|
||||
fallback_dir = os.path.join(os.getcwd(), 'gabbits')
|
||||
gabbi_paths = os.environ.get('GABBI_TEMPEST_PATH', fallback_dir)
|
||||
|
||||
top_suite = unittest.TestSuite()
|
||||
|
||||
for test_dir in gabbi_paths.split(':'):
|
||||
dotted_path = test_dir.replace(os.path.sep, '.')
|
||||
inner_suite = driver.build_tests(
|
||||
test_dir, unittest.TestLoader(), host=host, url=url,
|
||||
test_loader_name='tempest.scenario%s' % dotted_path)
|
||||
top_suite.addTest(inner_suite)
|
||||
|
||||
cls.tests = top_suite
|
||||
|
||||
@classmethod
|
||||
def _set_environ(cls):
|
||||
# Set test ENVIRON substitutions.
|
||||
endpoints, token = cls._get_service_auth()
|
||||
for service_type, url in endpoints.items():
|
||||
name = '%s_SERVICE' % service_type.upper()
|
||||
os.environ[name] = url
|
||||
|
@ -53,20 +62,6 @@ class GenericGabbiTest(tempest.test.BaseTestCase):
|
|||
os.environ['FLAVOR_REF'] = CONF.compute.flavor_ref
|
||||
os.environ['FLAVOR_REF_ALT'] = CONF.compute.flavor_ref_alt
|
||||
|
||||
if cls.service_type in endpoints:
|
||||
host = None
|
||||
url = endpoints[cls.service_type]
|
||||
else:
|
||||
host = 'stub'
|
||||
url = None
|
||||
|
||||
test_dir = os.path.join(os.path.dirname(__file__), 'gabbits',
|
||||
cls.service_type)
|
||||
cls.tests = driver.build_tests(
|
||||
test_dir, unittest.TestLoader(), host=host, url=url,
|
||||
test_loader_name='tempest.scenario.%s.%s' % (
|
||||
cls.__name__, cls.service_type))
|
||||
|
||||
@classmethod
|
||||
def clear_credentials(cls):
|
||||
# FIXME(sileht): We don't want the token to be invalided, but
|
||||
|
@ -104,21 +99,3 @@ class GenericGabbiTest(tempest.test.BaseTestCase):
|
|||
# NOTE(sileht): A fake test is needed to have the class loaded
|
||||
# by the test runner
|
||||
pass
|
||||
|
||||
|
||||
class NovaGabbiTest(GenericGabbiTest):
|
||||
credentials = ['admin']
|
||||
service_name = 'nova'
|
||||
service_type = 'compute'
|
||||
|
||||
|
||||
class GlanceGabbiTest(GenericGabbiTest):
|
||||
credentials = ['admin']
|
||||
service_name = 'glance'
|
||||
service_type = 'image'
|
||||
|
||||
|
||||
class PlacementNovaGabbiTest(GenericGabbiTest):
|
||||
credentials = ['admin']
|
||||
service_name = 'nova'
|
||||
service_type = 'multi'
|
||||
|
|
|
@ -1,76 +0,0 @@
|
|||
|
||||
defaults:
|
||||
request_headers:
|
||||
x-auth-token: $ENVIRON['SERVICE_TOKEN']
|
||||
|
||||
# NOTE(cdent): The use of complex JSONPath to determine URLs is bad
|
||||
# for readability.
|
||||
|
||||
tests:
|
||||
|
||||
- name: retrieve root
|
||||
GET: /
|
||||
response_json_paths:
|
||||
# $NETLOC contains the /v2.1 prefix
|
||||
$.version.links[?rel = "self"].href: /$ENVIRON['COMPUTE_SERVICE']/
|
||||
$.version.status: CURRENT
|
||||
|
||||
- name: retrieve empty servers
|
||||
GET: /servers
|
||||
response_json_paths:
|
||||
$.servers: []
|
||||
|
||||
- name: try bad accept
|
||||
desc: https://bugs.launchpad.net/nova/+bug/1567966
|
||||
xfail: True
|
||||
GET: /servers
|
||||
request_headers:
|
||||
accept: text/plain
|
||||
status: 406
|
||||
|
||||
- name: try bad method
|
||||
desc: https://bugs.launchpad.net/nova/+bug/1567970
|
||||
xfail: True
|
||||
DELETE: /servers
|
||||
status: 405
|
||||
|
||||
- name: post bad content-type
|
||||
desc: https://bugs.launchpad.net/nova/+bug/1567977
|
||||
xfail: True
|
||||
POST: /servers
|
||||
request_headers:
|
||||
content-type: text/plain
|
||||
data: I want a server so badly
|
||||
status: 415
|
||||
|
||||
- name: create server
|
||||
POST: /servers
|
||||
request_headers:
|
||||
content-type: application/json
|
||||
data:
|
||||
server:
|
||||
name: new-server-one
|
||||
imageRef: $ENVIRON['IMAGE_REF']
|
||||
flavorRef: $ENVIRON['FLAVOR_REF']
|
||||
status: 202
|
||||
response_headers:
|
||||
location: //servers/[a-f0-9-]+/
|
||||
|
||||
- name: wait for the server to be done
|
||||
GET: $LOCATION
|
||||
request_headers:
|
||||
content-type: application/json
|
||||
poll:
|
||||
count: 10
|
||||
delay: .5
|
||||
response_json_paths:
|
||||
$.server.status: ACTIVE
|
||||
|
||||
# bookmark link, whatever it is, is busted. Goes to bad version
|
||||
# - name: get bookmark
|
||||
# GET: $RESPONSE['$.server.links[?rel = "bookmark"].href']
|
||||
|
||||
- name: get server
|
||||
GET: $RESPONSE['$.server.links[?rel = "self"].href']
|
||||
response_json_paths:
|
||||
$.server.name: new-server-one
|
|
@ -1,34 +0,0 @@
|
|||
|
||||
defaults:
|
||||
request_headers:
|
||||
x-auth-token: $ENVIRON['SERVICE_TOKEN']
|
||||
|
||||
# NOTE(cdent): The use of complex JSONPath to determine URLs is bad
|
||||
# for readability.
|
||||
|
||||
tests:
|
||||
|
||||
- name: retrieve root
|
||||
GET: /
|
||||
status: 300
|
||||
|
||||
- name: choose current api
|
||||
desc: apparently the /v2/ url is a 404, despite being right there in discovery
|
||||
xfail: true
|
||||
GET: $RESPONSE['$.versions[?status = "CURRENT"].links[?rel = "self"].href']
|
||||
|
||||
- name: get images
|
||||
GET: /v2/images
|
||||
|
||||
- name: get one image
|
||||
GET: $RESPONSE['$.images[0].self']
|
||||
response_json_paths:
|
||||
$.status: active
|
||||
$.schema: /v2/schemas/image
|
||||
|
||||
- name: get the schema
|
||||
GET: $RESPONSE['$.schema']
|
||||
response_json_paths:
|
||||
$.name: image
|
||||
|
||||
# TODO(cdent): add some images
|
|
@ -1 +1,3 @@
|
|||
pbr
|
||||
gabbi
|
||||
six
|
||||
|
|
|
@ -89,7 +89,9 @@ tests:
|
|||
response_json_paths:
|
||||
$.server.status: VERIFY_RESIZE
|
||||
|
||||
# expected to fail because we don't double this way anymore
|
||||
- name: check for double allocations
|
||||
xfail: True
|
||||
GET: *s1_alloc
|
||||
poll:
|
||||
count: 10
|
|
@ -15,8 +15,8 @@ classifier =
|
|||
Programming Language :: Python :: 2
|
||||
Programming Language :: Python :: 2.7
|
||||
Programming Language :: Python :: 3
|
||||
Programming Language :: Python :: 3.4
|
||||
Programming Language :: Python :: 3.5
|
||||
Programming Language :: Python :: 3.6
|
||||
Topic :: Software Development :: Testing
|
||||
|
||||
[files]
|
||||
|
@ -31,3 +31,6 @@ source-dir = docs/source
|
|||
[entry_points]
|
||||
tempest.test_plugins =
|
||||
gabbi = gabbi_tempest.plugin:GabbiTempestPlugin
|
||||
|
||||
[bdist_wheel]
|
||||
universal=1
|
||||
|
|
Loading…
Reference in New Issue