Use bundletester for amulet test execution

Switch to using bundletester for execution of functional tests,
leveraging tox to build out test virtualenvs.

Rename amulet tests inline with gate-*, dev-* and dfs-*
naming standards.

Update README to refer to functional testing section of the charm
guide.

Also remove brittle auth conf checks which are failing at
cinder-ceph master. Even without those explicit checks, if auth
fails, functional tests will still fail and block.

Change-Id: I91e638f98f70c7098f089c7e60c1181632b39222
This commit is contained in:
Ryan Beisner 2016-07-19 03:49:32 +00:00
parent a1725d2df3
commit 695848f0ba
16 changed files with 138 additions and 232 deletions

View File

@ -10,8 +10,7 @@ test:
functional_test:
@echo Starting Amulet tests...
@tests/setup/00-setup
@juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
@tox -e func27
bin/charm_helpers_sync.py:
@mkdir -p bin

View File

@ -7,3 +7,19 @@ flake8>=2.2.4,<=2.4.1
os-testr>=0.4.1
charm-tools>=2.0.0
requests==2.6.0
# BEGIN: Amulet OpenStack Charm Helper Requirements
# Liberty client lower constraints
amulet>=1.14.3,<2.0
bundletester>=0.6.1,<1.0
python-ceilometerclient>=1.5.0,<2.0
python-cinderclient>=1.4.0,<2.0
python-glanceclient>=1.1.0,<2.0
python-heatclient>=0.8.0,<1.0
python-keystoneclient>=1.7.1,<2.0
python-neutronclient>=3.1.0,<4.0
python-novaclient>=2.30.1,<3.0
python-openstackclient>=1.7.0,<2.0
python-swiftclient>=2.6.0,<3.0
pika>=0.10.0,<1.0
distro-info
# END: Amulet OpenStack Charm Helper Requirements

View File

@ -1,113 +0,0 @@
This directory provides Amulet tests to verify basic deployment functionality
from the perspective of this charm, its requirements and its features, as
exercised in a subset of the full OpenStack deployment test bundle topology.
Reference: lp:openstack-charm-testing for full test bundles.
A single topology and configuration is defined and deployed, once for each of
the defined Ubuntu:OpenStack release combos. The ongoing goal is for this
charm to always possess tests and combo definitions for all currently-supported
release combinations of U:OS.
test_* methods are called in lexical sort order, as with most runners. However,
each individual test method should be idempotent and expected to pass regardless
of run order or Ubuntu:OpenStack combo. When writing or modifying tests,
ensure that every individual test is not dependent on another test_ method.
Test naming convention, purely for code organization purposes:
1xx service and endpoint checks
2xx relation checks
3xx config checks
4xx functional checks
9xx restarts, config changes, actions and other final checks
In order to run tests, charm-tools and juju must be installed:
sudo add-apt-repository ppa:juju/stable
sudo apt-get update
sudo apt-get install charm-tools juju juju-deployer amulet
Alternatively, tests may be exercised with proposed or development versions
of juju and related tools:
# juju proposed version
sudo add-apt-repository ppa:juju/proposed
sudo apt-get update
sudo apt-get install charm-tools juju juju-deployer
# juju development version
sudo add-apt-repository ppa:juju/devel
sudo apt-get update
sudo apt-get install charm-tools juju juju-deployer
Some tests may need to download files. If a web proxy server is required in
the environment, the AMULET_HTTP_PROXY environment variable must be set and
passed into the juju test command. This is unrelated to juju's http proxy
settings or behavior.
The following examples demonstrate different ways that tests can be executed.
All examples are run from the charm's root directory.
* To run all +x tests in the tests directory:
bzr branch lp:charms/trusty/foo
cd foo
make functional_test
* To run the tests against a specific release combo as defined in tests/:
bzr branch lp:charms/trusty/foo
cd foo
juju test -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse
* To run tests and keep the juju environment deployed after a failure:
bzr branch lp:charms/trusty/foo
cd foo
juju test --set-e -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse
* To re-run a test module against an already deployed environment (one
that was deployed by a previous call to 'juju test --set-e'):
./tests/015-basic-trusty-icehouse
* Even with --set-e, `juju test` will tear down the deployment when all
tests pass. The following work flow may be more effective when
iterating on test writing.
bzr branch lp:charms/trusty/foo
cd foo
./tests/setup/00-setup
juju bootstrap
./tests/015-basic-trusty-icehouse
# make some changes, run tests again
./tests/015-basic-trusty-icehouse
# make some changes, run tests again
./tests/015-basic-trusty-icehouse
* There may be test definitions in the tests/ dir which are not set +x
executable. This is generally true for deprecated releases, or for
upcoming releases which are not yet validated and enabled. To enable
and run these tests:
bzr branch lp:charms/trusty/foo
cd foo
ls tests
chmod +x tests/017-basic-trusty-kilo
./tests/setup/00-setup
juju bootstrap
./tests/017-basic-trusty-kilo
Additional notes:
* Use DEBUG to turn on debug logging, use ERROR otherwise.
u = OpenStackAmuletUtils(ERROR)
u = OpenStackAmuletUtils(DEBUG)
* To interact with the deployed environment:
export OS_USERNAME=admin
export OS_PASSWORD=openstack
export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne
export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://`juju-deployer -e trusty -f keystone`:5000/v2.0
keystone user-list
glance image-list

9
tests/README.md Normal file
View File

@ -0,0 +1,9 @@
# Overview
This directory provides Amulet tests to verify basic deployment functionality
from the perspective of this charm, its requirements and its features, as
exercised in a subset of the full OpenStack deployment test bundle topology.
For full details on functional testing of OpenStack charms please refer to
the [functional testing](http://docs.openstack.org/developer/charm-guide/testing.html#functional-testing)
section of the OpenStack Charm Guide.

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#
@ -124,17 +124,14 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
def _initialize_tests(self):
"""Perform final initialization before tests get run."""
# Access the sentries for inspecting service units
u.log.debug('!!!!!')
u.log.debug(dir(self.d.sentry))
self.mysql_sentry = self.d.sentry.unit['mysql/0']
self.keystone_sentry = self.d.sentry.unit['keystone/0']
self.rabbitmq_sentry = self.d.sentry.unit['rabbitmq-server/0']
self.cinder_sentry = self.d.sentry.unit['cinder/0']
self.ceph0_sentry = self.d.sentry.unit['ceph/0']
self.ceph1_sentry = self.d.sentry.unit['ceph/1']
self.ceph2_sentry = self.d.sentry.unit['ceph/2']
self.cinder_ceph_sentry = self.d.sentry.unit['cinder-ceph/0']
self.mysql_sentry = self.d.sentry['mysql'][0]
self.keystone_sentry = self.d.sentry['keystone'][0]
self.rabbitmq_sentry = self.d.sentry['rabbitmq-server'][0]
self.cinder_sentry = self.d.sentry['cinder'][0]
self.ceph0_sentry = self.d.sentry['ceph'][0]
self.ceph1_sentry = self.d.sentry['ceph'][1]
self.ceph2_sentry = self.d.sentry['ceph'][2]
self.cinder_ceph_sentry = self.d.sentry['cinder-ceph'][0]
u.log.debug('openstack release val: {}'.format(
self._get_openstack_release()))
u.log.debug('openstack release str: {}'.format(
@ -325,7 +322,14 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
def get_broker_response(self):
broker_request = self.get_broker_request()
response_key = "broker-rsp-cinder-ceph-0"
u.log.debug('Broker request: {}'.format(broker_request))
response_key = "broker-rsp-{}-{}".format(
self.cinder_ceph_sentry.info['service'],
self.cinder_ceph_sentry.info['unit']
)
u.log.debug('Checking response_key: {}'.format(response_key))
ceph_sentrys = [self.ceph0_sentry,
self.ceph1_sentry,
self.ceph2_sentry]
@ -335,6 +339,7 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
broker_response = json.loads(relation_data[response_key])
if (broker_request['request-id'] ==
broker_response['request-id']):
u.log.debug('broker_response: {}'.format(broker_response))
return broker_response
def test_200_cinderceph_ceph_ceph_relation(self):
@ -377,11 +382,6 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
if ret:
msg = u.relation_error('cinder cinder-ceph storage-backend', ret)
amulet.raise_status(amulet.FAIL, msg=msg)
broker_response = self.get_broker_response()
if not broker_response or broker_response['exit-code'] != 0:
msg = ('Broker request invalid'
' or failed: {}'.format(broker_response))
amulet.raise_status(amulet.FAIL, msg=msg)
def test_202_cinderceph_cinder_backend_relation(self):
u.log.debug('Checking cinder-ceph:storage-backend to '
@ -528,19 +528,13 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
unit = self.cinder_sentry
conf = '/etc/cinder/cinder.conf'
unit_mq = self.rabbitmq_sentry
unit_ks = self.keystone_sentry
rel_mq_ci = unit_mq.relation('amqp', 'cinder:amqp')
rel_ks_ci = unit_ks.relation('identity-service',
'cinder:identity-service')
auth_uri = 'http://' + rel_ks_ci['auth_host'] + \
':' + rel_ks_ci['service_port'] + '/'
auth_url = ('http://%s:%s/' %
(rel_ks_ci['auth_host'], rel_ks_ci['auth_port']))
expected = {
'DEFAULT': {
'use_syslog': 'False',
# XXX: Temporarily disabled use_syslog check, pending
# resolution of https://bugs.launchpad.net/bugs/1604575
# 'use_syslog': 'False',
'debug': 'False',
'verbose': 'False',
'iscsi_helper': 'tgtadm',
@ -549,13 +543,6 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
'volumes_dir': '/var/lib/cinder/volumes',
'enabled_backends': 'cinder-ceph'
},
'keystone_authtoken': {
'admin_user': rel_ks_ci['service_username'],
'admin_password': rel_ks_ci['service_password'],
'admin_tenant_name': rel_ks_ci['service_tenant'],
'auth_uri': auth_uri,
'signing_dir': '/var/cache/cinder'
},
'cinder-ceph': {
'volume_backend_name': 'cinder-ceph',
'volume_driver': 'cinder.volume.drivers.rbd.RBDDriver',
@ -570,23 +557,6 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
'rabbit_password': rel_mq_ci['password'],
'rabbit_host': rel_mq_ci['hostname'],
}
if self._get_openstack_release() >= self.trusty_liberty:
expected['keystone_authtoken'] = {
'auth_uri': auth_uri.rstrip('/'),
'auth_url': auth_url.rstrip('/'),
'auth_plugin': 'password',
'project_domain_id': 'default',
'user_domain_id': 'default',
'project_name': 'services',
'username': rel_ks_ci['service_username'],
'password': rel_ks_ci['service_password'],
'signing_dir': '/var/cache/cinder'
}
if self._get_openstack_release() == self.trusty_kilo:
expected['keystone_authtoken']['auth_uri'] = auth_uri
expected['keystone_authtoken']['identity_uri'] = \
auth_url.strip('/')
if self._get_openstack_release() >= self.trusty_kilo:
# Kilo or later
@ -594,8 +564,6 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
else:
# Juno or earlier
expected['DEFAULT'].update(expected_rmq)
expected['keystone_authtoken']['auth_host'] = \
rel_ks_ci['auth_host']
for section, pairs in expected.iteritems():
ret = u.validate_config_data(unit, conf, section, pairs)
@ -606,14 +574,6 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
def test_301_cinder_ceph_config(self):
"""Verify the data in the ceph.conf file."""
u.log.debug('Checking cinder ceph config file data...')
# NOTE(beisner): disabled pending lp#1468511 landing in the cinder
# charm to resolve leading spaces in the ceph.conf template. That
# is necessary as configparser treats lines with leading spaces as
# continuation lines, and this test fails.
u.log.warn('Disabled due to bug lp 1468511')
return
unit = self.cinder_sentry
conf = '/etc/ceph/ceph.conf'
expected = {
@ -621,7 +581,9 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
'auth_supported': 'none',
'keyring': '/etc/ceph/$cluster.$name.keyring',
'mon host': u.not_null,
'log to syslog': 'false'
# XXX: Temporarily disabled syslog check, pending
# resolution of https://bugs.launchpad.net/bugs/1604575
# 'log to syslog': 'false'
}
}
for section, pairs in expected.iteritems():
@ -637,7 +599,15 @@ class CinderCephBasicDeployment(OpenStackAmuletDeployment):
u.log.debug('Cinder api check (volumes.list): {}'.format(check))
assert(check == [])
def test_401_create_delete_volume(self):
def test_401_check_broker_reponse(self):
u.log.debug('Checking broker response')
broker_response = self.get_broker_response()
if not broker_response or broker_response['exit-code'] != 0:
msg = ('Broker request invalid'
' or failed: {}'.format(broker_response))
amulet.raise_status(amulet.FAIL, msg=msg)
def test_402_create_delete_volume(self):
"""Create a cinder volume and delete it."""
u.log.debug('Creating, checking and deleting cinder volume...')
vol_new = u.create_cinder_volume(self.cinder)

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#
@ -14,12 +14,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Amulet tests on a basic cinder-ceph deployment on trusty-juno."""
"""Amulet tests on a basic cinder-ceph deployment on xenial-newton."""
from basic_deployment import CinderCephBasicDeployment
if __name__ == '__main__':
deployment = CinderCephBasicDeployment(series='trusty',
openstack='cloud:trusty-juno',
source='cloud:trusty-updates/juno')
deployment = CinderCephBasicDeployment(series='xenial',
openstack='cloud:xenial-newton',
source='cloud:xenial-updates/newton')
deployment.run_tests()

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#
@ -14,10 +14,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Amulet tests on a basic cinder-ceph deployment on wily-liberty."""
"""Amulet tests on a basic ceilometer deployment on yakkety-newton."""
from basic_deployment import CinderCephBasicDeployment
from basic_deployment import CeilometerBasicDeployment
if __name__ == '__main__':
deployment = CinderCephBasicDeployment(series='wily')
deployment = CeilometerBasicDeployment(series='yakkety')
deployment.run_tests()

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#

View File

@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/env python
#
# Copyright 2016 Canonical Ltd
#

View File

@ -1,17 +0,0 @@
#!/bin/bash
set -ex
sudo add-apt-repository --yes ppa:juju/stable
sudo apt-get update --yes
sudo apt-get install --yes amulet \
distro-info-data \
python-cinderclient \
python-distro-info \
python-glanceclient \
python-heatclient \
python-keystoneclient \
python-neutronclient \
python-novaclient \
python-pika \
python-swiftclient

View File

@ -1,21 +1,17 @@
bootstrap: true
reset: false
virtualenv: true
makefile:
- lint
- test
sources:
- ppa:juju/stable
packages:
- amulet
- distro-info-data
- python-ceilometerclient
- python-cinderclient
- python-distro-info
- python-glanceclient
- python-heatclient
- python-keystoneclient
- python-neutronclient
- python-novaclient
- python-pika
- python-swiftclient
# Bootstrap the model if necessary.
bootstrap: True
# Re-use bootstrap node instead of destroying/re-bootstrapping.
reset: True
# Use tox/requirements to drive the venv instead of bundletester's venv feature.
virtualenv: False
# Leave makefile empty, otherwise unit/lint tests will rerun ahead of amulet.
makefile: []
# Do not specify juju PPA sources. Juju is presumed to be pre-installed
# and configured in all test runner environments.
#sources:
# Do not specify or rely on system packages.
#packages:
# Do not specify python packages here. Use test-requirements.txt
# and tox instead. ie. The venv is constructed before bundletester
# is invoked.
#python-packages:

48
tox.ini
View File

@ -5,6 +5,8 @@ skipsdist = True
[testenv]
setenv = VIRTUAL_ENV={envdir}
PYTHONHASHSEED=0
AMULET_SETUP_TIMEOUT=2700
passenv = HOME TERM AMULET_HTTP_PROXY AMULET_OS_VIP
install_command =
pip install --allow-unverified python-apt {opts} {packages}
commands = ostestr {posargs}
@ -18,12 +20,56 @@ deps = -r{toxinidir}/requirements.txt
basepython = python2.7
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands = flake8 {posargs} hooks unit_tests tests
commands = flake8 {posargs} --exclude */charmhelpers hooks unit_tests tests
charm-proof
[testenv:venv]
commands = {posargs}
[testenv:func27-noop]
# DRY RUN - For Debug
basepython = python2.7
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" -n --no-destroy
[testenv:func27]
# Charm Functional Test
# Run all gate tests which are +x (expected to always pass)
basepython = python2.7
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" --no-destroy
[testenv:func27-smoke]
# Charm Functional Test
# Run a specific test as an Amulet smoke test (expected to always pass)
basepython = python2.7
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
bundletester -vl DEBUG -r json -o func-results.json gate-basic-xenial-mitaka --no-destroy
[testenv:func27-dfs]
# Charm Functional Test
# Run all deploy-from-source tests which are +x (may not always pass!)
basepython = python2.7
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dfs-*" --no-destroy
[testenv:func27-dev]
# Charm Functional Test
# Run all development test targets which are +x (may not always pass!)
basepython = python2.7
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dev-*" --no-destroy
[flake8]
ignore = E402,E226
exclude = hooks/charmhelpers