From a222562cd26325bed94957289982865aaa5c41ce Mon Sep 17 00:00:00 2001 From: Ryan Beisner Date: Mon, 16 May 2016 21:02:50 +0000 Subject: [PATCH] Use bundletester for amulet test execution Switch to using bundletester for execution of functional tests, leveraging tox to build out test virtualenvs. Rename amulet tests inline with gate-*, dev-* and dfs-* naming standards. Update README to refer to functional testing section of the charm guide. Change-Id: Ife820a20179d2cad5a8511f23420b38224e61e25 --- Makefile | 5 +- test-requirements.txt | 16 +++ tests/README | 113 ------------------ tests/README.md | 9 ++ tests/basic_deployment.py | 25 ++-- ...ic-trusty-juno => dev-basic-xenial-newton} | 10 +- ...-wily-liberty => dev-basic-yakkety-newton} | 6 +- ...e-icehouse => gate-basic-precise-icehouse} | 2 +- ...ty-icehouse => gate-basic-trusty-icehouse} | 2 +- ...sic-trusty-kilo => gate-basic-trusty-kilo} | 2 +- ...usty-liberty => gate-basic-trusty-liberty} | 2 +- ...trusty-mitaka => gate-basic-trusty-mitaka} | 2 +- ...xenial-mitaka => gate-basic-xenial-mitaka} | 2 +- tests/setup/00-setup | 18 --- tests/tests.yaml | 38 +++--- tox.ini | 48 +++++++- 16 files changed, 121 insertions(+), 179 deletions(-) delete mode 100644 tests/README create mode 100644 tests/README.md rename tests/{016-basic-trusty-juno => dev-basic-xenial-newton} (68%) rename tests/{020-basic-wily-liberty => dev-basic-yakkety-newton} (81%) rename tests/{014-basic-precise-icehouse => gate-basic-precise-icehouse} (97%) rename tests/{015-basic-trusty-icehouse => gate-basic-trusty-icehouse} (97%) rename tests/{017-basic-trusty-kilo => gate-basic-trusty-kilo} (97%) rename tests/{018-basic-trusty-liberty => gate-basic-trusty-liberty} (97%) rename tests/{019-basic-trusty-mitaka => gate-basic-trusty-mitaka} (97%) rename tests/{021-basic-xenial-mitaka => gate-basic-xenial-mitaka} (97%) delete mode 100755 tests/setup/00-setup diff --git a/Makefile b/Makefile index f14f2b1..d7ad3c2 100644 --- a/Makefile +++ b/Makefile @@ -9,9 +9,8 @@ test: @tox -e py27 functional_test: - @echo Starting Amulet tests... - @tests/setup/00-setup - @juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700 + @echo Starting functional tests... + @tox -e func27 bin/charm_helpers_sync.py: @mkdir -p bin diff --git a/test-requirements.txt b/test-requirements.txt index 4faf254..74baa12 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -7,3 +7,19 @@ flake8>=2.2.4,<=2.4.1 os-testr>=0.4.1 charm-tools>=2.0.0 requests==2.6.0 +# BEGIN: Amulet OpenStack Charm Helper Requirements +# Liberty client lower constraints +amulet>=1.14.3,<2.0 +bundletester>=0.6.1,<1.0 +python-ceilometerclient>=1.5.0,<2.0 +python-cinderclient>=1.4.0,<2.0 +python-glanceclient>=1.1.0,<2.0 +python-heatclient>=0.8.0,<1.0 +python-keystoneclient>=1.7.1,<2.0 +python-neutronclient>=3.1.0,<4.0 +python-novaclient>=2.30.1,<3.0 +python-openstackclient>=1.7.0,<2.0 +python-swiftclient>=2.6.0,<3.0 +pika>=0.10.0,<1.0 +distro-info +# END: Amulet OpenStack Charm Helper Requirements diff --git a/tests/README b/tests/README deleted file mode 100644 index 79c5b06..0000000 --- a/tests/README +++ /dev/null @@ -1,113 +0,0 @@ -This directory provides Amulet tests to verify basic deployment functionality -from the perspective of this charm, its requirements and its features, as -exercised in a subset of the full OpenStack deployment test bundle topology. - -Reference: lp:openstack-charm-testing for full test bundles. - -A single topology and configuration is defined and deployed, once for each of -the defined Ubuntu:OpenStack release combos. The ongoing goal is for this -charm to always possess tests and combo definitions for all currently-supported -release combinations of U:OS. - -test_* methods are called in lexical sort order, as with most runners. However, -each individual test method should be idempotent and expected to pass regardless -of run order or Ubuntu:OpenStack combo. When writing or modifying tests, -ensure that every individual test is not dependent on another test_ method. - -Test naming convention, purely for code organization purposes: - 1xx service and endpoint checks - 2xx relation checks - 3xx config checks - 4xx functional checks - 9xx restarts, config changes, actions and other final checks - -In order to run tests, charm-tools and juju must be installed: - sudo add-apt-repository ppa:juju/stable - sudo apt-get update - sudo apt-get install charm-tools juju juju-deployer amulet - -Alternatively, tests may be exercised with proposed or development versions -of juju and related tools: - - # juju proposed version - sudo add-apt-repository ppa:juju/proposed - sudo apt-get update - sudo apt-get install charm-tools juju juju-deployer - - # juju development version - sudo add-apt-repository ppa:juju/devel - sudo apt-get update - sudo apt-get install charm-tools juju juju-deployer - -Some tests may need to download files. If a web proxy server is required in -the environment, the AMULET_HTTP_PROXY environment variable must be set and -passed into the juju test command. This is unrelated to juju's http proxy -settings or behavior. - -The following examples demonstrate different ways that tests can be executed. -All examples are run from the charm's root directory. - - * To run all +x tests in the tests directory: - - bzr branch lp:charms/trusty/foo - cd foo - make functional_test - - * To run the tests against a specific release combo as defined in tests/: - - bzr branch lp:charms/trusty/foo - cd foo - juju test -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse - - * To run tests and keep the juju environment deployed after a failure: - - bzr branch lp:charms/trusty/foo - cd foo - juju test --set-e -v -p AMULET_HTTP_PROXY 015-basic-trusty-icehouse - - * To re-run a test module against an already deployed environment (one - that was deployed by a previous call to 'juju test --set-e'): - - ./tests/015-basic-trusty-icehouse - - * Even with --set-e, `juju test` will tear down the deployment when all - tests pass. The following work flow may be more effective when - iterating on test writing. - - bzr branch lp:charms/trusty/foo - cd foo - ./tests/setup/00-setup - juju bootstrap - ./tests/015-basic-trusty-icehouse - # make some changes, run tests again - ./tests/015-basic-trusty-icehouse - # make some changes, run tests again - ./tests/015-basic-trusty-icehouse - - * There may be test definitions in the tests/ dir which are not set +x - executable. This is generally true for deprecated releases, or for - upcoming releases which are not yet validated and enabled. To enable - and run these tests: - bzr branch lp:charms/trusty/foo - cd foo - ls tests - chmod +x tests/017-basic-trusty-kilo - ./tests/setup/00-setup - juju bootstrap - ./tests/017-basic-trusty-kilo - - -Additional notes: - - * Use DEBUG to turn on debug logging, use ERROR otherwise. - u = OpenStackAmuletUtils(ERROR) - u = OpenStackAmuletUtils(DEBUG) - - * To interact with the deployed environment: - export OS_USERNAME=admin - export OS_PASSWORD=openstack - export OS_TENANT_NAME=admin - export OS_REGION_NAME=RegionOne - export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://`juju-deployer -e trusty -f keystone`:5000/v2.0 - keystone user-list - glance image-list diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..046be7f --- /dev/null +++ b/tests/README.md @@ -0,0 +1,9 @@ +# Overview + +This directory provides Amulet tests to verify basic deployment functionality +from the perspective of this charm, its requirements and its features, as +exercised in a subset of the full OpenStack deployment test bundle topology. + +For full details on functional testing of OpenStack charms please refer to +the [functional testing](http://docs.openstack.org/developer/charm-guide/testing.html#functional-testing) +section of the OpenStack Charm Guide. diff --git a/tests/basic_deployment.py b/tests/basic_deployment.py index ac32008..03540d9 100644 --- a/tests/basic_deployment.py +++ b/tests/basic_deployment.py @@ -103,13 +103,13 @@ class CeiloAgentBasicDeployment(OpenStackAmuletDeployment): def _initialize_tests(self): """Perform final initialization before tests get run.""" # Access the sentries for inspecting service units - self.ceil_sentry = self.d.sentry.unit['ceilometer/0'] - self.ceil_agent_sentry = self.d.sentry.unit['ceilometer-agent/0'] - self.mysql_sentry = self.d.sentry.unit['mysql/0'] - self.keystone_sentry = self.d.sentry.unit['keystone/0'] - self.rabbitmq_sentry = self.d.sentry.unit['rabbitmq-server/0'] - self.mongodb_sentry = self.d.sentry.unit['mongodb/0'] - self.nova_sentry = self.d.sentry.unit['nova-compute/0'] + self.ceil_sentry = self.d.sentry['ceilometer'][0] + self.ceil_agent_sentry = self.d.sentry['ceilometer-agent'][0] + self.mysql_sentry = self.d.sentry['mysql'][0] + self.keystone_sentry = self.d.sentry['keystone'][0] + self.rabbitmq_sentry = self.d.sentry['rabbitmq-server'][0] + self.mongodb_sentry = self.d.sentry['mongodb'][0] + self.nova_sentry = self.d.sentry['nova-compute'][0] u.log.debug('openstack release val: {}'.format( self._get_openstack_release())) u.log.debug('openstack release str: {}'.format( @@ -656,16 +656,23 @@ class CeiloAgentBasicDeployment(OpenStackAmuletDeployment): def test_910_pause_and_resume(self): """The services can be paused and resumed. """ u.log.debug('Checking pause and resume actions...') - unit_name = "ceilometer-agent/0" - unit = self.d.sentry.unit[unit_name] + unit = self.d.sentry['ceilometer-agent'][0] + unit_name = unit.info['unit_name'] + u.log.debug('Checking for active status on {}'.format(unit_name)) assert u.status_get(unit)[0] == "active" + u.log.debug('Running pause action on {}'.format(unit_name)) action_id = self._run_action(unit_name, "pause") + u.log.debug('Waiting on action {}'.format(action_id)) assert self._wait_on_action(action_id), "Pause action failed." + u.log.debug('Checking for maintenance status on {}'.format(unit_name)) assert u.status_get(unit)[0] == "maintenance" + u.log.debug('Running resume action on {}'.format(unit_name)) action_id = self._run_action(unit_name, "resume") + u.log.debug('Waiting on action {}'.format(action_id)) assert self._wait_on_action(action_id), "Resume action failed." + u.log.debug('Checking for active status on {}'.format(unit_name)) assert u.status_get(unit)[0] == "active" u.log.debug('OK') diff --git a/tests/016-basic-trusty-juno b/tests/dev-basic-xenial-newton similarity index 68% rename from tests/016-basic-trusty-juno rename to tests/dev-basic-xenial-newton index 1130b2f..a1732e4 100755 --- a/tests/016-basic-trusty-juno +++ b/tests/dev-basic-xenial-newton @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # @@ -14,12 +14,12 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Amulet tests on a basic ceilometer-agent deployment on trusty-juno.""" +"""Amulet tests on a basic ceilometer-agent deployment on xenial-newton.""" from basic_deployment import CeiloAgentBasicDeployment if __name__ == '__main__': - deployment = CeiloAgentBasicDeployment(series='trusty', - openstack='cloud:trusty-juno', - source='cloud:trusty-updates/juno') + deployment = CeiloAgentBasicDeployment(series='xenial', + openstack='cloud:xenial-newton', + source='cloud:xenial-updates/newton') deployment.run_tests() diff --git a/tests/020-basic-wily-liberty b/tests/dev-basic-yakkety-newton similarity index 81% rename from tests/020-basic-wily-liberty rename to tests/dev-basic-yakkety-newton index 4733143..1099697 100755 --- a/tests/020-basic-wily-liberty +++ b/tests/dev-basic-yakkety-newton @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # @@ -14,10 +14,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Amulet tests on a basic ceilometer-agent deployment on wily-liberty.""" +"""Amulet tests on a basic ceilometer-agent deployment on yakkety-newton.""" from basic_deployment import CeiloAgentBasicDeployment if __name__ == '__main__': - deployment = CeiloAgentBasicDeployment(series='wily') + deployment = CeiloAgentBasicDeployment(series='yakkety') deployment.run_tests() diff --git a/tests/014-basic-precise-icehouse b/tests/gate-basic-precise-icehouse similarity index 97% rename from tests/014-basic-precise-icehouse rename to tests/gate-basic-precise-icehouse index bc342e0..eedfb05 100755 --- a/tests/014-basic-precise-icehouse +++ b/tests/gate-basic-precise-icehouse @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # diff --git a/tests/015-basic-trusty-icehouse b/tests/gate-basic-trusty-icehouse similarity index 97% rename from tests/015-basic-trusty-icehouse rename to tests/gate-basic-trusty-icehouse index b4596be..4dfc5af 100755 --- a/tests/015-basic-trusty-icehouse +++ b/tests/gate-basic-trusty-icehouse @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # diff --git a/tests/017-basic-trusty-kilo b/tests/gate-basic-trusty-kilo similarity index 97% rename from tests/017-basic-trusty-kilo rename to tests/gate-basic-trusty-kilo index 824ac1b..61d632c 100755 --- a/tests/017-basic-trusty-kilo +++ b/tests/gate-basic-trusty-kilo @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # diff --git a/tests/018-basic-trusty-liberty b/tests/gate-basic-trusty-liberty similarity index 97% rename from tests/018-basic-trusty-liberty rename to tests/gate-basic-trusty-liberty index ce2c9fc..0a84608 100755 --- a/tests/018-basic-trusty-liberty +++ b/tests/gate-basic-trusty-liberty @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # diff --git a/tests/019-basic-trusty-mitaka b/tests/gate-basic-trusty-mitaka similarity index 97% rename from tests/019-basic-trusty-mitaka rename to tests/gate-basic-trusty-mitaka index 89afd2d..c1e96f5 100755 --- a/tests/019-basic-trusty-mitaka +++ b/tests/gate-basic-trusty-mitaka @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # diff --git a/tests/021-basic-xenial-mitaka b/tests/gate-basic-xenial-mitaka similarity index 97% rename from tests/021-basic-xenial-mitaka rename to tests/gate-basic-xenial-mitaka index 4e456a0..fed35f1 100755 --- a/tests/021-basic-xenial-mitaka +++ b/tests/gate-basic-xenial-mitaka @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/env python # # Copyright 2016 Canonical Ltd # diff --git a/tests/setup/00-setup b/tests/setup/00-setup deleted file mode 100755 index 658eb60..0000000 --- a/tests/setup/00-setup +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash - -set -ex - -sudo add-apt-repository --yes ppa:juju/stable -sudo apt-get update --yes -sudo apt-get install --yes amulet \ - distro-info-data \ - python-ceilometerclient \ - python-cinderclient \ - python-distro-info \ - python-glanceclient \ - python-heatclient \ - python-keystoneclient \ - python-neutronclient \ - python-novaclient \ - python-pika \ - python-swiftclient diff --git a/tests/tests.yaml b/tests/tests.yaml index 4d17631..e3185c6 100644 --- a/tests/tests.yaml +++ b/tests/tests.yaml @@ -1,21 +1,17 @@ -bootstrap: true -reset: false -virtualenv: true -makefile: - - lint - - test -sources: - - ppa:juju/stable -packages: - - amulet - - distro-info-data - - python-ceilometerclient - - python-cinderclient - - python-distro-info - - python-glanceclient - - python-heatclient - - python-keystoneclient - - python-neutronclient - - python-novaclient - - python-pika - - python-swiftclient +# Bootstrap the model if necessary. +bootstrap: True +# Re-use bootstrap node instead of destroying/re-bootstrapping. +reset: True +# Use tox/requirements to drive the venv instead of bundletester's venv feature. +virtualenv: False +# Leave makefile empty, otherwise unit/lint tests will rerun ahead of amulet. +makefile: [] +# Do not specify juju PPA sources. Juju is presumed to be pre-installed +# and configured in all test runner environments. +#sources: +# Do not specify or rely on system packages. +#packages: +# Do not specify python packages here. Use test-requirements.txt +# and tox instead. ie. The venv is constructed before bundletester +# is invoked. +#python-packages: diff --git a/tox.ini b/tox.ini index c051dba..b73c644 100644 --- a/tox.ini +++ b/tox.ini @@ -5,6 +5,8 @@ skipsdist = True [testenv] setenv = VIRTUAL_ENV={envdir} PYTHONHASHSEED=0 + AMULET_SETUP_TIMEOUT=2700 +passenv = HOME TERM AMULET_HTTP_PROXY AMULET_OS_VIP install_command = pip install --allow-unverified python-apt {opts} {packages} commands = ostestr {posargs} @@ -18,12 +20,56 @@ deps = -r{toxinidir}/requirements.txt basepython = python2.7 deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -commands = flake8 {posargs} hooks unit_tests tests actions +commands = flake8 {posargs} --exclude */charmhelpers hooks unit_tests tests actions charm-proof [testenv:venv] commands = {posargs} +[testenv:func27-noop] +# DRY RUN - For Debug +basepython = python2.7 +deps = -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt +commands = + bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" -n --no-destroy + +[testenv:func27] +# Charm Functional Test +# Run all gate tests which are +x (expected to always pass) +basepython = python2.7 +deps = -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt +commands = + bundletester -vl DEBUG -r json -o func-results.json --test-pattern "gate-*" --no-destroy + +[testenv:func27-smoke] +# Charm Functional Test +# Run a specific test as an Amulet smoke test (expected to always pass) +basepython = python2.7 +deps = -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt +commands = + bundletester -vl DEBUG -r json -o func-results.json gate-basic-xenial-mitaka --no-destroy + +[testenv:func27-dfs] +# Charm Functional Test +# Run all deploy-from-source tests which are +x (may not always pass!) +basepython = python2.7 +deps = -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt +commands = + bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dfs-*" --no-destroy + +[testenv:func27-dev] +# Charm Functional Test +# Run all development test targets which are +x (may not always pass!) +basepython = python2.7 +deps = -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt +commands = + bundletester -vl DEBUG -r json -o func-results.json --test-pattern "dev-*" --no-destroy + [flake8] ignore = E402,E226 exclude = hooks/charmhelpers