Added test documentation for Mistral fuel plugin.

Change-Id: Ife969ff0f2a1f48cfe8af7db2b504b657f412527
This commit is contained in:
Vitalii Yerys 2016-09-27 18:17:12 +03:00
parent c984e39886
commit de8400fa22
4 changed files with 471 additions and 0 deletions

View File

@ -0,0 +1,205 @@
=====================================================
Master test plan for Google cloud storage fuel plugin
=====================================================
1. Introduction
---------------
1.1 Purpose
###########
This document describes Master Test Plan for Mistral Fuel Plugin. The scope
of this plan defines the following objectives:
- describe testing activities;
- outline testing approach, test types, test cycles that will be used;
- test mission;
- deliverables;
1.2 Intended Audience
#####################
This document is intended for Mistral project team staff (QA and Dev
engineers and managers) all other persons who are interested in testing
results.
2. Governing Evaluation Mission
-------------------------------
Mistral plugin for Fuel provides the functionality to execute set of closely
related OpenStack API calls. It uses Fuel plugin architecture along
with pluggable architecture enhancements introduced in latest Mirantis
OpenStack Fuel.
The plugin must be compatible with the version 9.0 of Mirantis OpenStack.
2.1 Evaluation Test Mission
###########################
- Lab environment deployment.
- Deploy MOS with developed plugin installed.
- Create and run specific tests for plugin/deployment.
- Documentation
2.2 Test Items
##############
- Fuel Mistral plugin UI setup page and other UI scenarios will be tested
manually;
- Fuel CLI;
- Fuel API;
- Fuel UI;
- MOS;
- MOS API.
3. Test Approach
----------------
The project test approach consists of BVT, Functional, Destructive and
Acceptance test levels.
3.1 Criteria for test process starting
######################################
Before test process can be started it is needed to make some preparation
actions - to execute important preconditions. The following steps must be
executed successfully for starting test phase:
- all project requirements are reviewed and confirmed;
- implementation of testing features has finished (a new build is ready for
testing);
- implementation code is stored in GIT;
- bvt-tests are executed successfully (100% success);
- test environment is prepared with correct configuration;
- test environment contains the last delivered build for testing;
- test plan is ready and confirmed internally;
- implementation of manual tests and necessary autotests has finished.
3.2 Suspension Criteria
#######################
Testing of a particular feature is suspended if there is a blocking issue
which prevents tests execution. Blocking issue can be one of the following:
- Feature has a blocking defect, which prevents further usage of this feature
and there is no workaround available;
- CI test automation scripts failure.
3.3 Feature Testing Exit Criteria
#################################
Testing of a feature can be finished when:
- All planned tests (prepared before) for the feature are executed; no
defects are found during this run;
- All planned tests for the feature are executed; defects found during this
run are verified or confirmed to be acceptable (known issues);
- The time for testing of that feature according to the project plan has run
out and Project Manager confirms that no changes to the schedule are
possible.
4. Deliverables
---------------
4.1 List of deliverables
########################
Project testing activities are to be resulted in the following reporting
documents:
- Test plan;
- Test run report;
4.2 Acceptance criteria
#######################
90% of tests cases should be with status - passed. Critical and high issues
are fixed.
5. Test Cycle Structure
-----------------------
An ordinary test cycle for each iteration consists of the following steps:
- Smoke testing of each build ready for testing;
- Verification testing of each build ready for testing;
- Manual testing for functional and destructive tests;
- Creation of a new test case for covering of a new found bug (if such test
does not exist).
5.1.1 Smoke Testing
###################
Smoke testing is intended to check a correct work of a system after new
build delivery. Smoke tests allow to be sure that all main system
functions/features work correctly according to customer requirements.
5.1.2 Verification testing
##########################
Verification testing includes functional testing covering the following:
- new functionality (implemented in the current build);
- critical and major defect fixes (introduced in the current build).
5.1.3 Regression testing
########################
Regression testing includes execution of a set of test cases for features
implemented before current iteration to ensure that following modifications
of the system haven't introduced or uncovered software defects. It also
includes verification of minor defect fixes introduced in the current
iteration.
5.1.4 Bug coverage by new test case
###################################
Bug detection starts after all manual and automated tests are prepared and
test process initiated. Ideally, each bug must be clearly documented and
covered by test case. If a bug without a test coverage was found it must
be clearly documented and covered by custom test case to prevent occurrence
of this bug in future deployments/releases etc. All custom manual test
cases suppose to be added into TestRail and automated tests suppose to be
pushed to Git/Gerrit repo.
5.2 Metrics
###########
Test case metrics are aimed to estimate a quality of bug fixing; detect not
executed tests and schedule their execution. Passed / Failed test cases -
this metric shows results of test cases execution, especially, a ratio
between test cases passed successfully and failed ones. Such statistics must
be gathered after each delivered build test. This will help to identify a
progress in successful bugs fixing. Ideally, a count of failed test cases
should aim to a zero.
Not Run test cases - this metric shows a count of test cases which should be
run within a current test phase (have not run yet). Having such statistics,
there is an opportunity to detect and analyze a scope of not run test cases,
causes of their non execution and planning of their further execution
(detect time frames, responsible QA).
6. Test scope
-------------
.. include:: test_suite_smoke_bvt.rst
.. include:: test_suite_functional.rst
.. include:: test_suite_destructive.rst

View File

@ -0,0 +1,95 @@
===================
Destructive testing
===================
Verify master controller fail in HA cluster will not crash the system
----------------------------------------------------------------------
ID
##
mistral_controller_failover
Description
###########
Verify that after non-graceful shutoff of controller node, cluster stays
operational and after turning it back online, cluster is operational.
Complexity
##########
manual
Steps
#####
1. Create cluster
2. Install and configure Mistral plugin
3. Add 3 controller-mistral nodes
4. Add 1 compute and cinder nodes
5. Deploy cluster
6. Verify Cluster using OSTF
7. Power off main controller (non-gracefully)
8. Run OSTF
9. Power on controller which was powered off in step 6.
10. Run OSTF
Expected results
################
All steps except step 7 must be completed successfully, without any errors.
Step 7 one OSTF HA test will fail, because one of controllers is offline - this
is expected.
Verify mistral node fail in Non-HA cluster will not crush the system
--------------------------------------------------------------------
ID
##
mistral_node_failover
Description
###########
Verify that after non-graceful shutoff of mistral node cluster stays
operational and after turning it back online, cluster is operational.
Complexity
##########
manual
Steps
#####
1. Create cluster
2. Install and configure Mistral plugin
3. Add 1 controller-mistral, cinder and compute node
4. Add 1 mistral node
5. Deploy cluster
6. Run OSTF
7. Power off the mistral node (non-gracefully)
8. Run OSTF
9. Power on mistral node which was powered off in step 6
10. Run OSTF
Expected results
################
All steps except step 8 must be completed successfully, without any errors.
Step 8 one OSTF test will fail, because one of nodes is offline - this is
expected.

View File

@ -0,0 +1,91 @@
==================
Functional testing
==================
Check that Controller node can be deleted and added again
---------------------------------------------------------
ID
##
mistral_delete_add_controller
Description
###########
Verify that a controller-mistral node can be deleted and added after deploying
Complexity
##########
manual
Steps
#####
1. Create cluster
2. Enable and configure Mistral plugin
3. Add 3 controller-mistral nodes
4. Add 1 compute and 1 cinder node
5. Deploy cluster with plugin
6. Run OSTF tests
7. Delete a Controller-Mistral node and deploy changes
8. Run OSTF tests
9. Add a node with "Controller-Mistral" role and deploy changes
10. Run OSTF tests
Expected results
################
Step 6 will have failed cases because only 2 controller nodes are present in
cloud
Check that Compute node can be deleted and added again
------------------------------------------------------
ID
##
mistral_delete_add_mistral_node
Description
###########
Verify that a mistral node can be deleted and added after deploying
Complexity
##########
manual
Steps
#####
1. Create cluster
2. Enable and configure Mistral plugin
3. Add 1 controller, compute, cinder node
4. Add 1 mistral node
5. Deploy cluster with plugin
6. Run OSTF tests
7. Delete a mistral node and deploy changes
8. Run OSTF tests
9. Add a node with "mistral" role and deploy changes
10. Run OSTF tests
Expected results
################
All steps must be completed successfully, without any errors.

View File

@ -0,0 +1,80 @@
=========
BVT tests
=========
Smoke test
----------
ID
##
mistral_smoke
Description
###########
Smoke test for Mistral fuel plugin. Create cluster and install plugin. Deploy
cluster with controller-mistral, compute and cinder nodes
Complexity
##########
core
Steps
#####
1. Upload plugin to the master node
2. Create cluster
3. Install plugin
4. Add 1 nodes with controller-mistral role
5. Add 1 node with compute role
6. Add 1 node with cinder role
7. Deploy the cluster
Expected results
################
All steps must be completed successfully, without any errors.
BVT test
--------
ID
##
mistral_bvt
Description
###########
BVT test for Mistral fuel plugin. Deploy cluster in HA mode with
3 controllers-mistral and 2 compute nodes and install plugin.
Complexity
##########
core
Steps
#####
1. Upload plugin to the master node
2. Create cluster
3. Install plugin
4. Add 3 nodes with controller-mistral-ceph-osd role
5. Add 2 nodes with compute-ceph-osd role
6. Deploy the cluster
7. Run network verification
8. Run OSTF
Expected results
################
All steps must be completed successfully, without any errors.