From de8400fa22878d6864ae69ef58fbab92552ec08c Mon Sep 17 00:00:00 2001 From: Vitalii Yerys Date: Tue, 27 Sep 2016 18:17:12 +0300 Subject: [PATCH] Added test documentation for Mistral fuel plugin. Change-Id: Ife969ff0f2a1f48cfe8af7db2b504b657f412527 --- doc/testing/master_test_plan.rst | 205 +++++++++++++++++++++++++ doc/testing/test_suite_destructive.rst | 95 ++++++++++++ doc/testing/test_suite_functional.rst | 91 +++++++++++ doc/testing/test_suite_smoke_bvt.rst | 80 ++++++++++ 4 files changed, 471 insertions(+) create mode 100644 doc/testing/master_test_plan.rst create mode 100644 doc/testing/test_suite_destructive.rst create mode 100644 doc/testing/test_suite_functional.rst create mode 100644 doc/testing/test_suite_smoke_bvt.rst diff --git a/doc/testing/master_test_plan.rst b/doc/testing/master_test_plan.rst new file mode 100644 index 0000000..bbc8ad9 --- /dev/null +++ b/doc/testing/master_test_plan.rst @@ -0,0 +1,205 @@ +===================================================== +Master test plan for Google cloud storage fuel plugin +===================================================== + + +1. Introduction +--------------- + + +1.1 Purpose +########### + + This document describes Master Test Plan for Mistral Fuel Plugin. The scope + of this plan defines the following objectives: + + - describe testing activities; + - outline testing approach, test types, test cycles that will be used; + - test mission; + - deliverables; + + +1.2 Intended Audience +##################### + + This document is intended for Mistral project team staff (QA and Dev + engineers and managers) all other persons who are interested in testing + results. + + +2. Governing Evaluation Mission +------------------------------- + + +Mistral plugin for Fuel provides the functionality to execute set of closely +related OpenStack API calls. It uses Fuel plugin architecture along +with pluggable architecture enhancements introduced in latest Mirantis +OpenStack Fuel. +The plugin must be compatible with the version 9.0 of Mirantis OpenStack. + + +2.1 Evaluation Test Mission +########################### + + - Lab environment deployment. + - Deploy MOS with developed plugin installed. + - Create and run specific tests for plugin/deployment. + - Documentation + + +2.2 Test Items +############## + + - Fuel Mistral plugin UI setup page and other UI scenarios will be tested + manually; + - Fuel CLI; + - Fuel API; + - Fuel UI; + - MOS; + - MOS API. + + +3. Test Approach +---------------- + +The project test approach consists of BVT, Functional, Destructive and +Acceptance test levels. + + +3.1 Criteria for test process starting +###################################### + + Before test process can be started it is needed to make some preparation + actions - to execute important preconditions. The following steps must be + executed successfully for starting test phase: + + - all project requirements are reviewed and confirmed; + - implementation of testing features has finished (a new build is ready for + testing); + - implementation code is stored in GIT; + - bvt-tests are executed successfully (100% success); + - test environment is prepared with correct configuration; + - test environment contains the last delivered build for testing; + - test plan is ready and confirmed internally; + - implementation of manual tests and necessary autotests has finished. + + +3.2 Suspension Criteria +####################### + + Testing of a particular feature is suspended if there is a blocking issue + which prevents tests execution. Blocking issue can be one of the following: + + - Feature has a blocking defect, which prevents further usage of this feature + and there is no workaround available; + - CI test automation scripts failure. + + +3.3 Feature Testing Exit Criteria +################################# + + Testing of a feature can be finished when: + + - All planned tests (prepared before) for the feature are executed; no + defects are found during this run; + - All planned tests for the feature are executed; defects found during this + run are verified or confirmed to be acceptable (known issues); + - The time for testing of that feature according to the project plan has run + out and Project Manager confirms that no changes to the schedule are + possible. + + +4. Deliverables +--------------- + + +4.1 List of deliverables +######################## + + Project testing activities are to be resulted in the following reporting + documents: + + - Test plan; + - Test run report; + + +4.2 Acceptance criteria +####################### + + 90% of tests cases should be with status - passed. Critical and high issues + are fixed. + +5. Test Cycle Structure +----------------------- + +An ordinary test cycle for each iteration consists of the following steps: + + - Smoke testing of each build ready for testing; + - Verification testing of each build ready for testing; + - Manual testing for functional and destructive tests; + - Creation of a new test case for covering of a new found bug (if such test + does not exist). + + +5.1.1 Smoke Testing +################### + + Smoke testing is intended to check a correct work of a system after new + build delivery. Smoke tests allow to be sure that all main system + functions/features work correctly according to customer requirements. + + +5.1.2 Verification testing +########################## + + Verification testing includes functional testing covering the following: + + - new functionality (implemented in the current build); + - critical and major defect fixes (introduced in the current build). + + +5.1.3 Regression testing +######################## + + Regression testing includes execution of a set of test cases for features + implemented before current iteration to ensure that following modifications + of the system haven't introduced or uncovered software defects. It also + includes verification of minor defect fixes introduced in the current + iteration. + + +5.1.4 Bug coverage by new test case +################################### + + Bug detection starts after all manual and automated tests are prepared and + test process initiated. Ideally, each bug must be clearly documented and + covered by test case. If a bug without a test coverage was found it must + be clearly documented and covered by custom test case to prevent occurrence + of this bug in future deployments/releases etc. All custom manual test + cases suppose to be added into TestRail and automated tests suppose to be + pushed to Git/Gerrit repo. + + +5.2 Metrics +########### + + Test case metrics are aimed to estimate a quality of bug fixing; detect not + executed tests and schedule their execution. Passed / Failed test cases - + this metric shows results of test cases execution, especially, a ratio + between test cases passed successfully and failed ones. Such statistics must + be gathered after each delivered build test. This will help to identify a + progress in successful bugs fixing. Ideally, a count of failed test cases + should aim to a zero. + Not Run test cases - this metric shows a count of test cases which should be + run within a current test phase (have not run yet). Having such statistics, + there is an opportunity to detect and analyze a scope of not run test cases, + causes of their non execution and planning of their further execution + (detect time frames, responsible QA). + + +6. Test scope +------------- + +.. include:: test_suite_smoke_bvt.rst +.. include:: test_suite_functional.rst +.. include:: test_suite_destructive.rst diff --git a/doc/testing/test_suite_destructive.rst b/doc/testing/test_suite_destructive.rst new file mode 100644 index 0000000..b1b66d6 --- /dev/null +++ b/doc/testing/test_suite_destructive.rst @@ -0,0 +1,95 @@ +=================== +Destructive testing +=================== + + +Verify master controller fail in HA cluster will not crash the system +---------------------------------------------------------------------- + + +ID +## + +mistral_controller_failover + + +Description +########### + +Verify that after non-graceful shutoff of controller node, cluster stays +operational and after turning it back online, cluster is operational. + + +Complexity +########## + +manual + + +Steps +##### + + 1. Create cluster + 2. Install and configure Mistral plugin + 3. Add 3 controller-mistral nodes + 4. Add 1 compute and cinder nodes + 5. Deploy cluster + 6. Verify Cluster using OSTF + 7. Power off main controller (non-gracefully) + 8. Run OSTF + 9. Power on controller which was powered off in step 6. + 10. Run OSTF + + +Expected results +################ + +All steps except step 7 must be completed successfully, without any errors. +Step 7 one OSTF HA test will fail, because one of controllers is offline - this +is expected. + + +Verify mistral node fail in Non-HA cluster will not crush the system +-------------------------------------------------------------------- + + +ID +## + +mistral_node_failover + + +Description +########### + +Verify that after non-graceful shutoff of mistral node cluster stays +operational and after turning it back online, cluster is operational. + + +Complexity +########## + +manual + + +Steps +##### + + 1. Create cluster + 2. Install and configure Mistral plugin + 3. Add 1 controller-mistral, cinder and compute node + 4. Add 1 mistral node + 5. Deploy cluster + 6. Run OSTF + 7. Power off the mistral node (non-gracefully) + 8. Run OSTF + 9. Power on mistral node which was powered off in step 6 + 10. Run OSTF + + +Expected results +################ + +All steps except step 8 must be completed successfully, without any errors. +Step 8 one OSTF test will fail, because one of nodes is offline - this is +expected. diff --git a/doc/testing/test_suite_functional.rst b/doc/testing/test_suite_functional.rst new file mode 100644 index 0000000..0f23303 --- /dev/null +++ b/doc/testing/test_suite_functional.rst @@ -0,0 +1,91 @@ +================== +Functional testing +================== + + +Check that Controller node can be deleted and added again +--------------------------------------------------------- + + +ID +## + +mistral_delete_add_controller + + +Description +########### + +Verify that a controller-mistral node can be deleted and added after deploying + + +Complexity +########## + +manual + + +Steps +##### + + 1. Create cluster + 2. Enable and configure Mistral plugin + 3. Add 3 controller-mistral nodes + 4. Add 1 compute and 1 cinder node + 5. Deploy cluster with plugin + 6. Run OSTF tests + 7. Delete a Controller-Mistral node and deploy changes + 8. Run OSTF tests + 9. Add a node with "Controller-Mistral" role and deploy changes + 10. Run OSTF tests + + +Expected results +################ + +Step 6 will have failed cases because only 2 controller nodes are present in + cloud + + +Check that Compute node can be deleted and added again +------------------------------------------------------ + + +ID +## + +mistral_delete_add_mistral_node + + +Description +########### + +Verify that a mistral node can be deleted and added after deploying + + +Complexity +########## + +manual + + +Steps +##### + + 1. Create cluster + 2. Enable and configure Mistral plugin + 3. Add 1 controller, compute, cinder node + 4. Add 1 mistral node + 5. Deploy cluster with plugin + 6. Run OSTF tests + 7. Delete a mistral node and deploy changes + 8. Run OSTF tests + 9. Add a node with "mistral" role and deploy changes + 10. Run OSTF tests + + + +Expected results +################ + +All steps must be completed successfully, without any errors. diff --git a/doc/testing/test_suite_smoke_bvt.rst b/doc/testing/test_suite_smoke_bvt.rst new file mode 100644 index 0000000..47447fa --- /dev/null +++ b/doc/testing/test_suite_smoke_bvt.rst @@ -0,0 +1,80 @@ +========= +BVT tests +========= + + +Smoke test +---------- + + +ID +## + +mistral_smoke + +Description +########### + +Smoke test for Mistral fuel plugin. Create cluster and install plugin. Deploy +cluster with controller-mistral, compute and cinder nodes + +Complexity +########## + +core + +Steps +##### + + 1. Upload plugin to the master node + 2. Create cluster + 3. Install plugin + 4. Add 1 nodes with controller-mistral role + 5. Add 1 node with compute role + 6. Add 1 node with cinder role + 7. Deploy the cluster + +Expected results +################ + +All steps must be completed successfully, without any errors. + + + +BVT test +-------- + + +ID +## + + +mistral_bvt + +Description +########### + +BVT test for Mistral fuel plugin. Deploy cluster in HA mode with +3 controllers-mistral and 2 compute nodes and install plugin. + +Complexity +########## + +core + +Steps +##### + + 1. Upload plugin to the master node + 2. Create cluster + 3. Install plugin + 4. Add 3 nodes with controller-mistral-ceph-osd role + 5. Add 2 nodes with compute-ceph-osd role + 6. Deploy the cluster + 7. Run network verification + 8. Run OSTF + +Expected results +################ + +All steps must be completed successfully, without any errors.