fuel-noop-fixtures/doc/usage.rst

Typical use cases
=================

Let's discuss the most common use cases of the Noop test framework and how it
can be used.

Initial setup
-------------

In most cases you should setup the environment before using the Noop tests
framework. The setup consists of three parts:

- Fixtures repository clone
- Ruby gems installation
- Puppet modules download

There is a wrapper script **tests/noop/setup_and_diagnostics.sh** that will try
to do all these things. First, it will clone the fixtures repository unless
it have already been cloned, then it will run **tests/noop/noop_tests.sh**
with options **-b** and **-B** to create and use the bundle gems folder,
**-l** options will enable Puppet modules update and **-t** option will
initiate check procedures for paths and task library content.

If you are using *RVM* or are managing Ruby gems manually you are free to
bypass this stem and clone fixtures repository manually.

Running all tests using multiple processes
------------------------------------------

Running **tests/noop/noop_tests.sh** without any options will try to execute
all the spec tasks one by one. The list of the spec tasks will be generated by
combining all the possible combinations of specs, hiera files and facts files
that are allowed for this spec. Adding **-p** options allows you to review the
list of tasks that will be run without actually running them.

Running tasks one by one will be a very time consuming process, so you should
try to use multi-process run instead by providing the **-j** options with a
number of processes that can be started simultaneously. In this mode you
will not see the output of every RSpec process, but you will be able to get
the combined result report of all test runs at the end of the process.

You can also monitor the progress of tasks by using the debug option **-d**.
It will show you which tasks are starting and finishing and what shell commands
and environment variables are used to run them.::

  tests/noop/noop_tests.sh -j 24 -d

Will run all the spec task with debug output and keeping no more then 24
child processes all the time.::

  tests/noop/noop_tests.sh -p

Will output the list of tasks that is going to be run together with the facts
and yaml files used.

There is also the **run_all.sh** shortcut script for this action.

Running only a subset of tasks
------------------------------

In many cases you would want to run only a subset of tasks, up to only one task.
You can use filters to do so. By providing the **-s**, **-y** and **-f** will
allow you to set one or more specs, yams and facts that you want to use. The
list of tasks will be build by filtering out everything else that you have
not specified. Don't forget that you can use the **-p** option to review the
list of tasks before actually running them.

List options **-Y**, **-F**, **-S** and **-T** can be used to view the list of
all hiera yaml files, facts files, spec files and task files respectively.
These lists are very helpful for finding out correct values for the filter
options you want to use. Note, that using filter and list options together will
allow you to see the filtered lists.::

  tests/noop/noop_tests.sh -Y

Will list all available hiera yaml files.::

  tests/noop/noop_tests.sh -y neut_vlan.compute.ssl.yaml -p

Will show you which task are going to run with this yaml file.::

 tests/noop/noop_tests.sh -y neut_vlan.compute.ssl -s firewall/firewall -f ubuntu

Will run the *firewall/firewall* spec test with the provided yaml and facts
file, but it will only work if this combination is allowed for this spec.
Note, that you can either provide **.yaml**, **_spec.rb**, **.pp** extensions
for yaml and spec files, or you can omit them and they will be found out on
their own.::

  tests/noop/noop_tests.sh -y neut_vlan.compute.ssl,neut_vlan.compute.nossl -s firewall/firewall,netconfig/netconfig -p

Filters can use used with a list of elements or can be given as a regular
expression or a list of regular expressions::

  ./noop_tests.sh -p -s 'master/.*'

Will filter all tasks in the *master* group.::

  ./noop_tests.sh -p -s '^ceph.*,^heat.*,glance/db_spec'

Will filter all *ceph*, *heat* tasks and glance/db_spec task individually.::

  ./noop_tests.sh -p -s '^ceph.*' -y ceph

All *ceph* related tasks only on Hiera files which have Ceph enabled.

Recreating globals yaml files
-----------------------------

All globals files should already be precreated and commited to the fixtures
repository and there is no need for you to create them again in the most cases.
But, if you have made some changes to the existing yaml files or have
added a new one, you should create the globals yamls again.

You can do it by running *tests/noop/noop_tests.sh* with **-g** option.
It will set filters to run only the globals tasks as well as enabling the
option to save the generated yaml files. Using **-j** option will make the
process much faster.

There is also the **run_globals.sh** shortcut script for this action.

Spec file annotations
---------------------

The framework builds the list of tasks to run by combining all allowed facts
and yaml files for each spec file and creating a task for every combination.

By default, the list of yaml files, allowed for each spec will be determined
by the intersection of node roles the spec should be run on (obtained from the
*tasks.yaml* files used by the Nailgun to compile the deployment graph) and the
hiera file roles (obtained from the hiera files themselves). The facts file
will default to **ubuntu** value.

In most cases it would be better to manually specify the hiera files and,
possibly, the facts files for your spec file, because running a task for every
hiera file with the same roles would be redundant. On the other hand, if you
know which Hiera files can cause a different behaviour of your task, and you
want to test this behaviour in the different scenarios, you can explicitly
specify the list of yaml files and facts files you need.

The list of Hiera files can be set by using the **HIERA:** commented annotation
string followed by the list of hiera file names separated by the space
character.::

  # HIERA: neut_vlan.compute.ssl neut_vlan.compute.nossl

The list of facts files can be specified the same way using the **FACTS:**
annotation.::

  # FACTS: centos6 centos7

The list of task will contain this spec with all possible combinations of the
specified Hiera and facts files. If you need to enter only the precise list of
possible run combinations you can use the **RUN:** annotation.::

  # RUN: (hiera1) (facts1)
  # RUN: (hiera2) (facts2)

It can be specified many times an all entered combinations will be added to the
list.

You can use **ROLE** annotation to specify the list of node roles this
spec should be running at. It will find the list of Hiera yaml files
that have roles matching this list.::

  # ROLE: controller
  # ROLE: primary-controller

There is also a way to use the reverse logic. You can specify the Hiera
and facts yaml files that you want to exclude from the list instead of
providing the list of included files.::

  # SKIP_HIERA: neut_vlan.compute.ssl neut_vlan.compute.nossl
  # SKIP_FACTS: centos6

These yaml files will be excluded from the list of possible yaml files. If
you have used both include and exclude options, the exclude option will have
the priority over the include option. If there are no included Hiera files
the list of Hiera files will be generated from the node roles.

Note, that all these options can be combined all together. It will mean
to take all 'compute' yamls, add neut_vlan_l3ha.ceph.ceil-primary-controller
and remove 'neut_vlan.compute.ssl' and then add master/master_centos7 run.::

  # ROLE: compute
  # HIERA: neut_vlan_l3ha.ceph.ceil-primary-controller.yaml
  # RUN: master master_centos7
  # SKIP_HIERA: neut_vlan.compute.ssl.yaml

The final annotation **DISABLE_SPEC** allows you to temporarily disable the
spec from being seen the framework. It can use useful if you want to turn off
a spec with run problems and fix them later without breaking the tests.::

  # DISABLE_SPEC

The spec file with this annotation will be completely ignored.

Using hiera and facts overrides
-------------------------------

In some cases you need a special set of facts values or the Hiera data for
your task. If this values are very specific and are not useful for other tasks
you can use override system instead of creating the new Hiera or facts yaml.

There are *override* folders inside the Hiera and facts folders. If you place
a yaml file with the specific name to this folder, it will be used during the
spec catalog compilation as the top level of Hiera's hierarchy. The values which
are specified there will be used before the values in other yaml files. Hash
values will be merged on the basic values and the matching key will be
rewritten. Facts yamls work the same way by rewriting the basic values by the
values specified in the override file.

Both yaml files should be named after the task name with path separator changed
to the dash character. For example, the **firewall/firewall** task will use
the override file name *firewall-firewall.yaml* and
**openstack-controller/keystone** task will use the file name
*openstack-controller-keystone.yaml* if these files are found in the
override folders.

Working with report files
-------------------------

When the task manager runs the tasks they leave report files anf the manager
can collect them to generate a combined test report seen at the end of the test
process. These files can be found in the reports folder and re in json format.

You can use **-r** and **-R** options to load the saved reports from the
previous run and display the report again, or to load reports and run the tasks
that have previously failed after you have tried to somehow fix them.

You can use option **-o** to filter out only failed tasks and examples from
the report and **-O** options to show only tasks without showing the individual
examples. These options can be used together to show only failed tasks.

The task manager can also generate a test report in *jUnit XML* format using
the **-x** options. It will be saves to the **report.xml** file in the *reports*
folder of the fixtures repository. This file can be used by many tools to
visualize the tests results, notably by the Jenkins CI.

Catalog debugging
-----------------

There are several features that can be helpful during writing the initial spec
for a task or when you are debugging a spec failure. Running tasks with **-a**
options will show the report text about which files are being used in this task
run and what files are found in Hiera and facts hierarchies.

Using **-A** option will output the entire compiled catalog in the Puppet DSL
format. You can review its content and resource parameters to either find
out what resources and classes are there or to see what values the parameters
and properties actually have after all the catalog logic is processed. It's
very helpful when you are debugging a strange task behaviour or writing a spec
file.

The framework can also gather and report information about *File* resources
that are being installed by Puppet. Using *--save_file_resources* options
will dave the list of files that would be installed by the catalog and
description about their source or template. Using *--puppet_binary_files*
option will enable additional RSpec matcher that will fail if there are
files and, especially, binary files being installed. These ones should be
delivered by fuel packages.

Data-driven catalog tests
-------------------------

Usually the spec files try to repeat the logic found in the tested manifests,
receive the same set of resources and their parameters and compare them to
the set of resources found in the compiled catalog. Then the matchers are used
to check if the catalog contains what is expected from it to contain.

While this method works well in most cases it requires a lot of work and
extensive expertise in the tasks' domain to write a correct and comprehensive
set of spec for a task catalog. Specs also cannot detect if there are several
new resources or properties that have not been described in the spec file.

Data-driven tests can offer an alternative way to ensure that there are
no unwanted changes in the tasks catalogs. The idea behind them is building
catalogs in human-readable format before and after the changes are made. Then
these files can be compared and everything that have been changes will become
visible.

Using the **-V** options will save the current catalog to the *catalogs*
folder. These generated catalogs can be useful when reviewing complex patches
with major changes to the modules or manifests. A diff for data changes may
help a developer/reviewer examine the catalog contents and check
that every resource or class are receiving the correct property values.

You can also use **-v** option to enable automatic catalog checks. It should be
done after you have generated the initial versions and made some changes.
Running the tests with this option enabled will generate the catalogs again and
compare them to the saved version. If there are differences the test will be
failed and you will be able to locate the failed tasks. Here is an example
workflow one may use to examine a complex patch for data layer changes (we
assume she is a cool ruby developer and use the rvm manager and bundler):

.. code-block:: console

  $ git clone https://github.com/openstack/fuel-library
  $ cd fuel-library
  $ rvm use ruby-2.1.3
  $ PUPPET_GEM_VERSION=3.4.0
  $ PUPPET_VERSION=3.4.0
  $ ./tests/noop/setup_and_diagnostics.sh -B
  $ ./deployment/remove_modules.sh && ./deployment/update_modules.sh
  $ ./tests/noop/noop_tests.sh -V -j10 -b -s swift
  $ git review -d $swift_die_hard_patch_id
  $ ./tests/noop/noop_tests.sh -v -j10 -b -s swift

At this point, the reviewer will get the data diffs proposed to the swift
modules and finish her thorough review. Note that the command
`./tests/noop/setup_and_diagnostics.sh -B` gets a clean state and sets things
up, while removing and updating_modules is required to make things go smooth.

Using external environment variables and custom paths
-----------------------------------------------------

There are a number of environment variables used by either the task manager or
by the specs themselves which can alter their behaviour and override the
default or calculated values.

Paths related:

- **SPEC_ROOT_DIR** Set the path to the root folder of the framework. Many
  other folders are found relative to this path.
- **SPEC_SPEC_DIR** The path to the folder with the spec files. You can change
  it but it should be at the *spec/hosts* from the root folder or the
  rpsec-puppet will break.
- **SPEC_MODULE_PATH** or **SPEC_MODULEPATH** Set the path to the modules
  library.
- **SPEC_TASK_DIR** Set the path to the task manifests folder.
- **SPEC_DEPLOYMENT_DIR** Set the path to the *deployment* directory. It's
  actually use only to find the scripts to update and reset modules.
- **WORKSPACE** This variable is passed by the Jenkins jobs or will default to
  the *workspece* folder. Currently used only to store the Ruby gems installed
  by the *bundler* if *RVM* is not used.
- **SPEC_FACTS_DIR** The path to the folder with facts yaml files.
- **SPEC_HIERA_DIR** or **SPEC_YAML_DIR** The path to the folder with Hiera
  yaml files.

Spec related:

- **SPEC_FACTS_NAME** Set the name of the facts file that will be used by the
  spec process.
  It's set when the task is being run.
- **SPEC_HIERA_NAME** or **SPEC_ASTUTE_FILE_NAME** Set the name of the Hiera
  yaml file that will be used by the spec process.
  It's set when the task is being run.
- **SPEC_FILE_NAME** Set the spec/manifest file name for the spec process to
  test.
  It's set when the task is being run and even can override the internal value.
- **SPEC_BUNDLE_EXEC** Use *bundle exec* to run the *rspec* command by the
  task object.
- **SPEC_UPDATE_GLOBALS** Save the generated globals files instead of just
  checking that globals task's catalog is compiling without and error.
- **SPEC_CATALOG_SHOW** Ask the spec to output the catalog contents.
- **SPEC_SHOW_STATUS** Ask the spec to output the status text.

Debug related:

- **SPEC_TASK_CONSOLE** Run the pry console in the manager process.
- **SPEC_RSPEC_CONSOLE** Run the pry console in the RSpec process.
- **SPEC_PUPPET_DEBUG** Enable debug output of the Puppet's catalog compilation.
  This variable is also used by many other rspec suites of the Mirantis
  Puppet modules outside of the Noop tests framework to output the
  additional debug information.
- **SPEC_TASK_DEBUG** Enable the debug output of the task and manager objects.
- **SPEC_DEBUG_LOG** This variable can the the debug log destination file.

Fixtures source related:

- **NOOP_FIXTURES_REPO_URL** Fixtures repository. Defaults to
  `https://github.com/openstack/fuel-noop-fixtures.git`
- **NOOP_FIXTURES_BRANCH** Fixtures branch. Defaults to `origin/master`
- **NOOP_FIXTURES_GERRIT_URL** Gerrit repository. Defaults to
  `https://review.openstack.org/openstack/fuel-noop-fixtures`
- **NOOP_FIXTURES_GERRIT_COMMIT** Gerrit commit ref that should be
  cherry-picked. Could contain multiple refs, space separated. Defaults to
  `none`

Many of this variables can be set by the Noop manager CLI options, or you can
always export them externally.