CI: Add back ARA logging

Up till now the ARA plugin has been pinned to a very old version,
and is no longer functional.

This installs a much newer version of ARA and adds a README file
to guide developers on how to view the sqlite file.

The ARA plugin is installed by default, but not activated. This
is intended to catch a small amount of regressions and
integration failures.

Developers can enable the plugin by adding the string `#ara`
to their commit message. This avoids extra load on the CI.

Change-Id: Id8328e374c9590b1363026fa2b2b24e191183987

doc/source/contributor/ci.rst

@@ -2,5 +2,29 @@
 Continuous Integration (CI) & Testing
 =====================================
 
-This page is a placeholder for information on the Kolla Ansible Continuous
-Integration (CI) & testing setup.
+Kolla-Ansible uses
+`Zuul <https://zuul.openstack.org/buildsets?project=openstack%2Fkolla-ansible&branch=master&pipeline=check>`__
+for continuous integration. Similar to testing performed using
+`devstack <https://docs.openstack.org/devstack/latest/>`__, Kolla-Ansible is
+capable of integrating and testing pre-merged dependencies from many other
+projects.
+
+Debugging with ARA in CI
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Frequently, the need arises to obtain more verbose ansible logging in CI.
+`ARA <https://ara.recordsansible.org/>`__ is an ansible plugin that collects a
+large amount of execution information and can render it into a browser
+friendly format.
+
+This plugin is not enabled by default because there is a per-task overhead.
+However, it's possible to trigger it when trying to debug a failing job.
+
+By adding the text `#ara` to the git commit message of the review, the CI jobs
+will enable the plugin and generate a sqlite database containing comprehensive
+logging. It's possible to render an HTML version of this by using
+`#ara_verbose`. Generating the HTML is not very efficient, however, and
+consumes a large amount of logging resources.
+
+Please note that git usually strips lines beginning with `#` from the commit
+message. This can be avoided by preceding the string with a space.

tests/ara-readme.md.j2

@@ -0,0 +1,39 @@
+Viewing ARA logs
+================
+
+What is ARA?
+------------
+
+[ARA](https://ara.recordsansible.org/) is an ansible plugin that collects a
+large amount of execution information and can render it into a browser
+friendly format.
+
+kolla-Ansible CI enabled the plugin and logged the sqlite output into the log
+directory next to this file.
+
+How do I view the output?
+-------------------------
+
+You need a local installation of ARA: following the
+[getting started guide](https://ara.readthedocs.io/en/latest/getting-started.html)
+should help you with installing the `ara-manage` command.
+
+For small scale operations, you have a choice between using:
+* [a small embedded server](https://ara.readthedocs.io/en/latest/cli.html#ara-manage-runserver), or
+* [rendering the html to static files](https://ara.readthedocs.io/en/latest/cli.html#ara-manage-generate)
+
+For example, rendering a particular log to static html pages:
+```
+# (Use the "Download all logs" script from the Artifacts tab with DOWNLOAD_DIR=~/zuul-logs)
+python3 -m pip install --user "ara[server]=={{ ara_version.stdout.split(' ') | last }}"
+export ARA_DATABASE_NAME=~/zuul-logs/{{ inventory_hostname }}/ara-report/ansible.sqlite
+ara-manage generate ~/zuul-logs/{{ inventory_hostname }}/ara-report/ara-html
+```
+
+Can I get the CI to generate the HTML for me?
+---------------------------------------------
+
+Yes! If the commit message contains the string `#ara_verbose` then Zuul will
+render the html file and collect it in the logs. Beware: this adds hundreds of
+MiB to the log files. (Tip: the `#` shouldn't be the first character in the
+line)

tests/post.yml

@@ -52,25 +52,52 @@
 
 - hosts: primary
   environment:
-    PATH: "{{ ansible_env.HOME + '/.local/bin:' + ansible_env.PATH }}"
+    PATH: >-
+      {{ ansible_env.HOME + '/kolla-ansible-venv/bin:'
+      + ansible_env.PATH }}
+  vars:
+    ara_report_local_dir: "{{ zuul.executor.log_root }}/{{ inventory_hostname }}/ara-report"
+    kolla_ansible_local_src_dir: "{{ zuul.executor.work_root }}/src/{{ zuul.project.canonical_hostname }}/openstack/kolla-ansible"
   tasks:
     - name: check for existence of ara sqlite
       stat:
-        path: "{{ ansible_env.HOME }}/.ara/ansible.sqlite"
+        path: "{{ ansible_env.HOME }}/.ara/server/ansible.sqlite"
       register: ara_stat_result
 
     - block:
         - name: ensure ara-report folder existence
           file:
-            path: "{{ zuul.executor.log_root }}/{{ inventory_hostname }}/ara-report"
+            path: "{{ ara_report_local_dir }}"
             state: directory
           delegate_to: localhost
-          run_once: true
 
         - name: download ara sqlite
           synchronize:
-            src: "{{ ansible_env.HOME }}/.ara/ansible.sqlite"
-            dest: "{{ zuul.executor.log_root }}/{{ inventory_hostname }}/ara-report/"
+            src: "{{ ansible_env.HOME }}/.ara/server/ansible.sqlite"
+            dest: "{{ ara_report_local_dir }}/"
             mode: pull
 
+        - name: get ara version
+          command: "ara --version"
+          register: ara_version
+
+        - name: template ara readme
+          template:
+            src: "{{ kolla_ansible_local_src_dir }}/tests/ara-readme.md.j2"
+            dest: "{{ ara_report_local_dir }}/README.md"
+            mode: '0644'
+          delegate_to: localhost
+
+        - name: Generate HTML from ARA database
+          command: "ara-manage generate {{ ansible_env.HOME }}/ara-html"
+          when: (zuul.change_message | default('')) is search('#ara_verbose')
+
+        - name: Download ARA HTML
+          synchronize:
+            src: "{{ ansible_env.HOME }}/ara-html"
+            dest: "{{ ara_report_local_dir }}/"
+            mode: pull
+            rsync_opts:
+              - "--quiet"
+          when: (zuul.change_message | default('')) is search('#ara_verbose')
       when: ara_stat_result.stat.exists

tests/run.yml

@@ -260,7 +260,7 @@
           - "{{ kolla_ansible_src_dir }}"
           - "ansible-core{{ ansible_core_version_constraint }}"
           - "ansible{{ ansible_version_constraint }}"
-          - "ara<1.0.0"
+          - "ara[server]<1.7"
         virtualenv: "{{ kolla_ansible_venv_path }}"
 
     - name: install Ansible collections

tests/templates/ansible.cfg.j2

@@ -1,5 +1,8 @@
 [defaults]
+{% if (zuul.change_message | default('')) is search('#ara') %}
 callback_plugins = {{ ara_callback_plugins.stdout }}
+{% endif %}
+callbacks_enabled = default,profile_tasks,timer{{ ',ara_default' if (zuul.change_message | default('')) is search('#ara') else '' }}
 host_key_checking = False
 # Ensure that facts are referenced via ansible_facts.<fact>.
 inject_facts_as_vars = False