It removes the existing the TQE collect-logs role in order to
test with the new opendev/ansible-role-collect-logs role.
https://tree.taiga.io/project/tripleo-ci-board/task/1001
Change-Id: Ib7892fca145a8c1947f54bfa8f7a35675e625e4d
Signed-off-by: Chandan Kumar <chkumar@redhat.com>
ARA reports are generated automatically in infra when
they are in ara-report directory. Don't generate htmls in upstream
jobs to save a space.
Change-Id: I7aa81ae3b06878baeab471e340477ca8a20f8594
Standalone deployment log is not indexed by logstash which
makes impossible to investigate errors related to them.
Change-Id: Ia70bf9850fab5a3b92332b4885e895ed98653596
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
* The file getting generated is tempest.log under /home/zuul/
tempest.log not tempest-output.log that's why it is not able
to indexed in logstash.
* And tempest_log_file var is used twice in validate-tempest role
and tempest.log is used at each place which also leads that
tempest_output.log was never found in ci logs.
Related-Bug:#1802971
Change-Id: I9bb9f8bdd0a17d2a1481356caaf186ed6348f6ba
Looks like we where using yum at fedora28 job at build-test-packages, we
need to generalize code at build-test-package so it works with fedora
too.
Also install-build-repo was trying to use yum.
Change-Id: I8cea39a9923e23c5f0fceb895a1efe4cb8ec395d
Story: https://tree.taiga.io/project/tripleo-ci-board/task/319?kanban-status=1447275
Use atop[1] tool to monitor the whole job process.
Atop generates binary output that could be downloaded
and then investigated locally.
Using atop -r /path/to/atop.bin you can read the file
and by pressing "t" to move 10 seconds futher or by
pressing "b" to jump to specific time in job and to see
what happened on host in this time. It allows to track
all resources in specific time.
It allows also tracking of containers separately.
For more info you can visit the site[1]
If atop installation fails it shouldn't fail the job,
so ignore_errors is added.
Currently it's for undercloud in OVB and all nodes in
multinode.
[1] https://www.atoptool.nl/
Change-Id: I7e17db3e376218f620a18db7ea7ca82d7578f618
Depends-On: Ibcdcfb4d8c5c94e1a06c7e635b0b6778ad318094
Playbooks, roles and logs for the overcloud deployment
now are in /var/lib/mistral. These files should be
captured by default.
Change-Id: I00f7de1d1f6a4ac1c8785b92c6edef10c95bc6cd
ara_overcloud_db_path was undefined in collect-logs role and
undercloud data was collected twice, ansible didn't alert
about undefined variable if it's in "environment"
Close-Bug: #1794238
Change-Id: I1d982a129337188a884e366cdc56a07637107e4b
Allow undercloud installations/upgrades from custom t-h-t, similarly to
overcloud_templates_* et al.
Unify and share custom t-h-t install script for OC and UC and mixed
update/upgrade cases via new shared role tht-prep-config.
To diverge undercloud from overcloud install/update/upgrade cases, use
the following shared mappings (vars on the right correspond to UC
cases):
* tht_templates_path: undercloud_templates_path (defaults to RPM dir)
* tht_templates_repo: undercloud_templates_repo
* tht_templates_branch: undercloud_templates_branch
* tht_templates_refspec: undercloud_templates_refspec
and a 2nd group (defaults set to do nothing, but only disable the
t-h-t prepare script's code blocks under control of those):
* composable_scenario: undercloud_composable_scenario
* upgrade_composable_scenario: undercloud_upgrade_composable_scenario
* prep_post_hook_script: undercloud_prep_post_hook_script
* download_templates_rpm: download_undercloud_templates_rpm
* tht_rpm_url: undercloud_tht_rpm_url
Those in group 2 may be customized to mimic the overcloud mixed
upgrades logic for underclouds/all-in-one for future use.
And for overclouds respectively, use mappings:
* composable_scenario: remains unchanged
* upgrade_composable_scenario: remains unchanged
* prep_post_hook_script: overcloud_prep_post_hook_script
* download_templates_rpm: download_overcloud_templates_rpm
* tht_rpm_url: remains unchanged
* tht_templates_path: overcloud_templates_path
...
Related-bug: #1691467
Closes-bug: #1781227
Change-Id: I4e0dcf24b93b8e4afb8359cb5108c4b7b5ba3d55
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
Add log files from upgrades to indexing into logstash
This will add all files with a .log extension/suffix
containing "upgrade" or "update" in the filename to
the list of files for logstash indexing.
As some of the logging in tripleo-upgrade contains
release names and is in flux, this should ensure that
future filenames (with future release names embedded)
as well as newly added logfiles (planned) are indexed.
Change-Id: Iea2f38112ad2ecfb0bc0dae149b3a12fdd937d5f
Add some simple ODL diagnostics to the collect task,
and add the ODL log directory to the list of collection
locaions. The directory may change to /var/log at some
point in the future.
Change-Id: I37405e41ec0f85249cef87c09c966cbe0f9baddf
I'm trying to chase down a metadata parsing issue and the version.json
file is in home/browbeat/metadata not home/browbeat/results since the
rest of the folder is just a few python scripts and other log files
I figured collecting the whole thing would be fine.
Change-Id: I36f7b10bcdb803c4c5396aec1a2561f3aee0ecdf
output_dir defaults to $HOME in tripleoclient,
and the heat templates deployed by installer are located
in output_dir/tripleo-heat-installer-templates.
Use these defaults as expected directory to collect
as CI artifacts.
Needed-By: Ic7b8c36804124463a1b0582e3aeeb3019729686b
Change-Id: I638a1f9dd58bbefdca592d7f740a224f4d3eac37
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
This commit adds the pengine/pe-input* logs to assist with
debugging CI jobs
Change-Id: I6fa7b951ff9c41fbb83a9bc8e5767524bfa364f8
Partial-Bug: #1695237
This patch adds /var/lib/kolla/config_files to collected logs to help
developers to debug Pike and newer versions of TripleO. /var/lib was sorted
alphanumerically for readability reasons.
Change-Id: I29f55d0717d0980d18cb7e6ee2941b376a97fac4
This review adds functionality to create a reproducer
script in the logs. The reproducer script will allow
users to recreate failing OVB and multinode jobs
in personal cloud tenants.
User documentation for the reproducer-quickstart script
is added.
Change-Id: I9fe8550a75c3ffb6d1271b01b1144bfbdc82c95d
This add the heat agent notifications, and ensure we don’t miss any
new agent (heat-config-docker-cmd).
It include the deployed/<hash>.json as it contains information about
the resource that triggered the script.
Change-Id: I20a73f1ca761873b6897f32c07eec377d3769a79
Send ARA statistics of particular tasks to Graphite server
Depends-On: Ie5324b3328c1516d5a0e6af263da61b1d8692b4b
Change-Id: I7167b62dada67403faf1f5171d6cddef419e8da2
There is some ongoing work to deploy the overcloud steps from Ansible.
This patch aims to add the support for testing the feature.
Parameters additions:
- tripleo_config_download_log`: Sets the TripleO config-download log file path.
- ansible_steps_log`: Sets the TripleO Ansible steps log file path.
- config_download_args`: Sets the arguments to load config-download environment
in THT.
- step_tripleo_config_download`: false/true - whether to enable config-download.
- deploy_steps_ansible`: false/true - whether to deploy the overcloud with
config-download Ansible steps.
Log collection (temporary, until we have a Mistral workflow):
- /home/*/inventory
- /home/*/tripleo-config-download/
Steps added to the overcloud-deploy:
- Create config-downlad environment
- Download TripleO configuration with tripleo-config-download
- Add subnodes to known hosts
- Deploy overcloud steps with Ansible
Other:
templates/config-download.yaml.j2 file is added to configure
OS::TripleO::DeployedServer::ControlPlanePort resource_registry and also
HostnameMap + DeployedServerPortMap.
Some of the tasks implemented here are temporary and will be removed
once we have a clean Mistral workflow.
Co-Authored-By: James Slagle <jslagle@redhat.com>
Co-Authored-By: Emilien Macchi <emilien@redhat.com>
Implements: blueprint ansible-config-download
Change-Id: I3a66e4a22d7bf15eddc5be1604af2b5737d49055
It could happen that a user might need to use specific options to the
sosreport command, which by default uses just "--batch".
For example one might want to add "--all" to get all available logs
regardless of size or use specific plugins related to OpenStack.
Change-Id: I5f523222ff1f684f8b020f3843714dd867435e04
Because of the use of the PWD environment variable, it's quite tricky to
reuse validate-tempest out of the tripleo-quickstart context. This patch
replaces the variable with local_working_dir, which is already widely
used for this purpose.
The patch also adjust collect-logs's artcl_full_artifact_url for the same
reason.
Change-Id: I3647952dd298df6cb5f18cea45d7a30b7268ec85
Save /var/log/bootstrap-subnodes.log in logstash file so we can build
queries when bootstrap fails in OpenStack Infra.
Change-Id: Iec01038ba5508826a0e164aaed9ddb940d4e7290
The logs for the repo-setup are not included in collect-logs
We need to be able to debug failures here.
Closes-Bug: #1712910
Change-Id: I24224218d56303ffd9fdc6372556f72b1b7ee019
* Since Identity V2 api is deprecated since Mitaka, So we can
switch to Identity V3 api from pike onwards. We donot want
to break Newton and Ocata jobs so keep v2 for them.
* Since mistral does not work with Identity v3 api so
keep v2 for mistral.
* Collect overcloudrc.v3 file.
Change-Id: If66fc4df8d16bbbae45ac5f4bb643c62fff0f21e
* Run tempest cleanup --init-saved-state before running the tempest
tests to save resources before tempest execution.
* Run tempest cleanup --dry-run to see what new resources got created
after tempest run.
* use tempest_track_resources: true in defaults/main.yml to track
tempest resources before and after tempest run
Change-Id: I9787950996055bb2ca6876b5f42048be55028675
Collect subunit results from pingtest. It could be parsed and
converted to any result format like tempest tests results.
It's helpful for displaying it in various platforms like Jenkins.
Change-Id: I6dfd673df46a49fb29d4c99a107c8fce9676d51a
This change will create a README file with a simple job debug guide and
links to the frequently used but somewhat hidden files within the
collected logs.
Change-Id: I818067952017c88e855bfeee76fa438638cdd942
Sometimes we want to look at messages in the journal. In those
cases it is kind of a pain to download the binary journal and
inspect it. We can instead collect the end of the journal in text
form.
Change-Id: Ifc60aab30172f64b5e4a417116603931e7c9aed2
Instead of using a specific user name for the logs to be collected in
/home dir use a wildcard that will work with any user the deployment may
have specified
Change-Id: Icf115f18809d0fc694736942f794d4ddb03b3dfe
The missing variables were causing errors when the playbook generation
was disabled as Ansible is trying to check the condition for each loop
separately.
Change-Id: I0ba845ecb864d6406ac988fe11c0923fd08419ea
* Support excluding files during the copy step, avoiding
time consuming copy operations.
* Empty directories are no longer collected. This results in a cleaner
directory tree while browsing the collected files.
* The extra exclude items are just there to speed up rsync pattern
matching as there are a lot of subdirs in /dev, /proc, /run, /sys and
we generally do not want to collect files from there. It is possible
to override it if needed.
* Directory definitions now need to end with / or with * to get
collected.
Change-Id: Ie36073a0e7cda2c24d51305860049f2d8dcbb17e
In patch https://review.openstack.org/#/c/427074/ we collect the
whole directory which can contain file puppet-cib* which are
not needed and take a lot of space (about 3MB compressed), so
let's collect only cib* files from there.
Change-Id: I24ba89b8b8535d19fcd9fce4d846a263f541ae27
Debugging HA issues without having the CIBs around is extremely
challenging. Let's collect these files as well. A typical large
deployment will add about ~5k to the tar file.
This change is the oooq analogous of I01585425341d4518859e63db69c1799b9ed2450a
for tripleo-ci.
Change-Id: I21c643c4f01f00f61de19cb48a7e11eecdf8a791
Chandan Kumar