When I added the timeout I didn't move the lock file path properly. Fix
this by putting the lock file path before the timeout command to flock.
Change-Id: Iacf8d55f9cfe5b2c92569929bbdadebbe2f8de09
Add a four hour timeout to the generation of e-r graphs as these scripts
occasiaonlly run for days without exiting. We'd rather fail then try
again later to ensure the graphs are mostly up to date.
Change-Id: I7ee1c94c41edb9d6b5aeedebac64aeded1cf0e97
This was already changed in elastic-recheck in [0], so let's also use
this in our running config.
[0] https://review.openstack.org/330684
Change-Id: If9911f1ad2fa45afd6cfde1216f6178e87891028
From time to time the flock calls on the er system back up for an
unknown reason. However, when fixing another bug I found that flock
has a wait timeout. Set this to an hour so that if we fail to flock
after an hour we don't back up forever.
Change-Id: Ie574b994d2b5cc38f559d3d7f8dd980148cf0890
Replace URLs for workflow documentation to appropriate parts of the
OpenStack Project Infrastructure Manual.
Change-Id: I8aa32f5617a0331c116a703c45ae7f644f5b409a
* modules/elastic_recheck/files/elastic-recheck.init: It has in the
past been observed that the elastic-recheck IRC/Gerrit-commenting
bot does not always terminate cleanly. Use a retry with escalation
of kill signals in the initscript to help prevent additional old
processes from hanging around when restarts take place.
Change-Id: I9498eeeb935cb609c2c4c099c3552bed337fbe99
As of I7353af4c3f141d4bd617d6fd388d7957e0586ba8 we'll introduce
the ability to have a 2 layer config which includes the messages
that we'll send back to the user on gerrit outside of the
elastic-recheck code base. This makes it easier to make changes
to the UX consistent with what we want to change with the rest
of the config.
Change-Id: I9208123a4cb3be02c272cd8a6eba460f4130a960
Depends-On: I7353af4c3f141d4bd617d6fd388d7957e0586ba8
The elastic-recheck change: Ic3dd76bad94213c7152c29a99c00ed23a2c01a31
will add a new section to the recheckwatchbot yaml file to specify
which bug targets get reported to which channels. This commit preempts
that change by adding the new section to the -qa channel to preserve
the existing functionality.
Change-Id: I452040883027b260c0480b9d7ac2e2ca476b72b3
this adds the new elastic-recheck-unclassified report, which
generates an html page of all the unclassified gate failures.
it also runs this command, and the elastic-recheck-graph command
not only on cron, but also on updates to the repository. In order
to do this safely, we create a new er_safe_run.sh in /usr/local
that provides a flock front end to ensure we won't race on our
html writes.
Change-Id: I624f2629e2ac19a1394866f73f1aa9ba18e968ce
* modules/elastic_recheck/files/elastic-recheck.init: Minor
whitespace cleanup for space-before-tab indent on a comment line.
* modules/elastic_recheck/files/logging.config: New logging
configuration file for elastic-recheck IRC bot.
* modules/elastic_recheck/manifests/init.pp: Install the new logging
configuration file.
* modules/elastic_recheck/templates/elastic-recheck.conf.erb: Tell
the IRC bot to use the new logging configuration file.
Change-Id: I47cabb01df209c9f020846bcb272b9a2f796b0e5
* modules/elastic_recheck/files/elastic-recheck.init: Update the
daemon invocation now that it has an executable entrypoints wrapper.
* modules/elastic_recheck/manifests/init.pp: Correct the Puppet
dependency tree sufficiently to be able to apply the manifest
successfully.
Change-Id: Ib018564e66c29d73b30aaf50c9306dc6d67ba2b2
This elastic recheck thing that classifies failures based on logstash
queries is awesome. We should run it on the OpenStack developer
infrastructure. Deploy it on logstash.o.o to keep related services
together.
Change-Id: I16ae7a32c2c48474454df17931779d0f60c3ed85