When installing a custom initscript for a worker daemon, reload
systemd so it will pick that up before attempting to start the
service. Wrap it in a conditional check for Ubuntu 16.04 or later,
since earlier releases used upstart as their init instead of
systemd. Make sure the initscript template uses a distinct
"provides" key for each instance of the service as well.
This implementation is copied from the puppet-zuul module.
Change-Id: I4f7db205ac4ae29953757220dc0b9ae026ebbc71
When installing or upgrading subunit2sql, override pip's default
upgrade strategy to avoid updating preinstalled dependencies if they
already satisfy the requirements. This in particular avoids
attempting to replace the distro package of python-netifaces by
rebuilding a newer sdist (it is not distributed as a wheel on PyPI
and needs C extensions compiled, which in turn have a lot of
build-time deps).
This implementation is copied from the puppet-zuul module.
Change-Id: I885a61527ca9938200fb965ea97ff957fed3c480
The paho-mqtt package is only ever needed on the worker because that is
what emits mqtt. However by putting it in init this also gets installed
on the server node. Additionally this commit adds a duplication check
around the package because it's possible that something else in another
module will conflict with paho-mqtt as we start adding mqtt to more
services.
Change-Id: Id0308d2d4d1843fcca73f459cffa2ae944bebd0c
This commit makes sure that all the global definitions on the worker
class are only defined once. If we don't check for the previous
definition when running with multiple workers on a single node things
will fail.
Change-Id: I947d61f455778fe41d0a4723c99d029272d2155d
This commit switches the use of the /etc/logstash directory to
/etc/subunit2sql. This was a leftover from when we first got the
subunit workers up and running and it was largely a copy and paste
of the policy for the logstash workers. Changing this should make it
more clear to users and consistent with the naming of everything
else.
Change-Id: Ie425dcf53856edd873ff1d0185dcf8e4ae30e3f8
This commit updates the /etc/logstash config dir (where the
subunit2sql configs live, which makes so much sense) to be owned by
root. There is no logstash user ever created on the worker nodes so
we shouldn't try to assign it as the owner.
Change-Id: Ifd85cf4d33c85a958aa33fca50bdd847a651e560
In anticipation of puppet 4, start trying to deal with puppet 4 things
that can be helpfully predicted by puppet lint plugins. Also fix lint
errors caught by the puppet-lint-absolute_classname-check gem as well
as arrow alignment errors caught by the now bundle-installed
puppet-lint.
Change-Id: I6975c605e238046b3812b2fea20e65db8c890e42
This commit subscribes the worker daemon to the subunit2sql package
to make it restart when there is a new version released.
Change-Id: I27b2ad1ce4a02e369c52bc154670302e60bcb75f
This commit switches the subunit2sql_uri which get's passed into the
config file template to use decomposed values. The advantage of this
is that it enables using subsets of the information elsewhere.
Change-Id: I30b5117c12897fa3a1cb1e2579b4eec59ba57b2b
Previously the subunit2sql worker.pp was including the
logstash::indexer class, but this was really only done for the
/etc/logstash dir which is where the worker puts all it's files. This
commit corrects this by just ensuring the directory exists instead of
importing the other class which includes a lot of other unnecessary
things.
Change-Id: I3f00d1bb443ee0f1b2a8e9d393f46a7e9ebf148a
This commit moves all the pieces from worker.pp which are used by all
the worker process into init.pp. This was not done before because the
package definitions would conflict with those in logstash.o.o's
manifest. However with the if not defined checks we can have them in
init.pp, so let's move everything in there.
Change-Id: I5048604ce64bb52f463397c6a37b4a19a0bab8bb
We can only declare the packages once per node, but since we spin up
multiple workers per node, the definitions were being duplicated. This
fixes the worker.pp file to ensure we only define packages once.
Change-Id: Idc67f97b77b4f4be3864c9dacd3835bb8328d068
This adds a new gearman worker to process the subunit files from
the gate job runs. It will use subunit2sql to connect to a sql
server and process the data from the subunit file. The
log-gearman-client is modified to allow for pushing subunit jobs
to gearman, and the worker model for processsing logs is borrowed
to process the subunit files.
Change-Id: I83103eb6afc22d91f916583c36c0e956c23a64b3
Clark Boylan