The exec to install subunit2sql tries to compare the output of pip list
versus pip search to determine whether it is on the latest version.
Without this patch, the pip list command produces a string that has too
many spaces on the end and therefore doesn't match the result of the pip
search command. This means that it will always try to reinstall
subunit2sql even when it doesn't need to, which breaks idempotency
tests. This patch fixes the regex group to exclude trailing spaces.
Change-Id: I92f614db2da936d68d8f2305b348c65ba773b01c
The os-performance-tools library currently caps its statsd dependency at
<3.0[1]. If we try to install latest, puppet will succeed at installing
it in the first place, but upon the next run it will see that it is not
using the latest and try to upgrade and fail due to the cap. Add the pin
so that it doesn't try and fail to install something it can't.
[1] http://git.openstack.org/cgit/openstack/os-performance-tools/tree/requirements.txt?id=d96d5145fa5facdfd7a4e5ea2811948f0efd465e#n7
Change-Id: I37303d3b23cf5c387511292a8e128faa1f9df9ae
PyYAML is already a dependency of subunit2sql, so the distro package
doesn't need to be installed explicitly. If it is installed by the
distro, then on a system with pip 10 the subunit2sql installation will
fail.
Change-Id: Ie7be5cf01fcf9a630c7bf282549a862515c159d9
When testing to see whether the installed subunit2sql package
matches the version available on PyPI, the first argument can be
empty if the package has not yet been installed. Make sure these
arguments are quoted so that the test command gets a syntax it can
parse (rather than a string starting with a binary operator).
Change-Id: Ia2e7165f851085328e00431a739cce17d2e494b2
In at least some circumstances, our pip is installed into
/usr/local/bin to make sure that's added to the environment and
switch to dynamic path resolution when calling pip.
Change-Id: I4e53ce2aba1b8335025e180bd03a5f5e93461814
When installing a custom initscript for a worker daemon, reload
systemd so it will pick that up before attempting to start the
service. Wrap it in a conditional check for Ubuntu 16.04 or later,
since earlier releases used upstart as their init instead of
systemd. Make sure the initscript template uses a distinct
"provides" key for each instance of the service as well.
This implementation is copied from the puppet-zuul module.
Change-Id: I4f7db205ac4ae29953757220dc0b9ae026ebbc71
When installing or upgrading subunit2sql, override pip's default
upgrade strategy to avoid updating preinstalled dependencies if they
already satisfy the requirements. This in particular avoids
attempting to replace the distro package of python-netifaces by
rebuilding a newer sdist (it is not distributed as a wheel on PyPI
and needs C extensions compiled, which in turn have a lot of
build-time deps).
This implementation is copied from the puppet-zuul module.
Change-Id: I885a61527ca9938200fb965ea97ff957fed3c480
In order to make sure pip installation of subunit2sql does not
attempt to pip install (and subsequently build from sdist)
netifaces, set a dependency ensuring the distro package for
python-netifaces is installed before that.
Change-Id: I01118c2771aa4e4fa553fb04b4a0885bf977a738
Since subunit2sql requires oslo.db which requires oslo.utils which
requires netifaces which builds C extensions and does not provide
prebuilt wheels for any platform other than Windows, install
Ubuntu's python-netifaces package to avoid needing an entire build
toolchain. We're moving the servers to Xenial, which provides a
version compatible with the current oslo.utils requirements list.
Change-Id: I4c701f08cf11bc068f6ebb10534f7e6f865970bb
The paho-mqtt package is only ever needed on the worker because that is
what emits mqtt. However by putting it in init this also gets installed
on the server node. Additionally this commit adds a duplication check
around the package because it's possible that something else in another
module will conflict with paho-mqtt as we start adding mqtt to more
services.
Change-Id: Id0308d2d4d1843fcca73f459cffa2ae944bebd0c
This commit adds support to the gearman worker for publishing an mqtt
message when processing a gearman job succeeds or fails. By default this
is disabled since it requires extra configuration to tell the worker how
to talk to the mqtt broker.
Right now the payload of the message is just the build_uuid and whether
it was written to the db or not. Eventually some details about the
subunit2sql db entry will be added to the payload. But this requires
changes to either subunit2sql or how the worker calls the subunit2sql
api before it is feasible.
Change-Id: Ibd13b737eccf52863a69d20843cb7d50242f7bb9
With the new release of puppet-lint being used the puppet-lint jobs have
started failing on this repo. This commit fixes the issues identified,
to unblock ci.
Change-Id: I63ba93ecdac473aabf402d022505266371c471cb
This commit makes sure that all the global definitions on the worker
class are only defined once. If we don't check for the previous
definition when running with multiple workers on a single node things
will fail.
Change-Id: I947d61f455778fe41d0a4723c99d029272d2155d
This commit switches the use of the /etc/logstash directory to
/etc/subunit2sql. This was a leftover from when we first got the
subunit workers up and running and it was largely a copy and paste
of the policy for the logstash workers. Changing this should make it
more clear to users and consistent with the naming of everything
else.
Change-Id: Ie425dcf53856edd873ff1d0185dcf8e4ae30e3f8
This commit updates the /etc/logstash config dir (where the
subunit2sql configs live, which makes so much sense) to be owned by
root. There is no logstash user ever created on the worker nodes so
we shouldn't try to assign it as the owner.
Change-Id: Ifd85cf4d33c85a958aa33fca50bdd847a651e560
This commit fixes the incorrect usage of '' in the command for the
expiration cron job to use the proper "" instead. This should
hopefully fix the cron job so we actually expire old data.
Change-Id: I58a640f804313e1e4b80680f0e39b86d76cb29da
This commit adds a new cron job to the subunit2sql server manifest to
periodically run a data expiration on the subunit2sql db. This will
delete any data in the DB which is over 186 days old which is roughly
6 months. This should enable us to keep 1 releases worth of data in
the DB at a time.
Change-Id: I0e52a1943d75fedfefa8beb2b9f1fcdb406af220
Sometimes the subunit2sql DB migrations can take a *really* long time
it'd kinda be nice to know exactly how long. In an effort to actually
know the duration this commit simple runs the DB migrations under
time to record how much time we're actually spending on running these
migrations. That way it'll be logged in case anyone wants to bother to
check. (which is admittedly unlikely)
Change-Id: I31fe204f0544e9b7b58158a578552f907aa18543
The pre-migration mysqldump was added so we would have a backup to
restore to in case a migration went south. However, given the size of
the DB, and the times involved in both writing a backup and restoring
from one (which takes days) this isn't very practical. Also, whether
there is enough local disk on logstash.o.o where this will be run is
not entirely clear. Given this we should just abandon this idea before
we begin and just pray that a migration never goes south. (which is
probably a terrible assumption given that I've written most of them)
If/when we consider the data in the subunit2sql database valuable
enough to warrant backing up we can investigate using a real
multi-region periodic backup solution.
Change-Id: I59fb90811d2c9d300d6be7cb876997fd3f35d9bd
This will register itself as a plugin and use the statsd configuration
options added to send any counters seen in attachments to statsd.
Change-Id: If93c84781fde46852d65bac88ab500563f7978cc
Previously we were trying to run a mysql backup before we attempted
migrations, just in case something went wrong. However, the command
was failing because the $db_name variable wasn't being properly
interpolated. This commit fixes that by using " around the command
instead of '. Additionally, pipefail is set on the exec to prevent
this from succeeding but not working in the future.
Change-Id: I746a9f0ae722802419f36f80626c3a6f4186bad6
In anticipation of puppet 4, start trying to deal with puppet 4 things
that can be helpfully predicted by puppet lint plugins. Also fix lint
errors caught by the puppet-lint-absolute_classname-check gem as well
as arrow alignment errors caught by the now bundle-installed
puppet-lint.
Change-Id: I6975c605e238046b3812b2fea20e65db8c890e42
This commit fixes the syntax used for console redirection when running
the subunit2sql-db-manage command to upgrade the db schema. The
previous syntax used was a bashism that doesn't work with the shell
puppet is using to run the exec.
Change-Id: I6f76765ba23892ab082847b3b7f070c98314ff6a
This commit switches to running migrations in the background because
of timeout issues. As part of this it also adds an output file to
/var/log to store the output of the commands so it's easy to debug
if something goes wrong.
Change-Id: Id702df88d7eeb06bdb727a178bad232c9fa0d1f8
This commit subscribes the worker daemon to the subunit2sql package
to make it restart when there is a new version released.
Change-Id: I27b2ad1ce4a02e369c52bc154670302e60bcb75f
This commit is adding timeout fields to the 2 execs as part of a new
subunit2sql release. The first sets the timeout for the mysqldump to
be 30min which should be sufficient time for that. The timeout for the
actual db upgrade command is disabled however, because there is no
way to safely estimate how long the migration will take.
Change-Id: I7deb7abca9c97186343eb6c3ea30185c23c24c7a
Previously on new subunit2sql releases we would automatically run the
db upgrade command to cover any potential db migrations added in the
new release. However if there is a potential issue with a new migration
this could potentially corrupt the db. So to offset this risk this
commit runs a mysqldump to backup the db before we run the migration
command so in case things go bad we can restore the db from the backup.
Change-Id: I65d300699c457aeb4ead86c08077f7495eb24ed4
This commit switches the subunit2sql_uri which get's passed into the
config file template to use decomposed values. The advantage of this
is that it enables using subsets of the information elsewhere.
Change-Id: I30b5117c12897fa3a1cb1e2579b4eec59ba57b2b
Previously the subunit2sql worker.pp was including the
logstash::indexer class, but this was really only done for the
/etc/logstash dir which is where the worker puts all it's files. This
commit corrects this by just ensuring the directory exists instead of
importing the other class which includes a lot of other unnecessary
things.
Change-Id: I3f00d1bb443ee0f1b2a8e9d393f46a7e9ebf148a
This commit adds refreshonly to the 'upgrade_subunit2sql_db' exec.
The problem without this set is that subscribing to the Pip package
generates an event on each puppet refresh, which causes the db-manage
utility to run a migration update every 15 min, which is far from
ideal. Setting refreshonly means that the command is only run when
the resource changes, in this case when a new package version is
installed, which is the desired behavior.
Change-Id: Ib77d68f19edbf48a8d8cf4e757dbfb6463f1c6bb
This commit fixes the path for the gearman-worker script so it can
actually find the file. The script is located in the subunit2sql
module and not the log_processor module.
Change-Id: Ie3228d33ed3c8bdf50b6b2bdc46770fb38742b69
This commit moves all the pieces from worker.pp which are used by all
the worker process into init.pp. This was not done before because the
package definitions would conflict with those in logstash.o.o's
manifest. However with the if not defined checks we can have them in
init.pp, so let's move everything in there.
Change-Id: I5048604ce64bb52f463397c6a37b4a19a0bab8bb
We can only declare the packages once per node, but since we spin up
multiple workers per node, the definitions were being duplicated. This
fixes the worker.pp file to ensure we only define packages once.
Change-Id: Idc67f97b77b4f4be3864c9dacd3835bb8328d068
This commit adds the policy around keeping the schema for the
subunit2sql db up to date. It will run the db migrations on updates
to the version of subunit2sql installed.
Change-Id: Ice6c2607bb627b789bf265b38566b7b279ac0f9a
This adds a new gearman worker to process the subunit files from
the gate job runs. It will use subunit2sql to connect to a sql
server and process the data from the subunit file. The
log-gearman-client is modified to allow for pushing subunit jobs
to gearman, and the worker model for processsing logs is borrowed
to process the subunit files.
Change-Id: I83103eb6afc22d91f916583c36c0e956c23a64b3