Commit Graph

45 Commits

Author SHA1 Message Date
Clark Boylan ddb310c599 Retire this project
We've shutdown the subunit2sql processors and don't need to manage them
with puppet any more.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/839235
Change-Id: Id72f597c28807684fcfc8795a93ba0dc1b7e403c
2022-04-25 09:51:59 -07:00
Colleen Murphy 8360c894db Fix subunit2sql install guard for idempotency
The exec to install subunit2sql tries to compare the output of pip list
versus pip search to determine whether it is on the latest version.
Without this patch, the pip list command produces a string that has too
many spaces on the end and therefore doesn't match the result of the pip
search command. This means that it will always try to reinstall
subunit2sql even when it doesn't need to, which breaks idempotency
tests. This patch fixes the regex group to exclude trailing spaces.

Change-Id: I92f614db2da936d68d8f2305b348c65ba773b01c
2018-08-06 22:04:45 +02:00
Colleen Murphy 2d4e2178fe Pin statsd to version 2.1.2
The os-performance-tools library currently caps its statsd dependency at
<3.0[1]. If we try to install latest, puppet will succeed at installing
it in the first place, but upon the next run it will see that it is not
using the latest and try to upgrade and fail due to the cap. Add the pin
so that it doesn't try and fail to install something it can't.

[1] http://git.openstack.org/cgit/openstack/os-performance-tools/tree/requirements.txt?id=d96d5145fa5facdfd7a4e5ea2811948f0efd465e#n7

Change-Id: I37303d3b23cf5c387511292a8e128faa1f9df9ae
2018-07-10 13:55:33 +02:00
Colleen Murphy 3b44d63528 Don't install python-yaml package
PyYAML is already a dependency of subunit2sql, so the distro package
doesn't need to be installed explicitly. If it is installed by the
distro, then on a system with pip 10 the subunit2sql installation will
fail.

Change-Id: Ie7be5cf01fcf9a630c7bf282549a862515c159d9
2018-07-09 18:27:51 +02:00
Jeremy Stanley 22284188fb Quote arguments to the test command
When testing to see whether the installed subunit2sql package
matches the version available on PyPI, the first argument can be
empty if the package has not yet been installed. Make sure these
arguments are quoted so that the test command gets a syntax it can
parse (rather than a string starting with a binary operator).

Change-Id: Ia2e7165f851085328e00431a739cce17d2e494b2
2017-12-12 22:01:32 +00:00
Jeremy Stanley e02cfe94a3 Don't hardcode the path to pip
In at least some circumstances, our pip is installed into
/usr/local/bin to make sure that's added to the environment and
switch to dynamic path resolution when calling pip.

Change-Id: I4e53ce2aba1b8335025e180bd03a5f5e93461814
2017-12-12 17:06:12 +00:00
Jeremy Stanley e2743a3529 Register the worker daemon with systemd
When installing a custom initscript for a worker daemon, reload
systemd so it will pick that up before attempting to start the
service. Wrap it in a conditional check for Ubuntu 16.04 or later,
since earlier releases used upstart as their init instead of
systemd. Make sure the initscript template uses a distinct
"provides" key for each instance of the service as well.

This implementation is copied from the puppet-zuul module.

Change-Id: I4f7db205ac4ae29953757220dc0b9ae026ebbc71
2017-12-11 21:28:25 +00:00
Jeremy Stanley 9e2a5d4633 Install/upgrade subunit2sql safely
When installing or upgrading subunit2sql, override pip's default
upgrade strategy to avoid updating preinstalled dependencies if they
already satisfy the requirements. This in particular avoids
attempting to replace the distro package of python-netifaces by
rebuilding a newer sdist (it is not distributed as a wheel on PyPI
and needs C extensions compiled, which in turn have a lot of
build-time deps).

This implementation is copied from the puppet-zuul module.

Change-Id: I885a61527ca9938200fb965ea97ff957fed3c480
2017-12-11 21:28:24 +00:00
Jeremy Stanley 69150752a0 Make subunit2sql install depend on netifaces
In order to make sure pip installation of subunit2sql does not
attempt to pip install (and subsequently build from sdist)
netifaces, set a dependency ensuring the distro package for
python-netifaces is installed before that.

Change-Id: I01118c2771aa4e4fa553fb04b4a0885bf977a738
2017-10-29 12:43:21 +00:00
Jeremy Stanley 7e35c0b223 Install python-netifaces to avoid building
Since subunit2sql requires oslo.db which requires oslo.utils which
requires netifaces which builds C extensions and does not provide
prebuilt wheels for any platform other than Windows, install
Ubuntu's python-netifaces package to avoid needing an entire build
toolchain. We're moving the servers to Xenial, which provides a
version compatible with the current oslo.utils requirements list.

Change-Id: I4c701f08cf11bc068f6ebb10534f7e6f865970bb
2017-10-27 18:20:33 +00:00
Matthew Treinish 35025f8df5
Move paho-mqtt package install to worker node
The paho-mqtt package is only ever needed on the worker because that is
what emits mqtt. However by putting it in init this also gets installed
on the server node. Additionally this commit adds a duplication check
around the package because it's possible that something else in another
module will conflict with paho-mqtt as we start adding mqtt to more
services.

Change-Id: Id0308d2d4d1843fcca73f459cffa2ae944bebd0c
2017-04-27 10:05:47 -04:00
Matthew Treinish c06cdc8a8a
Add MQTT support to the gearman worker
This commit adds support to the gearman worker for publishing an mqtt
message when processing a gearman job succeeds or fails. By default this
is disabled since it requires extra configuration to tell the worker how
to talk to the mqtt broker.

Right now the payload of the message is just the build_uuid and whether
it was written to the db or not. Eventually some details about the
subunit2sql db entry will be added to the payload. But this requires
changes to either subunit2sql or how the worker calls the subunit2sql
api before it is feasible.

Change-Id: Ibd13b737eccf52863a69d20843cb7d50242f7bb9
2017-04-10 16:08:10 -04:00
Matthew Treinish 9b5753123e
Fix puppet-lint failures
With the new release of puppet-lint being used the puppet-lint jobs have
started failing on this repo. This commit fixes the issues identified,
to unblock ci.

Change-Id: I63ba93ecdac473aabf402d022505266371c471cb
2017-04-04 12:39:21 -04:00
Mikhail S Medvedev 4548b4114b Remove trailing slash so if-defined guard works
Both resource name in if statement and in declaration need to be exactly
the same.

Change-Id: Ie8635930e9d9520929865e59e3224819f487272a
2016-11-17 15:49:20 -06:00
Matthew Treinish 0e55e618ae Make sure global worker defs only done once
This commit makes sure that all the global definitions on the worker
class are only defined once. If we don't check for the previous
definition when running with multiple workers on a single node things
will fail.

Change-Id: I947d61f455778fe41d0a4723c99d029272d2155d
2016-07-26 15:23:45 -04:00
Philip Marc Schwartz 1c435df394 Update pip provider to use openstack_pip provider
Change-Id: I531d668bdc4f1bf080436b3f2648c7ff153dfe10
Signed-off-by: Philip Marc Schwartz <philip@progmad.com>
2016-06-16 10:20:56 -04:00
Matthew Treinish 34fe9cc284
Ensure worker log dir is created
This commit makes sure we create the dir that the worker tries to write
it's logs to.

Change-Id: Id899742ae3efc5e821f46aa10cc722e16d9bf8cb
2016-03-07 12:09:14 -05:00
Matthew Treinish 5025c20960
Switch /etc/logstash to /etc/subunit2sql
This commit switches the use of the /etc/logstash directory to
/etc/subunit2sql. This was a leftover from when we first got the
subunit workers up and running and it was largely a copy and paste
of the policy for the logstash workers. Changing this should make it
more clear to users and consistent with the naming of everything
else.

Change-Id: Ie425dcf53856edd873ff1d0185dcf8e4ae30e3f8
2016-03-04 15:29:15 -05:00
Matthew Treinish 50c21b8c3f
Change /etc/logstash dir ownership to root
This commit updates the /etc/logstash config dir (where the
subunit2sql configs live, which makes so much sense) to be owned by
root. There is no logstash user ever created on the worker nodes so
we shouldn't try to assign it as the owner.

Change-Id: Ifd85cf4d33c85a958aa33fca50bdd847a651e560
2016-03-04 12:02:31 -05:00
Matthew Treinish a8e4c8322d
Fix the expiration cron job
This commit fixes the incorrect usage of '' in the command for the
expiration cron job to use the proper "" instead. This should
hopefully fix the cron job so we actually expire old data.

Change-Id: I58a640f804313e1e4b80680f0e39b86d76cb29da
2016-02-12 14:46:54 -05:00
Jenkins 429e7fe60c Merge "Install os-performance-tools" 2016-01-14 01:01:14 +00:00
Matthew Treinish 78868c6950 Add cron job to run weekly prune the data set of db
This commit adds a new cron job to the subunit2sql server manifest to
periodically run a data expiration on the subunit2sql db. This will
delete any data in the DB which is over 186 days old which is roughly
6 months. This should enable us to keep 1 releases worth of data in
the DB at a time.

Change-Id: I0e52a1943d75fedfefa8beb2b9f1fcdb406af220
2016-01-13 16:23:04 +00:00
Matthew Treinish c38d9d37d4
Lets time the super long migrations we have to run
Sometimes the subunit2sql DB migrations can take a *really* long time
it'd kinda be nice to know exactly how long. In an effort to actually
know the duration this commit simple runs the DB migrations under
time to record how much time we're actually spending on running these
migrations. That way it'll be logged in case anyone wants to bother to
check. (which is admittedly unlikely)

Change-Id: I31fe204f0544e9b7b58158a578552f907aa18543
2015-11-24 16:41:20 -05:00
Matthew Treinish 8b8b088640
Stop attempting to run a mysqldump before migrations
The pre-migration mysqldump was added so we would have a backup to
restore to in case a migration went south. However, given the size of
the DB, and the times involved in both writing a backup and restoring
from one (which takes days) this isn't very practical. Also, whether
there is enough local disk on logstash.o.o where this will be run is
not entirely clear. Given this we should just abandon this idea before
we begin and just pray that a migration never goes south. (which is
probably a terrible assumption given that I've written most of them)

If/when we consider the data in the subunit2sql database valuable
enough to warrant backing up we can investigate using a real
multi-region periodic backup solution.

Change-Id: I59fb90811d2c9d300d6be7cb876997fd3f35d9bd
2015-11-17 20:33:30 -05:00
Clint Byrum c9d6cca1ee Install os-performance-tools
This will register itself as a plugin and use the statsd configuration
options added to send any counters seen in attachments to statsd.

Change-Id: If93c84781fde46852d65bac88ab500563f7978cc
2015-11-12 16:24:04 -08:00
Matthew Treinish 1cd42a389f
Use double quotes for mysql backup
Previously we were trying to run a mysql backup before we attempted
migrations, just in case something went wrong. However, the command
was failing because the $db_name variable wasn't being properly
interpolated. This commit fixes that by using " around the command
instead of '. Additionally, pipefail is set on the exec to prevent
this from succeeding but not working in the future.

Change-Id: I746a9f0ae722802419f36f80626c3a6f4186bad6
2015-10-09 16:06:36 -04:00
Colleen Murphy 237aa40927 Add Gemfile and puppet 4 checks
In anticipation of puppet 4, start trying to deal with puppet 4 things
that can be helpfully predicted by puppet lint plugins. Also fix lint
errors caught by the puppet-lint-absolute_classname-check gem as well
as arrow alignment errors caught by the now bundle-installed
puppet-lint.

Change-Id: I6975c605e238046b3812b2fea20e65db8c890e42
2015-08-13 18:38:02 -07:00
Jenkins 4378346e60 Merge "Subscribe worker service to subunit2sql package" 2015-04-30 01:17:13 +00:00
Matthew Treinish 0d033994c8 Fix db upgrade console redirection to be posix compliant
This commit fixes the syntax used for console redirection when running
the subunit2sql-db-manage command to upgrade the db schema. The
previous syntax used was a bashism that doesn't work with the shell
puppet is using to run the exec.

Change-Id: I6f76765ba23892ab082847b3b7f070c98314ff6a
2015-03-17 14:32:10 -04:00
Matthew Treinish 58d13898b5 Run subunit2sql migrations in the background
This commit switches to running migrations in the background because
of timeout issues. As part of this it also adds an output file to
/var/log to store the output of the commands so it's easy to debug
if something goes wrong.

Change-Id: Id702df88d7eeb06bdb727a178bad232c9fa0d1f8
2015-03-16 11:29:09 -04:00
Matthew Treinish a11d7763ce Subscribe worker service to subunit2sql package
This commit subscribes the worker daemon to the subunit2sql package
to make it restart when there is a new version released.

Change-Id: I27b2ad1ce4a02e369c52bc154670302e60bcb75f
2015-03-13 20:04:26 -04:00
Matthew Treinish a665f7b970 Set timeouts for db upgrade operations
This commit is adding timeout fields to the 2 execs as part of a new
subunit2sql release. The first sets the timeout for the mysqldump to
be 30min which should be sufficient time for that. The timeout for the
actual db upgrade command is disabled however, because there is no
way to safely estimate how long the migration will take.

Change-Id: I7deb7abca9c97186343eb6c3ea30185c23c24c7a
2015-03-13 16:14:53 -04:00
Matthew Treinish afc25cba89 Add backup before running db migrations
Previously on new subunit2sql releases we would automatically run the
db upgrade command to cover any potential db migrations added in the
new release. However if there is a potential issue with a new migration
this could potentially corrupt the db. So to offset this risk this
commit runs a mysqldump to backup the db before we run the migration
command so in case things go bad we can restore the db from the backup.

Change-Id: I65d300699c457aeb4ead86c08077f7495eb24ed4
2015-03-05 19:22:22 -05:00
Mikhail S Medvedev bf460ce2d1 Add standard puppet module files and .gitreview
This module has been split out from system-config, and additional
    files are added to create a proper puppet module.

    In addition, puppet-lint issues were fixed for manifests/*

    The spec:
    http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet-modules.html

Change-Id: I0d473b48a978727b294e1329b0176641afe2f02d
2015-01-29 08:13:02 -06:00
Matthew Treinish 80a9014589 Switch subunit2sql to use decomposed uri
This commit switches the subunit2sql_uri which get's passed into the
config file template to use decomposed values. The advantage of this
is that it enables using subsets of the information elsewhere.

Change-Id: I30b5117c12897fa3a1cb1e2579b4eec59ba57b2b
2014-12-05 14:27:00 -05:00
Matthew Treinish 624e0f6662 Remove unnecessary use of logstash::indexer from subunit_worker
Previously the subunit2sql worker.pp was including the
logstash::indexer class, but this was really only done for the
/etc/logstash dir which is where the worker puts all it's files. This
commit corrects this by just ensuring the directory exists instead of
importing the other class which includes a lot of other unnecessary
things.

Change-Id: I3f00d1bb443ee0f1b2a8e9d393f46a7e9ebf148a
2014-11-18 20:07:02 -05:00
Matthew Treinish dcb20f0c7e Use correct include in worker.pp
Change-Id: Ia80729ea2669cc3c8f411284065d0745da2552df
2014-11-18 18:22:46 -05:00
Jenkins cfc538c180 Merge "Add refreshonly to 'upgrade_subunit2sql_db'" 2014-11-18 22:12:22 +00:00
Matthew Treinish 93f40a7584 Add refreshonly to 'upgrade_subunit2sql_db'
This commit adds refreshonly to the 'upgrade_subunit2sql_db' exec.
The problem without this set is that subscribing to the Pip package
generates an event on each puppet refresh, which causes the db-manage
utility to run a migration update every 15 min, which is far from
ideal. Setting refreshonly means that the command is only run when
the resource changes, in this case when a new package version is
installed, which is the desired behavior.

Change-Id: Ib77d68f19edbf48a8d8cf4e757dbfb6463f1c6bb
2014-11-18 16:16:18 -05:00
Matthew Treinish 35a2f7e94d Fix subunit-gearman-worker.py path
This commit fixes the path for the gearman-worker script so it can
actually find the file. The script is located in the subunit2sql
module and not the log_processor module.

Change-Id: Ie3228d33ed3c8bdf50b6b2bdc46770fb38742b69
2014-11-18 10:36:36 -05:00
Matthew Treinish ac2fce5b05 Add missing logstash import
This commit adds a missing import to the subunit2sql worker manifest.

Change-Id: I23357d9404e4a4059d0b4943ccef3ec2bd7e1539
2014-11-17 20:24:05 -05:00
Matthew Treinish ede7123001 Move worker pieces that are only need once to init.pp
This commit moves all the pieces from worker.pp which are used by all
the worker process into init.pp. This was not done before because the
package definitions would conflict with those in logstash.o.o's
manifest. However with the if not defined checks we can have them in
init.pp, so let's move everything in there.

Change-Id: I5048604ce64bb52f463397c6a37b4a19a0bab8bb
2014-11-17 18:57:58 -05:00
Matthew Treinish 4d0b971483 Ensure we only define subunit worker packages once
We can only declare the packages once per node, but since we spin up
multiple workers per node, the definitions were being duplicated. This
fixes the worker.pp file to ensure we only define packages once.

Change-Id: Idc67f97b77b4f4be3864c9dacd3835bb8328d068
2014-11-17 15:55:42 -05:00
Matthew Treinish dbbe8129b4 Add subunit2sql server configuration
This commit adds the policy around keeping the schema for the
subunit2sql db up to date. It will run the db migrations on updates
to the version of subunit2sql installed.

Change-Id: Ice6c2607bb627b789bf265b38566b7b279ac0f9a
2014-10-29 15:39:16 -04:00
Matthew Treinish 9a06307c40 Add subunit2sql gearman workers
This adds a new gearman worker to process the subunit files from
the gate job runs. It will use subunit2sql to connect to a sql
server and process the data from the subunit file. The
log-gearman-client is modified to allow for pushing subunit jobs
to gearman, and the worker model for processsing logs is borrowed
to process the subunit files.

Change-Id: I83103eb6afc22d91f916583c36c0e956c23a64b3
2014-10-29 13:03:49 -04:00