When installing or upgrading subunit2sql, override pip's default
upgrade strategy to avoid updating preinstalled dependencies if they
already satisfy the requirements. This in particular avoids
attempting to replace the distro package of python-netifaces by
rebuilding a newer sdist (it is not distributed as a wheel on PyPI
and needs C extensions compiled, which in turn have a lot of
build-time deps).
This implementation is copied from the puppet-zuul module.
Change-Id: I885a61527ca9938200fb965ea97ff957fed3c480
With the new release of puppet-lint being used the puppet-lint jobs have
started failing on this repo. This commit fixes the issues identified,
to unblock ci.
Change-Id: I63ba93ecdac473aabf402d022505266371c471cb
This commit fixes the incorrect usage of '' in the command for the
expiration cron job to use the proper "" instead. This should
hopefully fix the cron job so we actually expire old data.
Change-Id: I58a640f804313e1e4b80680f0e39b86d76cb29da
This commit adds a new cron job to the subunit2sql server manifest to
periodically run a data expiration on the subunit2sql db. This will
delete any data in the DB which is over 186 days old which is roughly
6 months. This should enable us to keep 1 releases worth of data in
the DB at a time.
Change-Id: I0e52a1943d75fedfefa8beb2b9f1fcdb406af220
Sometimes the subunit2sql DB migrations can take a *really* long time
it'd kinda be nice to know exactly how long. In an effort to actually
know the duration this commit simple runs the DB migrations under
time to record how much time we're actually spending on running these
migrations. That way it'll be logged in case anyone wants to bother to
check. (which is admittedly unlikely)
Change-Id: I31fe204f0544e9b7b58158a578552f907aa18543
The pre-migration mysqldump was added so we would have a backup to
restore to in case a migration went south. However, given the size of
the DB, and the times involved in both writing a backup and restoring
from one (which takes days) this isn't very practical. Also, whether
there is enough local disk on logstash.o.o where this will be run is
not entirely clear. Given this we should just abandon this idea before
we begin and just pray that a migration never goes south. (which is
probably a terrible assumption given that I've written most of them)
If/when we consider the data in the subunit2sql database valuable
enough to warrant backing up we can investigate using a real
multi-region periodic backup solution.
Change-Id: I59fb90811d2c9d300d6be7cb876997fd3f35d9bd
Previously we were trying to run a mysql backup before we attempted
migrations, just in case something went wrong. However, the command
was failing because the $db_name variable wasn't being properly
interpolated. This commit fixes that by using " around the command
instead of '. Additionally, pipefail is set on the exec to prevent
this from succeeding but not working in the future.
Change-Id: I746a9f0ae722802419f36f80626c3a6f4186bad6
This commit fixes the syntax used for console redirection when running
the subunit2sql-db-manage command to upgrade the db schema. The
previous syntax used was a bashism that doesn't work with the shell
puppet is using to run the exec.
Change-Id: I6f76765ba23892ab082847b3b7f070c98314ff6a
This commit switches to running migrations in the background because
of timeout issues. As part of this it also adds an output file to
/var/log to store the output of the commands so it's easy to debug
if something goes wrong.
Change-Id: Id702df88d7eeb06bdb727a178bad232c9fa0d1f8
This commit is adding timeout fields to the 2 execs as part of a new
subunit2sql release. The first sets the timeout for the mysqldump to
be 30min which should be sufficient time for that. The timeout for the
actual db upgrade command is disabled however, because there is no
way to safely estimate how long the migration will take.
Change-Id: I7deb7abca9c97186343eb6c3ea30185c23c24c7a
Previously on new subunit2sql releases we would automatically run the
db upgrade command to cover any potential db migrations added in the
new release. However if there is a potential issue with a new migration
this could potentially corrupt the db. So to offset this risk this
commit runs a mysqldump to backup the db before we run the migration
command so in case things go bad we can restore the db from the backup.
Change-Id: I65d300699c457aeb4ead86c08077f7495eb24ed4
This commit switches the subunit2sql_uri which get's passed into the
config file template to use decomposed values. The advantage of this
is that it enables using subsets of the information elsewhere.
Change-Id: I30b5117c12897fa3a1cb1e2579b4eec59ba57b2b
This commit adds refreshonly to the 'upgrade_subunit2sql_db' exec.
The problem without this set is that subscribing to the Pip package
generates an event on each puppet refresh, which causes the db-manage
utility to run a migration update every 15 min, which is far from
ideal. Setting refreshonly means that the command is only run when
the resource changes, in this case when a new package version is
installed, which is the desired behavior.
Change-Id: Ib77d68f19edbf48a8d8cf4e757dbfb6463f1c6bb
This commit adds the policy around keeping the schema for the
subunit2sql db up to date. It will run the db migrations on updates
to the version of subunit2sql installed.
Change-Id: Ice6c2607bb627b789bf265b38566b7b279ac0f9a
Clark Boylan