Commit Graph

98 Commits

Author SHA1 Message Date
Clark Boylan ddb310c599 Retire this project
We've shutdown the subunit2sql processors and don't need to manage them
with puppet any more.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/839235
Change-Id: Id72f597c28807684fcfc8795a93ba0dc1b7e403c
2022-04-25 09:51:59 -07:00
OpenDev Sysadmins bc6edbe3d5 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:08 +00:00
Ian Wienand 35764a1e39 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I7ad3e225464d21116d448df13d8de7fb842b3745
2019-03-24 20:35:45 +00:00
Zuul 11005605dc Merge "Add beaker tests" 2019-02-04 16:39:20 +00:00
Zuul d7ff514c0c Merge "Fix subunit2sql install guard for idempotency" 2018-11-21 22:49:11 +00:00
Colleen Murphy f5d762f98e Add beaker tests
Add tests to ensure the subunit2sql server and worker classes work and
that the worker service can start.

Change-Id: I842ecbcd5988484043643fdae1c49dd95bba90f2
2018-08-06 23:01:14 +02:00
Colleen Murphy dcecea024c Update Gemfile for Zuulv3
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.

Change-Id: I363a112ee856a9f5eb986fd878730db082c26a81
2018-08-06 22:07:14 +02:00
Colleen Murphy 8360c894db Fix subunit2sql install guard for idempotency
The exec to install subunit2sql tries to compare the output of pip list
versus pip search to determine whether it is on the latest version.
Without this patch, the pip list command produces a string that has too
many spaces on the end and therefore doesn't match the result of the pip
search command. This means that it will always try to reinstall
subunit2sql even when it doesn't need to, which breaks idempotency
tests. This patch fixes the regex group to exclude trailing spaces.

Change-Id: I92f614db2da936d68d8f2305b348c65ba773b01c
2018-08-06 22:04:45 +02:00
Colleen Murphy 2d4e2178fe Pin statsd to version 2.1.2
The os-performance-tools library currently caps its statsd dependency at
<3.0[1]. If we try to install latest, puppet will succeed at installing
it in the first place, but upon the next run it will see that it is not
using the latest and try to upgrade and fail due to the cap. Add the pin
so that it doesn't try and fail to install something it can't.

[1] http://git.openstack.org/cgit/openstack/os-performance-tools/tree/requirements.txt?id=d96d5145fa5facdfd7a4e5ea2811948f0efd465e#n7

Change-Id: I37303d3b23cf5c387511292a8e128faa1f9df9ae
2018-07-10 13:55:33 +02:00
Colleen Murphy 3b44d63528 Don't install python-yaml package
PyYAML is already a dependency of subunit2sql, so the distro package
doesn't need to be installed explicitly. If it is installed by the
distro, then on a system with pip 10 the subunit2sql installation will
fail.

Change-Id: Ie7be5cf01fcf9a630c7bf282549a862515c159d9
2018-07-09 18:27:51 +02:00
Colleen Murphy bb2f9fc027 Use instance variables in puppet template
Using local ruby variables in puppet ERB templates was deprecated in
puppet 3 and doesn't work any more in puppet 4. Access these values as
ruby instance variables instead.

Change-Id: I6ea7b30d7a890b73f8a4ce6572751d0ecfb35667
2018-04-26 21:45:46 +02:00
Jeremy Stanley 22284188fb Quote arguments to the test command
When testing to see whether the installed subunit2sql package
matches the version available on PyPI, the first argument can be
empty if the package has not yet been installed. Make sure these
arguments are quoted so that the test command gets a syntax it can
parse (rather than a string starting with a binary operator).

Change-Id: Ia2e7165f851085328e00431a739cce17d2e494b2
2017-12-12 22:01:32 +00:00
Jeremy Stanley e02cfe94a3 Don't hardcode the path to pip
In at least some circumstances, our pip is installed into
/usr/local/bin to make sure that's added to the environment and
switch to dynamic path resolution when calling pip.

Change-Id: I4e53ce2aba1b8335025e180bd03a5f5e93461814
2017-12-12 17:06:12 +00:00
Jeremy Stanley e2743a3529 Register the worker daemon with systemd
When installing a custom initscript for a worker daemon, reload
systemd so it will pick that up before attempting to start the
service. Wrap it in a conditional check for Ubuntu 16.04 or later,
since earlier releases used upstart as their init instead of
systemd. Make sure the initscript template uses a distinct
"provides" key for each instance of the service as well.

This implementation is copied from the puppet-zuul module.

Change-Id: I4f7db205ac4ae29953757220dc0b9ae026ebbc71
2017-12-11 21:28:25 +00:00
Jeremy Stanley 9e2a5d4633 Install/upgrade subunit2sql safely
When installing or upgrading subunit2sql, override pip's default
upgrade strategy to avoid updating preinstalled dependencies if they
already satisfy the requirements. This in particular avoids
attempting to replace the distro package of python-netifaces by
rebuilding a newer sdist (it is not distributed as a wheel on PyPI
and needs C extensions compiled, which in turn have a lot of
build-time deps).

This implementation is copied from the puppet-zuul module.

Change-Id: I885a61527ca9938200fb965ea97ff957fed3c480
2017-12-11 21:28:24 +00:00
Jeremy Stanley 69150752a0 Make subunit2sql install depend on netifaces
In order to make sure pip installation of subunit2sql does not
attempt to pip install (and subsequently build from sdist)
netifaces, set a dependency ensuring the distro package for
python-netifaces is installed before that.

Change-Id: I01118c2771aa4e4fa553fb04b4a0885bf977a738
2017-10-29 12:43:21 +00:00
Jeremy Stanley 7e35c0b223 Install python-netifaces to avoid building
Since subunit2sql requires oslo.db which requires oslo.utils which
requires netifaces which builds C extensions and does not provide
prebuilt wheels for any platform other than Windows, install
Ubuntu's python-netifaces package to avoid needing an entire build
toolchain. We're moving the servers to Xenial, which provides a
version compatible with the current oslo.utils requirements list.

Change-Id: I4c701f08cf11bc068f6ebb10534f7e6f865970bb
2017-10-27 18:20:33 +00:00
Matthew Treinish fc0a11ebc4
Strip legacy prefix if it's present
Migrated jobs have the legacy- prefix which makes tracking things across
the migration a bit difficult. This commit strips that from the job
name on db insertion.

Change-Id: Ibfca7a7f4ff66ec6d3809f0ee96a98d598e27e91
2017-10-03 15:35:05 -04:00
Matthew Treinish 2a7070995e
Ensure that build_names are unique per project
With the migration to zuul v3 project names for common jobs, like unit
tests, no longer are unique per project. This causes issues for
openstack-health and subunit2sql because we can longer easily filter a
unit test job between projects. This commit adds a function to ensure
that the project name is in the build_name in the metadata. (assuming
devstack or tempest is not in the job_name) This way we can still filter
based on the job name per project.

Change-Id: I5231105af975d987d5f5e5f77c6ac038e10cd832
2017-10-03 15:34:18 -04:00
Colleen Murphy ebf675e02c Depend on helper gem for spec_helper_acceptance
Instead of keeping a local copy of spec_helper_acceptance.rb and
requiring updates to all modules for any change, we can move it into the
common helper gem and require it from there. This will make it easier to
create and review changes that affect all puppet modules. Also change
the Gemfile to look for the gem in the local workspace if running in a
zuul environment.

Change-Id: Ic6295486c9ec3c68c6dae90d5d22b01335c10895
2017-08-18 10:41:44 +02:00
Colleen Murphy 3f9d1f434a Update beaker setup for xenial
Add a xenial nodeset and update the spec helper to install puppet 3 from
the Ubuntu repos instead of from puppetlabs.

Change-Id: If76200210cf96861523979344d18637744dfdf5b
2017-06-11 20:23:26 +02:00
Jenkins c2beb42105 Merge "Add source_url to mqtt payload" 2017-04-28 21:05:06 +00:00
Matthew Treinish 35025f8df5
Move paho-mqtt package install to worker node
The paho-mqtt package is only ever needed on the worker because that is
what emits mqtt. However by putting it in init this also gets installed
on the server node. Additionally this commit adds a duplication check
around the package because it's possible that something else in another
module will conflict with paho-mqtt as we start adding mqtt to more
services.

Change-Id: Id0308d2d4d1843fcca73f459cffa2ae944bebd0c
2017-04-27 10:05:47 -04:00
Matthew Treinish c8b965bde6
Add source_url to mqtt payload
This commit adds the source_url to the mqtt payload for events emitted
by the subunit worker. Just the build_uuid isn't enough to figure out
what's going on because we typically do 2 events for each uuid, one with
a .gz extension and the other not. We need to distinguish between the 2
to figure out what's actually going on.

Change-Id: I56f0b9fb1128b412cd88e49164eb91c3e9e3e4cb
2017-04-26 17:01:25 -04:00
Matthew Treinish 4037e7b1ba
Handle mqtt errors on failures before out_event is set
There are some conditions where we fail before out_event is defined. In
those cases the mqtt publish won't work and will raise an exception
because out_event is not defined. This commit addresses that by using
fields instead of out_event. (which is the original source for
out_event) Fields exists earlier and build_uuid always remains the same
across the 2 dicts. Additionally we define fields to an empty dict
outside of the try to ensure that in the off chance of a failure before
fields is defined we don't fail sending a failure on mqtt.

Change-Id: I546a2ee42ea0f67f35c5e132d51f239a32a5582e
2017-04-25 14:18:09 -04:00
Matthew Treinish 5e8ec7ad3f
Config is a dict so use get not getattr
The config file is a yaml and parsed into a dict. However there was a
mistake in the recent patch that treated it as a ConfigParser object and
used getattr to access the mqtt options. This doesn't work and needs to
just be a get to access the key from the dict. This commit does that so
now it should work without crashing.

Change-Id: Ia3613adb6a037c9d406e6e33361d6d5ee826c9cd
2017-04-13 22:33:10 -04:00
Matthew Treinish c06cdc8a8a
Add MQTT support to the gearman worker
This commit adds support to the gearman worker for publishing an mqtt
message when processing a gearman job succeeds or fails. By default this
is disabled since it requires extra configuration to tell the worker how
to talk to the mqtt broker.

Right now the payload of the message is just the build_uuid and whether
it was written to the db or not. Eventually some details about the
subunit2sql db entry will be added to the payload. But this requires
changes to either subunit2sql or how the worker calls the subunit2sql
api before it is feasible.

Change-Id: Ibd13b737eccf52863a69d20843cb7d50242f7bb9
2017-04-10 16:08:10 -04:00
Matthew Treinish 9b5753123e
Fix puppet-lint failures
With the new release of puppet-lint being used the puppet-lint jobs have
started failing on this repo. This commit fixes the issues identified,
to unblock ci.

Change-Id: I63ba93ecdac473aabf402d022505266371c471cb
2017-04-04 12:39:21 -04:00
Jenkins 8fd36b64c4 Merge "Switch to storing run wall time in runs database" 2017-01-03 16:33:54 +00:00
Mikhail S Medvedev 4548b4114b Remove trailing slash so if-defined guard works
Both resource name in if statement and in declaration need to be exactly
the same.

Change-Id: Ie8635930e9d9520929865e59e3224819f487272a
2016-11-17 15:49:20 -06:00
Matthew Treinish 7f701e51ee
Switch to storing run wall time in runs database
This commit uses an option added in the subunit2sql 1.8.0 release to
set the run_time column in the runs table to be the wall time for the
subunit stream, in other words the stop time of the last test - the
start time of the first test. By default subunit2sql stores the sum of
all the individual test run times as the value for the run_time column
which is of questionable value for doing analysis. By switching what we
store for this value in the infra DB hopefully this data will be of more
use to people.

Change-Id: Ief4cfa91f9661444b3680b428da0a3c4ca5dedd0
2016-09-26 09:37:26 -04:00
Matthew Treinish 0e55e618ae Make sure global worker defs only done once
This commit makes sure that all the global definitions on the worker
class are only defined once. If we don't check for the previous
definition when running with multiple workers on a single node things
will fail.

Change-Id: I947d61f455778fe41d0a4723c99d029272d2155d
2016-07-26 15:23:45 -04:00
Spencer Krum a2e16a0f58 Use new infra_spec_helper for gem dependencies
Change-Id: I0d69168b0772cd9559870812f26683bb7e1f38d1
2016-06-21 18:40:25 -07:00
Philip Marc Schwartz 1c435df394 Update pip provider to use openstack_pip provider
Change-Id: I531d668bdc4f1bf080436b3f2648c7ff153dfe10
Signed-off-by: Philip Marc Schwartz <philip@progmad.com>
2016-06-16 10:20:56 -04:00
Matthew Treinish fa6b116168
Stop using a seperate retreiver thread
Since we recently switched from having 2 threads, 1 to retreive
subunit and 1 to process it, to having a single thread do both there
isn't any reason to launch to run that in a separate thread anymore.
This commit just removes the use of threading and runs everything in
the single process.

Change-Id: I5205fc73178b7d5a4bbee61e68b16b63499f5dd8
2016-04-20 16:13:26 -04:00
James E. Blair fa17a37a52 Make main thread non-daemon
This way it will wait to exit.

Change-Id: I2ee353b2c25e67bd87a42abf6b725867d4727b67
2016-04-20 13:09:11 -07:00
Jenkins b6bf50c001 Merge "Ensure all jobs send a completion packet" 2016-04-15 23:48:51 +00:00
Jenkins c078843f31 Merge "Make subunit2sql population single threaded" 2016-04-15 23:45:28 +00:00
Spencer Krum 458db8990c Pin google-api-client; sanitize Gemfile
Change-Id: I8d16b85915ceb2c02e55bbcdb5ccea100599e603
2016-04-14 15:24:32 -07:00
James E. Blair b8b253cfaf Ensure all jobs send a completion packet
In the job handler, aborted jobs would never send a work_complete
packet, which would cause the gearman server to track them
indefinitely.  Update the handler so that all jobs send either
a work_complete or work_exception.

Change-Id: Ia3b914762d46b1873888d5025e4ba86f9d042895
2016-04-13 16:14:09 -07:00
Matthew Treinish d31b5d9710
Make subunit2sql population single threaded
This commit switches the gearman worker to handle both retrieving the
subunit file and populating the DB with that data in a single thread.
This is as opposed to pushing it on an in memory queue and processing
the streams in a separate thread. This should have 2 major benefits
first it should make the behavior much more consistent, we'll only
send work completed to the gearman server after data is already in
the db. It should also significantly improve our memory consumption.

Change-Id: Ifaac2f150e26158b0a22e1c33e0c5d39c10182a4
2016-04-08 14:59:07 -04:00
Jenkins 224ed54b1e Merge "Fix log_url parsing for new job types" 2016-04-08 18:21:24 +00:00
Jenkins aff061f15c Merge "Remove inaccurate comment" 2016-03-10 20:59:06 +00:00
Jenkins 630ba7fad2 Merge "Switch use of cStringIO.StringIO to io.BytesIO" 2016-03-10 20:59:00 +00:00
Tim Buckley 3663cc2be0 Fix log_url parsing for new job types
Previously, the log_dir was determined by taking the dirname() of the
log_url twice, which works when all collected 'testrepository.subunit'
files were located under 'logs/'. Now that the worker is collecting
subunit files from all jobs, this assumption can result in the Zuul
UUID being cut off from the generated log_dir. This change adds a
check to make sure path segments are only removed when the log_dir
is 'logs/' or 'logs/old/'.

Change-Id: I75c53c498261e44989cdb7bf49d909ebde2b2699
2016-03-09 12:46:17 -07:00
Matthew Treinish 34fe9cc284
Ensure worker log dir is created
This commit makes sure we create the dir that the worker tries to write
it's logs to.

Change-Id: Id899742ae3efc5e821f46aa10cc722e16d9bf8cb
2016-03-07 12:09:14 -05:00
Matthew Treinish 5025c20960
Switch /etc/logstash to /etc/subunit2sql
This commit switches the use of the /etc/logstash directory to
/etc/subunit2sql. This was a leftover from when we first got the
subunit workers up and running and it was largely a copy and paste
of the policy for the logstash workers. Changing this should make it
more clear to users and consistent with the naming of everything
else.

Change-Id: Ie425dcf53856edd873ff1d0185dcf8e4ae30e3f8
2016-03-04 15:29:15 -05:00
Matthew Treinish 50c21b8c3f
Change /etc/logstash dir ownership to root
This commit updates the /etc/logstash config dir (where the
subunit2sql configs live, which makes so much sense) to be owned by
root. There is no logstash user ever created on the worker nodes so
we shouldn't try to assign it as the owner.

Change-Id: Ifd85cf4d33c85a958aa33fca50bdd847a651e560
2016-03-04 12:02:31 -05:00
Matthew Treinish 5d769819f5
Remove inaccurate comment
This commit removes an inaccurate comment that got copy and pasted
over when the script was originally created. The StdOut processor from
the logstash gearman client only works in foreground mode (for obvious
reasons) and since it was the simplest example that was used as a basis
for the subunit processor. However, the subunit processor doesn't share
this limitation and can (and has been) run as a daemon in the
background.

Change-Id: I332523567d16b0994b06e278b68ffe8bcb7d9bfa
2016-03-01 20:51:23 -05:00
Matthew Treinish ad3cc21c67
Switch use of cStringIO.StringIO to io.BytesIO
This commit switches all our uses of cStringIO.StringIO to io.BytesIO.
The StringIO class is the wrong class to use for this, especially
considering that subunit v2 is binary and not a string. This commit
fixes it so we're using the proper type.

Change-Id: I5d21f7ca2f40cbd5c2659f6cee1570ebe2d4a983
2016-03-01 19:59:30 -05:00