This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: I7ad3e225464d21116d448df13d8de7fb842b3745
Add tests to ensure the subunit2sql server and worker classes work and
that the worker service can start.
Change-Id: I842ecbcd5988484043643fdae1c49dd95bba90f2
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.
Change-Id: I363a112ee856a9f5eb986fd878730db082c26a81
The exec to install subunit2sql tries to compare the output of pip list
versus pip search to determine whether it is on the latest version.
Without this patch, the pip list command produces a string that has too
many spaces on the end and therefore doesn't match the result of the pip
search command. This means that it will always try to reinstall
subunit2sql even when it doesn't need to, which breaks idempotency
tests. This patch fixes the regex group to exclude trailing spaces.
Change-Id: I92f614db2da936d68d8f2305b348c65ba773b01c
The os-performance-tools library currently caps its statsd dependency at
<3.0[1]. If we try to install latest, puppet will succeed at installing
it in the first place, but upon the next run it will see that it is not
using the latest and try to upgrade and fail due to the cap. Add the pin
so that it doesn't try and fail to install something it can't.
[1] http://git.openstack.org/cgit/openstack/os-performance-tools/tree/requirements.txt?id=d96d5145fa5facdfd7a4e5ea2811948f0efd465e#n7
Change-Id: I37303d3b23cf5c387511292a8e128faa1f9df9ae
PyYAML is already a dependency of subunit2sql, so the distro package
doesn't need to be installed explicitly. If it is installed by the
distro, then on a system with pip 10 the subunit2sql installation will
fail.
Change-Id: Ie7be5cf01fcf9a630c7bf282549a862515c159d9
Using local ruby variables in puppet ERB templates was deprecated in
puppet 3 and doesn't work any more in puppet 4. Access these values as
ruby instance variables instead.
Change-Id: I6ea7b30d7a890b73f8a4ce6572751d0ecfb35667
When testing to see whether the installed subunit2sql package
matches the version available on PyPI, the first argument can be
empty if the package has not yet been installed. Make sure these
arguments are quoted so that the test command gets a syntax it can
parse (rather than a string starting with a binary operator).
Change-Id: Ia2e7165f851085328e00431a739cce17d2e494b2
In at least some circumstances, our pip is installed into
/usr/local/bin to make sure that's added to the environment and
switch to dynamic path resolution when calling pip.
Change-Id: I4e53ce2aba1b8335025e180bd03a5f5e93461814
When installing a custom initscript for a worker daemon, reload
systemd so it will pick that up before attempting to start the
service. Wrap it in a conditional check for Ubuntu 16.04 or later,
since earlier releases used upstart as their init instead of
systemd. Make sure the initscript template uses a distinct
"provides" key for each instance of the service as well.
This implementation is copied from the puppet-zuul module.
Change-Id: I4f7db205ac4ae29953757220dc0b9ae026ebbc71
When installing or upgrading subunit2sql, override pip's default
upgrade strategy to avoid updating preinstalled dependencies if they
already satisfy the requirements. This in particular avoids
attempting to replace the distro package of python-netifaces by
rebuilding a newer sdist (it is not distributed as a wheel on PyPI
and needs C extensions compiled, which in turn have a lot of
build-time deps).
This implementation is copied from the puppet-zuul module.
Change-Id: I885a61527ca9938200fb965ea97ff957fed3c480
In order to make sure pip installation of subunit2sql does not
attempt to pip install (and subsequently build from sdist)
netifaces, set a dependency ensuring the distro package for
python-netifaces is installed before that.
Change-Id: I01118c2771aa4e4fa553fb04b4a0885bf977a738
Since subunit2sql requires oslo.db which requires oslo.utils which
requires netifaces which builds C extensions and does not provide
prebuilt wheels for any platform other than Windows, install
Ubuntu's python-netifaces package to avoid needing an entire build
toolchain. We're moving the servers to Xenial, which provides a
version compatible with the current oslo.utils requirements list.
Change-Id: I4c701f08cf11bc068f6ebb10534f7e6f865970bb
Migrated jobs have the legacy- prefix which makes tracking things across
the migration a bit difficult. This commit strips that from the job
name on db insertion.
Change-Id: Ibfca7a7f4ff66ec6d3809f0ee96a98d598e27e91
With the migration to zuul v3 project names for common jobs, like unit
tests, no longer are unique per project. This causes issues for
openstack-health and subunit2sql because we can longer easily filter a
unit test job between projects. This commit adds a function to ensure
that the project name is in the build_name in the metadata. (assuming
devstack or tempest is not in the job_name) This way we can still filter
based on the job name per project.
Change-Id: I5231105af975d987d5f5e5f77c6ac038e10cd832
Instead of keeping a local copy of spec_helper_acceptance.rb and
requiring updates to all modules for any change, we can move it into the
common helper gem and require it from there. This will make it easier to
create and review changes that affect all puppet modules. Also change
the Gemfile to look for the gem in the local workspace if running in a
zuul environment.
Change-Id: Ic6295486c9ec3c68c6dae90d5d22b01335c10895
Add a xenial nodeset and update the spec helper to install puppet 3 from
the Ubuntu repos instead of from puppetlabs.
Change-Id: If76200210cf96861523979344d18637744dfdf5b
The paho-mqtt package is only ever needed on the worker because that is
what emits mqtt. However by putting it in init this also gets installed
on the server node. Additionally this commit adds a duplication check
around the package because it's possible that something else in another
module will conflict with paho-mqtt as we start adding mqtt to more
services.
Change-Id: Id0308d2d4d1843fcca73f459cffa2ae944bebd0c
This commit adds the source_url to the mqtt payload for events emitted
by the subunit worker. Just the build_uuid isn't enough to figure out
what's going on because we typically do 2 events for each uuid, one with
a .gz extension and the other not. We need to distinguish between the 2
to figure out what's actually going on.
Change-Id: I56f0b9fb1128b412cd88e49164eb91c3e9e3e4cb
There are some conditions where we fail before out_event is defined. In
those cases the mqtt publish won't work and will raise an exception
because out_event is not defined. This commit addresses that by using
fields instead of out_event. (which is the original source for
out_event) Fields exists earlier and build_uuid always remains the same
across the 2 dicts. Additionally we define fields to an empty dict
outside of the try to ensure that in the off chance of a failure before
fields is defined we don't fail sending a failure on mqtt.
Change-Id: I546a2ee42ea0f67f35c5e132d51f239a32a5582e
The config file is a yaml and parsed into a dict. However there was a
mistake in the recent patch that treated it as a ConfigParser object and
used getattr to access the mqtt options. This doesn't work and needs to
just be a get to access the key from the dict. This commit does that so
now it should work without crashing.
Change-Id: Ia3613adb6a037c9d406e6e33361d6d5ee826c9cd
This commit adds support to the gearman worker for publishing an mqtt
message when processing a gearman job succeeds or fails. By default this
is disabled since it requires extra configuration to tell the worker how
to talk to the mqtt broker.
Right now the payload of the message is just the build_uuid and whether
it was written to the db or not. Eventually some details about the
subunit2sql db entry will be added to the payload. But this requires
changes to either subunit2sql or how the worker calls the subunit2sql
api before it is feasible.
Change-Id: Ibd13b737eccf52863a69d20843cb7d50242f7bb9
With the new release of puppet-lint being used the puppet-lint jobs have
started failing on this repo. This commit fixes the issues identified,
to unblock ci.
Change-Id: I63ba93ecdac473aabf402d022505266371c471cb
This commit uses an option added in the subunit2sql 1.8.0 release to
set the run_time column in the runs table to be the wall time for the
subunit stream, in other words the stop time of the last test - the
start time of the first test. By default subunit2sql stores the sum of
all the individual test run times as the value for the run_time column
which is of questionable value for doing analysis. By switching what we
store for this value in the infra DB hopefully this data will be of more
use to people.
Change-Id: Ief4cfa91f9661444b3680b428da0a3c4ca5dedd0
This commit makes sure that all the global definitions on the worker
class are only defined once. If we don't check for the previous
definition when running with multiple workers on a single node things
will fail.
Change-Id: I947d61f455778fe41d0a4723c99d029272d2155d
Since we recently switched from having 2 threads, 1 to retreive
subunit and 1 to process it, to having a single thread do both there
isn't any reason to launch to run that in a separate thread anymore.
This commit just removes the use of threading and runs everything in
the single process.
Change-Id: I5205fc73178b7d5a4bbee61e68b16b63499f5dd8
In the job handler, aborted jobs would never send a work_complete
packet, which would cause the gearman server to track them
indefinitely. Update the handler so that all jobs send either
a work_complete or work_exception.
Change-Id: Ia3b914762d46b1873888d5025e4ba86f9d042895
This commit switches the gearman worker to handle both retrieving the
subunit file and populating the DB with that data in a single thread.
This is as opposed to pushing it on an in memory queue and processing
the streams in a separate thread. This should have 2 major benefits
first it should make the behavior much more consistent, we'll only
send work completed to the gearman server after data is already in
the db. It should also significantly improve our memory consumption.
Change-Id: Ifaac2f150e26158b0a22e1c33e0c5d39c10182a4
Previously, the log_dir was determined by taking the dirname() of the
log_url twice, which works when all collected 'testrepository.subunit'
files were located under 'logs/'. Now that the worker is collecting
subunit files from all jobs, this assumption can result in the Zuul
UUID being cut off from the generated log_dir. This change adds a
check to make sure path segments are only removed when the log_dir
is 'logs/' or 'logs/old/'.
Change-Id: I75c53c498261e44989cdb7bf49d909ebde2b2699
This commit switches the use of the /etc/logstash directory to
/etc/subunit2sql. This was a leftover from when we first got the
subunit workers up and running and it was largely a copy and paste
of the policy for the logstash workers. Changing this should make it
more clear to users and consistent with the naming of everything
else.
Change-Id: Ie425dcf53856edd873ff1d0185dcf8e4ae30e3f8
This commit updates the /etc/logstash config dir (where the
subunit2sql configs live, which makes so much sense) to be owned by
root. There is no logstash user ever created on the worker nodes so
we shouldn't try to assign it as the owner.
Change-Id: Ifd85cf4d33c85a958aa33fca50bdd847a651e560
This commit removes an inaccurate comment that got copy and pasted
over when the script was originally created. The StdOut processor from
the logstash gearman client only works in foreground mode (for obvious
reasons) and since it was the simplest example that was used as a basis
for the subunit processor. However, the subunit processor doesn't share
this limitation and can (and has been) run as a daemon in the
background.
Change-Id: I332523567d16b0994b06e278b68ffe8bcb7d9bfa
This commit switches all our uses of cStringIO.StringIO to io.BytesIO.
The StringIO class is the wrong class to use for this, especially
considering that subunit v2 is binary and not a string. This commit
fixes it so we're using the proper type.
Change-Id: I5d21f7ca2f40cbd5c2659f6cee1570ebe2d4a983