Commit Graph

202 Commits

Author SHA1 Message Date
Clark Boylan 278862b3f2 Retire this repo
The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.

Change-Id: I1d3a8a3999e75be00844a314fa7f52c37e28f9d4
2022-02-17 13:02:43 -08:00
Clark Boylan fb45aaaeec Add option for LE provisioned certs to zuul::web
We need to communicate to apache vhost config that we want LE certs to
be used in some cases. Add a vhosts_flags flag to signal this then plumb
the correct data into the vhost when set.

Note this requires the vhost name and the le cert name to match as that
is our mapping between puppet and ansibe here.

Change-Id: I40131525c10152ebb0a7495588f137e3c019bbc5
2020-01-06 12:53:25 -08:00
Clark Boylan b7d49bf548 Fix linter errors
Using rooted names (prefix with ::) is now a linter error because puppet
4 doesn't need this (see
  https://github.com/voxpupuli/puppet-lint-absolute_classname-check).
Fix occurences of this.

Change-Id: I4a3819ade0d9afcdc09a1a1ce4e975a4261e3cec
2020-01-06 12:53:25 -08:00
James E. Blair 67dc98c908 Add web_root option to zuul.conf
Change-Id: I8020c166b4896041abd063ddabc62780c5237915
2019-08-09 08:12:31 -07:00
James E. Blair 311e638d8a Update to new zuul JS content url
Also, remove the idempotency check since it's not compatible with
install-ara-safely.

Change-Id: If0d1f127644b0a897f2fd4869b635435b2f60e81
2019-07-24 16:18:40 -07:00
David Moreau Simard 7eca8bc3ee
Pin ARA to <1.0.0
ARA 1.0 will be released in the near future and isn't backwards
compatible. Pin it so it doesn't break things unexpectedly.

Change-Id: I6e239bb4b445163a3f0dd8c7557119b057da4ff0
2019-05-24 11:11:34 -04:00
David Shrewsbury 8d7153dbb3 Install objgraph package
The objgraph library is used to grab object counts for memory leak
detection. We already have this installed on zuul01, but that appears
to have been done manually. This cements the install.

Since puppet-nodepool requires the same packages (yappi and objgraph),
wrap both of the installs in a conditional check to prevent puppet
errors about being defined in multiple places.

Change-Id: I63ab5a21f7b98341bb203883f40c2df5404687ba
2019-04-22 14:22:10 -04:00
James E. Blair 318aadb095 Install gear into executor Ansible environment
This is required by the subunit/logstash jobs

Change-Id: I73fff8fa61497eaf174a3a2bbd207c76b3c6915b
2019-03-19 14:40:43 -07:00
James E. Blair 5e9a106e6d Update executor ansible on installation
There are two deployment options for the new multi-ansible support:
the executor can install Ansible on startup (the default), or we
can pre-install ansible when we install Zuul, so that it's ready
for the next executor restart.  This change implements the latter.

Change-Id: I0c75724b4376f68128590eaabdd8a10ed63dcda0
2019-03-18 08:35:35 -07:00
Clark Boylan e3fb3c27f2 Allow optional status_uri to be set for status backups
This updates the status_backups resource definition to allow for
specific uris to be set. This allows you to do per tenant backups
without tenant white labeling in the web server.

Change-Id: Ia94a16ef8b2a1069b9265bae2773cd323ac9a188
2019-03-12 14:34:37 -07:00
David Moreau Simard a6d9a5e4d4
Use $tenant_name instead of tenant_name in condition
This is probably why we are ending up with extra crons for the http
vhosts.

Change-Id: Ia3b261de051772ab765b3d7bc4b77a2a14e84397
2019-03-12 16:54:41 -04:00
Jeremy Stanley 7fdd302fd7 Use jemalloc on Zuul v3 executors
FreeBSD libc's malloc implementation, jemalloc, performs better than
glibc's for some tasks. In the case of our Zuul executors, we've
seen useful improvements in memory efficiency. Update the Puppet
module to install the libjemalloc1 package on Zuul executor servers
and export the necessary LD_PRELOAD envvar to use it when launching
executor daemons.

Change-Id: I8d8de3c118bd09d408b23a16482f71198a1232fc
2019-02-19 23:15:40 +00:00
Clark Boylan f85574113e Allow for both global zuul web and whitelabeled tenant
This updates the zuul web config to manage both global web server and
whitelabeled per tenant webserver configs. A new set of config hashes is
introduced to do this. For backward compatibility we construct the
hashes from existing parameters if the new hashes are unset.

Change-Id: Ie8ba46111530e74b8b1d0ec1746df2e09754fe67
2019-01-28 16:25:48 -08:00
James E. Blair 8d7b9c8ab4 Add relative_priority scheduler option
Change-Id: Ide901da7bc3068f3aafe3b35a110cc7c2de445b0
2018-11-30 08:42:07 -08:00
James E. Blair 4815bbbc98 Install openstacksdk on zuul executors
For use in uploading logs to swift.

Change-Id: I1302d960951b9358cd5d4c878194e9ad68036076
2018-07-27 15:24:06 -07:00
Paul Belanger c9b1ba6754
Make sure python3-yaml isn't installed
This is to deal with pip10 issue around python-yaml.

 | Jul 23 16:52:29 ze11 puppet-user[14344]: (/Stage[main]/Zuul/Exec[install_zuul]/returns) Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Change-Id: I3f7b209f4b8b5ed7f826369c9f8ad171cb530925
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-07-23 13:08:05 -04:00
Paul Belanger 7b94b3c7fc
Don't install python-yaml package
PyYAML is already a dependency of zuul, so the distro package doesn't
need to be installed. This attempts to clean pip10 issues with OS
packages.

Change-Id: Ia17aa61ab3c153c82aeae7b99747d142dd75ff61
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-07-20 20:50:55 -04:00
Zuul d2d648ca6c Merge "Fix operatingsystem major release fact name" 2018-06-05 22:22:16 +00:00
James E. Blair 8e006bcab5 Add web listen arguments to zuul class
These are used by the zuul.conf template to write the zuul-web
configuration.  The template compares the values to the empty string
to decide if they should be written, since they are undefined, they
are always written, but with no value.  This adds them to the class
with empty-string defaults which should cause them to not appear in
the config file unless needed.

Change-Id: I065d18a0743179cb2c9827a8e6b1b70c169d9199
2018-06-04 10:31:26 -07:00
Colleen Murphy dabe03d77f Fix defined type namespacing
Without this patch, puppet 4 apply tests can't find the mod defined type
and fail with:

  Evaluation Error: Resource type not found: Mod at /etc/puppetlabs/code/modules/zuul/manifests/web.pp:284:15

This patch ensures the resource reference is namespaced just like all
the other resource references in the manifest already are.

Change-Id: Iac45cd1fe28fa9cb42751d2d879224ddc7f255ce
2018-04-27 17:22:23 +02:00
Colleen Murphy 7e57e2e6eb Install libre2-dev package
Commit 0c3b8fb96 of zuul added a dependency on fb-re2 which requires the
libre2-dev package to build. Ensure the puppet module installs it.

Change-Id: I5e66a5a2176012663add76ea5d68a22b04b93cf9
2018-04-26 23:29:19 +02:00
Colleen Murphy 2df532deaa Fix operatingsystem major release fact name
There is no such default fact as 'operatingsystemmajversion'. On puppet
3, this would not raise an error, but on puppet 4 this causes an error
when used in the versioncmp function which expects a string and not
undef. The correct name of the fact is 'operatingsystemmajrelease'.

Change-Id: Iafc4554cffd4a5406535e00e158af7bd04ca2f13
2018-04-26 20:40:10 +02:00
Monty Taylor b94e292457
Support updated REST and HTML layout
The previous layout meant we couldn't really do apache offloading of
static content and instead were proxying them to the aiohttp server.

The new update moves api calls below /api. Update the rewrite rules to
deal with this. Also, update the document root to point to where we
deploy the zuul dashboard so that it's picked up by apache.

This also updates the apache config to work for non-whitelabeled deploys
as well.

Depends-On: https://review.openstack.org/556967
Change-Id: I6e662e7bc97de96969828801bcb18c5f354965c5
2018-03-27 19:04:22 -05:00
Monty Taylor fa66845b5b
Split v2 and v3 web config
In v3, we need to be able to use some variables from zuul/web.pp in the
vhost template, but the vhost is created in ::zuul because of how v2
worked.

Split the config, and split where we're defining it so that in the next
patch we can make use of variables from zuul/web.pp.

Change-Id: I3e1c72bb773be050854f5563d09f6c19af8bc6dc
Depends-On: https://review.openstack.org/557085
2018-03-27 19:04:13 -05:00
Monty Taylor ad7b7953d0
Add new parameters to zuul::web
In the next patch we'll start using them, but we need to be able to add
them to system-config so that the acceptance tests will run.

Add the parameters but don't do anything with them yet.

Change-Id: I4a2062fc35f8861267f8b8ce3e6d0e9f1d6cffdc
2018-03-27 17:40:40 -05:00
Ian Wienand 08a3c5f772 Don't remove libjs-jquery
Other packages, such as bup (for backups) depend on this, so removing
it also removes them.  Just leave it alone.

Change-Id: I4d3d69f943cf5f625ff1d2af475e6a6ed0dfa411
2018-03-16 18:06:27 +11:00
Monty Taylor 2fb12bb339
Update zuul-web install for javascript stack
With the update to the use of yarn/webpack for the zuul-web javascript
stack, there is now a pre-built tarball of the web assets. Update the
code to use it.

An alternative to this approach would be to install nodejs and yarn on
the machine and just do a pip install . in the zuul source dir which
would also work ... but since there is an option that doesn't require
the need to run the build tools, why not use it?

Because spec/acceptance/fixtures/default.pp is hard-coded to the
zuulv3 install path, we can remove the now unnecessary checks from it.

Move the status backups to /var/lib/zuul/backup, instead of www/backup
as the www directory no longer exists.  Also remove it as the
documentroot.

Depends-On: https://review.openstack.org/547790
Change-Id: Ib88b48ddea26ef5d8d0d152774112844b180f77e
2018-03-07 18:22:47 -06:00
Monty Taylor 7e99dca02d Update apache and backups for json suffix removal
Zuul is about to remove .json suffixes from the REST endpoints. Update
the puppet to deal with that.

Change-Id: Ic1df0c01eec57f163ce332518ebc560e77b92bf3
Depends-On: https://review.openstack.org/537010
2018-02-19 12:08:34 -08:00
Monty Taylor eb2a5f81b1
Add support for specifying a tenant explicitly
OpenStack white-labels the dashboard specifically at the OpenStack
level, but there are some URLs, like the github connection payload
webhook receiver, that are global and not per-tenant. It's conceivable
that we might want to add richer support for single-tenant whitelabels
as well.

Change-Id: I03ea09e86168ed4e896f74db1b715131ad61f600
2018-02-15 17:13:17 -06:00
Clark Boylan ab65041a1a Fix whitespace in ara install version checking
We have extra whitespace in the pip installed package listing that we
don't have in the upstream pypi package search so our string comparison
of the versions breaks. Fix this by trimming extra whitespace.

Change-Id: I04b28508ff9371397d42957f0d086aa9bb63a4f7
2018-01-20 18:23:33 +00:00
James E. Blair 241a72bbd6 Add TODO about fixing ara idempotency
Change-Id: If5ca46e09a59108ec6e5b68c8e23ed3a410d0fca
2018-01-20 18:23:29 +00:00
James E. Blair 73ee7872cc Redirect curl stderr to /dev/null
It's chatty.

Change-Id: I06a167e07fa8e2ac78d7c0bd3a1c43cd56788009
2018-01-16 08:56:58 -08:00
Zuul 86034f28a6 Merge "Periodically retrieve and back up Zuul's status.json" 2018-01-16 15:25:32 +00:00
Zuul 8bf71446c4 Merge "Update merger / scheduler init script for zuulv3" 2018-01-16 15:18:29 +00:00
David Moreau Simard 6df6a3ab81
Periodically retrieve and back up Zuul's status.json
Since the contents of the status.json is only available in-memory from
the Zuul scheduler, the contents are lost forever if the scheduler,
or the server it is hosted on, crash.

This implements two crons:
1) A curl to download a timestamped status.json, if it is available
2) File deletion so we keep no more than 180 timestamped status files
   at any time.

The files are stored at a location within the scope of the zuul-web
vhost in order to make it easily possible to dump queues with something
like:

    /opt/zuul/tools/zuul-changes.py http://zuulv3.openstack.org/backup/status_timestamp.json

Change-Id: Ia1923bfae0222559b8d2dcb5e0618848b3a618d0
2018-01-15 18:10:31 -05:00
James E. Blair 22c76841cb Add support for zuul-fingergw
This is a new service which centrally proxies log streaming.

Change-Id: I1129be61c0a77dbd46e402204ea8c4da2889b016
2017-12-21 13:20:36 -08:00
Zuul cf62a0f430 Merge "Ensure ara is updated on executors" 2017-12-18 21:12:53 +00:00
James E. Blair b44e05bc11 Ensure ara is updated on executors
Depends-On: Id4f6ae8cc4128083f0db72fb6e2dc81b9968e884
Change-Id: I0feb48b8b12d2346c328ab9c16aaa45dd36ae0bf
2017-12-18 19:31:32 +00:00
James E. Blair a405aafdf5 Update Zuulv3 url rewrites
This updates the rewrite rules to the current state of the zuulv3
dashboard, and removes the overlap with zuul v2 urls.

Change-Id: Idd84facca4a0e0170234bf7d4ee2b42811f52b67
2017-12-08 14:48:15 -08:00
Monty Taylor 18afb9dd6f Install angular.js and other static files for zuul web
Until we have the more generalized javascript tooling in place for zuul,
we need to add libraries directly here. The zuul web console requires
angular, so add it.

Some of the same dependencies are used in zuulv2 and zuulv3, but in a
different location.  Symlink them as appropriate

Change-Id: I75d5263ab2af3666d162b66394a537dfbe88010c
2017-12-08 14:17:38 -08:00
Paul Belanger 701acbfdbf
Update merger / scheduler init script for zuulv3
Zuulv3 now suport command socket to stop both merger and schedulers,
add logic to install new init scripts for them depending on zuulv2 or
zuulv3.

Change-Id: Iecc8c03896c145d29e42e6be6194d7db893c8515
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-12-07 15:31:05 -05:00
James E. Blair 0bfe6062a0 Zuul v3: add per-service default files
We're moving the default pidfile location in
https://review.openstack.org/517381

This change creates default files so that when running v3, the
new location is used.  The init scripts themselves are still
compatible with v2 and v3.

Change-Id: I57c486f4506114eb0447be2014baa42fc22cc8a2
2017-11-20 15:27:45 -08:00
Monty Taylor f410a47ae7
Split zuul and requirements install
We have some git-based requirements which pip installing zuul itself
doesn't get quite right.

Change-Id: I42a961d325bda9271d627c2ae4f3a6b61d8e601d
2017-10-12 17:44:24 -05:00
James E. Blair 0ec00b15bc Add support for zookeeper timeout
Change-Id: I743bfa28d2faaa93bba6fc975d575696bd8a5a70
2017-09-28 10:40:14 -07:00
James E. Blair 1f43ca60a9 Add gerrit_strip_branch_ref option
Change-Id: Id388274a7b628ad40a0e943087ec82596791c982
See: I012485065e0fd8692e4f2ef787dd0a62be303314
2017-09-19 07:51:34 -07:00
Monty Taylor d9cefce112
Add python-bpo-27945-backport PPA to executor
The python-bpo-27945-backport contains a backport of the patch that
fixes http://bugs.python.org/issue27945 and is a stop-gap while we wait
for the SRU
https://bugs.launchpad.net/ubuntu/+source/python3.5/+bug/1711724 to land
in Xenial.

Change-Id: Iaea633b8672110e49928a20d596b884c609d6345
2017-08-18 12:58:15 -05:00
Jenkins 43cf9a5990 Merge "Install ara on the executors" 2017-08-17 18:29:34 +00:00
James E. Blair a7ef39c2c0 Zuulv3: move the job dir under /var/lib/zuul
We want to ensure that our jobdir and the git repo cache are
on the same filesystem.  Configure the executors to use a dir
under /var/lib/zuul for the jobdir.  We will mount a new (large)
filesystem at /var/lib/zuul to accomodate both.

Change-Id: I91472fa609274aadcf4945aad44f25014c07b105
2017-08-16 10:10:28 -07:00
Monty Taylor c93f2cba85
Install ara on the executors
zuul v3 supports optionally collecting data via ara if it's installed.
Install it so that we can produce ara html reports for all of our runs.

The current implementation is similar to the devstack-gate
implementation - it uses a sqlite db per job. The bubblewrap invocation
sets $HOME to the work_dir, so the sqlite db for each job goes in to the
root of the work_dir with no further config.

Change-Id: Iefeda37338c57466d671ea07e7d318fa92aafc35
Depends-On: I8facdf0b95b83d43c337058d70fe6bf71e17d570
2017-08-16 11:14:46 -05:00
Paul Belanger 0cc4c05cbd Move zuul site variables file into init.pp
Because we need to update zuul.conf with the path to the file, we
actually need to configure this variable within init.pp.

Change-Id: I734f2c338ceab2b1da2a3245423cb912b7dd8c00
Depends-On: I7983ff9cfaea1ee6d3b099824b5f31df98ce72a5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-03 21:02:54 -04:00