Commit Graph

323 Commits

Author SHA1 Message Date
Clark Boylan 278862b3f2 Retire this repo
The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.

Change-Id: I1d3a8a3999e75be00844a314fa7f52c37e28f9d4
2022-02-17 13:02:43 -08:00
Jeremy Stanley 63126bdbb8 Make v3 pidfile directory creation dynamic
Addition of Zuul v3 initscripts raced the dynamic pidfile directory
creation fix Idfd5d19e5171278b9bb7005edb4ae927a01f3615. Port it to
them so that if the PIDFILE variable is overridden for a custom
path, we create the correct parent directory.

Change-Id: I3af4e0eb280eed5dfeab909ad4e3fb24ede75a5a
2020-01-21 22:51:37 +00:00
Clark Boylan fb45aaaeec Add option for LE provisioned certs to zuul::web
We need to communicate to apache vhost config that we want LE certs to
be used in some cases. Add a vhosts_flags flag to signal this then plumb
the correct data into the vhost when set.

Note this requires the vhost name and the le cert name to match as that
is our mapping between puppet and ansibe here.

Change-Id: I40131525c10152ebb0a7495588f137e3c019bbc5
2020-01-06 12:53:25 -08:00
Clark Boylan b7d49bf548 Fix linter errors
Using rooted names (prefix with ::) is now a linter error because puppet
4 doesn't need this (see
  https://github.com/voxpupuli/puppet-lint-absolute_classname-check).
Fix occurences of this.

Change-Id: I4a3819ade0d9afcdc09a1a1ce4e975a4261e3cec
2020-01-06 12:53:25 -08:00
James E. Blair 9f5225752d Fix a typo with web_root
Apparently more @ is needed.

Change-Id: I9580873f1e4995662ee7616a7e80808e6ced9816
2019-08-09 09:12:58 -07:00
James E. Blair 67dc98c908 Add web_root option to zuul.conf
Change-Id: I8020c166b4896041abd063ddabc62780c5237915
2019-08-09 08:12:31 -07:00
James E. Blair 311e638d8a Update to new zuul JS content url
Also, remove the idempotency check since it's not compatible with
install-ara-safely.

Change-Id: If0d1f127644b0a897f2fd4869b635435b2f60e81
2019-07-24 16:18:40 -07:00
David Moreau Simard 7eca8bc3ee
Pin ARA to <1.0.0
ARA 1.0 will be released in the near future and isn't backwards
compatible. Pin it so it doesn't break things unexpectedly.

Change-Id: I6e239bb4b445163a3f0dd8c7557119b057da4ff0
2019-05-24 11:11:34 -04:00
David Shrewsbury 8d7153dbb3 Install objgraph package
The objgraph library is used to grab object counts for memory leak
detection. We already have this installed on zuul01, but that appears
to have been done manually. This cements the install.

Since puppet-nodepool requires the same packages (yappi and objgraph),
wrap both of the installs in a conditional check to prevent puppet
errors about being defined in multiple places.

Change-Id: I63ab5a21f7b98341bb203883f40c2df5404687ba
2019-04-22 14:22:10 -04:00
OpenDev Sysadmins eba82d5a99 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:25:38 +00:00
Ian Wienand a1d11f90fe Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Ic33f4903f6ca065b733960e3c476f591d5774044
2019-03-24 20:35:47 +00:00
James E. Blair 318aadb095 Install gear into executor Ansible environment
This is required by the subunit/logstash jobs

Change-Id: I73fff8fa61497eaf174a3a2bbd207c76b3c6915b
2019-03-19 14:40:43 -07:00
James E. Blair 5e9a106e6d Update executor ansible on installation
There are two deployment options for the new multi-ansible support:
the executor can install Ansible on startup (the default), or we
can pre-install ansible when we install Zuul, so that it's ready
for the next executor restart.  This change implements the latter.

Change-Id: I0c75724b4376f68128590eaabdd8a10ed63dcda0
2019-03-18 08:35:35 -07:00
James E. Blair 96641b0edb Set fingergw user to zuul
The user field no longer has a default, so if we want fingergw
to drop root privs, we need to set it.

Change-Id: I0bd16cbd23a30d2c643d1ac1840868e63f7adb4e
2019-03-18 07:13:07 -07:00
Clark Boylan e3fb3c27f2 Allow optional status_uri to be set for status backups
This updates the status_backups resource definition to allow for
specific uris to be set. This allows you to do per tenant backups
without tenant white labeling in the web server.

Change-Id: Ia94a16ef8b2a1069b9265bae2773cd323ac9a188
2019-03-12 14:34:37 -07:00
David Moreau Simard a6d9a5e4d4
Use $tenant_name instead of tenant_name in condition
This is probably why we are ending up with extra crons for the http
vhosts.

Change-Id: Ia3b261de051772ab765b3d7bc4b77a2a14e84397
2019-03-12 16:54:41 -04:00
Clark Boylan 375befe790 Don't rewrite t/ prefix on global zuul vhost
We want the t/ prefix to survive to requests to the backend when we are
hosting a global tenant UI. This rewrite appears to be causing 404s when
/t/tenant/status is rewritten to /tenant/status.

The upstream docs [0] also don't include this rewrite.

[0] https://zuul-ci.org/docs/zuul/admin/installation.html#static-offload

Change-Id: Id8f94581b9a1c8a383c7a0d40fb89ce49c50b2e3
2019-03-05 08:38:09 -08:00
Jeremy Stanley 7fdd302fd7 Use jemalloc on Zuul v3 executors
FreeBSD libc's malloc implementation, jemalloc, performs better than
glibc's for some tasks. In the case of our Zuul executors, we've
seen useful improvements in memory efficiency. Update the Puppet
module to install the libjemalloc1 package on Zuul executor servers
and export the necessary LD_PRELOAD envvar to use it when launching
executor daemons.

Change-Id: I8d8de3c118bd09d408b23a16482f71198a1232fc
2019-02-19 23:15:40 +00:00
Clark Boylan ebdb5fa51a Log github3 at DEBUG level
We are figuring out some github API interaction slowness and additional
api logging will help tremendously.

Change-Id: I80959b31d0fb8200a924d689e1bc27b9cde0ca31
2019-02-12 14:29:04 -08:00
Clark Boylan f85574113e Allow for both global zuul web and whitelabeled tenant
This updates the zuul web config to manage both global web server and
whitelabeled per tenant webserver configs. A new set of config hashes is
introduced to do this. For backward compatibility we construct the
hashes from existing parameters if the new hashes are unset.

Change-Id: Ie8ba46111530e74b8b1d0ec1746df2e09754fe67
2019-01-28 16:25:48 -08:00
James E. Blair 8d7b9c8ab4 Add relative_priority scheduler option
Change-Id: Ide901da7bc3068f3aafe3b35a110cc7c2de445b0
2018-11-30 08:42:07 -08:00
Tristan Cacqueray 425f80b2e2
Revert "Revert "Revert "Revert "Add rewrite rules for HTML5 url""""
This change also removes backward compat /status rewrite
The new URL scheme actually uses /status for the status page,
and it should be rewritten to the index.html below instead.

Change-Id: I7ff5a8387ec6fb65bda55d3233b670180682d08d
2018-10-09 05:58:06 +02:00
James E. Blair 002efe1bbd Revert "Revert "Revert "Add rewrite rules for HTML5 url"""
This reverts commit 0bdd3c9a15.

The js tarball publication job is broken:

http://logs.openstack.org/c7/c7148815378b9f26751ec0729a96894e7cd6b4c3/post/publish-openstack-javascript-content/05c6626/ara-report/result/72fa45d7-c12e-4af5-b520-5e1e73e4853b/

Change-Id: I76a50203968de07b5449fbf46c2dd2d227cc2cfe
2018-10-08 14:40:43 -07:00
James E. Blair 0bdd3c9a15 Revert "Revert "Add rewrite rules for HTML5 url""
This reverts commit ba6e0e77dc.

Change-Id: I8b06980c7b052e90422bc3ca0913e45eb3a5031c
2018-10-08 08:41:57 -07:00
James E. Blair ba6e0e77dc Revert "Add rewrite rules for HTML5 url"
This reverts commit ff1f91895d.

See https://review.openstack.org/606611 which reverted the react
rewrite.

Change-Id: I41f12b0733a7ba3f0dfff265f806f3562879a160
2018-09-29 11:52:30 -07:00
Tristan Cacqueray ff1f91895d Add rewrite rules for HTML5 url
This change makes the ProxyPass rewrites Last to stop as soon as they
match. Then any requests that doesn't match a file is served the
index.html.

Depends-On: https://review.openstack.org/591604
Change-Id: I363c2c4d3924074c8ff5c7fd9a410e357ed71b5d
2018-09-21 03:16:11 +00:00
Zuul 63784f1cf5 Merge "Install openstacksdk on zuul executors" 2018-07-30 16:30:51 +00:00
James E. Blair 4815bbbc98 Install openstacksdk on zuul executors
For use in uploading logs to swift.

Change-Id: I1302d960951b9358cd5d4c878194e9ad68036076
2018-07-27 15:24:06 -07:00
Paul Belanger 9f5d36b007
Stop deleting PIDFILE on stop
This is no longer needed as if zuul is properly configured, will
create / delete the pidfile directly.

Change-Id: I51e100a293411695a31d438b5450e52498df7800
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-07-27 17:22:37 -04:00
Paul Belanger c9b1ba6754
Make sure python3-yaml isn't installed
This is to deal with pip10 issue around python-yaml.

 | Jul 23 16:52:29 ze11 puppet-user[14344]: (/Stage[main]/Zuul/Exec[install_zuul]/returns) Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Change-Id: I3f7b209f4b8b5ed7f826369c9f8ad171cb530925
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-07-23 13:08:05 -04:00
Paul Belanger 7b94b3c7fc
Don't install python-yaml package
PyYAML is already a dependency of zuul, so the distro package doesn't
need to be installed. This attempts to clean pip10 issues with OS
packages.

Change-Id: Ia17aa61ab3c153c82aeae7b99747d142dd75ff61
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-07-20 20:50:55 -04:00
Colleen Murphy 707d10f47c Update Gemfile for Zuulv3
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.

Change-Id: I8d7185268ccde4d7fac867d9b92101294d1871bb
2018-07-12 09:57:46 +02:00
James E. Blair 3d46444494 Increase apache disk cache max size
This allows us to cache larger status.json files (which is needed
at OpenStack-infra scale).

Change-Id: I462ea29c9b5e45ced8110750a96f53a666e8623e
2018-06-06 21:01:59 +00:00
Zuul 3009251154 Merge "Fix scope of vhost template variables" 2018-06-05 22:27:05 +00:00
Zuul d2d648ca6c Merge "Fix operatingsystem major release fact name" 2018-06-05 22:22:16 +00:00
Clark Boylan 7f61fc38b0 Configure cherrypy logging
Now that zuul-web runs with cherrypy we need to update our logging to
include cherrypy logs. We start by only including errors for now.

This is based on the config details at:
https://docs.cherrypy.org/en/latest/basics.html#play-along-with-your-other-loggers

Change-Id: I870da2272f8cfe5ed0a862e5ac6aa1920ddaa2f7
2018-06-04 13:01:57 -07:00
James E. Blair 8e006bcab5 Add web listen arguments to zuul class
These are used by the zuul.conf template to write the zuul-web
configuration.  The template compares the values to the empty string
to decide if they should be written, since they are undefined, they
are always written, but with no value.  This adds them to the class
with empty-string defaults which should cause them to not appear in
the config file unless needed.

Change-Id: I065d18a0743179cb2c9827a8e6b1b70c169d9199
2018-06-04 10:31:26 -07:00
Colleen Murphy ced484d8c2 Fix scope of vhost template variables
Dynamic scoping for variables in ERB templates was removed in puppet
4[1] which means that the variables defined in the zuul and zuul::web
classes cannot be found when it is referenced in the httpd::vhost
defined type and will be evaluated as nil when puppet runs. Use
scope.lookupvar instead to be explicit about the variable's source.

[1] https://puppet.com/docs/puppet/4.10/lang_updating_manifests.html#dynamic-scoping-in-erb

Change-Id: I321cc7215387723c787ebc5d359b27866da26995
2018-04-27 19:17:48 +02:00
Colleen Murphy dabe03d77f Fix defined type namespacing
Without this patch, puppet 4 apply tests can't find the mod defined type
and fail with:

  Evaluation Error: Resource type not found: Mod at /etc/puppetlabs/code/modules/zuul/manifests/web.pp:284:15

This patch ensures the resource reference is namespaced just like all
the other resource references in the manifest already are.

Change-Id: Iac45cd1fe28fa9cb42751d2d879224ddc7f255ce
2018-04-27 17:22:23 +02:00
Colleen Murphy 7e57e2e6eb Install libre2-dev package
Commit 0c3b8fb96 of zuul added a dependency on fb-re2 which requires the
libre2-dev package to build. Ensure the puppet module installs it.

Change-Id: I5e66a5a2176012663add76ea5d68a22b04b93cf9
2018-04-26 23:29:19 +02:00
Colleen Murphy 2df532deaa Fix operatingsystem major release fact name
There is no such default fact as 'operatingsystemmajversion'. On puppet
3, this would not raise an error, but on puppet 4 this causes an error
when used in the versioncmp function which expects a string and not
undef. The correct name of the fact is 'operatingsystemmajrelease'.

Change-Id: Iafc4554cffd4a5406535e00e158af7bd04ca2f13
2018-04-26 20:40:10 +02:00
Monty Taylor b94e292457
Support updated REST and HTML layout
The previous layout meant we couldn't really do apache offloading of
static content and instead were proxying them to the aiohttp server.

The new update moves api calls below /api. Update the rewrite rules to
deal with this. Also, update the document root to point to where we
deploy the zuul dashboard so that it's picked up by apache.

This also updates the apache config to work for non-whitelabeled deploys
as well.

Depends-On: https://review.openstack.org/556967
Change-Id: I6e662e7bc97de96969828801bcb18c5f354965c5
2018-03-27 19:04:22 -05:00
Monty Taylor fa66845b5b
Split v2 and v3 web config
In v3, we need to be able to use some variables from zuul/web.pp in the
vhost template, but the vhost is created in ::zuul because of how v2
worked.

Split the config, and split where we're defining it so that in the next
patch we can make use of variables from zuul/web.pp.

Change-Id: I3e1c72bb773be050854f5563d09f6c19af8bc6dc
Depends-On: https://review.openstack.org/557085
2018-03-27 19:04:13 -05:00
Monty Taylor ad7b7953d0
Add new parameters to zuul::web
In the next patch we'll start using them, but we need to be able to add
them to system-config so that the acceptance tests will run.

Add the parameters but don't do anything with them yet.

Change-Id: I4a2062fc35f8861267f8b8ce3e6d0e9f1d6cffdc
2018-03-27 17:40:40 -05:00
Ian Wienand 08a3c5f772 Don't remove libjs-jquery
Other packages, such as bup (for backups) depend on this, so removing
it also removes them.  Just leave it alone.

Change-Id: I4d3d69f943cf5f625ff1d2af475e6a6ed0dfa411
2018-03-16 18:06:27 +11:00
Zuul b9382ddaf2 Merge "Remove 'restart' for v3 executor" 2018-03-12 18:45:06 +00:00
Monty Taylor 2fb12bb339
Update zuul-web install for javascript stack
With the update to the use of yarn/webpack for the zuul-web javascript
stack, there is now a pre-built tarball of the web assets. Update the
code to use it.

An alternative to this approach would be to install nodejs and yarn on
the machine and just do a pip install . in the zuul source dir which
would also work ... but since there is an option that doesn't require
the need to run the build tools, why not use it?

Because spec/acceptance/fixtures/default.pp is hard-coded to the
zuulv3 install path, we can remove the now unnecessary checks from it.

Move the status backups to /var/lib/zuul/backup, instead of www/backup
as the www directory no longer exists.  Also remove it as the
documentroot.

Depends-On: https://review.openstack.org/547790
Change-Id: Ib88b48ddea26ef5d8d0d152774112844b180f77e
2018-03-07 18:22:47 -06:00
Monty Taylor 8f68109b2e
Move unanchored general rules after specific ones
The match ^/jobs doesn't have an ending anchor, so it matches /jobs
and /jobs.html - which makes the url
https://zuul.openstack.org/jobs.html return JSON. While the JSON is
valid job data, raw JSON is less pleasant for a human to read than HTML
(go figure)

Move the less specific entries to after the more specific entries.

Change-Id: I80f7fbe54d96cc151ef8da28465445f93092aadd
2018-02-20 06:38:01 -06:00
Monty Taylor 7e99dca02d Update apache and backups for json suffix removal
Zuul is about to remove .json suffixes from the REST endpoints. Update
the puppet to deal with that.

Change-Id: Ic1df0c01eec57f163ce332518ebc560e77b92bf3
Depends-On: https://review.openstack.org/537010
2018-02-19 12:08:34 -08:00
Monty Taylor eb2a5f81b1
Add support for specifying a tenant explicitly
OpenStack white-labels the dashboard specifically at the OpenStack
level, but there are some URLs, like the github connection payload
webhook receiver, that are global and not per-tenant. It's conceivable
that we might want to add richer support for single-tenant whitelabels
as well.

Change-Id: I03ea09e86168ed4e896f74db1b715131ad61f600
2018-02-15 17:13:17 -06:00