Commit Graph

224 Commits

Author SHA1 Message Date
Clark Boylan 86a69a5b4d Retire this repo
The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.

Change-Id: I46ccab87f063c51dd3236757dbb0567738803fae
2022-02-17 12:46:57 -08:00
Clark Boylan 2c4f42cecd Add a warning to the third party CI doc
The third party ci doc is out of date and not currently working. It
needs help. Point to our spec that provides guidance on how to apply
that help as well as poitning to Zuul's documentation for setting up
Zuul.

Change-Id: I7365185f15f6f2cb2e3a9f6757897e793de98181
2019-10-17 19:27:53 +00:00
Andreas Jaeger f494aa8762 Handle robots.txt on logs.o.o
We don't want to delete the top-level robots.txt file - but everything
that is part of the logs hierarchy.

Change-Id: I369565a780eb749eab95e4bf8e7d8d7fd5bf0630
2019-08-07 14:04:57 +00:00
Ian Wienand 329f330b3e Log archive maintenance; also clear out symlinks
Currently if a log directory has a symlink it is skipped in the find;
thus the directory does not become empty and it is never removed.
Also match symlinks.

Change-Id: Id34c74d848638182819a9c81d051b83820244463
2019-08-06 19:13:44 +10:00
Jeremy Stanley f4c637c738 Pin ara at 0.16.5 for logserver
With the ARA 1.0.0 release, it has been rearchitected in ways which
require more extensive changes to the way it's deployed. For now,
pin it to use the previous version.

Change-Id: Iadfc699e6a4ce1de1a90bf4accb916454b788df0
2019-06-14 18:44:12 +00:00
Clark Boylan e266e84982 Remove idempotency spec checks
The idempotency checks race external changes like updates to
project-config or nodepool. This often causes this job to fail when we
are busy updating other parts of our sytems.

Idempotency is nice but not necessary. Lets remove these checks to make
testing more reliable.

Change-Id: I1092b3a2f4b781465c2148336076677d3c07211e
2019-05-24 08:40:38 -07:00
Clark Boylan fd385b1210 The logs for a job my contain dotfiles, index them
By default our mod auto index setup isn't indexing dotfiles on the logs
server. Some jobs my need to log dotfiles for valid reasons so we should
index them. We can do this by resetting the index ignore rules then
setting it to the upstream apache default of just '.'.

Change-Id: I64e381a4e0bd41df678c6f9f09955063dbd473ff
2019-05-23 09:31:35 -07:00
Monty Taylor b42b3d4e5b Pin keyring to 18.0.1
keyring 19.0.0 only supports python3. While we should obviously
get things updated, we're going to need to unwind some things
to do that.

Change-Id: I2ae4fe3f43e5e0e606a58f58a4f6e2bafd3a3930
2019-05-09 16:15:41 +00:00
Monty Taylor 05d6bd00bf Fix opendev paths
Change-Id: I9c8e8c6bcc8f58f044a0772b0ebfcc5a985a74bf
2019-04-20 15:26:42 +00:00
OpenDev Sysadmins 388a2767df OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:00 +00:00
Clark Boylan 23425ea940 Use mod_expires to set cache headers for log files
Our CI jobs' logserver wasn't setting cache headers so browsers were
always downloading the log files on each request. Reduce load on the
logserver and improve performance by users via the setting of a 2 week
ttl on log data.

This should be safe because each job writes to a uuid identified dir and
that data doesn't get overwritten. Once written the data is static.

Change-Id: Ic99df4e4e64a6bf64e68e6f1bc297c91134ab716
2019-04-05 14:09:00 -07:00
Ian Wienand 19a6b1bea8 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I0333048d837e4b793351f6a471edeefba0a2358d
2019-03-24 20:35:43 +00:00
Tristan Cacqueray 20ae86a8f4 logserver: set CORS header
This change enables cross request to fetch job logs artifacts.

Change-Id: I88ef4d559bb9b8fa3b67b51ec750bf9136a62ffb
2019-01-02 00:11:41 +00:00
Zuul 3bc07944d5 Merge "Add beaker tests for nodepool" 2018-11-21 22:49:10 +00:00
Zuul ffb407d841 Merge "Update Gemfile for Zuulv3" 2018-11-21 22:49:10 +00:00
Clark Boylan f5ac2b3944 Make oscc_file_contents optional
The openstack infra team is moving the managmeent of clouds.yaml files
from puppet to ansible. To keep puppet and ansible from fighting with
each other over the contents of this file make the contents optional
here. If provided puppet will manage them otherwise puppet doesn't touch
the clouds.yaml file.

Change-Id: I037abf2fb64a63af160ea6e25ce300197f7f513f
2018-08-30 15:20:28 -07:00
Colleen Murphy d0ee5c2cf4 Add beaker tests for nodepool
Add tests for the nodepool launcher and builder. Since by default this
uses the nodepool configuration found in openstack-infra/project-config,
fudge the hostnames so that it can find a real config file.

Depends-On: https://review.openstack.org/577519
Change-Id: I036c1afc0414f846c9112ef5bc5f3abd486f3147
2018-08-14 13:03:40 +00:00
Colleen Murphy e385661db3 Update Gemfile for Zuulv3
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.

Change-Id: I1c006fd05da4e51a454c3018fd30216cb7f1146a
2018-08-06 20:08:47 +00:00
Zuul 173d2cf49d Merge "Fix template variables" 2018-07-21 13:15:39 +00:00
Colleen Murphy 47a1005731 Fix template variables
Fix the scope for the template variables for the httpd vhost, and fix
the ServerAlias value when serveraliases is undefined.

Depends-On: https://review.openstack.org/578559

Change-Id: Ia0f9b704d95facd4668d972a66f94f741a99bca2
2018-07-12 09:27:03 +02:00
Colleen Murphy 4c0cc92a78 Add beaker-rspec bindep dependencies
Without this bindep file, the openstackci-beaker* jobs for the
puppet-openstackci module fail trying to build the gem dependencies.

Change-Id: Ie079e9344121c931f2d60beae59a3e881b6e81a9
2018-06-28 08:13:31 +00:00
Zuul f3f7b47cb4 Merge "Fix scope of vhost template variables" 2018-06-05 22:27:05 +00:00
Zuul 821256b02d Merge "Fix puppet syntax for nodepool builder logic" 2018-05-21 03:58:28 +00:00
Ian Wienand cae194830f nodepool-builder: pass through ssl arguments
Change-Id: I4a54b49cef46c04a374dfd788c2d5986d4566ffb
Depends-On: https://review.openstack.org/569005
2018-05-17 13:52:14 +10:00
Colleen Murphy ef5e2f1034 Fix scope of vhost template variables
Dynamic scoping for variables in ERB templates was removed in puppet
4[1] which means that the variables defined in the
openstackci::logserver class cannot be found when it is referenced in
the httpd::vhost defined type and will be evaluated as nil when
puppet runs. Use scope.lookupvar instead to be explicit about the
variable's source.

[1] https://puppet.com/docs/puppet/4.10/lang_updating_manifests.html#dynamic-scoping-in-erb

Change-Id: Iea37fb14bc332cbf2d598b908bce997786814924
2018-04-27 19:16:46 +02:00
Mikhail S Medvedev 930e6e59ff Fix puppet syntax for nodepool builder logic
install_nodepool_builder is a variable and needs a $ to get an expected
result. Right now the logical condition always evaluates to true which
is wrong.

Change-Id: I6a94865cf356a4a8802650fe7b76e56204b351ee
2018-04-11 11:44:09 -05:00
David Moreau Simard 5cc7d0afff Enable tuned WSGI settings for logs.openstack.org
After being tested in the parent patch, this will improve the
performance of the logs and ara report browsing.

Change-Id: Ib0dd51e24e7e4cc0d4308639d27a2aba18a9e96c
2018-04-04 15:48:05 +00:00
David Moreau Simard 6b351d7fa1
Synchronize vhost configuration between logs.o.o and logs-dev.o.o
It seems the vhost configurations have slightly diverged over time.
Let's synchronize them so we're able to test things accurately.

Change-Id: Ic921fcb3a310fbb9f728d73390d7740455ca4414
2018-04-04 11:46:16 -04:00
David Moreau Simard 7a4e625ca1
Setup WSGIDaemonProcess, ProcessGroup + ApplicationGroup for logs-dev
ARA running as a WSGI application highlighted the fact that we don't
currently fork succifient processes to handle the different calls.

In our specific context, logs.o.o's mod_wsgi is always busy because
logstash is hitting it. The default settings with the increased load
from ARA aren't able to keep up.

These values are being inspired by the puppet-openstacklib work for
their WSGI implementation [1].

[1]: http://git.openstack.org/cgit/openstack/puppet-openstacklib/tree/manifests/wsgi/apache.pp

Change-Id: If3e338f412771c178f064602081256b6abeffc28
2018-04-03 23:35:47 -04:00
David Moreau Simard 8a826a0fd4
Promote /ara-report/ middleware redirection to logs.openstack.org
We've successfully tested the ARA sqlite middleware on logs-dev.o.o.
We're ready to use this on logs.openstack.org.

Change-Id: I88c06366b8d5837ad6fc50163e5d58fe1b5eaa8f
2018-03-29 15:33:13 -04:00
David Moreau Simard 8d373a8468
Don't redirect/rewrite direct requests to the ARA database
We want /ara-report/ to load the report dynamically but requests to
/ara-report/ansible.sqlite should allow users to download the raw
database if they want/need to.

Add an exclusion to allow for that and improve the docs a little bit.

Change-Id: I739a3e339f40c76311a99b1d9c452f1aebc01eae
2018-03-29 15:25:55 -04:00
David Moreau-Simard 5c14e86d37
Add support for installing ARA wsgi middleware for sqlite databases
ARA is going to bundle a WSGI middleware that allows an operator
to route an URL (by default ^.*/ara-report) to the ARA web application.
This URL is expected to contain an 'ansible.sqlite' database that
would have been uploaded as part of log collection.

There are no static files generated as part of this process. The
web application reads directly from the database. It is meant
to avoid having to generate a static report for each job which does
not scale very well for the amount of jobs we are running.

For more details, see this code review in ARA [1].

[1]: https://review.openstack.org/#/c/511992/

Change-Id: I3b10c93b4902a9b45e23c227863e472697f662ef
2018-03-05 15:08:40 -05:00
Zuul 8c0380f50f Merge "Enable build logs and webapp" 2018-02-26 00:00:07 +00:00
Zuul e84dd711d1 Merge "Remove nodepool_builder::image_log_document_root" 2018-02-25 23:21:54 +00:00
Ian Wienand da5a45878a nodepool-builder: look for fqdn config first
$::project_config::nodepool_config_file_zuulv3 is set as the fqdn of
the host.  So far it is only used for nodepool-launchers.

We wish to have the ability to create custom configuration files for
nodepool-builders too.  Add the fqdn variable as the first match for
the builder config.  If no more specific config, the default should
remain.

Change-Id: Ibdd8bca3f1e0a1f6709be29c37fe9a696a7bb2d4
2018-02-20 14:47:05 +11:00
Ian Wienand dd141f6ab9 Enable build logs and webapp
The dependent change adds a flag to the builder for exporting build
logs, and another flag to the launcher to enable the webapp.  Plumb
through so we can enable both.

Also remove deprecated build/upload log arguments that are no longer
relevant.

Change-Id: I1644e41629507b1a7082e70afaba1baf63d620a4
Depends-On: https://review.openstack.org/543671
2018-02-13 14:37:04 +11:00
Ian Wienand 6c8ebce346 Remove nodepool_builder::image_log_document_root
This arugment doesn't do anything in ::nodepool::builder, and is being
removed.

Depends-On: https://review.openstack.org/543669
Change-Id: I3eaa8ae0cbb3ed039726d3ca47dc81332539f889
2018-02-13 14:14:33 +11:00
Ian Wienand 3eacddb2d7 Allow viewing of .log.gz files
I'm not sure why, but previously we have not passed .log.gz files
through to the htmlify web-app.  This means clicking on it in the
browser prompts to download the file.  This is unlikley to be what
devlopers investigating logs interactively want.

Traditionally in devstack-gate there's been a bunch of renaming
log->log.txt that happens to make the files presentable [1].  With
zuulv3 native jobs this happens less and everyone has to be aware of
this behaviour (e.g. Ia7415d2fbbb320f8eddc4e46c3a055414df5f997 is a
good example of raw .log files being added).  I think it is fine to
assume a .log file is text that is safe to pass through os-loganalyze.

From my reading of os-loganalyze [2], a .log.gz file would get mached
by util.use_passthrough_view and by default be filtered via a HTMLView
-- thus the file_conditions.yaml update is perhaps unnecessary, but
provided just to keep things in sync with .txt which has always been
there (I9cda9c05f42c5658d1c1dde8aa2efbc0abcc48a2)

[1] https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n869
[2] https://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/os_loganalyze/view.py#n237

Change-Id: I72e4f752713e7947bd66dfe8164976f0a4e3af24
2018-02-12 10:05:15 +11:00
Zuul d6888706c1 Merge "Stop expiring contents for docs-draft.o.o" 2018-02-05 21:34:03 +00:00
Paul Belanger e58949737b
Expose zuulv3 setting for nodepool_builder.pp
Allow a user to toggle zuulv3 settings for nodepool.

Change-Id: Ifdcfadca1a56a48b958cd612fba67bdc9d8d3653
Depends-On: I7fff2565f9e77d737c247f204925f66a0e4aabe8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-01-17 14:40:15 -05:00
Mikhail S Medvedev 8d4eea2bee Lay groundwork for zuulv2/v3 coexisting
Both v2 and v3 deployments need to be supported for a bit after zuulv3
feature is merged. The architectures are different, with some parts
missing (jenkins), while new parts being added (zuul-exector). It is
logical to split the configuration into two using zuulv2 flag.

Setting zuulv2 to true would use the old configuration. The choice to
use zuulv2 = true instead of zuulv3 = false is made so that it would be
easier to remove the flag once v2 support is removed.

The support for v3 is not added with this patch. It merely allows for
feature/zuulv3 to be merged to master without forcing users of
puppet-openstackci to pickup the new version.

For zuulv2 using zuul@2.6.0 and nodepool@0.5.0 which represent the
latest master state as of this patch. The default used to be 'master'
for both in single_node_ci class, so this should minimize unintended
consequences for those who used the class with defaults.

Change-Id: I2e7cd8d1c8e662b7e21fa80d4fa62846e2d44a70
Story: #2001367
Task: #5951
2018-01-09 16:22:13 -06:00
Jeremy Stanley b3649db8fc Stop expiring contents for docs-draft.o.o
Since static.openstack.org is no longer serving a
docs-draft.openstack.org vhost, and downstream consumers of this
module are unlikely to have been doing so themselves either, stop
bothering to attempt to expire content for it.

Change-Id: I413ac425d752f4fe839648808c0df8c620bef1a5
Depends-On: Ib44df24100192f7903eb60c6fc93feeea0894b90
2017-12-07 21:03:10 +00:00
Clark Boylan 9978a409fe Check for presence of Zuul with Zuul v3
Zuul v3 compat layer isn't going to set a ZUUL_REF var because we aren't
fetching refs from a git repo server. Instead just check if the
ZUUL_UUID is set which is a unique id for every job run by Zuul to see
if we are running under Zuul.

Change-Id: If6c122852d796427b69fc01f41c5587155c039e9
2017-11-20 09:23:50 -08:00
James E. Blair 0d389f404f Pass through gerrit_strip_branch_ref
Change-Id: I5c8e6f95a78c70b861255ecbf55493efb7b6cc10
Depends-On: Id388274a7b628ad40a0e943087ec82596791c982
2017-09-19 07:56:29 -07:00
Jenkins 71fbde489c Merge "Depend on helper gem for spec_helper_acceptance" 2017-08-22 23:37:22 +00:00
Emilien Macchi 5fe1f3d2d5 Iterate readmes as an hash
- In vhosts templates, iterates each key/value from the hash
  by using the proper Ruby syntax: https://ruby-doc.org/core-2.2.0/Hash.html
- In logserver.pp, default readmes to an empty hash instead of an empty
  array.

Change-Id: Ib1c70cfd9254bb4da9b6f7477baa48918de16cc2
2017-07-31 21:59:30 -07:00
Emilien Macchi 13f3e35d98 logserver: cleanup legacy resources, not needed anymore
Depends-On: I18cc3431f32b169cb0d950ff25ef24a3a6550404
Change-Id: I798a48c3029563d84e0a729ecabb27e21eee9baa
2017-07-25 22:00:19 +00:00
Jeremy Stanley 03899eb98d logserver: disable legacy mode
Set the legacy parameter to default false, now that the
openstack_project::static class provides the same resources. This
way the parameter can be removed gracefully in a subsequent change.

Change-Id: I4656a511aa21b1096e60155e5f7de1ae089a95ae
Depends-On: I354639545c9cbc12f1b844946962946aa2bb4fb0
2017-07-25 22:00:19 +00:00
Emilien Macchi fdc4e9cc04 logserver: add legacy parameter
Add legacy parameter for logserver class, to install logserver resources
specific to OpenStack Infra only when needed (true by default for
backward compatibility).

Change-Id: I65e8ecf39130377c088c053e5623212f7db60f7a
2017-07-25 22:00:17 +00:00
David Shrewsbury 26ad94ec52 Add support for python 3 for nodepool
Allow our nodepool installations to be installed under python 3.
Goal is to eventually remove py2 support from nodepool for zuulv3.

Change-Id: Icac77f88e411a3ac08fc54afb4e9746039f908e8
Depends-On: Ie2c223d74b32798e5f0cb0ca8b76d9fa6a4f3095
2017-07-19 11:30:48 -04:00