The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.
Change-Id: I46ccab87f063c51dd3236757dbb0567738803fae
The third party ci doc is out of date and not currently working. It
needs help. Point to our spec that provides guidance on how to apply
that help as well as poitning to Zuul's documentation for setting up
Zuul.
Change-Id: I7365185f15f6f2cb2e3a9f6757897e793de98181
We don't want to delete the top-level robots.txt file - but everything
that is part of the logs hierarchy.
Change-Id: I369565a780eb749eab95e4bf8e7d8d7fd5bf0630
Currently if a log directory has a symlink it is skipped in the find;
thus the directory does not become empty and it is never removed.
Also match symlinks.
Change-Id: Id34c74d848638182819a9c81d051b83820244463
With the ARA 1.0.0 release, it has been rearchitected in ways which
require more extensive changes to the way it's deployed. For now,
pin it to use the previous version.
Change-Id: Iadfc699e6a4ce1de1a90bf4accb916454b788df0
The idempotency checks race external changes like updates to
project-config or nodepool. This often causes this job to fail when we
are busy updating other parts of our sytems.
Idempotency is nice but not necessary. Lets remove these checks to make
testing more reliable.
Change-Id: I1092b3a2f4b781465c2148336076677d3c07211e
By default our mod auto index setup isn't indexing dotfiles on the logs
server. Some jobs my need to log dotfiles for valid reasons so we should
index them. We can do this by resetting the index ignore rules then
setting it to the upstream apache default of just '.'.
Change-Id: I64e381a4e0bd41df678c6f9f09955063dbd473ff
keyring 19.0.0 only supports python3. While we should obviously
get things updated, we're going to need to unwind some things
to do that.
Change-Id: I2ae4fe3f43e5e0e606a58f58a4f6e2bafd3a3930
Our CI jobs' logserver wasn't setting cache headers so browsers were
always downloading the log files on each request. Reduce load on the
logserver and improve performance by users via the setting of a 2 week
ttl on log data.
This should be safe because each job writes to a uuid identified dir and
that data doesn't get overwritten. Once written the data is static.
Change-Id: Ic99df4e4e64a6bf64e68e6f1bc297c91134ab716
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: I0333048d837e4b793351f6a471edeefba0a2358d
The openstack infra team is moving the managmeent of clouds.yaml files
from puppet to ansible. To keep puppet and ansible from fighting with
each other over the contents of this file make the contents optional
here. If provided puppet will manage them otherwise puppet doesn't touch
the clouds.yaml file.
Change-Id: I037abf2fb64a63af160ea6e25ce300197f7f513f
Add tests for the nodepool launcher and builder. Since by default this
uses the nodepool configuration found in openstack-infra/project-config,
fudge the hostnames so that it can find a real config file.
Depends-On: https://review.openstack.org/577519
Change-Id: I036c1afc0414f846c9112ef5bc5f3abd486f3147
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.
Change-Id: I1c006fd05da4e51a454c3018fd30216cb7f1146a
Fix the scope for the template variables for the httpd vhost, and fix
the ServerAlias value when serveraliases is undefined.
Depends-On: https://review.openstack.org/578559
Change-Id: Ia0f9b704d95facd4668d972a66f94f741a99bca2
Without this bindep file, the openstackci-beaker* jobs for the
puppet-openstackci module fail trying to build the gem dependencies.
Change-Id: Ie079e9344121c931f2d60beae59a3e881b6e81a9
Dynamic scoping for variables in ERB templates was removed in puppet
4[1] which means that the variables defined in the
openstackci::logserver class cannot be found when it is referenced in
the httpd::vhost defined type and will be evaluated as nil when
puppet runs. Use scope.lookupvar instead to be explicit about the
variable's source.
[1] https://puppet.com/docs/puppet/4.10/lang_updating_manifests.html#dynamic-scoping-in-erb
Change-Id: Iea37fb14bc332cbf2d598b908bce997786814924
install_nodepool_builder is a variable and needs a $ to get an expected
result. Right now the logical condition always evaluates to true which
is wrong.
Change-Id: I6a94865cf356a4a8802650fe7b76e56204b351ee
After being tested in the parent patch, this will improve the
performance of the logs and ara report browsing.
Change-Id: Ib0dd51e24e7e4cc0d4308639d27a2aba18a9e96c
It seems the vhost configurations have slightly diverged over time.
Let's synchronize them so we're able to test things accurately.
Change-Id: Ic921fcb3a310fbb9f728d73390d7740455ca4414
ARA running as a WSGI application highlighted the fact that we don't
currently fork succifient processes to handle the different calls.
In our specific context, logs.o.o's mod_wsgi is always busy because
logstash is hitting it. The default settings with the increased load
from ARA aren't able to keep up.
These values are being inspired by the puppet-openstacklib work for
their WSGI implementation [1].
[1]: http://git.openstack.org/cgit/openstack/puppet-openstacklib/tree/manifests/wsgi/apache.pp
Change-Id: If3e338f412771c178f064602081256b6abeffc28
We've successfully tested the ARA sqlite middleware on logs-dev.o.o.
We're ready to use this on logs.openstack.org.
Change-Id: I88c06366b8d5837ad6fc50163e5d58fe1b5eaa8f
We want /ara-report/ to load the report dynamically but requests to
/ara-report/ansible.sqlite should allow users to download the raw
database if they want/need to.
Add an exclusion to allow for that and improve the docs a little bit.
Change-Id: I739a3e339f40c76311a99b1d9c452f1aebc01eae
ARA is going to bundle a WSGI middleware that allows an operator
to route an URL (by default ^.*/ara-report) to the ARA web application.
This URL is expected to contain an 'ansible.sqlite' database that
would have been uploaded as part of log collection.
There are no static files generated as part of this process. The
web application reads directly from the database. It is meant
to avoid having to generate a static report for each job which does
not scale very well for the amount of jobs we are running.
For more details, see this code review in ARA [1].
[1]: https://review.openstack.org/#/c/511992/
Change-Id: I3b10c93b4902a9b45e23c227863e472697f662ef
$::project_config::nodepool_config_file_zuulv3 is set as the fqdn of
the host. So far it is only used for nodepool-launchers.
We wish to have the ability to create custom configuration files for
nodepool-builders too. Add the fqdn variable as the first match for
the builder config. If no more specific config, the default should
remain.
Change-Id: Ibdd8bca3f1e0a1f6709be29c37fe9a696a7bb2d4
The dependent change adds a flag to the builder for exporting build
logs, and another flag to the launcher to enable the webapp. Plumb
through so we can enable both.
Also remove deprecated build/upload log arguments that are no longer
relevant.
Change-Id: I1644e41629507b1a7082e70afaba1baf63d620a4
Depends-On: https://review.openstack.org/543671
This arugment doesn't do anything in ::nodepool::builder, and is being
removed.
Depends-On: https://review.openstack.org/543669
Change-Id: I3eaa8ae0cbb3ed039726d3ca47dc81332539f889
I'm not sure why, but previously we have not passed .log.gz files
through to the htmlify web-app. This means clicking on it in the
browser prompts to download the file. This is unlikley to be what
devlopers investigating logs interactively want.
Traditionally in devstack-gate there's been a bunch of renaming
log->log.txt that happens to make the files presentable [1]. With
zuulv3 native jobs this happens less and everyone has to be aware of
this behaviour (e.g. Ia7415d2fbbb320f8eddc4e46c3a055414df5f997 is a
good example of raw .log files being added). I think it is fine to
assume a .log file is text that is safe to pass through os-loganalyze.
From my reading of os-loganalyze [2], a .log.gz file would get mached
by util.use_passthrough_view and by default be filtered via a HTMLView
-- thus the file_conditions.yaml update is perhaps unnecessary, but
provided just to keep things in sync with .txt which has always been
there (I9cda9c05f42c5658d1c1dde8aa2efbc0abcc48a2)
[1] https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n869
[2] https://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/os_loganalyze/view.py#n237
Change-Id: I72e4f752713e7947bd66dfe8164976f0a4e3af24
Allow a user to toggle zuulv3 settings for nodepool.
Change-Id: Ifdcfadca1a56a48b958cd612fba67bdc9d8d3653
Depends-On: I7fff2565f9e77d737c247f204925f66a0e4aabe8
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Both v2 and v3 deployments need to be supported for a bit after zuulv3
feature is merged. The architectures are different, with some parts
missing (jenkins), while new parts being added (zuul-exector). It is
logical to split the configuration into two using zuulv2 flag.
Setting zuulv2 to true would use the old configuration. The choice to
use zuulv2 = true instead of zuulv3 = false is made so that it would be
easier to remove the flag once v2 support is removed.
The support for v3 is not added with this patch. It merely allows for
feature/zuulv3 to be merged to master without forcing users of
puppet-openstackci to pickup the new version.
For zuulv2 using zuul@2.6.0 and nodepool@0.5.0 which represent the
latest master state as of this patch. The default used to be 'master'
for both in single_node_ci class, so this should minimize unintended
consequences for those who used the class with defaults.
Change-Id: I2e7cd8d1c8e662b7e21fa80d4fa62846e2d44a70
Story: #2001367
Task: #5951
Since static.openstack.org is no longer serving a
docs-draft.openstack.org vhost, and downstream consumers of this
module are unlikely to have been doing so themselves either, stop
bothering to attempt to expire content for it.
Change-Id: I413ac425d752f4fe839648808c0df8c620bef1a5
Depends-On: Ib44df24100192f7903eb60c6fc93feeea0894b90
Zuul v3 compat layer isn't going to set a ZUUL_REF var because we aren't
fetching refs from a git repo server. Instead just check if the
ZUUL_UUID is set which is a unique id for every job run by Zuul to see
if we are running under Zuul.
Change-Id: If6c122852d796427b69fc01f41c5587155c039e9
- In vhosts templates, iterates each key/value from the hash
by using the proper Ruby syntax: https://ruby-doc.org/core-2.2.0/Hash.html
- In logserver.pp, default readmes to an empty hash instead of an empty
array.
Change-Id: Ib1c70cfd9254bb4da9b6f7477baa48918de16cc2
Set the legacy parameter to default false, now that the
openstack_project::static class provides the same resources. This
way the parameter can be removed gracefully in a subsequent change.
Change-Id: I4656a511aa21b1096e60155e5f7de1ae089a95ae
Depends-On: I354639545c9cbc12f1b844946962946aa2bb4fb0
Add legacy parameter for logserver class, to install logserver resources
specific to OpenStack Infra only when needed (true by default for
backward compatibility).
Change-Id: I65e8ecf39130377c088c053e5623212f7db60f7a
Allow our nodepool installations to be installed under python 3.
Goal is to eventually remove py2 support from nodepool for zuulv3.
Change-Id: Icac77f88e411a3ac08fc54afb4e9746039f908e8
Depends-On: Ie2c223d74b32798e5f0cb0ca8b76d9fa6a4f3095