The opendev project has been moving away from puppet and this is one of
the puppet modules that is no longer used. To simplify things for us we
are taking the extra step of retiring this repo.
Change-Id: I46ccab87f063c51dd3236757dbb0567738803fae
By default our mod auto index setup isn't indexing dotfiles on the logs
server. Some jobs my need to log dotfiles for valid reasons so we should
index them. We can do this by resetting the index ignore rules then
setting it to the upstream apache default of just '.'.
Change-Id: I64e381a4e0bd41df678c6f9f09955063dbd473ff
Our CI jobs' logserver wasn't setting cache headers so browsers were
always downloading the log files on each request. Reduce load on the
logserver and improve performance by users via the setting of a 2 week
ttl on log data.
This should be safe because each job writes to a uuid identified dir and
that data doesn't get overwritten. Once written the data is static.
Change-Id: Ic99df4e4e64a6bf64e68e6f1bc297c91134ab716
Fix the scope for the template variables for the httpd vhost, and fix
the ServerAlias value when serveraliases is undefined.
Depends-On: https://review.openstack.org/578559
Change-Id: Ia0f9b704d95facd4668d972a66f94f741a99bca2
Dynamic scoping for variables in ERB templates was removed in puppet
4[1] which means that the variables defined in the
openstackci::logserver class cannot be found when it is referenced in
the httpd::vhost defined type and will be evaluated as nil when
puppet runs. Use scope.lookupvar instead to be explicit about the
variable's source.
[1] https://puppet.com/docs/puppet/4.10/lang_updating_manifests.html#dynamic-scoping-in-erb
Change-Id: Iea37fb14bc332cbf2d598b908bce997786814924
After being tested in the parent patch, this will improve the
performance of the logs and ara report browsing.
Change-Id: Ib0dd51e24e7e4cc0d4308639d27a2aba18a9e96c
It seems the vhost configurations have slightly diverged over time.
Let's synchronize them so we're able to test things accurately.
Change-Id: Ic921fcb3a310fbb9f728d73390d7740455ca4414
ARA running as a WSGI application highlighted the fact that we don't
currently fork succifient processes to handle the different calls.
In our specific context, logs.o.o's mod_wsgi is always busy because
logstash is hitting it. The default settings with the increased load
from ARA aren't able to keep up.
These values are being inspired by the puppet-openstacklib work for
their WSGI implementation [1].
[1]: http://git.openstack.org/cgit/openstack/puppet-openstacklib/tree/manifests/wsgi/apache.pp
Change-Id: If3e338f412771c178f064602081256b6abeffc28
We've successfully tested the ARA sqlite middleware on logs-dev.o.o.
We're ready to use this on logs.openstack.org.
Change-Id: I88c06366b8d5837ad6fc50163e5d58fe1b5eaa8f
We want /ara-report/ to load the report dynamically but requests to
/ara-report/ansible.sqlite should allow users to download the raw
database if they want/need to.
Add an exclusion to allow for that and improve the docs a little bit.
Change-Id: I739a3e339f40c76311a99b1d9c452f1aebc01eae
ARA is going to bundle a WSGI middleware that allows an operator
to route an URL (by default ^.*/ara-report) to the ARA web application.
This URL is expected to contain an 'ansible.sqlite' database that
would have been uploaded as part of log collection.
There are no static files generated as part of this process. The
web application reads directly from the database. It is meant
to avoid having to generate a static report for each job which does
not scale very well for the amount of jobs we are running.
For more details, see this code review in ARA [1].
[1]: https://review.openstack.org/#/c/511992/
Change-Id: I3b10c93b4902a9b45e23c227863e472697f662ef
I'm not sure why, but previously we have not passed .log.gz files
through to the htmlify web-app. This means clicking on it in the
browser prompts to download the file. This is unlikley to be what
devlopers investigating logs interactively want.
Traditionally in devstack-gate there's been a bunch of renaming
log->log.txt that happens to make the files presentable [1]. With
zuulv3 native jobs this happens less and everyone has to be aware of
this behaviour (e.g. Ia7415d2fbbb320f8eddc4e46c3a055414df5f997 is a
good example of raw .log files being added). I think it is fine to
assume a .log file is text that is safe to pass through os-loganalyze.
From my reading of os-loganalyze [2], a .log.gz file would get mached
by util.use_passthrough_view and by default be filtered via a HTMLView
-- thus the file_conditions.yaml update is perhaps unnecessary, but
provided just to keep things in sync with .txt which has always been
there (I9cda9c05f42c5658d1c1dde8aa2efbc0abcc48a2)
[1] https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n869
[2] https://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/os_loganalyze/view.py#n237
Change-Id: I72e4f752713e7947bd66dfe8164976f0a4e3af24
- In vhosts templates, iterates each key/value from the hash
by using the proper Ruby syntax: https://ruby-doc.org/core-2.2.0/Hash.html
- In logserver.pp, default readmes to an empty hash instead of an empty
array.
Change-Id: Ib1c70cfd9254bb4da9b6f7477baa48918de16cc2
Add legacy parameter for logserver class, to install logserver resources
specific to OpenStack Infra only when needed (true by default for
backward compatibility).
Change-Id: I65e8ecf39130377c088c053e5623212f7db60f7a
This sets the json.gz files' mimetype to application/json so that we can
compress stackviz report data and still have it render properly in your
browser. This will allow us to save space on the log server.
Change-Id: I42d7d607e5864a81d52e72ff695331eb2d67d2c0
We tested an addition to the logs-dev.o.o vhost in
https://review.openstack.org/#/c/435566/ in order to properly
detect and serve .svg.gz files and it works well.
Let's do it on logs.o.o.
Change-Id: I1305f7e1c4fe02269eccd60fe8d49419d8372b59
In https://review.openstack.org/#/c/396428/ we improved logs.o.o
to be able to serve statically generated html content/web applications
that have been gzipped.
The svg format was missed in that patch so that we can fully compress
output before uploading/storing it to logs.o.o.
We are first testing it on logs-dev.o.o and we will later apply it
to logs.o.o if it works well.
Change-Id: I0354f61f24176874a6590d32cfa06bb9cf3a02e3
In I5e5d2da0ef3c385f836507e69a686821aba925a1 we applied new
rewrite rules and mimetypes on logs-dev.o.o to try and make
sure that rich html content (css/js/fonts) would work either
when gzipped prior to initial log upload or when automatically
gzipped after some time.
This brings the change from logs-dev.o.o to logs.o.o.
Change-Id: I20db0aa12eda8157d68a7b27dbbfe3ca729347c7
If gate jobs gzip their builds before they are uploaded or if the
builds are automatically gzipped after a while on logs.o.o, the
current configuration does not try to seek out the gzipped files
with the proper mimetypes.
This commit adds missing mimetypes for common files:
- css.gz
- js.gz
- ttf.gz (fonts)
It also adds a rewrite rule that will check if there is a .gz
equivalent for the file the user is trying to seek, if there is,
it will serve it - otherwise it's business as usual.
Change-Id: I5e5d2da0ef3c385f836507e69a686821aba925a1
We have a vars.yaml file that is output as part of jobs. It's useful to
be able to browse its content in the web.
Change-Id: I618b25cde4118ae3eadfdf88ad4e7f2f004d6b71
When someone looks at the logs for a periodic job, they are almost
certainly looking for the results of the latest run. Order the
listing by date descending so that is right up the top, and you don't
have to search through all the entries to find the latest one.
Change-Id: Ie37343728a90a805b20b2b125e17cd3b11dc6b40
Automatically create the jenkins SCP repository host
for jenkins job log files. Otherwise it needs to be setup
manually.
Change-Id: I54ecccc3d3d9e668d989ec52a822fc194722e619
This reverts commit fededacaf0.
Reverting because os-loganalyze doesn't list file sizes from disk. This
is an important piece of info for people debugging failures.
Change-Id: Ibecc3372c3040432db03056bd681dec182dd2d03
os-loganalyze now has the ability to provide indexes for both on
disk and in swift files. Handle all requests through osloganalyze
so that paths containing both disk and swift files can have all
of their assets listed.
Change-Id: I495ce20d56b04fdc5c2051f2bbcb834049ac6bda
This will enable us to use different filters/views for various files.
By explicitly setting txt+html, we allow other formats (such as images
and tarballs) to be streamed from swift and hence downloaded.
Change-Id: I9cda9c05f42c5658d1c1dde8aa2efbc0abcc48a2
Remove some old path patterns which are a year past their expiration
date, and also sync up the logs-dev template comments to match the
production version (for ease of diffing).
Change-Id: Ibfd4fc1b16c3eac2c6093f1b031bf7ae237eaf04
Warning: Variable access via 'swift_authurl' is deprecated. Use
'@swift_authurl' instead.
Change-Id: Ic6580bdd5f5ecba972ab757141216788ca406c83
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Clark Boylan