Commit Graph

87 Commits

Author SHA1 Message Date
Clark Boylan fb7c8790dd Retire this project
We've shutdown the log processing service and don't need to manage it
with puppet any more.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/839235
Change-Id: I451488faf6a7502a5171d2a4299d7a4e40d96072
2022-04-25 09:48:21 -07:00
melanie witt 89bfe00dda Stream log files instead of loading full files into memory
For awhile now lately, we have been seeing Elastic Search indexing
quickly fall behind as some log files generated in the gate have become
larger. Currently, we download a full log file into memory and then
emit it line-by-line to be received by a logstash listener. When log
files are large (example: 40M) logstash gets bogged down processing
them.

Instead of downloading full files into memory, we can stream the files
and emit their lines on-the-fly to try to alleviate load on the log
processor.

This:

  * Replaces use of urllib2.urlopen with requests with stream=True
  * Removes manual decoding of gzip and deflate compression
    formats as these are decoded automatically by requests.iter_lines
  * Removes unrelated unused imports
  * Removes an unused arg 'retry' from the log retrieval method

Change-Id: I6d32036566834da75f3a73f2d086475ef3431165
2020-11-09 09:49:07 -08:00
Clark Boylan a9da328436 Fix puppet linter complaints about :: prefixes
This is a thing that puppet has gone back and forth on and now we are on
the wrong side of it. Fix it like we've fixed it elsewhere.

Change-Id: I6d514b2345ff284c57409cc508786b76258d9f4a
2020-10-23 11:13:25 -07:00
Zuul 09b0ed74f7 Merge "Handle case where content encoding isn't set" 2019-08-27 18:01:38 +00:00
Zuul c0f67cef05 Merge "Don't try to get .gz suffixed files in addition to base url" 2019-08-24 02:25:13 +00:00
Clark Boylan 972b6355b0 Handle case where content encoding isn't set
Belts and suspenders for cases where content encoding may not be
present. I believe this is possible if the content is served with the
identity encoding. In that case setting the encoding header isn't
required.

Change-Id: If18670d4fd3656a35f818247539b7afad39493e6
2019-08-23 18:45:06 -07:00
Clark Boylan 15991cfded Don't try to get .gz suffixed files in addition to base url
Zuul gives us the source url to index. Previously we tried to fetch
url + .gz because in many cases we uploaded the file as a gzip file but
logically treated it as unzipped. Now with logs in swift we compress
files without the .gz suffix. This means we should be able to always
fetch the url that zuul provides unmodified.

Depends-On: https://review.opendev.org/678303
Change-Id: I0ea4d9daa905ccb50372b73b5035758fc0963716
2019-08-23 21:55:17 +00:00
Clark Boylan b9063a7e7e Fix systemd severity filter input data
The severity filters are passed the entire json event and not just a
string. Update the systemd filter to access the message string out of
the event json dict.

Prior to this we get a type error:

  2019-08-19 17:18:48,055 Exception handling log event.
  Traceback (most recent call last):
    File "/usr/local/bin/log-gearman-worker.py", line 255, in
  _handle_event
      keep_line = f.process(out_event)
    File "/usr/local/bin/log-gearman-worker.py", line 183, in process
      m = self.SYSTEMDRE.match(msg)
  TypeError: expected string or buffer

Change-Id: I7ab56ac397133f00539d9d3374fa400363ef12d6
2019-08-19 10:27:45 -07:00
Ian Wienand 3119c0cddd log-gearman-worker: remove obsolete GET debug filter, add local filter
Now that logs have moved into swift, the os-loganalyze middleware that
stripped DEBUG level logs when the URL was given a ?level= parameter
no longer functions.

We move to filtering DEBUG statements directly.  Because services in
devstack now run as systemd services, their log files are actually
journalctl dumps.  Thus we add a new filter for systemd style
timestamps and messages (this is loosely based on the zuul log viewer
at [1]).

[1] 8c1f4e9d6b/web/src/actions/logfile.js

Change-Id: I54087c95c809612758139136d5b3e86b1a6372be
2019-08-19 17:15:25 +10:00
Ian Wienand 5b30a3a6c0 log-gearman-worker: Remove jenkins streaming workaround
We don't need to worry about the file changing under us any more; this
was all pre-zuul, let alone pre-using-swift for logs.  Remove this
workaround.

Change-Id: I5938dcef5550d4c62c8158c5f89ace75ae99aedc
2019-08-19 13:06:58 +10:00
Ian Wienand bca04e3155 log-gearman-worker: handle deflate encoded values
We are now logging to swift and store the objects as deflate encoded
data [1].  This means that we get back "Content-Encoding: deflate"
data when downloading the logs (even despite us not accepting that).

So put in a path for deflate encoding to the extant code with zlib.
For completeness we also update our Accept-Encoding: header to show we
accept deflate.

[1] 60e7542875/roles/upload-logs-swift/library/zuul_swift_upload.py (L608)

Change-Id: I328bafea3ddae858fd77af043f16c499ddd5a30e
2019-08-19 13:02:32 +10:00
OpenDev Sysadmins f985b87197 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:25:38 +00:00
Ian Wienand a4955ffcda Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I218bec35df3cb1ee1a65f05639756a5cfe9a56b6
2019-03-24 20:35:40 +00:00
Zuul e3f9f291b2 Merge "Add beaker tests" 2018-11-21 22:49:09 +00:00
Clark Boylan 772a94ff6d Force geard to listen on ::
By default geard only listens on ipv4 0.0.0.0 which means ipv6
connectiosn don't work. Because we run dual stack and things expect ipv6
to work (we have AAAA dns records after all) force geard to listen on ::
which will accept ipv6 and ipv4 connections.

Change-Id: Ibf3bfc5f80ca139b375ee2902dc3149ac791ef96
2018-10-18 15:47:14 -07:00
Clark Boylan 79ef165477 Install statsd version 2.1.2
The logstash log processing and subunit2sql tooling is often paired
together on servers. Currently subunit2sql depends on
os-performance-tools which depend on statsd<3.0. That means our package
resource here for statsd that tried to install latest conflicts with our
usage of subunit2sql on the same server.

Avoid this conflict by installing 2.1.2 for geard and then subunit2sql
will be happy too.

Change-Id: I3ac04cb93025ae2e2115ed23ba4927c2060f6dc8
2018-10-18 11:38:47 -07:00
Zuul 8c748b0cd5 Merge "Add support for running a standalone geard" 2018-10-18 16:44:21 +00:00
Colleen Murphy 6d20f37d26 Add beaker tests
Add basic tests to ensure the classes can apply cleanly and the services
start.

Change-Id: I51033cee31c58c8e5a4b1b16c731844c8e26ef46
2018-08-06 22:12:30 +02:00
James E. Blair 625bb48d13 Add severity info to logstash and filter out DEBUG lines
This adds severity as a logstash field for every oslo formatted
log line, and does not add any lines which are at DEBUG level.

This means we no longer rely on the level=INFO query paremeter
in order to remove DEBUG lines, so we will avoid sending them
to logstash regardless of whether os-loganalyze is used.

Change-Id: I8c4ac76a7fa0c3badd82fc7c54959ef6eb052732
2018-08-06 10:00:40 -07:00
Colleen Murphy 8a8693dabd Update Gemfile for Zuulv3
The logic in the Gemfile was relying on Zuulv2 variables to find out
whether the spec helper gem was already available on disk, and since
Zuulv3 has changed things it was failing to find it and downloading the
master version instead. This patch ensures the Gemfile looks for the gem
in the right place when running in CI.

Change-Id: If0864708efebd16058707ae102ef1b06b9332c42
2018-07-12 09:57:44 +02:00
Clark Boylan ffb50e9e8e Add support for running a standalone geard
We don't use the jenkins log client 0mq events anymore with zuulv3.
Instead zuul jobs submit the log indexing jobs directly to the gearman
queue for log processing. This means we only need a geard to be running
so add support for running just that daemon.

Change-Id: Iedcb5b29875494b8e18fa125adb08ec2e34d0064
2017-12-22 10:21:52 -08:00
Jens Harbott c646602e20 Fix multiple workers for systemd
systemd evaluated the "# Provides:" comment, need unique keys there.

Change-Id: I1ac6032edf9e14d45bc922221c4e7dd2c221fe11
2017-12-11 17:55:10 +00:00
David Moreau Simard b87b76f27c
Run daemon-reload after installing log processor init script
Now that we are upgrading to Xenial, we need to take into account
that we're running with systemd and reload it so that it picks up the
new service.

Change-Id: Id02ac2bc51132a8d8d4a77cb05d41fa902765b28
2017-12-11 09:47:36 -05:00
Clark Boylan 54eb1a0785 Collapse logically identical filenames for crm114
Log files come with many names while still containing the same logical
content. That may be because the path to them differs (eg /var/log/foo.log
and /opt/stack/log/foo.log) or due to file rotations (eg
/var/log/foo.log and /var/log/foo.log.1) or due to compression (eg
/var/log/foo.log and /var/log/foo.log.gz). At the end of the day these
are all the same foo.log log file.

This means when we do machine learning on the log files we can collapse
all these different cases down into a single case that we learn on. This
has become more important with the recent running out of disk space due
to all the non unique log paths out there for our log files but should
also result in better learning.

Change-Id: I4ba276870b73640909ac469b336a436eb127f611
2017-11-22 23:05:35 +00:00
Clark Boylan f35b4e2490 Reduce log worker internal queue size
We are having OOM problems with larger log files. Attempt to make this
more robust by having much smaller internal log line message queues (we
reduce the queue size to about 10% the original size). The idea here is
that if we have the old 130k queue size full then grab a large log file
the overhead there is fairly significant whereas if we have a small 16k
queue size and grab a large log file we really only have to worry about
the size of the logfile itself.

Depends-On: Iddbbab9ea5996df4922bf7927deb8f0354378ab7
Change-Id: I761fabaa1b5aae64790def721980151f9fdc720d
2017-11-18 23:40:46 +00:00
Colleen Murphy 22c22b0356 Depend on helper gem for spec_helper_acceptance
Instead of keeping a local copy of spec_helper_acceptance.rb and
requiring updates to all modules for any change, we can move it into the
common helper gem and require it from there. This will make it easier to
create and review changes that affect all puppet modules. Also change
the Gemfile to look for the gem in the local workspace if running in a
zuul environment.

Change-Id: I39394696ddb30d4c9ab67c7b3598c7a723c89b14
2017-08-18 10:41:43 +02:00
Clark Boylan 88e0d21347 Only send mqtt events for processed files
We were previously sending events for every file we attempted to
process, not just those that were processed and also for every single
log line event. This effectively doubled the io performed by the
logstash workers which seemed to slow the whole pipeline down. Trim it
down to only recording events for log files that are processed which
should significantly trim down the total number of events.

Change-Id: I0daf3eb2e2b3240e3efa4f2c7bac57de99505df0
2017-08-03 15:28:45 -07:00
Clark Boylan becc05e0aa else needs to be else:
Fix missing ':' syntax error.

Change-Id: I65d26db42eb871c230fd880457e12a25016baf1e
2017-08-03 09:44:46 -07:00
Matthew Treinish 662ae3777c
Handle cases without a build_change field
Previously the mqtt topic generation always assumed a build_change was
present. However there are some cases where the isn't a build_change in
the metadata, like periodic, post, and release jobs. This commit handles
those edge cases so it uses the build queue in the topic instead of the
build_change. If that doesn't work the topic is just the project.

Change-Id: I26dba76e3475749d00a45b076d981778f885c339
2017-08-03 11:46:55 -04:00
Matthew Treinish 541d226b36
Ensure we use paho 1.2.3
The new paho-mqtt 1.3.0 release brings
https://github.com/eclipse/paho.mqtt.python/commit/0a8cccc
which prevents its use on Ubuntu Trusty's default Python interpreter.
Until we upgrade to a newer Python there, stay on paho-mqtt 1.2.3 so
that things keep working.

Change-Id: I4ffcd8c7906c86a40f3cd8f8d83fb8208944d189
2017-08-02 18:34:40 -04:00
Clark Boylan 8a55071b10 Fix syntax errors
There was bad indentation and a missing '.' in config.get.
_generate_topic())_ is an object method not global function and it takes
an action argument.

Change-Id: I01c4af83cf98f0d7191041a864618a1608f97647
2017-08-02 15:23:29 -07:00
Jenkins 1505f2e3b1 Merge "Add MQTT support to the gearman worker" 2017-08-02 18:16:30 +00:00
Colleen Murphy c6ab9bd82c Fix beaker on xenial
Add a xenial nodeset and update the spec helper to install puppet 3 from
the Ubuntu repos instead of from puppetlabs.

Change-Id: I0a747faa3b9964227b0f2f2ca4a7bb895eca95e9
2017-06-24 15:52:02 +02:00
Matthew Treinish b1a4357058
Add MQTT support to the gearman worker
This commit adds support to the gearman worker for publishing an mqtt
message when processing a gearman job succeeds or fails. It also adds
a message for when the processor passes the logs to logstash either via
stdout or over a socket. By default this is disabled since it requires
extra configuration to tell the worker how to talk to the mqtt broker.

Depends-On: Id0308d2d4d1843fcca73f459cffa2ae944bebd0c
Change-Id: I43be3562780c61591ebede61f3a8929e8217f199
2017-04-27 10:06:34 -04:00
Jenkins 3aa0ef4305 Merge "Fix deprecated warning for suffix" 2017-03-05 21:57:50 +00:00
Clark Boylan 1f0f91fdbb Reduce log client logging by default
We had been running at debug level which is incredibly verbose. Remove
the -d flag. This will cause the logs which are logged to go to
stdout/err which should mean that upstart (or whatever init system) will
deal with them for us.

We should properly clean this up so that debug logging is useful again
in the long term.

Change-Id: I613c135ea56507d083df8c66e8846c6fbfa8b2ed
2016-09-27 17:18:23 -07:00
Jenkins 277cc29e50 Merge "Check known pid file when checking service status" 2016-06-28 23:07:27 +00:00
Spencer Krum b1ea4d82f1 Use new infra_spec_helper for gem dependencies
Change-Id: Iae01ed3d94e2e3495c117556c26d8f6ade990e1b
2016-06-21 18:38:56 -07:00
K Jonathan Harker d2de15f0e6 Check known pid file when checking service status
This is needed to support running multiple services on the same host,
otherwise all services will report as running if any single service is
running.

Change-Id: Ie6b7918af846af2189324b0b177b45ac858eadba
2016-06-17 08:47:37 -07:00
Philip Marc Schwartz c06b6c8414 Update pip provider to use openstack_pip provider
Change-Id: Id70b84e6656312c8e4850c5066511c7854c7485a
Signed-off-by: Philip Marc Schwartz <philip@progmad.com>
2016-06-16 10:19:52 -04:00
Paul Belanger 693d42c9a6
Fix deprecated warning for suffix
Change-Id: I5a0ed09fe881e857a66a1d52f5a0117f912be39f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-05-26 10:50:38 -04:00
Spencer Krum 6244933588 Pin google-api-client; sanitize Gemfile
Change-Id: I5b2640c7f5ebfe61a4ecad47e80061a82cf54d30
2016-04-14 15:20:23 -07:00
Clark Boylan 8491d10d26 Decouple log processing from logstash
As part of the move to logstash 2.0 we are relying on upstream packaging
for logstash. This packaging replaces a lot of the micromanagement of
users and groups and dirs that was done in puppet for logstash. This is
great news because its less work for us but means that the log
processors can't rely on puppet resources for those items and we don't
actually want to install logstash package everywhere we run log
processor daemons.

Since the log processors don't need a logstash service running and
actually don't need any of the logstash stuff at all decouple them
completely and have the log processor daemons use their own user, group,
log dir, config dir, etc. With this in place we can easily switch to
using the logstash packages only where we actually need logstash to be
running.

Change-Id: I2354fbe9d3ab25134c52bfe58f562dfdf9ff6786
2016-03-09 13:52:35 -08:00
Andrea Frittoli (andreaf) 08274f5f35 Allow for python-daemon already defined
log_processor class may be applied to a server along with
other classes also declaring python-daemon as a dependency.

As puppet cannmot handle this, add a if defined check.

Change-Id: I40dc68bd93f113912373cb10b376819d30eb3087
2016-01-13 15:28:21 +00:00
James E. Blair d7d9d50ee2 Change node_region to node_provider
This matches nodepool terminology to reduce confusion.

Change-Id: I3a8776010dcaf6677a450d0a9cb770313e604019
2015-12-17 14:51:59 -08:00
K Jonathan Harker 84c7e72312 Revert "Switch to using the new log_processor project"
This reverts commit b548b141ce.

b548b141ce was supposed to depend-on
https://review.openstack.org/248868

Change-Id: If3d4ad8a1cd45e6e63155a76dc1477ab38b156e3
2015-12-07 16:21:00 -08:00
K Jonathan Harker b548b141ce Switch to using the new log_processor project
The python scripts have been moved to their own project at
openstack-infra/log_processor. Delete the files here and start
installing that project from source. As a part of this split, the
.py extension has been dropped from the filename of the installed
executables.

Change-Id: Ied3025df46b5014a092be0c26e43d4f90699a43f
2015-11-25 15:23:26 -08:00
Matthew Treinish b2190b1a42
Add a node_region field to the job metadata
The node region can be figured out from the build_node very easily and
having a discrete field will make filtering to a single region much
simpler. This commit adds a new metadata field 'node_region' which is
the cloud region that the build_node ran in.

Change-Id: I06bbb62d21871ee61dbfb911143efff376992b98
2015-11-19 19:16:13 -05:00
Joshua Hesketh cd55cdf7d7 Revert "Create subunit proccessor subclass"
This reverts commit 135ac1809d.

EventProcessor was called before being defined. The code also doesn't
look entirely right. Reverting this to fix up the logstash servers

Change-Id: I2fb8081426646565814090c152d04d7349c16945
2015-11-19 11:05:53 +00:00
Jenkins 75ed9aca88 Merge "Create subunit proccessor subclass" 2015-11-18 14:23:41 +00:00