Commit Graph

32 Commits

Author SHA1 Message Date
OpenDev Sysadmins 2751aa44bb OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:03 +00:00
Ian Wienand 36e9b3db55 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I087216b631beea811cdee6cc4b4d5e26067be673
2019-03-24 20:34:05 +00:00
K Jonathan Harker 2c63950e34 Add project boilerplate
Update the .gitreview file for the new project name.
Add skeleton docs, setup, and testing.

Most of this boilerplate has been copied from nodepool.

In order to make testing easier, the EventProcessor class has been
slightly refactored to allow FakeEventProcessor to subclass and
mock out the socket object with a Queue.Queue object.

Change-Id: I137849a16807b9c56cd6f31de57b2b99506c8219
2015-11-23 15:32:06 -08:00
Matthew Treinish 17e2e0946c Add a node_region field to the job metadata
The node region can be figured out from the build_node very easily and
having a discrete field will make filtering to a single region much
simpler. This commit adds a new metadata field 'node_region' which is
the cloud region that the build_node ran in.

Change-Id: I06bbb62d21871ee61dbfb911143efff376992b98
2015-11-19 19:16:13 -05:00
Joshua Hesketh 48500b4462 Revert "Create subunit proccessor subclass"
This reverts commit 135ac1809d0182e04c2362458568de14a3cf948d.

EventProcessor was called before being defined. The code also doesn't
look entirely right. Reverting this to fix up the logstash servers

Change-Id: I2fb8081426646565814090c152d04d7349c16945
2015-11-19 11:05:53 +00:00
Jenkins 2287ca2cb2 Merge "Create subunit proccessor subclass" 2015-11-18 14:23:41 +00:00
Jenkins 9417c3a5f8 Merge "Add the ability to filter on project" 2015-11-15 13:49:48 +00:00
Jenkins f53df91a4b Merge "Process ZUUL_VOTING parameter" 2015-10-08 15:35:29 +00:00
Matt Riedemann c026c7a3f0 Process ZUUL_VOTING parameter
Read the ZUUL_VOTING parameter and add to the event before posting for
log processing.

The plan is that elastic-recheck will eventually use this field for
filtering out non-voting jobs from the e-r uncategorized bugs page.

Depends-On: I40746bb77aab900c1dd2637f940c14f72a904a61

Change-Id: I1f3c2a65104db39fdd7d786d421cded1b436a5f6
2015-09-16 09:04:27 -07:00
Anita Kuno c58d5b02f6 Add build_zuul_url parameter
Currently logstash does not track the zuul url. The zuul url
contains the zm (zuul merger) node identifier.

While trying to troubleshoot a zuul cloning issue, I noticed all
faiures were coming from the same zm (zuul merger) node. Tracking
the build_zuul_url can be helpful. This patch adds the
build_zuul_url parameter.

Change-Id: I83358dc0d9b27852df2395a9c52d2daaaeda712b
2015-09-01 14:22:16 -04:00
Clark Boylan 5badcb64d0 Import socket so we can use it to get name info
Previously this was using socket.getaddrinfo() without importing socket
and causing the daemon to fail. Running in the foreground did not use
statsd thus did not attempt to resolve the statsd host which is how this
got past manual testing. Import socket to get everything working agian.

Change-Id: I280973bdcdf472736a07d19173559b062ed74d3c
2015-07-17 11:15:19 -07:00
Jenkins 104fcc2a90 Merge "Lazily connect to logstash" 2015-07-15 22:50:58 +00:00
Jenkins 701fc86f81 Merge "Retry on EAI_AGAIN name resolution failures" 2015-07-15 22:50:57 +00:00
K Jonathan Harker 700b58517a Create subunit proccessor subclass
This allows for subunit files that do not include subunit in the name.

Change-Id: I8504fad6a4dea98700c204984cf00fea95de8369
2015-06-11 11:18:06 -07:00
K Jonathan Harker 3fc1dfc05a Add the ability to filter on project
Implement a project-filter option to gearman client config alongside the
job-filter and build-queue-filter options.

Change-Id: Ia71f216f4acc9de145eb9124df691393d2a86808
2015-06-11 09:13:41 -07:00
Clark Boylan 77d4f7cac2 Lazily connect to logstash
Because boot order is such a mess we will lazily connect to the logstash
TCP/UDP ports to allow for logstash to come up before we start writing
to it. This takes advantage of existing logstash restart handler code in
the log processors.

Change-Id: I836c55806c88cc86b7973b3d40f4bfce076970f5
2015-03-05 11:14:15 -08:00
Clark Boylan e397381251 Retry on EAI_AGAIN name resolution failures
There is no sane way to convince Ubuntu to start these services after
name resolution is working (because sysv init is horribly broken on
Ubuntu). Work around this by catching EAI_AGAIN errors during name
resolution and retrying until we can resolve names.

This logs each failed resolution attempt so that users are aware of the
issue if investigating logs.

Change-Id: If94d4f04d0e1cfedc358fd9d678a36fc9cd8aa7b
2015-03-05 10:29:48 -08:00
Ramy Asselin 158a75ac6e Add puppet module files for log processor
Change-Id: Ic87d2ae7259ad75f087f8999a4643abd1982b80b
2015-01-29 14:52:17 -08:00
Matthew Treinish bd559b0390 Add support to log gearman client to filter on build-queue
This commit adds a new job filter to the gearman client to filter
based on the build queue. This is used for the subunit jobs which
we don't want to run on check jobs.

Change-Id: If81fe98d8d67bb718c53a963695a7d06f5f6625d
2014-11-19 09:42:47 -05:00
Matthew Treinish 294627ba72 Add subunit2sql gearman workers
This adds a new gearman worker to process the subunit files from
the gate job runs. It will use subunit2sql to connect to a sql
server and process the data from the subunit file. The
log-gearman-client is modified to allow for pushing subunit jobs
to gearman, and the worker model for processsing logs is borrowed
to process the subunit files.

Change-Id: I83103eb6afc22d91f916583c36c0e956c23a64b3
2014-10-29 13:03:49 -04:00
Clark Boylan 30dcf06946 Handle log processing subprocess cleanup better
We are leaking file descriptors in our log worker processes because we
are are not catch all possible errors leaving some actions left behind
to do. More aggressively catch errors so that all cleanup happens

Change-Id: I7a73a36c6fc42d4eba636cf36c8cfffcea48a318
2014-09-03 17:03:27 -07:00
Christian Berendt 837d4f8265 Use except x as y instead of except x, y
According to https://docs.python.org/3/howto/pyporting.html the
syntax changed in Python 3.x. The new syntax is usable with
Python >= 2.6 and should be preferred to be compatible with Python3.

Enabled hacking check H231.

Change-Id: I4c20a04bc7732efc2d4bbcbc3d285107b244e5fa
2014-05-29 23:55:41 +02:00
Clark Boylan 92b0ab208a Better logstash field data.
We are currently using a lot of wildcard searches in elasticsearch which
are slow. Provide better field data so that we can replace those
wildcard searches with filters. In particular add a short uuid field and
make the filename tag field the basename of the filepath so that grenade
and non grenade files all end up with the same tags.

Change-Id: If558017fceae96bcf197e611ab5cac1cfe7ae9bf
2014-03-13 14:42:58 -07:00
James E. Blair 0a0b13c04b Use statsd in logstash client
Have the log-gearman-client (aka jenkins-log-client) initialize
the statsd parameters when starting the geard server.  Also, make
sure that the python statsd package is installed on the host.

Change-Id: I04fe1a7609f08bc710891b6a3b92d0f4d156d86c
2014-02-24 15:34:48 -08:00
Clark Boylan 7390aeb828 Handle log filter exceptions more gracefully.
If there is an exception filtering a log event handle that by removing
the filter and continuing to process the remaining log events for the
assocaited file. This prevents non filter data from being lost when the
filters have an exception.

Change-Id: I65141daf21a873096829c41fdc2c77cbeecde2e3
2014-02-10 10:20:12 -08:00
Clark Boylan 7dea09588f Close unneeded fds before execing CRM 114.
CRM 114 is being forked off of the gearman worker processes and as a
result has open fds for log files and tcp connections. CRM 114 should be
isolated from the fds so that it doesn't crash when they change
unexpectedly. Close the fds using the subprocess.Popen close_fds flag.

Change-Id: I4fbdf3564771be7d7a7e4c518e571634de576253
2014-02-05 09:44:55 -08:00
Clark Boylan 8a2ca3c689 Add grenade logs to logstash.
Add grenade new/ and old/ logs to logstash. To do this without tripling
HTTP GETs for every finished job add a job filter to the log gearman
client that when present only attempts to grab files if the job name
matches the job filter.

Change-Id: Ia33722bf71d482f2fd6b655b28090a10bf46af54
2014-01-31 14:14:12 -08:00
Jenkins 1b285dbab8 Merge "Add jenkins master name to the logstash records" 2014-01-28 17:50:56 +00:00
Davanum Srinivas 0857280e60 Add jenkins master name to the logstash records
Please see corresponding review for zmq-event-publisher:
https://review.openstack.org/#/c/67495/

This will help track down jenkins host/slave issues

Change-Id: I660252dc79f074b52587298120b4d6ceeedaf9a3
2014-01-28 00:58:17 +00:00
Joe Gordon cab484e84c Record build_branch in logstash
Since some bugs are branch specific we want to write logstash queries
that use the branch, so log ZUUL_BRANCH as build_branch in logstash.

From zuul's launchers doc:
ZUUL_BRANCH: The target branch for the change that triggered this build

Change-Id: Ic408afb235be5716231c663616c17a98ef6f8870
2014-01-17 10:35:36 -05:00
Davanum Srinivas 16e7cd5b0e Add jenkins slave name to the logstash records
zmq publisher already has the computer/node name information
per commit:
openstack-infra/zmq-event-publisher 36ca349

we just need to pull it out of the event and pass it along as
a new field

Change-Id: Iddefdf74ddf170eaafcd82c5e1f5b0389651cf89
2014-01-09 16:33:19 -05:00
James E. Blair 7b908f8d65 Refactor logstash_worker into log_processor module
Separate the jenkins log client and worker bits into a new module
called log_processor with ::client and ::worker classes.

Instantiate two workers on each logstash worker node.

Change-Id: I7cfec410983c25633e6b555f22a85e9435884cfb
2013-12-18 00:40:49 +00:00