Commit Graph

10112 Commits

Author SHA1 Message Date
Simon Westphahl 9aea549305
Log Zookeeper session on connection state change
In order to identify e.g. the ephemeral owner of a node in Zookeeper it
is helpful to log the Zookeeper session ID when the connection is
established or changes.

Change-Id: I8a3645497ff73541b15d18ba32c20a7629792418
2024-03-27 12:17:49 +01:00
Zuul e1ad2443a3 Merge "Correctly limit buildsets with multiple refs" 2024-03-27 10:48:20 +00:00
Zuul 1561fe775e Merge "Web UI: substring search for builds, buildsets" 2024-03-27 10:27:49 +00:00
Zuul 15e41212c9 Merge "Fix Gantt chart overlap on dup build names" 2024-03-26 09:04:59 +00:00
Simon Westphahl 8bcfb8bc4a Correctly limit buildsets with multiple refs
The current SQL query will not correctly limit the number of buildsets
when some of the buildsets are related to multiple refs (circular
dependencies). The problem is that LIMTI works on number of rows, but we
want to limit only on the number of buildsets.

This corrects the problem by using a subquery to identify the distinct
buildsets, limiting that, and then querying for all of the information
about those buildsets.

This also updates the methods which perform the same function for builds,
even though we are not yet seeing an issue in practice.  It is
theoretically possible to call the getBuilds method with 'provides' and
'limit' arguments, which would produce the same problem as the buildsets
query.  That is not possible in practice, as the REST API doesn't support
provides, and the scheduler which does pass the provides argument doesn't
pass limit.  However, that could easily change in the future.  Additionally,
this future-proofs us if we add more queryable one-to-many relationships
to builds in the future (such as if we linked builds to multiple refs).
Also, it's easier to maintain these methods if they follow the same pattern.

There does not appear to be a performance loss in either mysql or postgres
with local testing on large data sets.  There may actually be an improvement
(but it's within the margin of error, so hard to say).

The index hints previously needed for mysql appear to no longer be
necessary and are removed.

Change-Id: Ib19e4cb8171f5d4d2873fb6b9c0301eb5d4ee43d
Co-Authored-By: James E. Blair <jim@acmegating.com>
2024-03-25 13:31:19 -07:00
Zuul 3a6208f342 Merge "Make ansible package check more robust" 2024-03-25 19:53:20 +00:00
Zuul 56a73f86eb Merge "Fix github docs for pull_request_review.state" 2024-03-25 19:53:18 +00:00
Zuul a3abea408b Merge "Emit per-branch queue stats separately" 2024-03-25 19:22:37 +00:00
Zuul 3b19ca9cb3 Merge "Add zuul_unreachable ansible host group" 2024-03-25 18:26:14 +00:00
James E. Blair 8821dbaa13 Make ansible package check more robust
The JSON parser will fail if given the empty string, so make sure
that if we don't get any output back, we don't try to parse it.

Additionally, if no extra packages are required, then don't bother
running the command in the first place.

Finally, include the missing package list in the error log message
rather than in a separate debug log entry, for easier correlation.

Change-Id: I0c39c74fdf05611439b35cd72b8ab70b836f4c1a
2024-03-25 11:25:35 -07:00
Zuul 0496c249be Merge "Reset jobs behind non-mergeable cycle" 2024-03-25 18:21:43 +00:00
Zuul ad1ef8e862 Merge "Add python 3.12 testing to Zuul" 2024-03-25 17:37:19 +00:00
Zuul 16af37e0c9 Merge "Replace pkg_resource dep resolution with pip" 2024-03-25 17:13:57 +00:00
Zuul 3ebe0cdc96 Merge "Convert pkg_resources usage to importlib" 2024-03-25 17:11:41 +00:00
Zuul 3917a4160d Merge "Use importlib instead of pkg_resources for ansible conf" 2024-03-25 17:11:38 +00:00
Zuul b2c0d69cbc Merge "Add a zuul.buildset_refs variable" 2024-03-25 10:09:16 +00:00
Zuul b0a7ed2899 Merge "Attempt to preserve triggering event across re-enqueues" 2024-03-25 10:09:13 +00:00
Zuul 51ec0d9159 Merge "Use the triggering change as the zuul change" 2024-03-25 09:56:06 +00:00
Zuul e8ff27ebe7 Merge "Don't reset buildset when cycle dependency merged" 2024-03-25 09:03:47 +00:00
James E. Blair 632839804c Add a zuul.buildset_refs variable
This adds information about the changes associated with a
circular dependency queue item.  Currently the bundle_id can be
used to identify which of the items in zuul.items is related to
the current dependency cycle.  That variable is deprecated, so
zuul.buildset_refs can be used to replace that functionality.

Since it repeats some of the information at the top level (eg
zuul.change, zuul.project, etc), the code is refactored so they
can share the dictionary construction.  That is also used by
zuul.items.  This results in a few extra fields in zuul.items
now, such as the change message, but that is relatively
inconsequential, so is not called out in the release notes.

The src_dir is similarly included in all of these places.  In
writing this change it was discovered that
zuul.items.project.src_dir always used the golang scheme, but
zuul.project.src_dir used the correct per-job workspace scheme.
This has been corrected so that they both use the per-job scheme
now.

A significant reorganization of the job variable documentation is
included.  Previously we had a section with additional variables
for each item type, but since many of these are duplicated at the
top level, in the item list, and now in the refs list, that
structure became difficult to work with.  Instead, the
documentation for each of these sections now exhaustively lists
all of the possible variables.  This makes for some repitition,
but it also means that a user can see at a glance what variables
are available, and we can deep-link to each one.  To address the
variation between different item types, the variables that mutate
based on item type now contain a definition list indicating what
types they are valid for and their respective meanings.

Change-Id: Iab8f99d4c4f40c44d630120c458539060cc725b5
2024-03-22 06:41:36 -07:00
James E. Blair dff9bba490 Add container image sanity checks
Since we're building skopeo ourselves in the container image, and
we don't have any functional testing of that, perform a sanity check.

Change-Id: I362e97d1120f3ce6c6be85ccf6948ced763b6cc1
2024-03-22 06:32:56 -07:00
Simon Westphahl 349c6a029d Don't reset buildset when cycle dependency merged
In case a live change depends on a cycle and the cycle is merged while
the item is still active the scheduler will detect the cycle as changed
and re-enqueue the dependent change.

The reason for this behavior is that we don't consider dependencies of
merged changes when building the dependency graph.

Change-Id: Ibc952886b56655c0705882497511b120e5a731cd
2024-03-21 13:35:50 -07:00
James E. Blair 179fa02ed0 Build a new skopeo for the zuul-executor container image
New versions of docker are no longer compatible with old versions
of skopeo.  To correct this, build a new version of skopeo for
the container images.  We need 1.14+ which is not available in
debian yet, so we build 1.15 (the latest tagged release) from
source.

Change-Id: I5a5c351e90b06d3acdd02f3117aa29eafb72445e
2024-03-21 12:48:32 -07:00
Clark Boylan 20657c13bd Add python 3.12 testing to Zuul
We use pyenv to install python3.12 on ubuntu jammy (which doesn't have
native packages for this version of python). We don't update the
container images just yet as we want to build confidence through testing
before end users see the new python version in production.

Change-Id: I20a57f6d29699886867ce912e9943c59aa9963f4
2024-03-20 09:17:19 -07:00
Clark Boylan dfe96519b1 Replace pkg_resource dep resolution with pip
Python 3.12 has more fully excised setuptools and as a result
pkg_resources. Since we are using pip to install things we can instead
rely on pip to resolve whether or not packages are installed. Do this so
that we don't need to explicitly install setuptools where it may not be
needed.

Change-Id: I8ee189c828914fd648847b5650b5fb2fb255ff17
2024-03-20 09:17:19 -07:00
Clark Boylan b52af834cc Convert pkg_resources usage to importlib
Importlib is the modern replacement for pkg_resources and is bundled in
python itself. Meanwhile pkg_resources is part of setuptools which is no
longer included in python as of python3.12. Do this transition to be
ready for python3.12 but also to modernize our package introspection.

Change-Id: I9a404e34ae2a833a925dcc156073e0f3f0680a11
2024-03-20 09:17:19 -07:00
James E. Blair 341e8d8ccd Use importlib instead of pkg_resources for ansible conf
Using importlib to read resource strings is said to be much faster
than pkg_resources[1].  See if we see a performance improvement.

Thanks to Clark Boylan for the suggestion.

[1] https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

Change-Id: I0b7d16d14989918682e137a56da7b0c65b14c336
2024-03-20 09:17:19 -07:00
Zuul 3bcd923a68 Merge "Handle dependency limit errors more gracefully" 2024-03-20 15:05:41 +00:00
Zuul c4d1db0093 Merge "Fix web dequeue of post items" 2024-03-19 18:00:48 +00:00
James E. Blair f695820f2e Fix Gantt chart overlap on dup build names
If a buildset has more than one build with the same name, the
Gantt chart on the buildset page will render both of them on the
same line which can obscure one of them.

Correct this by telling the chart library that the independent
variable is the build uuid (which is unique) rather than the name,
and then override the axis labels to show the name.

Change-Id: I07b4059ca75748910633f72458db1b5fb8a84041
2024-03-19 10:54:07 -07:00
Simon Westphahl 305d4dbab9
Handle dependency limit errors more gracefully
When the dependency graph exceeds the configured size we will raise an
exception. Currently we don't handle those exceptions and let them
bubble up to the pipeline processing loop in the scheduler.

When this happens during trigger event processing this is only aborting
the current pipeline handling run and the next scheduler will continue
processing the pipeline as usual.

However, in case where the item is already enqueued this exception can
block the pipeline processor and lead to a hanging pipeline:

ERROR zuul.Scheduler: Exception in pipeline processing:
Traceback (most recent call last):
  File "/opt/zuul/lib/python3.11/site-packages/zuul/scheduler.py", line 2370, in _process_pipeline
    while not self._stopped and pipeline.manager.processQueue():
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1800, in processQueue
    item_changed, nnfi = self._processOneItem(
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1624, in _processOneItem
    self.getDependencyGraph(item.changes[0], dependency_graph, item.event,
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
    self.getDependencyGraph(needed_change, dependency_graph,
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
    self.getDependencyGraph(needed_change, dependency_graph,
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
    self.getDependencyGraph(needed_change, dependency_graph,
  [Previous line repeated 8 more times]
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 813, in getDependencyGraph
    raise Exception("Dependency graph is too large")
Exception: Dependency graph is too large

To fix this, we'll handle the exception and remove the affected item.
We'll also handle the exception during enqueue and ignore the trigger
event in this case.

Change-Id: I210c5fa4c568f2bf03eedc18b3e9c9a022628dc3
2024-03-19 14:37:26 +01:00
Flavio Percoco d7b79f05a6 Use the pipeline from the buildset
There's no pipeline key in the ref object. This causes the queue
action to fail with a 400 error due to the missing pipeline key
in the request payload.

This change switches to using the buildset.pipeline value, which was
being used in the past and changed in 24e1cbc26d

Signed-off-by: Flavio Percoco <flavio@pacerevenue.com>
Change-Id: Icaf65d798b03c083944280140d25267351d61f78
2024-03-19 09:19:36 +00:00
Benjamin Schanzel 00d55851f9
Web UI: substring search for builds, buildsets
Allow to search for builds and buildsets using substrings of job_name,
project, branch, and pipeline.

This is the user-facing part of 908420.

Change-Id: If3c8b870a4f9feb8c50a19cda8115aef048f75b9
2024-03-19 09:41:35 +01:00
Zuul 4d06f081bd Merge "Zuul-Web: substring search for builds, buildsets" 2024-03-19 08:34:42 +00:00
Zuul fb796cc59a Merge "Use ProjectNotFoundError" 2024-03-19 08:22:24 +00:00
Zuul bdebb8471f Merge "Use NodesetNotFoundError class" 2024-03-19 08:16:49 +00:00
James E. Blair 99b3c11ce2 Use ProjectNotFoundError
This error is used in some places, but not all.  Correct that to
improve config error structured data.

Change-Id: Ice4fbee679ff8e7ab05042452bbd4f45ca8f1122
2024-03-18 15:09:47 -07:00
James E. Blair 1350ce8ad6 Use NodesetNotFoundError class
This error exception class went unused, likely due to complications
from circular imports.

To resolve this, move all of the configuration error exceptions
into the exceptions.py file so they can be imported in both
model.py and configloader.py.

Change-Id: I19b0f078f4d215a2e14c2c7ed893ab225d1e1084
2024-03-18 15:03:58 -07:00
Zuul 72849b2daa Merge "Add script to generate openapi spec" 2024-03-18 19:12:29 +00:00
James E. Blair c92b2c2221 Fix web dequeue of post items
This is a partial revert of 6c1c91f8ba
which relied on change.id being absent in the case that a queue item
was for a non-change ref; however, all queue items have an id.  This
reverts that section of code back to the previous regex-based
heuristic.

Change-Id: I07ff7c2925b44a7e85529491755b2f3d3f73eaf8
2024-03-15 07:37:24 -07:00
James E. Blair 91e22dfa96 Fix github docs for pull_request_review.state
The schema validation for these values is correct (if a little
generous) but the docs had some incorrect example values.  Update
them to match the schema.

Also, add comments to the schema validation indicating the source
of these somewhat mysterious values.

Change-Id: Idf26848a8c121d311590d07e003b53b8e867affd
2024-03-14 08:00:32 -07:00
Simon Westphahl 4680c58a27
Allow rerequested action for Github triggers
The 'requested' action is deprecated in favor of 'rerequested', but the
new schema did not permit the new action name.

Change-Id: I047d2676f44151e7569d38bc1df3d26ffee83202
2024-03-14 14:48:05 +01:00
Simon Westphahl 382e9d386c
Use Github label schema for 'unlabeled' actions
The schema validation for Github trigger events did not use the label
schema for 'unlabeled' actions leading to bogus config warnings.

Change-Id: I6c888d990047e611b560491be9bc784eb1981ada
2024-03-14 12:39:34 +01:00
James E. Blair 6ccbdacdf2 Attempt to preserve triggering event across re-enqueues
When a dependency cycle is updated, we will re-enqueue the changes
in the cycle so that each of the changes goes thorugh the process
of being added to the queue with the updated contents of the cycle.
That may mean omitting changes from the cycle, or adding new ones,
or even splitting into two cycles.

In that case, in order to preserve the idea of the
"triggering change", carry over the triggering change from when the
cycle was originally enqueued.

Note that this now exposes us to the novel idea that the triggering
change may not be part of the queue item.

Change-Id: I9e00009040f91d7edc31f4928e632edde4b2745f
2024-03-13 13:07:08 -07:00
James E. Blair 802c5a8ca6 Use the triggering change as the zuul change
In the case of circular dependencies with job deduplication, we
arbitrarily pick one of the changes as the zuul change (to use
when setting the zuul.change job variable and friends).  In
theory, it shouldn't matter which change we use, but in practice,
users may be surprised if it is something other than the triggering
change. Since it doesn't really matter to Zuul, let's set the zuul
change to the triggering change when possible.  It still needs to
be one of the changes for the job, so if the triggering change
itself doesn't actually run the job (easily possible if the job is
only run on dependent changes), then we will fall back to the
current behavior.  And of course the change must be one of the
item's changes, so in the case of linear dependencies, we're not
going to start setting it to some other queue item's change.

If we are unable to set it to the triggering change, then the
behavior remains undefined beyond setting it to one of the job's
changes arbitrarily.

Included in this change is a cleanup of a no-longer-needed api
migration from 12->13 related to EventInfo objects that was
missed due to a missing MODEL_API tag.

Information about the triggering change is added to the EventInfo
object to implement this feature.

Because the fallback behavior and the model upgrade behavior are
the same, we don't need to add any conditional api behavior or
upgrade testing -- in both cases we will simply use the current
behavior.

Change-Id: Iee5a7d975fea1f7491b652c406c24d73ada7a1a1
2024-03-13 13:07:08 -07:00
James E. Blair c2103f7058 Reset jobs behind non-mergeable cycle
In the case of a dependency cycle, we check the mergeability of
each change in the item before we try to merge any of them, and
dequeue the item if it looks like one of them won't be able to
merge.  However, that bypasses the normal behavior where we reset
changes behind failing items, which could lead to merging changes
that were tested with changes ahead that did not merge.

To correct this, update the cycle-can-not-be-merged dequeue stanza
with a reset, to mirror the stanza below which handles the failure
of any individual change to merge.

Change-Id: I52a9fc2da4dd89131722d69d2b5dea886eb3d51c
2024-03-13 09:03:16 -07:00
Zuul da07d5ff5e Merge "Report topic to jobs as zuul.topic" 2024-03-12 22:38:31 +00:00
Zuul 93d2118ecf Merge "Replace special characters in MQTT topic" 2024-03-12 14:32:49 +00:00
Benjamin Schanzel f9ebf6a1c9
Zuul-Web: substring search for builds, buildsets
Allow to search for builds and buildsets using substrings of job_name,
project, branch, and pipeline. This is done by placing wildcard
characters (*) into the filter string which get translated to SQL
wildcards (%), representing zero, one, or multiple characters.

Translating SQL style wildcards (%) to asterisks is done because
asterisks as wildcard chars might feel more intuitive, cf. shell file
globbing or regexp.

The SQL LIKE operator is only used if a wildcard is present in the
filter string. This is to not rely on the underlying SQL implementation
of optimizing queries with a LIKE op but no wildcard (ie. exact match),
so that we don't introduce unnecessary performance penalties.

Change-Id: I827a27915308f78fc01019bd988b34ea987c90ea
2024-03-12 13:58:01 +01:00
Zuul 1242e1b5f0 Merge "Include job dependency UUIDs in MQTT payload" 2024-03-12 11:06:52 +00:00