So far, when getting parent jobs recursively we've kept a set of tuples
with the job UUID and the dependency type (hard/soft) that needed to be
processed.
This worked as long as all relevant jobs used the same type of
dependency to a job. However, when hard/soft dependencies are mixed this
could lead to a job being processed twice. Zuul wrongly detected this as
a job dependency cycle.
This problem also lead to exceptions in pipeline processing as some
schedulers detected a dependency cycle whereas other did not. The reason
here is the non-deterministic odering of the set data type.
Since we don't need the information about the dependency type, we'll
just keep a set of job UUIDs to be processed, which prevents Zuul from
processing a job with a different type of dependency twice.
Note: The provided regression test did not fail consistently due to the
set non-determinism mentioned above.
Change-Id: I6459cbc90bf745ddda2da9dbc25bcee97005d933
Since squash merges will not create a merge commit Zuul is creating one
separately. However, this step can fail when there are no changes to
commit. This might for example happen in change-triggered post
pipelines.
To fix this issue we will restore the behavior used prior to
I3379b19d77badbe2a2ec8347ddacc50a2551e505 and allow creating empty
merge commits.
Change-Id: Ic18cb98f12260338799e963b8dc915c8be1d421b
Fixes the way that missing labels are handled in Gerrit. The intention
is that labels provided by Zuul are removed from the set of missing
labels on the change (and thus ignored). The original code was using the
">" set comparison operator to do this, but this operator is actually
"issuperset()". This means that if there was any disjoint members in the
allow_needs set (that is allow_needs had labels that were not missing),
the comparison would be False, and any actual missing labels would be
ignored.
The fix is to use set difference to calculate the missing labels and
remove the allow_needs set. If any labels are left after this, they are
actually missing and the change cannot be merged
Change-Id: Ibdb5df44e80d75198493f8287443ed19bcf269f1
When a change that is part of a circular dependency is abandoned we'd
set the item status to dequeued needing change. This will set all builds
as skipped, overwriting exiting builds.
This means that when the item was removed, we did not cancel any of the
builds. For normal builds this mainly waste resources, but if there are
paused builds, those will be leaked and continue running until the
executor is force-restarted.
The fix here is to cancel the jobs before setting it as dequeued needing
change.
Change-Id: If111fe1a21a1c944abcf460a6601293c255376d6
The only way to get reasonable performance out of all three databases
(mariadb, mysql, postgres) appears to be to make the following changes:
* Use group-by instead of distinct.
Some versions of mysql produce the wrong output with distinct;
this is more descriptive of what we want anyway.
* Split the query into two statements instead of joining on a subquery.
Mariadb seems to be unable to produce a good query plan for the
buildset list query. Neither mariadb nor mysql support using
"IN" with a subquery (so the idea of using IN instead of JOIN for
the subquery is out).
These methods now perform a query to get the ids of the builds or
buildsets that match the criteria, then perform a second query to load
the ORM objects that match those ids. This appears to be quite
fast for all three queries with the latest versions of all three database
systems.
Change-Id: I30bb3214807dfa8b26a848f85bb7a7bc660c6c1d
Zuul packs refs directly rather than rely on git to do so. The reason
for this is it greatly speeds up repo resetting. Typically there are two
pieces of information for each packged ref (sha and refname). Git
annotated and signed tags are special because they have the sha of the
tag object proper, the tag refname, and finally the sha of the object
the tag refers to.
Update Zuul's ref packing to handle this extra piece of information for
git tags.
Co-Authored-By: James E. Blair <jim@acmegating.com>
Change-Id: I828ab924a918e3ded2cd64deadf8ad0b4726eb1e
https://review.opendev.org/c/908420 introduced a bug where literal
filters on builds and buildsets with sql wildcard characters (_, %) get
escaped, altering the search string. That leads to missing filter
results, e.g. when the user searches for a job_name with an underscore
in the name, because zuul escapes the underscore, searching for
`foo$_bar` instead of `foo_bar` (where `$` is the escape char). Fix this
by only applying the sanitation if we're going to filter with the sql
`LIKE` operator.
Change-Id: Iccef71fdf8a5a1150ff957be68358882e16e9da8
This adds information about the changes associated with a
circular dependency queue item. Currently the bundle_id can be
used to identify which of the items in zuul.items is related to
the current dependency cycle. That variable is deprecated, so
zuul.buildset_refs can be used to replace that functionality.
Since it repeats some of the information at the top level (eg
zuul.change, zuul.project, etc), the code is refactored so they
can share the dictionary construction. That is also used by
zuul.items. This results in a few extra fields in zuul.items
now, such as the change message, but that is relatively
inconsequential, so is not called out in the release notes.
The src_dir is similarly included in all of these places. In
writing this change it was discovered that
zuul.items.project.src_dir always used the golang scheme, but
zuul.project.src_dir used the correct per-job workspace scheme.
This has been corrected so that they both use the per-job scheme
now.
A significant reorganization of the job variable documentation is
included. Previously we had a section with additional variables
for each item type, but since many of these are duplicated at the
top level, in the item list, and now in the refs list, that
structure became difficult to work with. Instead, the
documentation for each of these sections now exhaustively lists
all of the possible variables. This makes for some repitition,
but it also means that a user can see at a glance what variables
are available, and we can deep-link to each one. To address the
variation between different item types, the variables that mutate
based on item type now contain a definition list indicating what
types they are valid for and their respective meanings.
Change-Id: Iab8f99d4c4f40c44d630120c458539060cc725b5
In case a live change depends on a cycle and the cycle is merged while
the item is still active the scheduler will detect the cycle as changed
and re-enqueue the dependent change.
The reason for this behavior is that we don't consider dependencies of
merged changes when building the dependency graph.
Change-Id: Ibc952886b56655c0705882497511b120e5a731cd
When the dependency graph exceeds the configured size we will raise an
exception. Currently we don't handle those exceptions and let them
bubble up to the pipeline processing loop in the scheduler.
When this happens during trigger event processing this is only aborting
the current pipeline handling run and the next scheduler will continue
processing the pipeline as usual.
However, in case where the item is already enqueued this exception can
block the pipeline processor and lead to a hanging pipeline:
ERROR zuul.Scheduler: Exception in pipeline processing:
Traceback (most recent call last):
File "/opt/zuul/lib/python3.11/site-packages/zuul/scheduler.py", line 2370, in _process_pipeline
while not self._stopped and pipeline.manager.processQueue():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1800, in processQueue
item_changed, nnfi = self._processOneItem(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1624, in _processOneItem
self.getDependencyGraph(item.changes[0], dependency_graph, item.event,
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
self.getDependencyGraph(needed_change, dependency_graph,
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
self.getDependencyGraph(needed_change, dependency_graph,
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
self.getDependencyGraph(needed_change, dependency_graph,
[Previous line repeated 8 more times]
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 813, in getDependencyGraph
raise Exception("Dependency graph is too large")
Exception: Dependency graph is too large
To fix this, we'll handle the exception and remove the affected item.
We'll also handle the exception during enqueue and ignore the trigger
event in this case.
Change-Id: I210c5fa4c568f2bf03eedc18b3e9c9a022628dc3
This error is used in some places, but not all. Correct that to
improve config error structured data.
Change-Id: Ice4fbee679ff8e7ab05042452bbd4f45ca8f1122
This error exception class went unused, likely due to complications
from circular imports.
To resolve this, move all of the configuration error exceptions
into the exceptions.py file so they can be imported in both
model.py and configloader.py.
Change-Id: I19b0f078f4d215a2e14c2c7ed893ab225d1e1084
The 'requested' action is deprecated in favor of 'rerequested', but the
new schema did not permit the new action name.
Change-Id: I047d2676f44151e7569d38bc1df3d26ffee83202
The schema validation for Github trigger events did not use the label
schema for 'unlabeled' actions leading to bogus config warnings.
Change-Id: I6c888d990047e611b560491be9bc784eb1981ada
When a dependency cycle is updated, we will re-enqueue the changes
in the cycle so that each of the changes goes thorugh the process
of being added to the queue with the updated contents of the cycle.
That may mean omitting changes from the cycle, or adding new ones,
or even splitting into two cycles.
In that case, in order to preserve the idea of the
"triggering change", carry over the triggering change from when the
cycle was originally enqueued.
Note that this now exposes us to the novel idea that the triggering
change may not be part of the queue item.
Change-Id: I9e00009040f91d7edc31f4928e632edde4b2745f
In the case of circular dependencies with job deduplication, we
arbitrarily pick one of the changes as the zuul change (to use
when setting the zuul.change job variable and friends). In
theory, it shouldn't matter which change we use, but in practice,
users may be surprised if it is something other than the triggering
change. Since it doesn't really matter to Zuul, let's set the zuul
change to the triggering change when possible. It still needs to
be one of the changes for the job, so if the triggering change
itself doesn't actually run the job (easily possible if the job is
only run on dependent changes), then we will fall back to the
current behavior. And of course the change must be one of the
item's changes, so in the case of linear dependencies, we're not
going to start setting it to some other queue item's change.
If we are unable to set it to the triggering change, then the
behavior remains undefined beyond setting it to one of the job's
changes arbitrarily.
Included in this change is a cleanup of a no-longer-needed api
migration from 12->13 related to EventInfo objects that was
missed due to a missing MODEL_API tag.
Information about the triggering change is added to the EventInfo
object to implement this feature.
Because the fallback behavior and the model upgrade behavior are
the same, we don't need to add any conditional api behavior or
upgrade testing -- in both cases we will simply use the current
behavior.
Change-Id: Iee5a7d975fea1f7491b652c406c24d73ada7a1a1
In the case of a dependency cycle, we check the mergeability of
each change in the item before we try to merge any of them, and
dequeue the item if it looks like one of them won't be able to
merge. However, that bypasses the normal behavior where we reset
changes behind failing items, which could lead to merging changes
that were tested with changes ahead that did not merge.
To correct this, update the cycle-can-not-be-merged dequeue stanza
with a reset, to mirror the stanza below which handles the failure
of any individual change to merge.
Change-Id: I52a9fc2da4dd89131722d69d2b5dea886eb3d51c
Allow to search for builds and buildsets using substrings of job_name,
project, branch, and pipeline. This is done by placing wildcard
characters (*) into the filter string which get translated to SQL
wildcards (%), representing zero, one, or multiple characters.
Translating SQL style wildcards (%) to asterisks is done because
asterisks as wildcard chars might feel more intuitive, cf. shell file
globbing or regexp.
The SQL LIKE operator is only used if a wildcard is present in the
filter string. This is to not rely on the underlying SQL implementation
of optimizing queries with a LIKE op but no wildcard (ie. exact match),
so that we don't introduce unnecessary performance penalties.
Change-Id: I827a27915308f78fc01019bd988b34ea987c90ea
Since jobs are no longer identified by name but by UUID we also need to
reference job dependencies in the MQTT payload by UUID.
For backward-compatibility we'll keep the old "dependencies" field and
add a new "job_dependencies" mapping with the job names and UUIDs.
Change-Id: Ib74b11faf72602e1708ea6364cc4a1000e3f0d3b
The characters '+' and '#' have a special meaning (wildcards) and are
not allowed when publishing messages.
ERROR zuul.MQTTConnection: Could not publish message to topic 'foobar/zuul/c++-test' via mqtt
Traceback (most recent call last):
File "/opt/zuul/lib/python3.11/site-packages/zuul/driver/mqtt/mqttconnection.py", line 97, in publish
self.client.publish(topic, payload=json.dumps(message), qos=qos)
File "/opt/zuul/lib/python3.11/site-packages/paho/mqtt/client.py", line 1233, in publish
raise ValueError('Publish topic cannot contain wildcards.')
ValueError: Publish topic cannot contain wildcards.
Change-Id: Iad2ad551151284910de076cec15b3ac6b1cbda52
Reports the change topic to jobs as an ansible variable. This can be
useful for jobs that either want to name artifact output based on a
topic, or enforce that a topic is set using a zuul job.
Change-Id: I678404523d228947541160554623bf4066a729c4
With the circular dependency refactoring we also removed the job name
from the requestor data in the node request. However, this could
previously be used as part of the dynamic-tags in Nodepool which might
be useful for billing and cost calculations.
Add back the job name so those use-cases start working again.
Change-Id: Ie3be39819bf84d05a7427cd0e859f485de90835d
We currently emit 4 statsd metrics for each shared queue, but in
the case that a queue is configured as per-branch, we disregard
the branch and emit the stats under the same hierarchy for any
branch of that queue. This means that if we have a queue for
integrated-master and a queue for integrated-stable at the same
time, we would emit the stats for the master queue, then
immediately emit the same stats for the stable queue, overwriting
the master stats.
To correct this, move the metrics down a level in the case that
the queue is configured per-branch, and include the branch name
in the key.
Change-Id: I2f4b22394bc3774410a02ae76281eddf080e5c7f
In addition to the safeguard in
Iebf49a9efe193788199197bf7846e336d96edf19 we will only return the final
config for a project-branch as part of the merge result.
Change-Id: I1eb3b75d8762aff4e1ebd057661869df985a79e2
There are two issues with supercedent pipelines related to circular deps:
1) When operating in a post-merge configuration on changes (not refs), the
pipeline manager would throw an exception starting with 10.0.0 because
any time it operates on change objects, it attempts to collect the
dependency cycle before enqueing a change, and starting with 10.0.0,
the supercedent manager raises an exception in that case.
2) When operating in a pre-merge configuration on changes, the behavior
regarding circular dependencies was undefined before 10.0.0. It is
likely that they were ignored because the manager creates a dynamic
queue based on the project-ref, but it wasn't explicitly documented
or tested.
To correct both of these:
Override the cycleForChange method in the supercedent manager so that it
always returns an empty cycle.
Document the expected behavior.
Add tests that cover the cases described above.
Change-Id: Icf30d488334d40a929f31c2f390e18ae599a3c42
Since circular dependencies are now modelled as multiple changes in a
single item there is no good reason anymore not to deduplicate noop
jobs.
So far the noop job was listed separately for each change that was part
of a cycle and with that cluttered the UI, especially for large
dependency cycles.
Change-Id: Ic8d447df7d9040a767c88127ba361badc9ef016a
If the test suite is run in an environment with yappi or objgraph
installed, the stack_dump_handler test will activate them, and so
we need to call the handler a second time to turn them off.
Otherwise, test performance or errors can occur.
Change-Id: If073f0e46b24fc4e9f1281f911ce287f5c23d4dd
When an item updates config we will schedule a merge for the proposed
change and its dependencies.
The merger will return a list of config files for each merged change.
The scheduler upon receiving the merge result will combine the collected
config files for a project-branch from all involved changes.
This lead to the problem that the old content of renamed config files
were still used when building the dynamic layout.
Since the config we receive from the merger is always exhaustive, we
just need to keep the latest config files.
Another (or additional) fix would be to only return the latest config
files for a project-branch from the mergers. However, in case of
circular dependencies it could make sense in the future to get the
update config per change to report errors more precisely.
Change-Id: Iebf49a9efe193788199197bf7846e336d96edf19
Zuul