So far, when getting parent jobs recursively we've kept a set of tuples
with the job UUID and the dependency type (hard/soft) that needed to be
processed.
This worked as long as all relevant jobs used the same type of
dependency to a job. However, when hard/soft dependencies are mixed this
could lead to a job being processed twice. Zuul wrongly detected this as
a job dependency cycle.
This problem also lead to exceptions in pipeline processing as some
schedulers detected a dependency cycle whereas other did not. The reason
here is the non-deterministic odering of the set data type.
Since we don't need the information about the dependency type, we'll
just keep a set of job UUIDs to be processed, which prevents Zuul from
processing a job with a different type of dependency twice.
Note: The provided regression test did not fail consistently due to the
set non-determinism mentioned above.
Change-Id: I6459cbc90bf745ddda2da9dbc25bcee97005d933
Since squash merges will not create a merge commit Zuul is creating one
separately. However, this step can fail when there are no changes to
commit. This might for example happen in change-triggered post
pipelines.
To fix this issue we will restore the behavior used prior to
I3379b19d77badbe2a2ec8347ddacc50a2551e505 and allow creating empty
merge commits.
Change-Id: Ic18cb98f12260338799e963b8dc915c8be1d421b
Fixes the way that missing labels are handled in Gerrit. The intention
is that labels provided by Zuul are removed from the set of missing
labels on the change (and thus ignored). The original code was using the
">" set comparison operator to do this, but this operator is actually
"issuperset()". This means that if there was any disjoint members in the
allow_needs set (that is allow_needs had labels that were not missing),
the comparison would be False, and any actual missing labels would be
ignored.
The fix is to use set difference to calculate the missing labels and
remove the allow_needs set. If any labels are left after this, they are
actually missing and the change cannot be merged
Change-Id: Ibdb5df44e80d75198493f8287443ed19bcf269f1
When a change that is part of a circular dependency is abandoned we'd
set the item status to dequeued needing change. This will set all builds
as skipped, overwriting exiting builds.
This means that when the item was removed, we did not cancel any of the
builds. For normal builds this mainly waste resources, but if there are
paused builds, those will be leaked and continue running until the
executor is force-restarted.
The fix here is to cancel the jobs before setting it as dequeued needing
change.
Change-Id: If111fe1a21a1c944abcf460a6601293c255376d6
The only way to get reasonable performance out of all three databases
(mariadb, mysql, postgres) appears to be to make the following changes:
* Use group-by instead of distinct.
Some versions of mysql produce the wrong output with distinct;
this is more descriptive of what we want anyway.
* Split the query into two statements instead of joining on a subquery.
Mariadb seems to be unable to produce a good query plan for the
buildset list query. Neither mariadb nor mysql support using
"IN" with a subquery (so the idea of using IN instead of JOIN for
the subquery is out).
These methods now perform a query to get the ids of the builds or
buildsets that match the criteria, then perform a second query to load
the ORM objects that match those ids. This appears to be quite
fast for all three queries with the latest versions of all three database
systems.
Change-Id: I30bb3214807dfa8b26a848f85bb7a7bc660c6c1d
Artifacts processing assumed it was always operating on Change objects
as speculative artifact state only makes sense for speculative Changes.
However, this was not enforced and we would attempt to process artifacts
for refs/branches/tag/etc leading to this exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1065, in _executeJobs
self.sched.executor.execute(
File "/usr/local/lib/python3.11/site-packages/zuul/executor/client.py", line 63, in execute
params = zuul.executor.common.construct_build_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zuul/executor/common.py", line 97, in construct_build_params
) = item.getJobParentData(job)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zuul/model.py", line 5431, in getJobParentData
artifact_data = job.artifact_data or self.getArtifactData(job)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/zuul/model.py", line 5414, in getArtifactData
self.providesRequirements(job, data)
File "/usr/local/lib/python3.11/site-packages/zuul/model.py", line 5377, in providesRequirements
'change': change.number,
^^^^^^^^^^^^^
AttributeError: 'Tag' object has no attribute 'number'
Address this by checking that the object type is Change before we handle
artifacts for it. If it isn't a Change we report that there are no
satisfied provides/requires for this dependency by the currently
processed queue item.
Change-Id: I5ba7f30e56fc890847b86ee776e74272ec04046d
This restores the conditional around adding the mysql index hint
for the builds query. It was thought (incorrectly) that the hint
is now safe in all cases, but it is still the case that it can
slow down the query if some columns appear in the where clause.
Change-Id: Ieafab7e529e305f084ebf8c7844c8fa78da86902
Zuul packs refs directly rather than rely on git to do so. The reason
for this is it greatly speeds up repo resetting. Typically there are two
pieces of information for each packged ref (sha and refname). Git
annotated and signed tags are special because they have the sha of the
tag object proper, the tag refname, and finally the sha of the object
the tag refers to.
Update Zuul's ref packing to handle this extra piece of information for
git tags.
Co-Authored-By: James E. Blair <jim@acmegating.com>
Change-Id: I828ab924a918e3ded2cd64deadf8ad0b4726eb1e
We previously included this mysql index hint under some circumstances
for both build and buildset queries. It initially appeared to no
longer be necessary after recent table changes, however that does not
seem to be the case, and in fact, it is necessary for some simple
build table queries.
It does no longer appear to be necessary for the buildset table, so
it is not added back here.
When applied to MySQL 5.7, these hints result in the build and buildset
queries returning the first N rows rather than the last N rows as
expected.
We should be able to resolve that issue as well, but it will require
further changes. We're going to merge this as a short-term fix while
we work on a better solution. The Zuul community has been notified
that master is currently a work in progress and to temporarily avoid
continuous deployment under mysql/mariadb.
Change-Id: Ia6d962957af3cac87d174145e69571a405ed505b
https://review.opendev.org/c/908420 introduced a bug where literal
filters on builds and buildsets with sql wildcard characters (_, %) get
escaped, altering the search string. That leads to missing filter
results, e.g. when the user searches for a job_name with an underscore
in the name, because zuul escapes the underscore, searching for
`foo$_bar` instead of `foo_bar` (where `$` is the escape char). Fix this
by only applying the sanitation if we're going to filter with the sql
`LIKE` operator.
Change-Id: Iccef71fdf8a5a1150ff957be68358882e16e9da8
In order to identify e.g. the ephemeral owner of a node in Zookeeper it
is helpful to log the Zookeeper session ID when the connection is
established or changes.
Change-Id: I8a3645497ff73541b15d18ba32c20a7629792418
The current SQL query will not correctly limit the number of buildsets
when some of the buildsets are related to multiple refs (circular
dependencies). The problem is that LIMTI works on number of rows, but we
want to limit only on the number of buildsets.
This corrects the problem by using a subquery to identify the distinct
buildsets, limiting that, and then querying for all of the information
about those buildsets.
This also updates the methods which perform the same function for builds,
even though we are not yet seeing an issue in practice. It is
theoretically possible to call the getBuilds method with 'provides' and
'limit' arguments, which would produce the same problem as the buildsets
query. That is not possible in practice, as the REST API doesn't support
provides, and the scheduler which does pass the provides argument doesn't
pass limit. However, that could easily change in the future. Additionally,
this future-proofs us if we add more queryable one-to-many relationships
to builds in the future (such as if we linked builds to multiple refs).
Also, it's easier to maintain these methods if they follow the same pattern.
There does not appear to be a performance loss in either mysql or postgres
with local testing on large data sets. There may actually be an improvement
(but it's within the margin of error, so hard to say).
The index hints previously needed for mysql appear to no longer be
necessary and are removed.
Change-Id: Ib19e4cb8171f5d4d2873fb6b9c0301eb5d4ee43d
Co-Authored-By: James E. Blair <jim@acmegating.com>
The JSON parser will fail if given the empty string, so make sure
that if we don't get any output back, we don't try to parse it.
Additionally, if no extra packages are required, then don't bother
running the command in the first place.
Finally, include the missing package list in the error log message
rather than in a separate debug log entry, for easier correlation.
Change-Id: I0c39c74fdf05611439b35cd72b8ab70b836f4c1a
This adds information about the changes associated with a
circular dependency queue item. Currently the bundle_id can be
used to identify which of the items in zuul.items is related to
the current dependency cycle. That variable is deprecated, so
zuul.buildset_refs can be used to replace that functionality.
Since it repeats some of the information at the top level (eg
zuul.change, zuul.project, etc), the code is refactored so they
can share the dictionary construction. That is also used by
zuul.items. This results in a few extra fields in zuul.items
now, such as the change message, but that is relatively
inconsequential, so is not called out in the release notes.
The src_dir is similarly included in all of these places. In
writing this change it was discovered that
zuul.items.project.src_dir always used the golang scheme, but
zuul.project.src_dir used the correct per-job workspace scheme.
This has been corrected so that they both use the per-job scheme
now.
A significant reorganization of the job variable documentation is
included. Previously we had a section with additional variables
for each item type, but since many of these are duplicated at the
top level, in the item list, and now in the refs list, that
structure became difficult to work with. Instead, the
documentation for each of these sections now exhaustively lists
all of the possible variables. This makes for some repitition,
but it also means that a user can see at a glance what variables
are available, and we can deep-link to each one. To address the
variation between different item types, the variables that mutate
based on item type now contain a definition list indicating what
types they are valid for and their respective meanings.
Change-Id: Iab8f99d4c4f40c44d630120c458539060cc725b5
In case a live change depends on a cycle and the cycle is merged while
the item is still active the scheduler will detect the cycle as changed
and re-enqueue the dependent change.
The reason for this behavior is that we don't consider dependencies of
merged changes when building the dependency graph.
Change-Id: Ibc952886b56655c0705882497511b120e5a731cd
New versions of docker are no longer compatible with old versions
of skopeo. To correct this, build a new version of skopeo for
the container images. We need 1.14+ which is not available in
debian yet, so we build 1.15 (the latest tagged release) from
source.
Change-Id: I5a5c351e90b06d3acdd02f3117aa29eafb72445e
Python 3.12 has more fully excised setuptools and as a result
pkg_resources. Since we are using pip to install things we can instead
rely on pip to resolve whether or not packages are installed. Do this so
that we don't need to explicitly install setuptools where it may not be
needed.
Change-Id: I8ee189c828914fd648847b5650b5fb2fb255ff17
Importlib is the modern replacement for pkg_resources and is bundled in
python itself. Meanwhile pkg_resources is part of setuptools which is no
longer included in python as of python3.12. Do this transition to be
ready for python3.12 but also to modernize our package introspection.
Change-Id: I9a404e34ae2a833a925dcc156073e0f3f0680a11
When the dependency graph exceeds the configured size we will raise an
exception. Currently we don't handle those exceptions and let them
bubble up to the pipeline processing loop in the scheduler.
When this happens during trigger event processing this is only aborting
the current pipeline handling run and the next scheduler will continue
processing the pipeline as usual.
However, in case where the item is already enqueued this exception can
block the pipeline processor and lead to a hanging pipeline:
ERROR zuul.Scheduler: Exception in pipeline processing:
Traceback (most recent call last):
File "/opt/zuul/lib/python3.11/site-packages/zuul/scheduler.py", line 2370, in _process_pipeline
while not self._stopped and pipeline.manager.processQueue():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1800, in processQueue
item_changed, nnfi = self._processOneItem(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1624, in _processOneItem
self.getDependencyGraph(item.changes[0], dependency_graph, item.event,
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
self.getDependencyGraph(needed_change, dependency_graph,
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
self.getDependencyGraph(needed_change, dependency_graph,
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
self.getDependencyGraph(needed_change, dependency_graph,
[Previous line repeated 8 more times]
File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 813, in getDependencyGraph
raise Exception("Dependency graph is too large")
Exception: Dependency graph is too large
To fix this, we'll handle the exception and remove the affected item.
We'll also handle the exception during enqueue and ignore the trigger
event in this case.
Change-Id: I210c5fa4c568f2bf03eedc18b3e9c9a022628dc3
This error is used in some places, but not all. Correct that to
improve config error structured data.
Change-Id: Ice4fbee679ff8e7ab05042452bbd4f45ca8f1122
This error exception class went unused, likely due to complications
from circular imports.
To resolve this, move all of the configuration error exceptions
into the exceptions.py file so they can be imported in both
model.py and configloader.py.
Change-Id: I19b0f078f4d215a2e14c2c7ed893ab225d1e1084
The schema validation for these values is correct (if a little
generous) but the docs had some incorrect example values. Update
them to match the schema.
Also, add comments to the schema validation indicating the source
of these somewhat mysterious values.
Change-Id: Idf26848a8c121d311590d07e003b53b8e867affd
The 'requested' action is deprecated in favor of 'rerequested', but the
new schema did not permit the new action name.
Change-Id: I047d2676f44151e7569d38bc1df3d26ffee83202
The schema validation for Github trigger events did not use the label
schema for 'unlabeled' actions leading to bogus config warnings.
Change-Id: I6c888d990047e611b560491be9bc784eb1981ada
When a dependency cycle is updated, we will re-enqueue the changes
in the cycle so that each of the changes goes thorugh the process
of being added to the queue with the updated contents of the cycle.
That may mean omitting changes from the cycle, or adding new ones,
or even splitting into two cycles.
In that case, in order to preserve the idea of the
"triggering change", carry over the triggering change from when the
cycle was originally enqueued.
Note that this now exposes us to the novel idea that the triggering
change may not be part of the queue item.
Change-Id: I9e00009040f91d7edc31f4928e632edde4b2745f
Zuul