This change completes the circular dependency refactor.
The principal change is that queue items may now include
more than one change simultaneously in the case of circular
dependencies.
In dependent pipelines, the two-phase reporting process is
simplified because it happens during processing of a single
item.
In independent pipelines, non-live items are still used for
linear depnedencies, but multi-change items are used for
circular dependencies.
Previously changes were enqueued recursively and then
bundles were made out of the resulting items. Since we now
need to enqueue entire cycles in one queue item, the
dependency graph generation is performed at the start of
enqueing the first change in a cycle.
Some tests exercise situations where Zuul is processing
events for old patchsets of changes. The new change query
sequence mentioned in the previous paragraph necessitates
more accurate information about out-of-date patchsets than
the previous sequence, therefore the Gerrit driver has been
updated to query and return more data about non-current
patchsets.
This change is not backwards compatible with the existing
ZK schema, and will require Zuul systems delete all pipeline
states during the upgrade. A later change will implement
a helper command for this.
All backwards compatability handling for the last several
model_api versions which were added to prepare for this
upgrade have been removed. In general, all model data
structures involving frozen jobs are now indexed by the
frozen job's uuid and no longer include the job name since
a job name no longer uniquely identifies a job in a buildset
(either the uuid or the (job name, change) tuple must be
used to identify it).
Job deduplication is simplified and now only needs to
consider jobs within the same buildset.
The fake github driver had a bug (fakegithub.py line 694) where
it did not correctly increment the check run counter, so our
tests that verified that we closed out obsolete check runs
when re-enqueing were not valid. This has been corrected, and
in doing so, has necessitated some changes around quiet dequeing
when we re-enqueue a change.
The reporting in several drivers has been updated to support
reporting information about multiple changes in a queue item.
Change-Id: I0b9e4d3f9936b1e66a08142fc36866269dc287f1
Depends-On: https://review.opendev.org/907627
This is part of the circular dependency refactor.
This updates the buildset object in memory (and zk) to store builds
indexed by frozen job uuid rather than job name. This also updates
everal related fields and also temporary dictionaries to do the same.
This will allow us, in the future, to have more than one job/build
in a buildset with the same name (for different changes/refs).
Change-Id: I70865ec8d70fb9105633f0d03ba7c7e3e6cd147d
If a configuration error existed for a project on one branch
and a change was proposed to update the config on another branch,
that would activate a code path in the manager which attempts to
determine whether errors are relevant. An error (or warning) is
relevant if it is not in a parent change, and is on the same
project+branch as the current patch. This is pretty generous.
This means that if a patch touches Zuul configuration with a
warning, all warnings on that branch must be updated. This was
not the intended behavior.
To correct that, we no longer consider warnings in any of the
places where we check that a queue item is failing due to
configuration errors.
An existing test is updated to include sufficient setup to trigger
the case where a new valid configuration is added to a project
with existing errors and warnings.
A new test case is added to show that we can add new deprecations
as well, and that they are reported to users as warnings.
Change-Id: Id901a540fce7be6fedae668390418aca06a950af
We recently added a severity field to configuration errors (but all
errors are currently at the "error" severity). To prepare for
"warning" severity, update the pipeline managers and reporters to
expect both warnings and errors.
Errors will still trigger buildset failures, but warnings will not.
Both will be reported as comments.
Change-Id: Ia24e91f5ddff7d9869e9e83886f996e4f425e110
Most of a change's attributes are tenant-independent. This however is
different for topic dependencies, which should only be considered in
tenants where the dependencies-by-topic feature is enabled.
This is mainly a problem when a project is part of multiple tenants as
the dependencies-by-topic setting might be different for each tenant. To
fix this we will only return the topic dependencies for a change in
tenants where the feature has been activated.
Since the `needs_changes` property is now a method called
`getNeedsChanges()`, we also changed `needed_by_changes` to
`getNeededByChanges()` so they match.
Change-Id: I343306db0abbe2fbf98ddb3f81b6d509eaf4a2bf
In the recent change to omit skipped jobs when reporting, we may
have swung the pendulum too far. While it seems that users may
not want to see a list of hundreds of skipped child_jobs, they may
want to see a list of a small number of skipped jobs due to failed
dependencies.
To try to thread the needle, we omit skipped jobs from the report
iff they were skipped due to zuul_return child_jobs; otherwise
we include them.
Change-Id: I66a223da344a93b4691a969876e887b5eec0e67c
This adds a config-error pipeline reporter configuration option and
now also reports config errors and merge conflicts to the database
as buildset failures.
The driving use case is that if periodic pipelines encounter config
errors (such as being unable to freeze a job graph), they might send
email if configured to send email on merge conflicts, but otherwise
their results are not reported to the database.
To make this more visible, first we need Zuul pipelines to report
buildset ends to the database in more cases -- currently we typically
only report a buildset end if there are jobs (and so a buildset start),
or in some other special cases. This change adds config errors and
merge conflicts to the set of cases where we report a buildset end.
Because of some shortcuts previously taken, that would end up reporting
a merge conflict message to the database instead of the actual error
message. To resolve this, we add a new config-error reporter action
and adjust the config error reporter handling path to use it instead
of the merge-conflicts action.
Tests of this as well as the merge-conflicts code path are added.
Finally, a small debug aid is added to the GerritReporter so that we
can easily see in the logs which reporter action was used.
Change-Id: I805c26a88675bf15ae9d0d6c8999b178185e4f1f
The report message "This change depends on a change that failed to
merge" (and a similar change for circular dependency bundles) is
famously vague. To help users identify the actual problem, include
URLs for which change(s) caused the problem so that users may more
easily resolve the issue.
Change-Id: Id8b9f8cf2c108703e9209e30bdc9a3933f074652
For heavy users of "dispatch jobs" (where many jobs are declared as
dependencies of a single job which then mutates the child_jobs
return value to indicate which few of those should be run), there
may be large numbers of "SKIPPED" jobs in the status page and in the
final job report, which reduces the usability of both of those.
Yet it is important for users to be able to see skipped jobs since
they may represent an error (they may be inadvertently skipped).
To address this, we remove "SKIPPED" jobs from the status page by
default, but add a button at the bottom of the change box which
can toggle their display.
We remove "SKIPPED" jobs from the report, but add a note at the
bottom which says "Skipped X jobs". Users can follow the buildset
link to see which ones were skipped.
The buildset page will continue to show all the jobs for the buildset.
Change-Id: Ie297168cdf5b39d1d6f219e9b2efc44c01e87f35
Previously support for Gerrit's submitWholeTopic feature was added
so that when it is enabled, changes that are submitted together are
treated as circular dependencies in Zuul. However, this feature did
not work in a gating pipeline because when that setting is enabled,
Gerrit requires all changes to be mergable at once so that it can
attempt to atomically merge all of them. That means that every
change would need a Verified+2 vote before any change can be
submitted. Zuul leaves the vote and submits each change one at a
time.
(Note, this does not affect the emulated submitWholeTopic feature
in Zuul, since in that case, Gerrit will merge each change alone.)
To correct this, a two-phase option is added to reporters. In phase1,
reporters will report all information about the change but not submit.
In phase2, they will only submit. In the cases where we are about
to submit a successful bundle, we enable the two-phase option and
report the entire bundle without submitting first, then proceed to
submit each change in the bundle in sequence as normal. In Gerrit,
if submitWholeTopic is enabled, this will cause all changes to be
submitted as soon as the first one is, but that's okay because we
handle the case where we try to submit a change and it is already
submitted.
The fake Gerrit used in the Zuul unit tests is updated to match
the real Gerrit in these cases. If submitWholeTopic is enabled,
it will return a 409 to submit requests if the whole topic is not
able to be submitted.
One unit test of failed bundle reporting is adjusted since we will
now report the buildset result to all changes before potentially
reporting a second time if the bundle later fails to merge.
While this does mean that some changes will have extra report
messages, it actually makes the behavior consistent (before, some
changes would have 2 reports and some would have only 1; now all
changes will have 2 reports: the expected result and then a second
report of the unexpected merge failure).
All reporters are updated to handle the two-phase reporting. Since
all reporters have different API methods to leave comments and merge
changes, this won't actually cause any extra API calls even for
reporters which don't need two-phase reporting.
Non-merging reporters (MQTT, SMTP, etc) simply ignore phase2.
Change-Id: Ibf377ab5b7141fe60ecfd5cbbb296bb4f9c24265
formatStatusUrl() return build set URL when the
item.current_build_set.result exist (zuul/model.py)
Also updated the quick-start to continue to look for the
build URL instead of the buildset URL.
Change-Id: I5f8433e2926da5a8d14b966d89cc943be1ecfca9
When Zuul dequeues a change because it no longer meets the code
review systems merge requirements, it says it was dequeued
"due to a missing requirement". To clarify it's the code review
system's merge requirements (as opposed to some kind of job,
artifact, or other change), update the text to "due to a
missing merge requirement".
Change-Id: Ia76c0564726c0e1317302e76f3f2bdae2c1ff431
We do not allow items to be enqueued into dependent pipelines if they
do not meet the approval/mergability requirements of the code review system.
For example, an "Approved" review in gerrit, or a required status check in
github. However, we do not check those once the item has been enqueued.
This means that if someone revokes approval, the item may remain in the queue
until it finishes, at which point it will fail to merge and be dequeued.
To avoid this, perform the canMerge check (which operates on local data)
each time we process the item.
Note: this does not perform Zuul pipeline requirement checks, but that would
be a natural follow-on from this change.
Change-Id: Iaf0ff530a9cb70052bf6a0908b28e2794dd110ae
We now report a distinct buildset result to the database if the upstream code
review system is unable to merge a change. Previously it was reported as
MERGE_CONFLICT which makes it difficult to distinguish from merge conflicts.
Essentially, the two states we're interested in are when Zuul's merger is
unable to prepare a git checkout of the change (99% of the time, this is
a merge conflict). This is the MERGE_CONFLICT result.
The second state is when Zuul asks Gerrit/Github/etc to submit/merge a change
and the remote system is unable (or refuses) to do so. This is MERGE_FAILURE.
Change-Id: I47af242aeb19ae4b4aedde353fcc76ff82d90fbe
This is a prelude to a change which will report a distinct buildset result
to the database if the upstream code review system is unable to merge a change.
Currently it is reported as MERGER_FAILURE which makes it difficult to
distinguish from merge conflicts.
Essentially, the two states we're interested in are when Zuul's merger is
unable to prepare a git checkout of the change (99% of the time, this is
a merge conflict). This will be known as MERGE_CONFLICT now.
The second state is when Zuul asks Gerrit/Github/etc to submit/merge a change
and the remote system is unable (or refuses) to do so. In a future change,
that will be reported as MERGE_FAILURE.
To avoid confusion and use names which better reflect the situation, this change
performs the rename to MERGE_CONFLICT.
Because there are pipeline configuration options tied to the MERGER_FAILURE
status (which start with 'merge-failure') they are also renamed to 'merge-conflict'.
The old names are supported for backwards compatibility.
A SQL migration takes care of updating values in the database.
The upgrade procedure is noted as being special because of the db value updates.
If an operator doesn't follow the recommended procedure, however, the consequences
are minimal (builds which won't be easily queried in the web ui; that can be
manually corrected if desired).
A model API change is not needed since the only place where we receive this value
from ZK can be updated to accept both values.
Change-Id: I3050409ed68805c748efe7a176b9755fa281536f
Provide separate methods for resolving change key and references to
change objects as suggested in
https://review.opendev.org/c/zuul/zuul/+/816460
Change-Id: I433b1cba3b0eec362a686682ce508fb82c3813ae
This stores necessary attributes from a Project directly on the
SourceContext instead of the whole Project.
This will help us to serialize the SourceContext without serialising the
Project as well as the Project also contains references to the
Connection and Source.
This is part of a bigger change to serialize the JobGraph (including all
frozen jobs).
Change-Id: Ib4037da2f7a0f803aca24ce880dbc262375db6a4
When there is a long list of build results, it is difficult quickly
to see which jobs are failed.
This change add emoji of check mark and cross in front of each job
to solve this problem.
Change-Id: I38177824767d475b0c4881ef0ab24ca1164d3187
In order to cache changes in Zookeeper we need to make change objects
JSON serializable. This means that we can no longer reference other
change objects directly. Instead we will use a cache key consisting of
the connection name and a connection specific cache key.
Those cache keys can be resolved by getting the source instance using
the connection name and then retrieving the change instance via the new
`getChangeByKey()` interface.
The pipeline manager provides a helper method for resolving a list of
cache keys. Cache keys that where resolved once are also cached by the
manager as long as the reference is needed by any change in the
pipeline. The cache will be cleaned up at the end of a run of the queue
processor.
Until we can run multiple schedulers the change cache in the pipeline
manager will avoid hitting Zookeeper every time we resolve a cache key.
Later on when we have the pipeline state in Zookeeper we probably want
to clear the change cache in the pipeline manager at the end of the
queue processor. This way we make sure the change is recent enough when
we start processing a pipeline.
Change-Id: I09845d65864edc0e54af5f24d3c7be8fe2f7a919
This change introduces a new class for storing runtime related system
attributes that are coming from the Zuul config. The system attributes
will be updated when the config changes instead of directly reading them
from the config on demand.
This gives us a clear interface for updating those values and storing
them in Zookeeper later on.
System config attributes currently are:
* [scheduler].relative_priority
* [scheduler].max_hold_expiration
* [scheduler].default_hold_expiration
* [scheduler].default_ansible_version
* [web].root
* [web].status_url
* [web].websocket_url
Change-Id: Iefc7857eddc3e0c068b10d01489aa6ae75a6374b
This moves some functions of the SQL reporter into the pipeline
manager, so that builds and buildsets are always recorded in the
database when started and when completed. The 'final' flag is
used to indicate whether a build or buildset result is user-visible
or not.
Change-Id: I053e195d120ecbb2fd89cf7e1e9fc7eccc9dcd2f
In order to reduce the chance of a partially merged cycle we make sure
all items in the cycle can still be merged prior to starting reporting.
Change-Id: I22a9e8b76325e09b6a0f6bc563840f264c12dfae
Allow Zuul to process circular dependencies between changes. Gating of
circular dependencies must be explicitly enabled on a per tenant or
project basis.
In case Zuul detects a dependency cycle it will make sure that every
change also include all other changes that are part of the cycle. However
each change will still be a normal item in the queue with its own jobs.
When it comes to reporting, all items in the cycle are treated as one
unit that determines the success/failure of those changes.
Changes with cross-repo circular dependencies are required to share the
same change queue.
Depends-On: https://review.opendev.org/#/c/643309/
Change-Id: Ic121b2d8d057a7dc4448ae70045853347f265c6c
We set url to the job name if it's missing, but a job name is not a url
nor is it additional information. Then, in dashboard displays
we have display code that sets url to null if it matches the job name.
Instead of fixing the data at the display layer, let's set it to null in
the first place. So that we can update the dashboard code to remove the
workarounds, let's also run a migration to update the previously saved
build data.
Change-Id: I80ce26de4abc15720d7e37aee73049423584d1b9
This patch provides the general functionality to allow reporting of
dequeued items and makes use of that in the Github checks API.
This reporting will only apply if the item wasn't a success or failure.
Change-Id: I1297da4d1708908c6b179110479fe0450e5550fe
Zuul is already providing these file comments via the zuul_return value.
So far, the Github reporter wasn't able to use those, but with the help
of the checks API we can add so called "annotations" to each check run.
Change-Id: Iff10172f95dc0430bec8e4dafb9a6c09bbe06077
This facilitates integration with the gerrit checks API (and may
prove useful for other similar APIs). It will allow us to report
that a change has no jobs in a particular pipeline. A Zuul
pipeline will correspond to a Gerrit check, which means we can
update the status for that check from "SCHEDULED" to "NOT_RELEVANT"
if we determine that no jobs should run for the change. This
closes out the status of the check in Gerrit when a project is
configured to participate in a check/pipeline but no jobs are
actually configured.
Test coverage for this will be added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.
Change-Id: Ide2a332b294d7efe23601d80eeb92b5af1d4c21b
This facilitates integration with the gerrit checks API (and may
prove useful for other similar APIs). It will allow us to report
that a change has been enqueued in a particular pipeline. A Zuul
pipeline will correspond to a Gerrit check, which means we can
update the status for that check from "NOT_STARTED" to "SCHEDULED"
when it enters the pipeline. This is important for our check
polling loop, and it will cause that check to stop appearing in
the list of pending checks.
Test coverage for this is added in change
Ida0cdef682ca2ce117617eacfb67f371426a3131.
Change-Id: I9ec329b446fa51e0911d4d9ff67eea7ddd55ab5d
By adding that attribute in the formater it is then possible to
report the buildset status URL as reporter start message.
For instance:
start-message: Build started. Ephemeral buildset status {status_url}
/{pipeline.tenant.name}/status/change/{change.number},
{change.patchset}.
Change-Id: I5f7503e4babc6b84b20292f2063ffd90cb6065d9
After a build finishes, if it returned file comments, the executor
will use the repo in the workspace (if it exists) to map the
supplied line numbers to the original lines in the change (in case
an intervening change has altered the files).
A new facility for reporting warning messages is added, and if the
executor is unable to perform the mapping, or the file comment syntax
is incorrect, a warning is reported.
Change-Id: Iad48168d41df034f575b66976744dbe94ec289bc
Only add line comments for configuration errors if they are for
files in the current change.
Put this in the reporter base class so that all drivers can use
it.
Add a test case for config errors in line comments in general,
and for this case as well.
Change-Id: I441474b93ba70c055f7f839ecf76547ac3356e0e
This change make the job name in the job report comments of github
driver as a hotlink to instead of using a log url directly. This can
make the report comments more brief.
Change-Id: I19fb8ffbc153230b7f8eedfcd5ac15ec81a66c72
This may be set by a project to help debug why a job is or is not
running. It works speculatively, and so can be used to debug
a single change.
Change-Id: I1957d21fe7775f786935e5d7d4bdf65b86eb5e4d
There are some errors that the executor may encounter where it will
be unable to, or refuse to, run a job. We know that these errors
will not be corrected by retrying the build, so return them as
errors to the user. The build result will be "ERROR" and the message
which is brief, but hopefully sufficient to illuminate the problem,
will be added to the job report.
Change-Id: Iad486199de19583eb1e9f67c89a8ed8dac75dea1
Story: 2001105
Story: 2001106
This is the last entry standing in the 'zuul' section, it seems
like it may be more appropriate here.
Change-Id: I48d2c4d69025a93a61c4d4fc64574545e3215e2c
It exists only for py2/py3 compat. We do not need it any more.
This will explicitly break Zuul v3 for python2, which is different than
simply ceasing to test it and no longer declaring we support it. Since
we're not testing it any longer, it's bound to degrade overtime without
us noticing, so hopefully a clean and explicit break will prevent people
from running under python2 and it working for a minute, then breaking
later.
Change-Id: Ia16bb399a2869ab37a183f3f2197275bb3acafee
This option was to maintain backwards compatability with the report
format used by the jenkins gerrit trigger plugin. We no longer
need to worry about that. With the addition of various reporters
which may need to report information in different formats, we
should probably either move this to the gerrit driver, or make it
customizable.
For now, let's go ahead and remove the option and use the behavior
we would have if it were set to true, which has been the case for
OpenStack infra's and all of our third-party ci users for some
time.
Change-Id: I8f040e03b7074cf59eced6fca38e6b221051d93e
The pipeline can be obtained by way of the item which was also being
passed in.
Change-Id: Ibcafa7daef3567c840a23defcab6fd40a9c6b206
Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
Source is available as part of the item so we don't need to pass it as a
parameter.
Change-Id: If871823f60f28665c87a705fb00cc30d7b849c12
Signed-off-by: Jamie Lennox <jamielennox@gmail.com>
E.g. https://zuul-server/#333 will set filter to 333
Introduce related configuration option
And always pass the full change url in github status payloads
Allow the status_url to be used in a pipeline start-message.
Change-Id: I7b9eccb1d218d65d8c6b01e042e6cce4f8aec89a
This has been moved to the job definition (with the expectation
that it will be set by a site-local base job).
Change-Id: Ibdb5968a090c335d5a8449466599da38d321d55e
James E. Blair