Commit Graph

1321 Commits

Author SHA1 Message Date
Simon Westphahl 8bcfb8bc4a Correctly limit buildsets with multiple refs
The current SQL query will not correctly limit the number of buildsets
when some of the buildsets are related to multiple refs (circular
dependencies). The problem is that LIMTI works on number of rows, but we
want to limit only on the number of buildsets.

This corrects the problem by using a subquery to identify the distinct
buildsets, limiting that, and then querying for all of the information
about those buildsets.

This also updates the methods which perform the same function for builds,
even though we are not yet seeing an issue in practice.  It is
theoretically possible to call the getBuilds method with 'provides' and
'limit' arguments, which would produce the same problem as the buildsets
query.  That is not possible in practice, as the REST API doesn't support
provides, and the scheduler which does pass the provides argument doesn't
pass limit.  However, that could easily change in the future.  Additionally,
this future-proofs us if we add more queryable one-to-many relationships
to builds in the future (such as if we linked builds to multiple refs).
Also, it's easier to maintain these methods if they follow the same pattern.

There does not appear to be a performance loss in either mysql or postgres
with local testing on large data sets.  There may actually be an improvement
(but it's within the margin of error, so hard to say).

The index hints previously needed for mysql appear to no longer be
necessary and are removed.

Change-Id: Ib19e4cb8171f5d4d2873fb6b9c0301eb5d4ee43d
Co-Authored-By: James E. Blair <jim@acmegating.com>
2024-03-25 13:31:19 -07:00
Zuul 56a73f86eb Merge "Fix github docs for pull_request_review.state" 2024-03-25 19:53:18 +00:00
Zuul 3ebe0cdc96 Merge "Convert pkg_resources usage to importlib" 2024-03-25 17:11:41 +00:00
James E. Blair 179fa02ed0 Build a new skopeo for the zuul-executor container image
New versions of docker are no longer compatible with old versions
of skopeo.  To correct this, build a new version of skopeo for
the container images.  We need 1.14+ which is not available in
debian yet, so we build 1.15 (the latest tagged release) from
source.

Change-Id: I5a5c351e90b06d3acdd02f3117aa29eafb72445e
2024-03-21 12:48:32 -07:00
Clark Boylan b52af834cc Convert pkg_resources usage to importlib
Importlib is the modern replacement for pkg_resources and is bundled in
python itself. Meanwhile pkg_resources is part of setuptools which is no
longer included in python as of python3.12. Do this transition to be
ready for python3.12 but also to modernize our package introspection.

Change-Id: I9a404e34ae2a833a925dcc156073e0f3f0680a11
2024-03-20 09:17:19 -07:00
Zuul 4d06f081bd Merge "Zuul-Web: substring search for builds, buildsets" 2024-03-19 08:34:42 +00:00
James E. Blair 1350ce8ad6 Use NodesetNotFoundError class
This error exception class went unused, likely due to complications
from circular imports.

To resolve this, move all of the configuration error exceptions
into the exceptions.py file so they can be imported in both
model.py and configloader.py.

Change-Id: I19b0f078f4d215a2e14c2c7ed893ab225d1e1084
2024-03-18 15:03:58 -07:00
James E. Blair 91e22dfa96 Fix github docs for pull_request_review.state
The schema validation for these values is correct (if a little
generous) but the docs had some incorrect example values.  Update
them to match the schema.

Also, add comments to the schema validation indicating the source
of these somewhat mysterious values.

Change-Id: Idf26848a8c121d311590d07e003b53b8e867affd
2024-03-14 08:00:32 -07:00
Simon Westphahl 4680c58a27
Allow rerequested action for Github triggers
The 'requested' action is deprecated in favor of 'rerequested', but the
new schema did not permit the new action name.

Change-Id: I047d2676f44151e7569d38bc1df3d26ffee83202
2024-03-14 14:48:05 +01:00
Simon Westphahl 382e9d386c
Use Github label schema for 'unlabeled' actions
The schema validation for Github trigger events did not use the label
schema for 'unlabeled' actions leading to bogus config warnings.

Change-Id: I6c888d990047e611b560491be9bc784eb1981ada
2024-03-14 12:39:34 +01:00
Zuul 93d2118ecf Merge "Replace special characters in MQTT topic" 2024-03-12 14:32:49 +00:00
Benjamin Schanzel f9ebf6a1c9
Zuul-Web: substring search for builds, buildsets
Allow to search for builds and buildsets using substrings of job_name,
project, branch, and pipeline. This is done by placing wildcard
characters (*) into the filter string which get translated to SQL
wildcards (%), representing zero, one, or multiple characters.

Translating SQL style wildcards (%) to asterisks is done because
asterisks as wildcard chars might feel more intuitive, cf. shell file
globbing or regexp.

The SQL LIKE operator is only used if a wildcard is present in the
filter string. This is to not rely on the underlying SQL implementation
of optimizing queries with a LIKE op but no wildcard (ie. exact match),
so that we don't introduce unnecessary performance penalties.

Change-Id: I827a27915308f78fc01019bd988b34ea987c90ea
2024-03-12 13:58:01 +01:00
Zuul 1242e1b5f0 Merge "Include job dependency UUIDs in MQTT payload" 2024-03-12 11:06:52 +00:00
Simon Westphahl ba19e1fa6d
Fix retried build result and URL in MQTT payload
The wrong build object was used when formatting the result and web-URL
for a retried builds.

Change-Id: I17e2caac833ab7969382257791d6160b2e25ade8
2024-03-11 15:54:50 +01:00
Simon Westphahl 7fd84658e3
Include job dependency UUIDs in MQTT payload
Since jobs are no longer identified by name but by UUID we also need to
reference job dependencies in the MQTT payload by UUID.

For backward-compatibility we'll keep the old "dependencies" field and
add a new "job_dependencies" mapping with the job names and UUIDs.

Change-Id: Ib74b11faf72602e1708ea6364cc4a1000e3f0d3b
2024-03-11 14:19:11 +01:00
Simon Westphahl c24314a47f
Replace special characters in MQTT topic
The characters '+' and '#' have a special meaning (wildcards) and are
not allowed when publishing messages.

ERROR zuul.MQTTConnection: Could not publish message to topic 'foobar/zuul/c++-test' via mqtt
Traceback (most recent call last):
  File "/opt/zuul/lib/python3.11/site-packages/zuul/driver/mqtt/mqttconnection.py", line 97, in publish
    self.client.publish(topic, payload=json.dumps(message), qos=qos)
  File "/opt/zuul/lib/python3.11/site-packages/paho/mqtt/client.py", line 1233, in publish
    raise ValueError('Publish topic cannot contain wildcards.')
ValueError: Publish topic cannot contain wildcards.

Change-Id: Iad2ad551151284910de076cec15b3ac6b1cbda52
2024-03-11 07:32:29 +01:00
Zuul 3d30928d39 Merge "Add some github configuration deprecations" 2024-03-01 18:54:10 +00:00
James E. Blair 171d4c56b1 Add some github configuration deprecations
The "event" trigger attribute can currently be a list.  Technically,
it is possible to construct a trigger with an event list, such as:

    trigger:
      github:
        - event:
            - pull_request
            - pull_request_review
          branch: master

Which would trigger on any pull_request or pull_request_review event
on the master branch.  However in practice users typically have much
more narrow event selections, such as only triggering on pull_request
events with the opened action, or a pull_request event with a certain
comment.  It is not possible to construct that example with a single
trigger; the following is invalid:

    trigger:
      github:
        - event:
            - pull_request
            - pull_request_review
          actions:
            - opened
            - commented
          branch: master
          comment: recheck

That will pass syntax validation but would only fire on a recheck
comment; it would never fire on a PR opened event because that event
won't have a comment.

To help users avoid these problems, or worse, let's limit the event
specifier to a single event (of course users can add more triggers for
other events).  That will allow us to inform users when they use
options incompatible with the event they selected.

For now, we make this a deprecation so that in the future we can
enforce it and improve feedback.

This adds syntax validation for each of the possible event/action
combinations in the case where the user has already specified a single
event.  This allows us to go ahead and issue warnings if users specify
incompatible options.  Later, all of these can become errors.

Some time ago (8.3.0) we deprecated the require-status attribute.  It
is eligible for removal now, but that predated the deprecation
warnings system.  Since we haven't yet removed it, and we now have
that system, let's add a deprecation warning for it and give users a
little more time to notice that and remove it before it becomes an
error.

When a Github user requests that a check run start again, Github emits
a "check_run" event with a "rerequested" action.  In zuul < 5.0.0, we
asked users to configure the check_run trigger with the "requested"
action and we silently translated the "rerequested" from github to the
zuul "requested".  In 5.0.0, we reversed that decision in order to
match our policy of passing through data from remote systems as
closely as possible to enable users to match the corresponding
documentation of zuul and the remote system.  We deprecated
"requested" and updated the examples in the documentation to say
"rerequested".  Unfortunately, we left the canonical documentation of
the value as "requested".  To correct this oversight, that
documentation is updated to say "rerequested" and a configuration
deprecation warning is added for uses of "requested".

The "unabel" trigger attribute is undocumented and unused.  Deprecate
it from syntax checking here so we can gracefully remove it later.

Some unit tests configs are updated since they passed validation
previously but no longer do, and the actual github pull request
review state constants ('approved', etc) are updated to match
what github sends.

Change-Id: I6bf7753d74ec0c5f19dad508c33762a7803fe805
2024-02-29 16:37:47 -08:00
James E. Blair ced306d5b1 Update gerrit changes more atomically
The following problem was observed:

Change A depends-on change B, which is in turn the tip of a
patch series of several changes.

Drivers warm the change cache on events by querying information
about changes related to those events.  But they don't process
depends-on headers, which means most drivers only warm one change,
and while the gerrit driver will follow other types of dependency
links which are unique to it, it stops at depends-on boundaries.

So in the example above, the only change in the cach which was warm
was change A.

The triggering event was enqueued, forwarded, and processed by
two responding pipelines simultaneously on two executors.

Each of them noticed the depends-on link and started querying gerrit
for change B and its dependencies.  One of the schedulers was about
1 second ahead of the other in this process.

In the gerrit driver, there is a two phase process for updating
changes.  First the change itself is updated in the normal way
common to all drivers, and then gerrit-specific dependency links
are updated.  That means the change is added to the change cache
with no dependencies, then mutated to add dependencies later.

The first scheduler added change B to the cache with no dependencies.
The second scheduler saw the update and refreshed its copy of B.
The second scheduler begin updating B, saw that the ltime of its
copy of B was sufficiently new it didn't need to update the cache
and stopped updating.
The second scheduler enqueued changes A and B, but no others in its
pipeline.
The first scheduler finished querying the stack of changes ending at
B, added them to the change cache, and mutated the entry for B in the
cache.
The first scheduler enqueued A, B, and the rest of the stack in its
pipeline.
The second scheduler updated its copy of B to include the new
dependencies.
The second scheduler ran a pipeline processor, noticed that B lacked
dependencies, and dequeued A and B, and reported an error to the user.

The good news is that Zuul noticed the mistake and dequeued the
changes.

To correct this, we will now collect all of the information about a
change and its gerrit-specific dependencies before writing any of
that information to the change cache.  This means that in our example
above, the second scheduler would not have aborted its queries.
Eventually, both schedulers would end up with the same information
before enqueing anything.

This process is still not quite optimal, in that we will have multiple
schedulers racing to update the same changes at the same time, but
they are designed to deal with collisions like that, so it should
at least be correct.

A future area of work might be to investigate whether we can optimize
this case further.

Change-Id: I647c2b54a55789e521fca71c8c3814907df65da6
2024-02-22 06:37:31 -08:00
Zuul 569fe78b5a Merge "Initialize github client manager if needed" 2024-02-16 17:41:06 +00:00
Zuul 1beac435ab Merge "Finish circular dependency refactor" 2024-02-10 21:27:22 +00:00
James E. Blair 1f026bd49c Finish circular dependency refactor
This change completes the circular dependency refactor.

The principal change is that queue items may now include
more than one change simultaneously in the case of circular
dependencies.

In dependent pipelines, the two-phase reporting process is
simplified because it happens during processing of a single
item.

In independent pipelines, non-live items are still used for
linear depnedencies, but multi-change items are used for
circular dependencies.

Previously changes were enqueued recursively and then
bundles were made out of the resulting items.  Since we now
need to enqueue entire cycles in one queue item, the
dependency graph generation is performed at the start of
enqueing the first change in a cycle.

Some tests exercise situations where Zuul is processing
events for old patchsets of changes.  The new change query
sequence mentioned in the previous paragraph necessitates
more accurate information about out-of-date patchsets than
the previous sequence, therefore the Gerrit driver has been
updated to query and return more data about non-current
patchsets.

This change is not backwards compatible with the existing
ZK schema, and will require Zuul systems delete all pipeline
states during the upgrade.  A later change will implement
a helper command for this.

All backwards compatability handling for the last several
model_api versions which were added to prepare for this
upgrade have been removed.  In general, all model data
structures involving frozen jobs are now indexed by the
frozen job's uuid and no longer include the job name since
a job name no longer uniquely identifies a job in a buildset
(either the uuid or the (job name, change) tuple must be
used to identify it).

Job deduplication is simplified and now only needs to
consider jobs within the same buildset.

The fake github driver had a bug (fakegithub.py line 694) where
it did not correctly increment the check run counter, so our
tests that verified that we closed out obsolete check runs
when re-enqueing were not valid.  This has been corrected, and
in doing so, has necessitated some changes around quiet dequeing
when we re-enqueue a change.

The reporting in several drivers has been updated to support
reporting information about multiple changes in a queue item.

Change-Id: I0b9e4d3f9936b1e66a08142fc36866269dc287f1
Depends-On: https://review.opendev.org/907627
2024-02-09 07:39:40 -08:00
Zuul 0e6b023a5f Merge "Optimize db prune query" 2024-01-30 22:39:21 +00:00
Artem Goncharov 7e7ce18e8c Check blocking_discussions_resolved in gitlab driver
In addition to `merge_status` attribute `blocking_discussions_resolved`
should be checked to know whether it makes sense to attempt merging.
In the project setting it is possible to enable "All discussions must be
resolved" check box what will result in the attribute to be set to false
once there are open discussions. With that (while merge_status is still
can_be_merged) the merge request can not be merged.

Sadly this is another badly documented case.

Change-Id: Iba3c7b424fb8acb3134622776eb1518ffddd5374
2024-01-24 20:35:28 +00:00
Zuul a1acf5f659 Merge "gitlab - avoid trigger build when reviewers added/removed" 2024-01-24 15:38:34 +00:00
Fabien Boucher ff94910877 gitlab - avoid trigger build when reviewers added/removed
This change fixes an unexpected behavior where Zuul triggers
the buildset when a reviewer is added or removed but also
in case of comment thread resolved.

Now the driver only rely on the 'update' action specifying
the 'oldrev' attribute for code change. Actually this is stated
in the doc that this attribute is set in case of code change.

https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html#merge-request-events

Change-Id: Ibee53e3e9ead9a0bbfdc0d60a35dcdd4b0a0dba7
2024-01-22 10:45:29 +00:00
Zuul 429c4da671 Merge "Fix Github protected check for renamed branches" 2024-01-16 16:27:43 +00:00
Simon Westphahl b6564e42ce
Fix Github protected check for renamed branches
When a branch is renamed in Github the REST API will redirect the
request to the endpoint for the new branch name. So far the the Github
client automatically followed those redirects and we did not check if
the branch name in the response matched our request.

This lead to cases where an old branch was added to the branch cache as
protected even though the branch no longer existed. This is not a
problem for the schedulers, but since there won't be any cached config
in Zookeeper, zuul-web will display a warning about missing config files
for the branch.

Since the REST endpoint also returns the (new) name of the branch we can
validate this against the requested branch name in addition to disabling
redirects.

However, this fix is not enough as the 'cachecontrol' adapter that we
use also caches the HTTP 301 redirects which is a problem when a new
branch with the same name as the renamed branch is created. To fix this
we will use a cache busting header (no-cache) to not return a cached
response in those cases.

Change-Id: I2670f951cac1bf41c6569f5495a60e9de262d4a4
2024-01-16 09:48:59 +01:00
Zuul 8d149846f3 Merge "Store builds on buildset by uuid" 2024-01-13 21:21:10 +00:00
James E. Blair 344ad5c3d3 Avoid joining the provides table in the builds query
The builds query includes an outer join on the provides table for
the purpose of allowing the scheduler to find previously completed
builds which provide artifacts for items currently in the queue.
Otherwise it is unused (it is not possible to query builds by
provides in the web api).

This join can sometimes produce very slow queries, especially
under postgres with certain data characteristics (usually when there
are too few rows to prompt the query planner to use indexes).  This
is similar to the problem that prompted the recent change to the
job runtime queries.

To avoid this, only join the provides table if required for the query.

Change-Id: I83810577c230bcd7365504e01e2cd1d1e642fa0e
2024-01-11 10:41:21 -08:00
James E. Blair a0cefe66c1 Optimize db prune query
We've added some more tables and the query used to delete old
buildsets suffered as a result.  The current approach performs
quite a lot of individual queries: one for each table multiplied
by each buildset that we inspect.

A faster approach is to use one query for each table, regardless
of the number of buildsets (note, however, that sqlalchemy will
shard this in order to manage query length; in practice, this means
one query for each table for every group of 500 buildsets -- so
a 1/500 reduction in the number of queries).

In local testing, this runs in 1/3 of the time as the current
code in postgres, and between 1/5 and 2/3 of the time with
mysql (performance appears more variable than postgres).

Change-Id: Ie93318ab12832516640ee9210ec071e18623eac4
2024-01-11 09:48:34 -08:00
Zuul 8a41a11885 Merge "Gerrit driver: fix for topics containing white space" 2024-01-10 16:00:23 +00:00
Zuul 9a387d0d85 Merge "Fix bug with cached merge modes in TPC" 2024-01-09 16:15:36 +00:00
Benjamin Schanzel 252b63f097
Gerrit driver: fix for topics containing white space
When using gerrit topics containing white spaces, zuul fails to find the
changes contained in the topic because the query it builds does not
enclose the topic in quotes. So only the first word of the topic is
considered by the gerrit driver. Fixing this by quoting the topic in the
query.

Change-Id: I99d2890d317fb8424740e25d166d17381f1319c8
2024-01-09 15:04:47 +01:00
Zuul cd310c9fde Merge "Set log_url for retried builds in MQTT payload correctly" 2024-01-09 08:41:21 +00:00
Felix Edel 733f02ae1e Set log_url for retried builds in MQTT payload correctly
Currently, the log_url for retried builds in the MQTT payload always
points to the build result page in Zuul web. As mqtt is meant to be
consumed by machines this breaks e.g. log post processing for those
builds.

To fix this, we do the same as for non-retried builds and provide a
dedicated web_url and log_url [1].

[1]: https://review.opendev.org/c/zuul/zuul/+/703983

Change-Id: I139a80d616d59e262a4f21772d7712fda3b5c03b
2024-01-08 17:11:49 +01:00
James E. Blair f99cee543e Use getBuildTimes for build time estimator
The existing build time estimator uses the normal getBuilds query
method.  We recently added an optimized query for a new build times
API endpoint.  As one would expect, the build time estimator has
limited query needs and should be able to use the new optimized
query as well.

The one thing missing is a search for a specific result (ie, SUCCESS),
so that is added.  The unused sort operator is removed.

From what I can tell, switching to sorting by end_time should not
produce a reduction in performance compared to sorting by id.

Change-Id: I1096d466accad5574b6cfa226e68b070f769128f
2024-01-04 06:30:52 -08:00
James E. Blair 50f068ee6d Add a build-times web endpoint
This endpoint runs an optimized query for returning information
suitable for displaying a graph of build times.

This includes a schema migration to add some indexes to aid
the query.

Change-Id: I56e8422a599c1ee51216f26fcae5a39013066e6b
2024-01-03 13:06:07 -08:00
Ahmon Dancy e916f151ff gitlabconnection.py: Handle 404 on unapprove
zuul/driver/gitlab/gitlabconnection.py:
  Handle a 404 response when attempting to unapprove an MR which does
  not have an existing approval.

  Ensure that an exception is raised in otherwise unexpected
  situations.

  The modified codepath is exercised by
  tests.unit.test_gitlab_driver.TestGitlabDriver.test_merge_request_commented

tests/fakegitlab.py:
  Make GitlabWebServer.post_mr_approve()/post_mr_unapprove() act more
  like real GitLab.

tests/fixtures/layouts/basic-gitlab.yaml:
  Add "approval: False" to the pipeline.check.start.gitlab to ensure
  that the test suite ends up trying to unapprove not-yet-approved MRs
  at the start of a pipeline.  This also makes the configuration more
  like the reference pipeline in the documentation.

Change-Id: Ia000b55e28c9628cf97682939215430baa78d065
2024-01-03 08:48:00 -08:00
Jeremy Stanley 5087f00ac5 Correct the timespec fields for the timer trigger
The parser rework for the new seeded PRNG jitter implementation in
change Idd9d3a8cfa791860e46e4fc508566417f5d5a9bf omitted the month
field and shifted the hour and day field by one. Correct this so
that the timer triggers continue to be parsed the same as before
that change.

Change-Id: If7988e28bd1e8fed7e21f2deb6eef9b51c8c2ae3
2023-12-17 15:39:15 +00:00
Zuul 3c1eaab8cc Merge "Use a seeded PRNG when setting timer triggers" 2023-12-13 10:38:41 +00:00
James E. Blair be4ab1bb38 Use a seeded PRNG when setting timer triggers
In the single scheduler days, we went to some effort to make sure
that we preserved apscheduler timer trigger jobs across reconfigurations
because to do otherwise risks missing zuul triggers if we reconfigure
within the jitter interval of a timer job.

In other words, if a timer trigger is set for midnight with a jitter
of 60s, and we reconfigure the tenant at midnight plus 10 seconds, if
the jitter would have caused the trigger to fire at midnight plus 30
seconds and the apscheduler job is canceled and recreated but this
time the jitter would cause it to fire at midnight plus 5 seconds,
the job would not fire.

Our mitigation for this was to preserve the apscheduler jobs across
reconfigurations, assuming nothing has changed.

In the multi-scheduler world, a similar problem arises when we perform
a rolling restart during a jitter interval.  Fixing this by recording
all of the jobs in ZK would be costly (as we create an apscheduler job
for each project-branch participating in a timer pipeline).  But we
can address the bulk of the problem just by ensuring that the same
jitter values are used on all schedulers.  This reduces the window for
missed jobs down to the time it takes for the schedulers to run the
timer trigger election (which, assuming an orderly shutdown, is very
fast).

In APScheduler the jitter value is calculated for every event.  This
new approach calculates a fixed jitter for a job, and will use that
value every time the job fires.  So if a job ends up running at
midnight plus 10 seconds one day, and no configuration changes,
it will run at midnight plus 10 seconds the next day.  This is
sufficient to achieve what we want to do with jitter: avoid starting
all the jobs at once.

We subclass the apscheduler CronTrigger to implement this custom
fixed jitter behavior and use a PRNG seeded with the tenant, project
and branch so that each scheduler will use the same value for each
combination, assuming they have the same configuration (and they
should).

Docs are updated to highlight that the same jitter value may be reused,
but no release note is added since the behavior change is not
significant.

Change-Id: Idd9d3a8cfa791860e46e4fc508566417f5d5a9bf
2023-12-12 13:25:14 -08:00
James E. Blair 9201f9ee28 Store builds on buildset by uuid
This is part of the circular dependency refactor.

This updates the buildset object in memory (and zk) to store builds
indexed by frozen job uuid rather than job name.  This also updates
everal related fields and also temporary dictionaries to do the same.

This will allow us, in the future, to have more than one job/build
in a buildset with the same name (for different changes/refs).

Change-Id: I70865ec8d70fb9105633f0d03ba7c7e3e6cd147d
2023-12-12 11:58:21 -08:00
Zuul 0396cd01c2 Merge "Add more zuul_ref indexes" 2023-12-08 07:33:46 +00:00
James E. Blair 164b1784c6 Add gerrit hashtags support
This adds support for the hashtags-changed trigger event as well
as using hashtags as pipeline and trigger requirements.

Change-Id: I1f6628d7c227d12355f651c3c822b06e2d5c5562
2023-12-07 07:07:14 -08:00
James E. Blair ef88c15405 Assign Gerrit change events to a patchset
We receive some events from Gerrit (such as the currently-unhandled
hashtags-changed or currently-partially-handled topic-changed) events
without a patchset, only a change number. This makes sense from a
data model perspective, but is not useful in Zuul since we must have
a patchset to actually enqueue something.

Currently we will handle these events by querying for the change and
storing the results in the change cache under the key (change, None).
Note that the change itself will have the current patchset assigned
as a result of the query.  But this "upgrade" of information applies
only to the change itself, not the change key, so this up-to-date
change object will never be used for anything productive in Zuul.

In order to actually trigger off of these events, let's "upgrade" the
change key as well after performing the query.  To do that, we will
use a new change key with patchset information when storing the change
object in the cache if our initial change key lacked patchset info but
the resulting change has it.  But only in the specific case where we
are performing our first query after receiving an event.  We will also
update the event with the same patchset information.  This should
mean that after receiving an event and performing the initial query,
we should be guaranteed to have patchset information about the change
and therefore Zuul should never see a (change, None) tuple for a
change key any more.

* (Change keys have more information than that tuple, but those are
   the relevant parts for this change.)

Change-Id: I6f077376044ffbbd3853e2050c507f449da77962
2023-12-07 07:07:08 -08:00
James E. Blair eaab8d6017 Add more zuul_ref indexes
This drops the (project, change) index in favor of several single-column
indexes that the dbms can combine as needed to quickly seek zuul_refs in queries.

Change-Id: I50ec1552e24c5aa634538ebb181da907898f1284
2023-11-28 07:55:18 -08:00
James E. Blair c071e729cc Fix some old entries in zuul_ref table
The recent db migration used existing build table entries to
populate the new zuul_ref table which is normalized so that it has
a single row for each (project, ref, change, patchset, oldrev, newrev)
combination.  Note that 'branch' is not included in that list because
a given ref implies a given branch.

However, some old (we're talking many, many, many years old) entries
in the builds table for periodic jobs were not made with a value in
the branch column.  This can cause problems with current jobs since
a periodic job today will use the same ref table entry as one from
10 years ago.  Periodic jobs have no change, patchset, oldrev, and
newrev fields, so the only unique attributes are project and ref.

When we collapsed the build table entries down to a single zuul_ref
row for each combination, if we picked one for a branch-tip ref as
described above with no value in the branch field, then as we
associate new builds with that ref, we won't be able to search for
them by branch (or even display the correct branch) since it will
be empty.

To correct this, update the zuul_ref table so that every row where
the branch is empty but the ref starts with 'refs/heads/' is updated
to include the branch (as read from the ref).

This migration can happen on-line with no coordination, and it should
take only a few seconds.

Change-Id: Idaa1ce91e9b763fd64e50f55bba4eb948ab21171
2023-11-21 12:21:55 -08:00
James E. Blair 4ac07c3a22 Log migration errors
If we encounter an error in sql migrations during startup, send it
to the python logger before we exit.

Change-Id: I574123b5e1eb27b75614eaf0a0b0235a4fd47adb
2023-11-17 09:48:30 -08:00
James E. Blair 489c477068 Fix bundle refactor sql to be compatible with mysql 5.7
Mysql 5.7 does not support "alter table drop constraint" so update
it to use "drop foreign key" instead.

Change-Id: Ied36b5868650391aa71df0bb108ef42113006588
2023-11-17 09:34:36 -08:00