Commit Graph

587 Commits

Author SHA1 Message Date
Jeremy Stanley dab0c7c606 Use Debian major version in Debian wheel filenames
Between September 4 and 5, our wheel cache publication began failing
for Debian distributions. The ansible_distribution_version fact
began to differentiate between Debian minor versions (so e.g.
bullseye went from "11" to "11.7"), and this does not match our
cache scheme in AFS so resulted in attempts to write to a
nonexistent directory. This corresponds to when
I9527fc847e18a80414139ebd6f19a9dbb9a5778e merged to switch the
tenant's default Ansible version to 8, so is very likely a behavior
change in Ansible itself.

Switch the logic in our wheel-mirror playbook to use
ansible_distribution_major_version like we do for Ubuntu, and rename
the tasks to be a little more generic.

Change-Id: Ie95e99bce1ed01aa9c284871b517e6c28c278d16
2023-10-06 13:45:03 +00:00
Ian Wienand 47d99524c5 wheel builds : move to individual releases
The publish jobs build and copy the wheels to the AFS r/w partitions,
but they are not released to the mirrors until a final "vos release"
of their volume.

Previously this happened in a final job that would not trigger if any
of the wheel builds failed.  This has meant thing like out-of-sync
centos mirrors or arm64 node failures have stopped all wheel
publishing previously.  There's really no need to block other
platforms publishing if one of these jobs fails.

This converts the jobs to have a release job syncrhonized by a
semaphore.  We don't want many "vos release" processes running all at
once, as it has been a source of failure with openafs before.  Since
releasing requires the admin key, we keep it in a separate job from
the building.

Change-Id: I10c307c2d46c0e5b86732943208c3167da28a694
2023-06-07 18:57:27 +02:00
Takashi Kajinami 8f183d1671 Periodically update Puppetfile_unit
This updates the script to automatically update puppet constraints so
that Puppetfile_unit is updated as well as Puppetfile. We introduced
this separate file[1] a while ago but have not been updated it
properly. Updating it periodically helps us catch up with any update
in the dependent modules more timely.

[1] 047ea125ddefa8151d6592d4772560a1b23a13dc

Depends-on: https://review.opendev.org/875301
Change-Id: I4d89a2985781f14d9ff7f76f412f0b8ccc4c6ddb
2023-02-26 01:48:56 +09:00
Ade Lee d322181322 Add base openstack FIPS job
A new job openstack-fips is created.  This job is expected to be
a base job for most OpenStack jobs, so that FIPS testing can be
easily enabled throughout OpenStack CI.

A base job is required here because, for Ubuntu nodes, we need
to enable an Ubuntu Advantage subscription, as Ubuntu considers
FIPS to be a UA feature.  The subscription key is stored here in
project-config.

Depends-On: I47a31f680172b47584510adb672b68498a85bd32
Change-Id: I8a88d6a9bcf5725986b00b063e03686d3225b48e
2023-02-01 09:59:36 +01:00
Ian Wienand 92d9cd7d8d
openstack-afs.yaml : correct indentation
This was found by the linter

Change-Id: I91f8b9264b17a27855942c5e375d81ce0b1c99d9
2023-01-11 08:57:11 +11:00
Jeremy Stanley 1ea6374489 Skip existing remote artifacts during PyPI upload
In order to be able to safely re-enqueue tags which previously
failed release jobs after successfully uploading at least some
artifacts to PyPI, instruct twine to treat "file already exists"
responses as benign and ignore them, proceeding to upload any others
which aren't yet there.

Depends-On: https://review.opendev.org/864004
Change-Id: Iab38df6386ce5219a52787fda5e64a8faab23a06
2022-11-08 15:41:47 +00:00
Dr. Jens Harbott 702037af47 Switch the requirements-constraints job to py310
Since the PTI switched the supported python versions for the current
cycle from 3.8+3.9 to 3.9+3.10, we want to update the constraints
generation accordingly. Since we don't have a stable platform where we
can run both py3.9 and py3.10 in parallel, generate constraints only for
py3.10 for now.

Change-Id: I4a4d9cb6e292bb693ae424be0928501fed81113f
2022-10-12 05:38:30 +00:00
Ian Wienand ae2b728536 linter: use spaces between filters in jinja
A later version of ansible-lint picks this up; fix lack of whitespace
around filters.  No operational change.

Change-Id: If042a0f92a74ea06a312aad68de21f9e44f16582
2022-07-28 12:03:59 +10:00
Ian Wienand 76037c434e linter: add name to blocks
A later version of ansible-lint enforces names on blocks.  This is
generally a good rule; fix a few missing blocks here.

Change-Id: Ia87a0c21ec0ed1662e37cbc9e17a0df344b54e57
2022-07-28 12:03:23 +10:00
Ian Wienand 8cdf719bb3 linter: update some YAML whitespace issues
These were found by a later version of ansible-lint.  This should have
no effect, but just fixes some inconsistent whitespace issues.

Change-Id: I7bcde4942c97cfe743e8aba74833aeb5844c8290
2022-07-28 12:02:45 +10:00
Ian Wienand 3bcb5fca79 trigger-rtd: fix role syntax
This doesn't need a name: when included like this.  I guess Ansible
accepts it, but it doesn't document this working anywhere.

Change-Id: Ibe6fa9296041af54f39a3ef69957d0e8a0df0259
2022-07-28 12:01:50 +10:00
Clark Boylan 55215ac11c Set ensure_pip_virtualenv_command in Focal proposal jobs
Previously we had this error:

  No key/value pairs provided, at least one is required for this action to succeed

Which appears to be due to the default(omit) condition when setting
ensure_pip_virtualenv_command in focal proposal jobs. The issue is
default(omit) doesn't do what you think it does when defining a
variable. It is really only useful for defining module parameters that
may be omitted.

Instead we need to call ensure-tox two different ways depending on
whether or not we want to override the defaults for
ensure_pip_virtualenv_command. One method for Bionic and another for
Focal keyed off of whether or not _venv_command is defined.

Change-Id: I0cbca64f4a31c8b4eacb5e1c50f2e9fb289ce18e
2022-07-27 09:08:04 -07:00
Ian Wienand 4616695be0 proposal updates: fix typo in release match
This fixes a typo in I683a61b1bae1a809caf724aee87a21af2e18fb0c -- it
should be "ansible_distribution_release"

Change-Id: I074f089371054856a8b96bd6e650496a0bd53685
2022-07-27 14:49:09 +10:00
Ian Wienand 6f1e9fb2b9 translation updates: refactor Python 3.8 workarounds
I think we have quite a tangle of fixes on fixes in the translation
jobs, and I hope this refactor makes things a bit better.

The fundamental problem seems to be that python3.6 can't work with
master requirements.txt.  But we can not move some jobs off bionic
(python 3.6) because of java version contsraints.

The first thing that happened was these jobs got "ensure-python" with
"python_version: 3.8" set to bring in python3.8 on Bionic.  This role
was not bringing in python3.8-venv, so we had no way to create 3.8
virtual environments.  Thus ensure-pip ended up getting modified to
drag this package in (reverted with notes in
372f3af706c796331578ef81c2680e8c4c03c3b6).  This has a fairly easy
solution; let's just make ensure-python bring in the venv packages for
the udpated python -- see dependency
Ie3c03fea82bcec80a897f0905c15f35405a50396.

The git-review install here uses 'ensure_pip_virtualenv_command' to
install git-review in a virtualenv.  git-review still supports bionic,
and does not need to be installed in any speical way.  However, by
overriding the 'ensure_pip_virtualenv_command' for the entire job,
this unrelated install now grew a dependency on python3.8 and created
more confusion over why it was failing.

I believe what we actually want to do on Bionic is just run tox under
python3.8.  So this pulls out all the job-level defintions and
encapsulates them in a single block scoped to run on bionic.  It will
pull in python3.8 packages, and then install tox under python3.8 with
an override of the virtulenv command just for that step.  The jobs
just maintain their node definition and note on why it is required.

The long-term future of these jobs is obviously a question; but I
believe this keeps them running on bionic in as sane a way as possible
for now.

The second part of this is the python3.9 interpreter required on
Focal.  Setting "python_version: 3.9" in the job definition is a bit
confusing, because the job actually requires both Python 3.8 AND 3.9.
This variable is passed to ensure-python, and the naming is a bit
unfortunate because it is so broad (but we can't change that without
zuul-jobs updates).  Here I've pulled out the Python 3.9 install into
a separate focal-gated section in the pre.yaml that explains what is
going on (note that Python 3.8 is just the base python on focal, so
needs no extra installation).  I've also updated the description on
the job node with this info.  There's no need to install the tox
environment any differently on focal; it just uses the base system
python (3.8) which is fine.

Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/850957
Change-Id: I547e5eb7faabcd9d3983ae85f3291b1a740bc77c
2022-07-27 07:48:43 +10:00
Ian Wienand 8a36fcb9c0 Remove testpypi references
These are vestigial parts of pypi upload work that are no longer used.

Change-Id: Ie0ac2a115630a6400927eb47449b3fc805c04485
2022-07-14 08:17:43 +10:00
yatinkarel be4b276729 Only install python3.8-venv in translation update jobs
Currently both python3.8 and python3.9 venv package
is being installed, but since python3.9-venv is not
available in bionic, python3.8-venv installation also
fails as mentioned in [1].
This patch moves ensure_pip_from_upstream_interpreters
var from proposal pre playbook to job definition and
override it in translation jobs to attempt installation
only for python3.8-venv.

Also switch back to "python3.8 -m venv" virtualenv_cmd
as with this patch python3.8-venv will get installed.

[1] https://review.opendev.org/c/zuul/zuul-jobs/+/846201
[2] https://review.opendev.org/c/openstack/project-config/+/846390

Change-Id: I9d9a423e3147d3f04c1951a796636e5520202a6d
2022-06-20 22:50:41 +05:30
yatinkarel 0ef308725f [proposal-updates] Run ensure-python before git-review setup
Install git-review task relies on ensure_pip_virtualenv_command
variable but with [1] this is set to python3.8 in translation
update jobs but it get's installed later by ensure-python role,
let's run it earlier to fix the issue.

[1] https://review.opendev.org/c/openstack/project-config/+/845907

Change-Id: Iecbdcb50063cb461b3a09ddfa205d071c7d77b75
2022-06-16 11:37:05 +05:30
Dr. Jens Harbott c7d42980c6 Fix propose-updates job for requirements (3rd attempt)
The previous attempts at fixing the job were not successful yet, it
seems the variable in the pre playbook isn't overridden by a job
variable. Since the other uses of this job are running with ubuntu-focal
anyway, set the list of available python interpreters in the playbook
directly and remove the modifications from the requirements specific job
again. Also pin the nodeset for this job to guard against changes in the
base job, which will likely require another change to the interpreter
list.

[0] I9735f34639777c928fbe06febc16dd24667b1405
[1] I52cc316ce9756c4411c18c8339d44c457cb59897

Change-Id: Iddd61e1be766f2bb6da4bf62766829247fa143ce
2022-05-26 23:00:13 +02:00
Dr. Jens Harbott 31714e97bb Build and publish wheel mirror for Ubuntu Jammy
Depends-on: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/842839
Change-Id: If43044419b68702fd540486aacd2aad75ea9e196
2022-05-21 14:42:00 +02:00
Matthew Thode 91bbf1fdc6
update generate constraints to py38,39
3.10 will come later when there is an image that supports all three

Change-Id: I663de65a971337fce65a51f0e5a7f959b8da2c03
Signed-off-by: Matthew Thode <mthode@mthode.org>
2022-04-14 10:58:33 -05:00
Zuul f42148b027 Merge "Build and publish wheel mirror for CentOS Stream 9" 2022-04-11 12:34:42 +00:00
Marcin Juszkiewicz 7e8a32266b Build and publish wheel mirror for CentOS Stream 9
Depends-on: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/836829

Change-Id: I4923b2bfaf7c5bbfa327aa64f9c3ae3fd658e0cf
2022-04-06 13:19:40 +00:00
Marcin Juszkiewicz 71b0e43491 Remove CentOS 8 wheel mirrors
Change-Id: I149cdd8635a1785e9e51757992d376a9e461ee6a
2022-04-06 12:37:12 +02:00
Jeremy Stanley 17aff0721b Match configure-mirrors for CentOS wheel URLs
The configure-mirrors role in zuul-jobs expects CentOS wheels will
be in URLs like distro-ver-arch and so doesn't include "-stream"
like our wheel publishing jobs are adding. This was a necessary
distinction in order to differentiate between CentOS Linux 8 and
CentOS Stream 8 wheels, but since configure-mirrors didn't know to
tell Stream nodes to look in the other location, this was actually
broken from the moment it was introduced and we simply never noticed
until we removed the old centos-8 volume (at which point
centos-8-stream nodes no longer found any wheels).

As we no longer have centos-8 nodes, rather than try to change how
configure-mirrors works (since it may be successfully used in other
sites already and would need advance warning or complex overrides in
base jobs), simply rearrange our mirrors to mount the volumes in the
paths it expects. Once this merges I'll adjust AFS accordingly.

This partially reverts 3d776f6734.

Change-Id: I07ee7ccdd10e68f6f38ec362ce7e5d333d5fc2f7
2022-04-01 16:04:57 +00:00
Jeremy Stanley 3c9b1c3bf1 Read tenant config with care in github_manager
The addition of ACLs to zuul/main.yaml broke the naive parsing done
by the maintain-github-mirror playbook, as it assumed all list items
were tenant definitions. Add a conditional check to allow it to skip
past any which aren't.

Change-Id: I31ab916a17fd116f00e94ea7bfa4d7243c0e1c2c
2022-02-15 20:24:24 +00:00
Ghanshyam Mann 3b3b8bf240 Remove 'publish-training-labs-scripts' definition
Job 'publish-training-labs-scripts' definition is moving to
openstack/training-labs stable branch in
- https://review.opendev.org/c/openstack/training-labs/+/818626

Change-Id: Ib89369ed602c48a3441d07398c37287b24ed8d16
2021-11-19 20:19:30 +00:00
Ian Wienand 3d776f6734 Add CentOS 8 Stream wheel publish jobs
Depends-On: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/802988
Change-Id: Id71888115597894d4dcb5c12731f819177fd9341
2021-08-05 08:20:04 +10:00
Ian Wienand 3a401e0dea wheel: don't try to release Xenial ARM64
This was missed with Ia44c384072c0482cfd11c642013fd51004f85c8b when we
removed the Xenial ARM64 jobs.

Also cleanup the grafana; additionally remove CentOS 7 ARM64
references which are not present.

Change-Id: If5fb2a1fe2cce058f1d516d5f5f7180a6cab06cf
2021-06-28 11:57:03 +10:00
Ian Wienand aaf59df628 Remove publish-irc-meetings job
The irc-meetings are now published to AFS
(I6f3a9970d907f2ae1420e1523cdf8a7c50647235) and served statically from
there.

Depends-On: https://review.opendev.org/c/opendev/system-config/+/794089
Change-Id: Ib852eac630f5aedcb4cd1d240fee63405d50188e
2021-06-02 13:45:45 +10:00
Dmitriy Rabotyagov 4ac828fb7b Add Debian Bullseye nodepool images and wheels
Since Debian Buster can not be used with nova 23.0.0 because of the
min required libvirt version, we should make Bullseye available for CI
to ensure that OpenStack Wallaby release will run on it smoothly.

Depends-On: https://review.opendev.org/c/openstack/diskimage-builder/+/783790
Change-Id: I9c1bb7aaa02ba60ee52e2d7b990e2e6e1212317f
2021-04-19 14:16:38 +00:00
Jeremy Stanley a4c873b51a Clean up pti-python-tarball python-branch-tarball
OpenStack's Python release jobs don't run tox, and don't install the
projects being packaged, not even in the branch tarball variants
these days. Remove the bindep, ensure-tox and collect-tox-logs roles
from pti-python-tarball and python-branch-tarball playbooks so that
the jobs using them can be more portable between distro versions
with less work. This should also make the jobs considerably faster.

Change-Id: Icc30a79f770ed78674354f6226d28907bab7eb1d
2020-11-13 18:11:44 +00:00
Zuul affb60c564 Merge "Updates Message on PR Close" 2020-09-02 22:49:35 +00:00
Kendall Nelson f493ac98b8 Updates Message on PR Close
Points to contributor guide instead of infra manual and mentions
the First Contact SIG.

Change-Id: Id06f5c36ee3688936d73f00520590198461514fc
2020-09-01 16:44:02 -07:00
Thierry Carrez 0f12fa2d5a [maintain-github-mirror] Skip abandoned gov files
Following the deletion of the foundation-board-repos.yaml file,
the script failed as it was expecting it.

Remove that file from the scanned list, and make the script more
resilient in case other extra files go missing in the future (like
user-committee.yaml which should soon be cleaned up).

Change-Id: I83cff14f19a829b5e56771442ffb3a0341e82d69
2020-08-10 11:13:15 +02:00
Zuul 36b665536c Merge "maintain-github-mirror: add requests dependency" 2020-07-14 15:26:29 +00:00
Zuul 131b76147c Merge "update-constraints: Install pip for all versions" 2020-07-14 14:43:17 +00:00
Thierry Carrez baa350ef23 maintain-github-mirror: add requests dependency
maintain-github-mirror makes direct use of the requests module. That
module is currently installed as a dependency of PyGithub, but that's
brittle, as PyGithub may well opt for a different library in the
future.

Let's proactively fix that before it bites us.

Change-Id: I8fc288e6c3978e20ddf48f905968931f89834e53
2020-07-13 17:07:07 +02:00
Andreas Jaeger 7a8ed31b71 Fix maintain-github-mirror
The script needs yaml, install it.

Change-Id: Ied64412a8814978d0c15acb7bc98f2d66a7ad18f
2020-07-12 08:56:01 +02:00
Sean McGinnis 848d080c1a
update-constraints: Install pip for all versions
Depends-on: https://review.opendev.org/739272

Change-Id: Ic6444e6c186c82a21e7fe2868c27fda3378c187a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-07-10 08:55:50 -05:00
Thierry Carrez 2f534a2d46 maintain-github-mirror: fix repository names
Job definition referenced opendev/governance and opendev/project_config
instead of openstack/governance and openstack/project-config, causing
the job to fail.

Also replace variable names so that they make more sense.

Change-Id: I7cd026a1d0bc1a8ad2d0c6b7bf3a5a0d37ecd423
2020-07-10 11:22:18 +02:00
Thierry Carrez 299017c496 maintain-github-mirror: fix dependency name
The script needs the 'github' module, but this is provided by the
'PyGithub' library on Pypi, not the 'github' library.

Change-Id: I7c6d4b30a11d38e060fb4e6b2cdf29301f6f03e3
2020-07-09 11:15:11 +02:00
Thierry Carrez 0197d9f92a Define maintain-github-openstack-mirror job
Opendev no longer automatically creates repositories on the
GitHub mirror, nor does it update descriptions or closes open PRs.

Add a playbook and a job for periodically maintaining the GitHub
mirror for the 'openstack' organization:

- updating descriptions based on Gerrit project descriptions
- creating on GitHub newly-added openstack repositories
- archiving from GitHub recently-retired openstack repositories
- closing any open PR with a healpful message

This job makes use of a GitHub API token (from the openstack-mirroring
user) and is defined to run periodically on project-config.

Change-Id: Ic02f436eb655dcbe84824b304ea2933e16312e67
2020-07-08 15:42:06 +02:00
Sean McGinnis 118a7ba3d4
Make tox global for update proposal jobs
If00afb50c37b45aefbb45070da4efef3e43e62b2 updated the requirements
update job to use tox as part of the update checking script. We have
ensure-tox as one of the roles in the pre playbook for this job, but it
is not set to be made available globally, which results in the script
failing with being unable to find tox.

This updates the role inclusion to set ensure_global_symlinks: true so
tox will be available in the script.

Change-Id: I5d3ed4ce62cd8df604802fedf522d644d5664698
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-06-24 10:34:18 -05:00
Zuul 0ff9b37818 Merge "Fix venv use in requirements propose_update.sh" 2020-06-23 16:04:49 +00:00
Sean McGinnis bfc06ae553
Fix venv use in requirements propose_update.sh
This script creates a virtualenv to run a requirements script to update
constraints. virtualenv is not globally available anymore. Rather than
trying to call it directly, this now uses tox to set up the virtual
environment.

Change-Id: If00afb50c37b45aefbb45070da4efef3e43e62b2
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-06-23 07:51:44 -05:00
Jens Harbott 540bdac7ff Fix wheel release playbook
Somehow an empty role invocation without setting the needed variable
ended up there.

Change-Id: I0a2ab8e4c119f1efcaf7e4231efb1804301637d8
2020-06-23 14:50:49 +02:00
Jeremy Stanley 3f72ab620c Add ensure-tox to pti-python-tarball
The publish-openstack-python-branch-tarball job run by many projects
uses tox (in python-branch-tarball/post.yaml), so needs it installed.

Change-Id: Iba219ad14523ce900d9562ef56c78cc6c15aa01d
2020-06-18 21:43:32 +00:00
Andreas Jaeger 6376aa1341 Fix translation jobs
The translation jobs use scripts that expect tox pre-installed, install
tox globally for them.

Change-Id: I633c868a96721f38f063f8e057b0b3c950d5d3e9
2020-06-18 20:05:02 +02:00
Sean McGinnis 9c91006304 Ensure pip is installed for propose-update-constraints
Pip is used by this task, make sure ensure-pip is used so it is
present.

While here, install git-review to a venv and symlink it globally to
keep things separate.

Change-Id: If6b1abd2545c68f727bddc5317797236f451b67c
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Signed-off-by: Ian Wienand <iwienand@redhat.com>
2020-06-17 16:02:46 +10:00
Zuul 6f2b723a9c Merge "Install git-review under python3 for proposed updates" 2020-06-11 23:45:48 +00:00