Between September 4 and 5, our wheel cache publication began failing
for Debian distributions. The ansible_distribution_version fact
began to differentiate between Debian minor versions (so e.g.
bullseye went from "11" to "11.7"), and this does not match our
cache scheme in AFS so resulted in attempts to write to a
nonexistent directory. This corresponds to when
I9527fc847e18a80414139ebd6f19a9dbb9a5778e merged to switch the
tenant's default Ansible version to 8, so is very likely a behavior
change in Ansible itself.
Switch the logic in our wheel-mirror playbook to use
ansible_distribution_major_version like we do for Ubuntu, and rename
the tasks to be a little more generic.
Change-Id: Ie95e99bce1ed01aa9c284871b517e6c28c278d16
In order to be able to safely re-enqueue tags which previously
failed release jobs after successfully uploading at least some
artifacts to PyPI, instruct twine to treat "file already exists"
responses as benign and ignore them, proceeding to upload any others
which aren't yet there.
Depends-On: https://review.opendev.org/864004
Change-Id: Iab38df6386ce5219a52787fda5e64a8faab23a06
A later version of ansible-lint picks this up; fix lack of whitespace
around filters. No operational change.
Change-Id: If042a0f92a74ea06a312aad68de21f9e44f16582
A later version of ansible-lint enforces names on blocks. This is
generally a good rule; fix a few missing blocks here.
Change-Id: Ia87a0c21ec0ed1662e37cbc9e17a0df344b54e57
This doesn't need a name: when included like this. I guess Ansible
accepts it, but it doesn't document this working anywhere.
Change-Id: Ibe6fa9296041af54f39a3ef69957d0e8a0df0259
The configure-mirrors role in zuul-jobs expects CentOS wheels will
be in URLs like distro-ver-arch and so doesn't include "-stream"
like our wheel publishing jobs are adding. This was a necessary
distinction in order to differentiate between CentOS Linux 8 and
CentOS Stream 8 wheels, but since configure-mirrors didn't know to
tell Stream nodes to look in the other location, this was actually
broken from the moment it was introduced and we simply never noticed
until we removed the old centos-8 volume (at which point
centos-8-stream nodes no longer found any wheels).
As we no longer have centos-8 nodes, rather than try to change how
configure-mirrors works (since it may be successfully used in other
sites already and would need advance warning or complex overrides in
base jobs), simply rearrange our mirrors to mount the volumes in the
paths it expects. Once this merges I'll adjust AFS accordingly.
This partially reverts 3d776f6734.
Change-Id: I07ee7ccdd10e68f6f38ec362ce7e5d333d5fc2f7
To use the write-root-marker role from opendev/base-jobs, set
the root_marker_dir to the same value that is currently set
by the openstack-zuul-jobs role.
Needed-By: https://review.opendev.org/731460
Change-Id: I169f069aa458bc7d02f9c0295702c1316fff825f
- Use zuul_return fake module to make ansible lint happy, this allows
to remove Zuul import.
- While ansible-lint 4.2.0 is now able to detect playbooks/roles, this
is broken, so don't use it
- move its config out of tox.ini so it can be used by any tools, or
without tox
Change-Id: Ie8935f47db855647e19ae091044e5ac1871f1551
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Co-Authored-By: Andreas Jaeger <aj@suse.com>
playbooks/publish/service-types.yaml is not used anymore, with AFS
publishing it got obsoleted and can now be removed.
site_specs and site_specs_promote are not used anymore, remove them.
Change-Id: I0c3ca2455b4f8b721d48d44780dbc397ddc9cb20
Remove publish-tox-docs-static, the last user has been converted to AFS
publishing.
Remove the parent base-publish-tox-docs-static as well, it's not used
anymore.
Remove playbook files and secrets used by these jobs.
Depends-On: https://review.opendev.org/708918
Change-Id: I68b3ab7e597e230617ec1eaa8217d4f5f4c5fb15
Add new job publish-tox-docs-releases for publishing of releases repo to
new AFS location to obsolete static.o.o publishing.
Task: #37724
Story: #2006598
Needed-By: https://review.opendev.org/708918
Change-Id: I804f6ddb0b10b4aa3fac6a3cf11a2d23b8b4e730
Following-on from Ia3a0358249e9ed3d766b1b61535f2f6d67d4eb2d, this
removes the publishing from the static site, which is now happening to
AFS.
The site_tarballs secret is no longer required. The two jobs still
using it have a parent of publish-openstack-artifacts so should not
need a separate secret.
Depends-On: https://review.opendev.org/706732
Change-Id: I1b1db7d2451d7fb1fdd7921e7c9efd0e020fbce6
This adds publishing of artifacts to AFS in parallel with the existing
publishing to the static volume.
The additional tasks added to the openstack-artifacts.yaml playbook
have been tested in the project-config-test-artifacts-to-afs test
(results at [1]) and also in an external project
ironic-python-agent-builder (see notes in [2]). Thus we can have high
confidence it will work.
After this merges, I will manually rsync the old static tarballs data
to the AFS directory (again; it has been sync'd previously) and from
that point on both should be exactly the same.
At that point we can switch the DNS entries for
static.<openstack|opendev>.org (the sites have been created with [3])
to the new server and we will be fully migrated to the AFS volume. We
can then remove the static publishing.
[1] http://files.openstack.org/project/tarballs.opendev.org/openstack/project-config/afs-copy-test/
[2] https://review.opendev.org/705161
[3] https://review.opendev.org/704913
Change-Id: Ia3a0358249e9ed3d766b1b61535f2f6d67d4eb2d
This role moves the synchronize logic for copying the files into the
upload-afs-synchronize role from the dependent change.
Depends-On: https://review.opendev.org/705368
Change-Id: I7408e0ec9c731f2ae1d40ef3a4526aa67f28d4e0
Handle the name change in the dependent change. Should have no
functional change.
Depends-On: https://review.opendev.org/705368
Change-Id: I5fdc858892b1cf31649c2a8bef3d018afe68cb22
The -a flag to rsync tries to maintain owner and group permissions
which doesn't work copying to AFS.
This sets the flags to the same as the upload-afs role at [1]
fc27907b98/roles/upload-afs/library/zuul_afs.py (L78)
Change-Id: Ib1ed516df24ee113322f791bbbeab6286d34967e
The upload-afs role has a range of features for publishing
documentation trees that we don't need -- we just want a straight copy
of the artifacts/ directory to the output location. Just use a
hard-coded synchronize: call.
Change-Id: Iea71bb8e583daa31ab2dcbe08b9a84cb3ba0863f
This is part of our efforts to get tarball publishing onto AFS volumes
[1].
Test a new artifact publishing job that puts output at
/afs/.openstack.org/project/tarballs.opendev.org. This is intended to
replace the existing publishing job when it is working.
[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html
Change-Id: Ied96194e1904fee232e144a4e89ec8ba2252e42e
Create new promote jobs for infra-index and docs-site.
Also, add promote-openstack-specs job that we can use for publishing
all specs site later.
Add the same file list (copied from openstack-zuul-jobs) for the two
project-config promote jobs.
Remove the now obsolete publish jobs.
Remove now obsolete playbooks.
Needed-By: https://review.opendev.org/682215
Change-Id: I3c227f4229a0572a532009f270d44059698f53e8
Since openstack-manuals, api-guide, api-ref, and openstack-tox-docs are
now publishing via promote jobs, the publish jobs are not needed
anymore, remove them and the playbooks they used.
Note that publish-openstack-tox-docs is used in a few places, update one
here (requirements) and mark the job as deprecated. Before this one can
be removed, update all users.
Remove the now unused afsdeveloper_secret as well.
Depends-On: https://review.opendev.org/677009
Change-Id: Id8069557d0cf788e6eb469c8141945f89ac462e6
Create new jobs that can be used in promote pipeline to publish
documents. These job is used by openstack-manuals repositories.
Also, change output directory to use publish-docs/html in the publish
job so that we can merge 676482 and then use the promote job.
Depends-On: https://review.opendev.org/676488
Needed-By: https://review.opendev.org/676482
Change-Id: Ie1d89dcf4d79c284b59182729b6278144c1dbafb
Use tox-docs based playbooks for publishing api-guide and api-ref
documents.
Depends-On: https://review.opendev.org/659975
Change-Id: I6ad83d9fc400615329d993a07e6a1a37ae8b28b8
Change Ia4680f24d78af1260f2f0106a458b78a079c1287 remove the
placement-api-ref job but forgot to remove a playbook - remove this
as cleanup.
Change-Id: I4fb060a7c37a680854ef6f577ab6ede00bf1ff00
Reworks the publish playbooks and the post-run jobs
for the Puppet release job.
Depends-On: https://review.openstack.org/#/c/635941/
Change-Id: Ia9b149d3e1e94331b047d9bc762359d09ce81b20
The site_logs secret moved into opendev/basejobs and we renamed the
site_logs secret here into site_logs_project_config and site_specs.
Unfortauntely many of our playbooks in this repo were not updated to use
the new secret name so they cause some jobs to fail.
Fix this by updated the playbooks to use site_logs_project_config and
site_specs as appropriate.
Change-Id: I852a6362c35dd6aa17b0f767fceea811402dd90c
Changes the hosts field from localhost to all
so that the publish playbook is executed in
the cloud instead of locally on the zuul executor
node.
This is required because when it's executed on the
zuul executor the security context supresses the
facts as a security measure which causes the job to fail.
Change-Id: Iab2252ca647bd6d8eaff6c48f458c566c0393cb2
This is a follow-on to I3761dd9d2ab752e66bcccdd2434405f11535b95f where
I misunderstood what "linear" was doing. What was really intended was
to stop both the python2 and python3 builds copying to AFS at the same
time. "linear" just means the *tasks* start at the same time across
hosts; not that they run serially. To do that you want the (aptly
named) "serial" specifier on the playbook.
Add this so we do the copies sequentially
Change-Id: I4f8a2f2c6c8164e7ea207f8e4b286e06df0b13dd
Publish using tox-docs instead of using sphinx directly.
Change-Id: If409fc54eb6777e63e3998e513ff1120d23cefca
Depends-On: https://review.openstack.org/608442
prepare-for-afs includes already the root-marker wirter, an extra
write-root-marker is not needed and wrong - we currently end with
stable/queens job having two root-markers:
html/.root-marker
html/queens/.root-marker
Remove the extra root-marker.
Change-Id: I9a7de9ec372266b3b6dcd650b69f1ec401ef0b0c
We saw what looks very much like the index generation racing, with
"mv" unable to close the index file ...
Move to a linear strategy so that each host finishes each task before
moving on. Then add run_once to the index generation so only one of
the hosts in a python2/3 situation creates the index file.
Change-Id: I3761dd9d2ab752e66bcccdd2434405f11535b95f
This job is to trigger readthedocs via their new API
The old API was quite simple in that anyone could hit the end-point
and trigger a new build. The v2 API now requires authentication and a
unique id for each project we wish to update.
If projects wish, they can directly use the new "trigger-readthedocs"
role (from the dependent change) and create their own jobs, providing
their username/password or webhook authentication token directly from
a secret in their project playbook.
This job, however, uses the infra secret for the "openstackci" user to
update the docs. Projects that wish to take this route require some
minor setup, but don't have to worry about managing new jobs or their
secrets and gain some benefits in having the openstackci user as a
backup admin.
To use the trigger-readthedocs-webhook job, projects will need to do
the following:
1) add openstackci user as an admin of their RTD project
2) generate an webhook via the "Integrations" dashboard page on the
RTD project
3) provide the id from that webhook URL in job or project variables.
This webhook id is not private; it is useless without
authentication (it appears to be a global monotonically increasing
integer for each webhook generated).
4) include the job. gerrit-dash-creator is populated with the new job
for initial testing. Other projects are currently using the
"docs-on-readthedocs" template. Change
I3b65813671961d07c0a743685c537ad1df4bf68 proposes switching the
template to use this new job.
Change-Id: Ic34ea5ead26a477a1cc5f8b25fd2ce1949c1dc13
Depends-On: https://review.openstack.org/579434
Depends-On: https://review.openstack.org/584230
Jeremy Stanley