This enables the nodepool delete-after-upload option with keep-formats
set to qcow2 on x86 image builders. This should clear out vhd and raw
files after uploads for those formats are completed keeping only qcow2
longer term. This should reduce disk space overhead while still enabling
us to convert from qcow2 to the other formats if that becomes necessary.
Note that we do not enable this for arm64 before arm64 builders
currently build raw images only and we still want at least one copy of
the image to be kept even if it is raw (and not qcow2).
Change-Id: I6cf481e0f9a5eaff35b5d961a084ae34a49ea6c6
This is the last step in cleaning centos-7 out of nodepool. The previous
change will have cleaned up uploads and now we can stop building the
images entirely.
Change-Id: Ie81d6d516cd6cd42ae9797025a39521ceede7b71
This removal of centos-7 image uploads should cause Nodepool to clean up
the existing images in the clouds. Once that is done we can completely
remove the image builds in a followup change.
We are performing this cleanup because CentOS 7 is near its EOL and
cleaning it up will create room on nodepool builders and our mirrors for
other more modern test platforms.
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/912786
Change-Id: I48f6845bc7c97e0a8feb75fc0d540bdbe067e769
This reverts commit eca3bde9cb.
This was successful, but we want to make the change without altering
the cloud name. So switch this back, and separately we will update
the config of the rax cloud.
Change-Id: I8cdbd7777a2da866e54ef9210aff2f913a7a0211
Switch the Rackspace region with the smallest quota to uploading
images and booting server instances with our account's API key
instead of its password, in preparation for their MFA transition. If
this works as expected, we'll make a similar switch for the
remaining two regions.
Change-Id: I97887063c735c96d200ce2cbd8950bbec0ef7240
Depends-On: https://review.opendev.org/911164
This should be landed after the parent chagne has landed and nodepool
has successfully deleted all debian-buster image uploads from our cloud
providers. At this point it should be safe to remove the image builds
entirely.
Change-Id: I7fae65204ca825665c2e168f85d3630686d0cc75
Debian buster has been replaced by bullseye and bookworm, both of which
are releases we have images for. It is time to remove the unused debian
buster images as a result.
This change follows the process in nodepool docs for removing a provider
[0] (which isn't quite what we are doing) to properly remove images so
that they can be deleted by nodepool before we remove nodepool's
knowledge of them. The followup change will remove the image builds from
nodepool.
[0] https://zuul-ci.org/docs/nodepool/latest/operation.html#removing-a-provider
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/910015
Change-Id: I37cb3779944ff9eb1b774ecaf6df3c6929596155
This should be landed after the parent chagne has landed and nodepool
has successfully deleted all opensuse-15 image uploads from our cloud
providers. At this point it should be safe to remove the image builds
entirely.
Change-Id: Icc870ce04b0f0b26df673f85dd6380234979906f
These images are old opensuse 15.2 and there doesn't seem to be interest
in keeping these images running (very few jobs ever ran on them and
rarely successfully and no one is trying to update to 15.5 or 15.6).
This change follows the process in nodepool docs for removing a provider
[0] (which isn't quite what we are doing) to properly remove images so
that they can be deleted by nodepool before we remove nodepool's
knowledge of them. The followup change will remove the image builds from
nodepool.
[0] https://zuul-ci.org/docs/nodepool/latest/operation.html#removing-a-provider
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/909773
Change-Id: Id9373762ed5de5c7c5131811cec989c2e6e51910
This was the old timeout then some refactoring happened and we ended up
with the openstacksdk timeout of one hour. Since then Nodepool added the
ability to configure the timeout so we set it back to the original six
hour value.
Change-Id: I29d0fa9d0077bd8e95f68f74143b2d18dc62014b
This removes the fedora image builds from nodepool. At this point
Nodepool should no longer have any knowledge of fedora.
There is potential for other cleanups for things like dib elements, but
leaving those in place doesn't hurt much.
Change-Id: I3e6984bc060e9d21f7ad851f3a64db8bb555b38a
This will stop providing the node label entirely and should result in
nodepool cleaning up the existing images for these images in our cloud
providers. It does not remove the diskimages for fedora which will
happen next.
Change-Id: Ic1361ff4e159509103a6436c88c9f3b5ca447777
In order to reduce the load on our builder nodes and reduce the strain
on our providers' image stores, build most images only once per week.
Exceptions are ubuntu-jammy, our most often used distro image, which we
keep rebuilding daily, and some other more frequently used images built
every 2 days.
Change-Id: Ibba7f864b15e478fda59c998843c3b2ace0022d8
Enable uploads for all images again for rax-iad. We have configured the
nodepool-builders to run with only 1 upload thread, so we will have at
most two parallel uploads (one per builder).
Change-Id: Ia2b737e197483f9080b719bab0ca23461850e157
This is a partial revert of d50921e66b.
We want to slowly re-enable image uploads for rax-iad, start with a
single image, choosing the one that is getting used most often.
Change-Id: I0816f7da73e66085fe6c52372531477e140cfb76
Depends-On: https://review.opendev.org/892056
Manual cleanup of approximately 1200 images in this region, some as
much as 4 years old, has completed. Start attempting uploads again
to see if they'll complete now.
This reverts commit 71d1f02164.
Change-Id: I850acb3926a3fdedad599767b99be466bf45daef
We're getting Glance task timeout errors when trying to upload new
images into rax-iad, which seems to be resulting in rapidly leaking
images and may be creating an ever-worsening feedback loop. Let's
pause uploads for now since they're not working anyway, and
hopefully that will allow us to clean up the mess that's been
created more rapidly as well.
Change-Id: I0cc93a80e2cfa2ef761c6f538e134505bf4dc53c
Release is done, mirror is in place, ready to go.
Adopt using systemd-timesyncd like we do for recent Ubuntu releases.
Change-Id: I3fbdc151177bf2dba81920a4a2e3966f271b50ad
Once the builders have a chance to clear out all uploaded images,
this will remove the remaining references in Nodepool. Then
system-config cleanup can proceed.
Change-Id: I69b96b690918a9145d2e7ccbc79968c5341480bb
The mirror in our Limestone Networks donor environment is now
unreachable, but we ceased using this region years ago due to
persistent networking trouble and the admin hasn't been around for
roughly as long, so it's probably time to go ahead and say goodbye
to it.
In preparation for cleanup of credentials in system-config, first
remove configuration here except leave the nodepool provider with an
empty diskimages list so that it will have a chance to pick up after
itself.
Change-Id: I504682884a1439fac84d514880757c2cd041ada6
This removes iweb configs from the project-config repo. We'll still have
a few system-config items to clean up in a separate change.
Change-Id: I7bd2f0f6fcd7449e724815ed0c0fe743702ae8f3
The element was missing from the configuration due to an oversight, this
change brings it in line with C9S as well as Rocky 8 to have EPEL
enabled.
Change-Id: I080f8118ccf411e363e0f7c2433dfaa9e911dd15
openEuler 20.03-LTS-SP2 was out of date in May 2022. 22.03 LTS
is the newest LTS version. It was release in March 2022 and
will be maintained for 2 years. This patch upgrades the LTS
version. It'll be used in Devstack, Kolla-ansible and so on
in CI jobs.
Change-Id: I23f2b397bc7f1d8c2a959e0e90f5058cf3bf104d
Recent changes in how nodepool deletes images mean that we need
a list of all diskimages on every builder (otherwise there is a
small window where a new image build might be deleted by a different
builder). To correct this, list all diskimages on all builders
(but they still won't build images they shouldn't because they have
different provider configurations).
Change-Id: I89cff1aa9d3b60ee24176cbfea9d1a81e2c976cd
This is a partial revert of [0]. For the bootstrap we only use the
release pocket, not updates or security, so the version of
ca-certificates we install initially is too old to verify current
LetsEncrypt protected hosts.
[0] I455568f1e261060aefe555e9f5b4ef0830b60157
Change-Id: I673bc0ec687df6db59911f17dae1d50cbc7e3dd6
We have noticed Jammy jobs failing with DISK_FULL returns from Zuul.
Upon further investigation, they have hundreds of megabytes of syslog
filled with traces. As noted inline we've debugged this down to SSBD
interactions with some cloud providers.
Set this to the later-kernel upstream default of "prctl" to avoid this
issue.
Change-Id: I09a495a9b50c55b65221be56337f52388a5d9d48
Debian is already doing this and according to ubuntu package search
apt-transport-https is a transitional package at least as far back as
bionic. We leave xenial with http as it isn't clear if apt there
supports https.
Change-Id: I455568f1e261060aefe555e9f5b4ef0830b60157
Once we've settled on a max-servers value of 0 without any servers this
is the nextstep in removing the provider. This should completely remove
the provider from the launcher and the image builders.
We keep the airship-citycloud nodepool provider for historical
information purposes. We can clean this up later.
Change-Id: Icfb8fc6d2b15714ecb58960d8e44b199bedd6b0d
We've been notified that these resources won't be provided any longer.
This is the first step of setting max-servers to 0 and removing images
from the cloud. Once that is in we can remove the cloud more completely.
Change-Id: Iabcd6487a5bb3ed7fb6aae5dadf23a8171abcb7f
This distro release reached its EOL December 31, 2021. We are removing
it from our CI system as people should really stop testing on it. They
can use CentOS 8 Stream or other alternatives instead.
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/827181
Change-Id: I13e8185b7839371a9f9043b715dc39c6baf907d5
In line with CentOS deprecation, http://mirrorlist.centos.org has
stopped returning mirror sources. As the dib builds have this in the
initial image clones, they won't build any more. Pause this to stop a
spiral of failing builds.
Since we setup local mirrors, things might keep working in our
ecosystem for a bit, so far the upstream mirrors haven't pulled the 8/
directory directly it seems.
Change-Id: I99ca38f346b3a5d19d7565c1f6114a51b7fe2da3
This removes the label, nodes, and images for opensuse-tumbleweed across
our cloud providers. We also update grafana to stop graphing stats for
the label.
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/824068
Change-Id: Ic311af5d667c01c1845251270fd2fdda7d99ebcb
The openEuler node can be setup successful now. While the
error `/opt/cache/files, No such file or directory ` will
raise when running devstack jobs on it.
This patch add the cache-devstack element for openEuler diskimage
to ensure devstack jobs work well.
Change-Id: Ie26f19bd845c348d8871ac18a6a4cea9a84ed9f8
Change I20e1a926d02a632450b8114d84a0fa738b7ec639 updated this for
the arm64 images, but we have amd64 images as well which will need
the same treatment.
Change-Id: I187b6f283d8c861c5f5a32c4c9b46f80bda19f94
DIB's tests suggest these images should build successfully again, so
let's see if we can get them to build and launch reachable nodes in
our providers.
Apparently Gentoo carries Python 3.9 now, so set that as well.
Change-Id: Id645cd6f15df6446ff81716c607ab4c8b17b7b22
Fedora-34 was paused until bug fixes could land in DIB and our nodepool
builder images could be updated. This happened about 2 weeks ago from
what I can tell so unpause and let those images build again.
Opensuse-15 was paused becuase the images failed to build. Unfortunately
that was almost a year ago and we don't have logs for why this was
broken. Unpause to generate new logs allowing us to debug them.
Change-Id: I7d1957304d90b9902f36eab8f45871e081373282
We are having issues talking to https://image.api.mtl01.cloud.iweb.com
as it appears to be missing an intermediate certificate.
This has been reported to the provider (Ticket #4526047)
In the mean time, disable uploads, which are in a failure loop.
Change-Id: If224a44eb3a0727cdb34c7f016c6407bac610549