Commit Graph

756 Commits

Author SHA1 Message Date
Clark Boylan 8c088d17f1 Enable nodepool delete after upload option
This enables the nodepool delete-after-upload option with keep-formats
set to qcow2 on x86 image builders. This should clear out vhd and raw
files after uploads for those formats are completed keeping only qcow2
longer term. This should reduce disk space overhead while still enabling
us to convert from qcow2 to the other formats if that becomes necessary.

Note that we do not enable this for arm64 before arm64 builders
currently build raw images only and we still want at least one copy of
the image to be kept even if it is raw (and not qcow2).

Change-Id: I6cf481e0f9a5eaff35b5d961a084ae34a49ea6c6
2024-03-26 15:10:36 -07:00
Clark Boylan aabaf95b49 Remove centos-7 nodepool image builds
This is the last step in cleaning centos-7 out of nodepool. The previous
change will have cleaned up uploads and now we can stop building the
images entirely.

Change-Id: Ie81d6d516cd6cd42ae9797025a39521ceede7b71
2024-03-13 08:30:16 -07:00
Clark Boylan b8c53b9c03 Remove centos-7 image uploads from Nodepool
This removal of centos-7 image uploads should cause Nodepool to clean up
the existing images in the clouds. Once that is done we can completely
remove the image builds in a followup change.

We are performing this cleanup because CentOS 7 is near its EOL and
cleaning it up will create room on nodepool builders and our mirrors for
other more modern test platforms.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/912786
Change-Id: I48f6845bc7c97e0a8feb75fc0d540bdbe067e769
2024-03-13 08:21:46 -07:00
James E. Blair f5c200181a Revert "Try switching Rackspace DFW to an API key"
This reverts commit eca3bde9cb.

This was successful, but we want to make the change without altering
the cloud name.  So switch this back, and separately we will update
the config of the rax cloud.

Change-Id: I8cdbd7777a2da866e54ef9210aff2f913a7a0211
2024-03-07 08:46:25 -08:00
Jeremy Stanley eca3bde9cb Try switching Rackspace DFW to an API key
Switch the Rackspace region with the smallest quota to uploading
images and booting server instances with our account's API key
instead of its password, in preparation for their MFA transition. If
this works as expected, we'll make a similar switch for the
remaining two regions.

Change-Id: I97887063c735c96d200ce2cbd8950bbec0ef7240
Depends-On: https://review.opendev.org/911164
2024-03-06 15:06:34 +00:00
Clark Boylan c41bc6e5c2 Remove debian-buster image builds from nodepool
This should be landed after the parent chagne has landed and nodepool
has successfully deleted all debian-buster image uploads from our cloud
providers. At this point it should be safe to remove the image builds
entirely.

Change-Id: I7fae65204ca825665c2e168f85d3630686d0cc75
2024-02-23 13:23:22 -08:00
Clark Boylan feff36e424 Drop debian-buster image uploads from nodepool
Debian buster has been replaced by bullseye and bookworm, both of which
are releases we have images for. It is time to remove the unused debian
buster images as a result.

This change follows the process in nodepool docs for removing a provider
[0] (which isn't quite what we are doing) to properly remove images so
that they can be deleted by nodepool before we remove nodepool's
knowledge of them. The followup change will remove the image builds from
nodepool.

[0] https://zuul-ci.org/docs/nodepool/latest/operation.html#removing-a-provider

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/910015
Change-Id: I37cb3779944ff9eb1b774ecaf6df3c6929596155
2024-02-23 13:19:49 -08:00
Clark Boylan 211fe14946 Remove opensuse-15 image builds from nodepool
This should be landed after the parent chagne has landed and nodepool
has successfully deleted all opensuse-15 image uploads from our cloud
providers. At this point it should be safe to remove the image builds
entirely.

Change-Id: Icc870ce04b0f0b26df673f85dd6380234979906f
2024-02-22 10:27:37 -08:00
Clark Boylan 5635e67866 Drop opensuse image uploads from nodepool
These images are old opensuse 15.2 and there doesn't seem to be interest
in keeping these images running (very few jobs ever ran on them and
rarely successfully and no one is trying to update to 15.5 or 15.6).

This change follows the process in nodepool docs for removing a provider
[0] (which isn't quite what we are doing) to properly remove images so
that they can be deleted by nodepool before we remove nodepool's
knowledge of them. The followup change will remove the image builds from
nodepool.

[0] https://zuul-ci.org/docs/nodepool/latest/operation.html#removing-a-provider

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/909773
Change-Id: Id9373762ed5de5c7c5131811cec989c2e6e51910
2024-02-22 10:25:15 -08:00
Clark Boylan 4a3c87dbcd Set a six hour nodepool image upload timeout
This was the old timeout then some refactoring happened and we ended up
with the openstacksdk timeout of one hour. Since then Nodepool added the
ability to configure the timeout so we set it back to the original six
hour value.

Change-Id: I29d0fa9d0077bd8e95f68f74143b2d18dc62014b
2023-09-15 12:57:25 -07:00
Clark Boylan 3b9c5d2f07 Remove fedora image builds
This removes the fedora image builds from nodepool. At this point
Nodepool should no longer have any knowledge of fedora.

There is potential for other cleanups for things like dib elements, but
leaving those in place doesn't hurt much.

Change-Id: I3e6984bc060e9d21f7ad851f3a64db8bb555b38a
2023-09-06 09:16:34 -07:00
Clark Boylan d83736575e Remove fedora-35 and fedora-36 from nodepool providers
This will stop providing the node label entirely and should result in
nodepool cleaning up the existing images for these images in our cloud
providers. It does not remove the diskimages for fedora which will
happen next.

Change-Id: Ic1361ff4e159509103a6436c88c9f3b5ca447777
2023-09-06 09:12:33 -07:00
Dr. Jens Harbott d0c0ddb977 Reduce frequency of image rebuilds
In order to reduce the load on our builder nodes and reduce the strain
on our providers' image stores, build most images only once per week.

Exceptions are ubuntu-jammy, our most often used distro image, which we
keep rebuilding daily, and some other more frequently used images built
every 2 days.

Change-Id: Ibba7f864b15e478fda59c998843c3b2ace0022d8
2023-09-02 13:18:19 +02:00
Dr. Jens Harbott 407f859232 Unpause image uploads for rax-iad part 2
Enable uploads for all images again for rax-iad. We have configured the
nodepool-builders to run with only 1 upload thread, so we will have at
most two parallel uploads (one per builder).

Change-Id: Ia2b737e197483f9080b719bab0ca23461850e157
2023-08-30 21:07:27 +02:00
Dr. Jens Harbott c8b1b1c3b6 Unpause image upload for rax-iad part 1
This is a partial revert of d50921e66b.

We want to slowly re-enable image uploads for rax-iad, start with a
single image, choosing the one that is getting used most often.

Change-Id: I0816f7da73e66085fe6c52372531477e140cfb76
Depends-On: https://review.opendev.org/892056
2023-08-19 20:12:40 +00:00
Dr. Jens Harbott d50921e66b Revert "Revert "Temporarily pause image uploads to rax-iad""
This reverts commit 27a3da2e53.

Reason for revert: Uploads are still not working properly

Change-Id: I2a75dd9ff0731a4113a362f9f17f510a9a236ebb
2023-08-10 07:24:07 +00:00
Jeremy Stanley 27a3da2e53 Revert "Temporarily pause image uploads to rax-iad"
Manual cleanup of approximately 1200 images in this region, some as
much as 4 years old, has completed. Start attempting uploads again
to see if they'll complete now.

This reverts commit 71d1f02164.

Change-Id: I850acb3926a3fdedad599767b99be466bf45daef
2023-08-09 11:43:54 +00:00
Jeremy Stanley 71d1f02164 Temporarily pause image uploads to rax-iad
We're getting Glance task timeout errors when trying to upload new
images into rax-iad, which seems to be resulting in rapidly leaking
images and may be creating an ever-worsening feedback loop. Let's
pause uploads for now since they're not working anyway, and
hopefully that will allow us to clean up the mess that's been
created more rapidly as well.

Change-Id: I0cc93a80e2cfa2ef761c6f538e134505bf4dc53c
2023-08-08 15:48:48 +00:00
Dr. Jens Harbott 4c16313ad2 Build debian bookworm images
Release is done, mirror is in place, ready to go.

Adopt using systemd-timesyncd like we do for recent Ubuntu releases.

Change-Id: I3fbdc151177bf2dba81920a4a2e3966f271b50ad
2023-07-03 06:05:36 +00:00
Jeremy Stanley 92814f9b71
Remove empty limestone nodepool providers
Once the builders have a chance to clear out all uploaded images,
this will remove the remaining references in Nodepool. Then
system-config cleanup can proceed.

Change-Id: I69b96b690918a9145d2e7ccbc79968c5341480bb
2023-02-14 08:25:25 +11:00
Jeremy Stanley 7c81cf6eda
Farewell limestone
The mirror in our Limestone Networks donor environment is now
unreachable, but we ceased using this region years ago due to
persistent networking trouble and the admin hasn't been around for
roughly as long, so it's probably time to go ahead and say goodbye
to it.

In preparation for cleanup of credentials in system-config, first
remove configuration here except leave the nodepool provider with an
empty diskimages list so that it will have a chance to pick up after
itself.

Change-Id: I504682884a1439fac84d514880757c2cd041ada6
2023-02-14 08:25:10 +11:00
Clark Boylan 7b0629e2cb Remove iweb nodepool configs
This removes iweb configs from the project-config repo. We'll still have
a few system-config items to clean up in a separate change.

Change-Id: I7bd2f0f6fcd7449e724815ed0c0fe743702ae8f3
2022-12-12 08:43:43 -08:00
Clark Boylan bf34c3ea5a Set iweb to empty labels and diskimages
This should allow nodepool to clean up what it can as part of the
removal of this provider.

Change-Id: Ia084f7f6b4a8988010b71e6242f580ec70f788c8
2022-12-12 08:35:48 -08:00
Michal Nasiadka d61182e6c5 Add rockylinux-9-arm64
Change-Id: I522bf00c737dd96cac337c37068545d168ecce32
2022-09-27 07:24:26 +00:00
Neil Hanlon ba93b3570a
Add epel element to Rocky Linux 9 configuration
The element was missing from the configuration due to an oversight, this
change brings it in line with C9S as well as Rocky 8 to have EPEL
enabled.

Change-Id: I080f8118ccf411e363e0f7c2433dfaa9e911dd15
2022-09-01 09:05:38 -04:00
Ian Wienand 36b9c302e5 nodepool: Add Fedora 36
Add Fedora 36 builds

Change-Id: I64fac34945ea5c6ec91ddd442281fcaba2c53271
2022-08-22 11:25:09 +10:00
Neil Hanlon 705e8420a2
Add rockylinux 9 to nodepool
Change-Id: Iedf1c8eb2898cfc5771a5a695a53a39f9396edc9
2022-08-05 08:58:27 -04:00
Dr. Jens Harbott 277f30bdc6 Fix mismatch for openEuler image name
One instance was missing the -LTS suffix.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I4ae1d93a13c47bdc5732854ab441a88e2f4e3473
2022-08-04 11:49:33 +02:00
wangxiyuan 30fca1376d Bump openEuler to 22.03 LTS
openEuler 20.03-LTS-SP2 was out of date in May 2022. 22.03 LTS
is the newest LTS version. It was release in March 2022 and
will be maintained for 2 years. This patch upgrades the LTS
version. It'll be used in Devstack, Kolla-ansible and so on
in CI jobs.

Change-Id: I23f2b397bc7f1d8c2a959e0e90f5058cf3bf104d
2022-08-03 14:40:34 +08:00
James E. Blair c1b692b0a0 Synchronize diskimages on builders
Recent changes in how nodepool deletes images mean that we need
a list of all diskimages on every builder (otherwise there is a
small window where a new image build might be deleted by a different
builder).  To correct this, list all diskimages on all builders
(but they still won't build images they shouldn't because they have
different provider configurations).

Change-Id: I89cff1aa9d3b60ee24176cbfea9d1a81e2c976cd
2022-07-28 10:55:21 -07:00
Dr. Jens Harbott ce68aa4e64 Don't use https in Ubuntu bionic and focal builds
This is a partial revert of [0]. For the bootstrap we only use the
release pocket, not updates or security, so the version of
ca-certificates we install initially is too old to verify current
LetsEncrypt protected hosts.

[0] I455568f1e261060aefe555e9f5b4ef0830b60157

Change-Id: I673bc0ec687df6db59911f17dae1d50cbc7e3dd6
2022-05-22 09:28:28 +02:00
Ian Wienand 9673eb9a76 nodepool: set spec_store_bypass_disable=prctl for Jammy images
We have noticed Jammy jobs failing with DISK_FULL returns from Zuul.
Upon further investigation, they have hundreds of megabytes of syslog
filled with traces.  As noted inline we've debugged this down to SSBD
interactions with some cloud providers.

Set this to the later-kernel upstream default of "prctl" to avoid this
issue.

Change-Id: I09a495a9b50c55b65221be56337f52388a5d9d48
2022-05-20 11:07:34 +10:00
Clark Boylan 322b261b20 Use https with apt in our ubuntu image builds
Debian is already doing this and according to ubuntu package search
apt-transport-https is a transitional package at least as far back as
bionic. We leave xenial with http as it isn't clear if apt there
supports https.

Change-Id: I455568f1e261060aefe555e9f5b4ef0830b60157
2022-05-03 16:28:25 -07:00
Dr. Jens Harbott f5b01d71b7 Start bulding ubuntu-jammy images
Jammy was released yesterday, we can start building images for it.

Change-Id: I5fd16df8bf0e3b74711875ad0573be69db12791b
2022-04-24 15:23:28 +02:00
Marcin Juszkiewicz c9474f7799 Add EPEL into CentOS Stream 9 images
disabled by default

Change-Id: I332d239e8ce971fe420391925d7ba12af463bfaa
2022-04-06 18:19:01 +02:00
Clark Boylan 3dc09d41d1 Remove airship-citycloud nodepool provider
Once we've settled on a max-servers value of 0 without any servers this
is the nextstep in removing the provider. This should completely remove
the provider from the launcher and the image builders.

We keep the airship-citycloud nodepool provider for historical
information purposes. We can clean this up later.

Change-Id: Icfb8fc6d2b15714ecb58960d8e44b199bedd6b0d
2022-03-01 08:18:08 -08:00
Clark Boylan 1c73cd826c Set airship-citycloud max-servers to 0
We've been notified that these resources won't be provided any longer.
This is the first step of setting max-servers to 0 and removing images
from the cloud. Once that is in we can remove the cloud more completely.

Change-Id: Iabcd6487a5bb3ed7fb6aae5dadf23a8171abcb7f
2022-03-01 08:10:23 -08:00
Neil Hanlon 2ccc5241c8
Add rockylinux-8 to nodepool configuration
Signed-off-by: Neil Hanlon <neil@rockylinux.org>
Change-Id: Ic3344bc47ca56c27f7ec3427a0865bd6ce3349d3
Depends-On: https://review.opendev.org/c/openstack/project-config/+/829405
Depends-On: https://review.opendev.org/c/openstack/project-config/+/829712
2022-02-17 09:12:00 -05:00
Ian Wienand dd58c496f8 Remove Fedora 34
The dependent changes should be the last references to Fedora 34 nodes

Depends-On: https://review.opendev.org/c/openstack/devstack/+/827576
Depends-On: https://review.opendev.org/c/openstack/devstack/+/827578
Depends-On: https://review.opendev.org/c/zuul/nodepool/+/827577

Change-Id: Ie14ea374808e5518588925de3a476f0bc6ff2ccf
2022-02-11 07:55:21 +11:00
Clark Boylan dce378a6b4 Remove centos-8
This distro release reached its EOL December 31, 2021. We are removing
it from our CI system as people should really stop testing on it. They
can use CentOS 8 Stream or other alternatives instead.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/827181
Change-Id: I13e8185b7839371a9f9043b715dc39c6baf907d5
2022-02-02 09:48:36 +11:00
Ian Wienand 39e51526e4 Stop building CentOS 8
In line with CentOS deprecation, http://mirrorlist.centos.org has
stopped returning mirror sources.  As the dib builds have this in the
initial image clones, they won't build any more.  Pause this to stop a
spiral of failing builds.

Since we setup local mirrors, things might keep working in our
ecosystem for a bit, so far the upstream mirrors haven't pulled the 8/
directory directly it seems.

Change-Id: I99ca38f346b3a5d19d7565c1f6114a51b7fe2da3
2022-02-01 10:35:54 +11:00
Clark Boylan 07e9803134 Remove opensuse-tumbleweed from nodepool
This removes the label, nodes, and images for opensuse-tumbleweed across
our cloud providers. We also update grafana to stop graphing stats for
the label.

Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/824068
Change-Id: Ic311af5d667c01c1845251270fd2fdda7d99ebcb
2022-01-10 14:02:55 -08:00
wangxiyuan 4a058ac8bf Add cache-devstack element for openEuler
The openEuler node can be setup successful now. While the
error `/opt/cache/files, No such file or directory ` will
raise when running devstack jobs on it.

This patch add the cache-devstack element for openEuler diskimage
to ensure devstack jobs work well.

Change-Id: Ie26f19bd845c348d8871ac18a6a4cea9a84ed9f8
2021-12-17 15:09:09 +08:00
Jeremy Stanley bd04590212 nodepool: Remove yet still more . from openEuler
Change I20e1a926d02a632450b8114d84a0fa738b7ec639 updated this for
the arm64 images, but we have amd64 images as well which will need
the same treatment.

Change-Id: I187b6f283d8c861c5f5a32c4c9b46f80bda19f94
2021-12-16 22:02:36 +00:00
Zuul 482d27e16a Merge "Add openEuler 20.03 LTS SP2 node" 2021-12-14 04:56:07 +00:00
Zuul 20da605994 Merge "Un-pause gentoo-17-0-systemd images" 2021-12-06 23:51:53 +00:00
Ian Wienand 89058761c8 Revert "iweb: disable image uploads"
This reverts commit 1580f43853.

The upstream certificate issues are fixed

Change-Id: I6184467da6261c9258fdf745c5111acee87481be
2021-12-07 08:37:08 +11:00
Jeremy Stanley 669dcaf0b5 Un-pause gentoo-17-0-systemd images
DIB's tests suggest these images should build successfully again, so
let's see if we can get them to build and launch reachable nodes in
our providers.

Apparently Gentoo carries Python 3.9 now, so set that as well.

Change-Id: Id645cd6f15df6446ff81716c607ab4c8b17b7b22
2021-12-06 20:13:38 +00:00
Clark Boylan db1810f925 Unpause fedora-34 and opensuse-15 image builds
Fedora-34 was paused until bug fixes could land in DIB and our nodepool
builder images could be updated. This happened about 2 weeks ago from
what I can tell so unpause and let those images build again.

Opensuse-15 was paused becuase the images failed to build. Unfortunately
that was almost a year ago and we don't have logs for why this was
broken. Unpause to generate new logs allowing us to debug them.

Change-Id: I7d1957304d90b9902f36eab8f45871e081373282
2021-12-06 08:48:17 -08:00
Ian Wienand 1580f43853 iweb: disable image uploads
We are having issues talking to https://image.api.mtl01.cloud.iweb.com
as it appears to be missing an intermediate certificate.

This has been reported to the provider (Ticket #4526047)

In the mean time, disable uploads, which are in a failure loop.

Change-Id: If224a44eb3a0727cdb34c7f016c6407bac610549
2021-12-06 15:25:54 +00:00