Commit Graph

77 Commits

Author SHA1 Message Date
Dr. Jens Harbott 6b1cfbe079 Cache new cirros images
The cirros project has released new images, add them to our cache prior
to actually using them in the CI. We can remove the old images once the
migration is completed and not too many stable branches using the old
images are still active, but comparing the size of these in relation to
the total size of our images, the impact of this shouldn't be too large
in comparison to the benefit in CI stability.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I6d6bcc0e9cfef059de70bbb19e4254e8d29d415b
2023-06-01 16:26:54 +00:00
Jeremy Stanley 5262094f9e Stop caching infrequently-used CirrOS images
According to Ic8b3e790fe332cf68bad7aaa3d5f85229600380b review
comments, OpenSearch indexing indicates jobs aren't often using
CirrOS 0.3.4, 0.3.5, 0.4.0 or 0.5.1 images any longer. If jobs
occasionally used them and have to retrieve them from the Internet
then that's fine, we really only need to cache images which are used
frequently. Remove the rest in order to shrink our node images
somewhat.

Change-Id: Ibada405e0c1183559f428c749d0e54d0a45a2223
2023-02-14 17:25:45 +00:00
yatinkarel 10abfbe573 Cache Cirros 0.6.1 images
0.6.1 is the latest cirros release and with [1][2]
is being used in neutron jobs.

Add these to nodepool images to avoid pulling it
in jobs and hit external connectivity issues.

[1] https://review.opendev.org/c/openstack/neutron/+/869154
[2] https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/869152

Change-Id: Ic8b3e790fe332cf68bad7aaa3d5f85229600380b
2023-02-09 17:03:31 +05:30
Dr. Jens Harbott eebe545182
Add py36 variant of get-pip.py
The common version of get-pip.py no longer supports python3.6 or older.
Devstack has amended it's use of this file to use a stable version of
the installer for python3.6 [0]. Pre-cache this version in our images,
too, so that it can be consumed in the CI.

[0] https://review.opendev.org/#/q/Iab2c391d5388461fe9e9037cee81884ce8032e72

Change-Id: Ied14dc7188e27e2a4b144d597b696ee85f25721d
2022-02-02 16:42:04 +01:00
Ian Wienand d54c2b46a7 cache-devstack: use dstat_graph from opendev
We have incoporated outstanding changes into a new opendev.org hosted
repository.

Change-Id: Ie4d31fa380573eddf6a5c31eeb6403d7c3b00159
2021-12-07 13:07:35 +11:00
Ian Wienand 30fd4b4549 dstat graph: update to version with fixes
This updates the on-disk cache version to the same as the default
version in zuul-jobs from I0d5e1b567c364a9e6c7aa0b95de17abffaef0434

Note we have a pull request open on this original project to
incorporate these changes, but the project has been dormat for a long
time.  If we don't have a response in the medium-term future, we can
bring this project into opendev.org git and maintain our enhancments
there.

Change-Id: I26b1a7d0dde12be7c493d44d754b9ea8f72e6e66
2021-10-29 09:36:33 +11:00
Lee Yarwood 388d227704 Add Cirros 0.5.2 to cache
Recently released [1] and including a fix [2] required by the
Iad1adbc23b31dd54a96299e7a8a4b622c15eed8d change introducing q35 testing
in the nova-next job this image needs to cached before we update
devstack.

[1] https://github.com/cirros-dev/cirros/releases/tag/0.5.2
[2] https://github.com/cirros-dev/cirros/pull/65

Change-Id: I064f1e4a8ba0bab5b759fb419fa9f9c52e122baa
2021-03-08 08:46:15 +00:00
Lee Yarwood cf2ba9c2e9 Revert "Add custom cirros image with ahci module enabled to cache"
This reverts commit 5ee0780486.

0.5.2 [1] was cut after another colleague asked for a release. I guess their release build issues have been resolved since I asked a few weeks ago. As a result this build is no longer required once we've bumped to 0.5.2.

[1] https://github.com/cirros-dev/cirros/releases/tag/0.5.2

Change-Id: I5332d0e47ad863ca9795a8b0b86b73156621622d
2021-03-08 08:43:53 +00:00
Lee Yarwood 5ee0780486 Add custom cirros image with ahci module enabled to cache
As discussed on the ML [1] the nova-next job is looking to start testing
the q35 machine type. In order to do this *before* the next Cirros
release a custom dev build of the Cirros image has been built with the
ahci module included, as is now required to allow for SATA based config
drives to work.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020823.html

Change-Id: I67912064487598c0e5b4ce3001276f42e0ebcad1
2021-03-03 22:28:05 +00:00
Dmitry Tantsur d30035b433 Cache cirros UEC images
Ironic uses them in its gate jobs, downloading every time. With
github broken all these jobs are failing now.

Change-Id: I8649d2cd530bdedcbd333991f7376fe9cd9bf267
2020-04-23 16:43:43 +02:00
Radosław Piliszek 624eec932b Cache CirrOS 0.5.1 for AArch64 too
OpenStack is getting tested on AArch64 CI nodes.
Let's cache CirrOS for it as well.

Change-Id: I3138bda523533bbbdc354d50c57179dca60d6d9c
2020-03-23 16:49:36 +01:00
Radosław Piliszek dc68586431 Cache CirrOS 0.5.1
We are moving towards CirrOS 0.5.1 usage in CI.
DevStack's patch is pending [1] and Kolla Ansible merged [2].

[1] https://review.opendev.org/711182
[2] https://review.opendev.org/711492

Change-Id: I4e82a955b59dfdcebbcd1f1687466e546781d28b
2020-03-23 16:27:36 +01:00
Clark Boylan 9e1c44d9a4 Statically cache devstack images and packages
We had been running a script to generate a list of things to cache for
devstack. Unfortunately, we've discovered that this attempts to perform
unsafe actions which illustrates that this is unsafe (and creates
unnecessary relationship between opendev images and openstack/devstack).

Address this by providing a static list of things to cache.

Note this does not do anything for arm64 images (that will need to be
addressed in a follow on but they are largely not running devstack there
yet).

On a Bionic node this is what we have in /opt/cache/files/:

cirros-0.3.2-i386-disk.vmdk
cirros-0.3.4-x86_64-disk.img
cirros-0.3.4-x86_64-disk.vhd.tgz
cirros-0.3.4-x86_64-uec.tar.gz
cirros-0.3.5-x86_64-disk.img
cirros-0.3.5-x86_64-disk.vhd.tgz
cirros-0.3.5-x86_64-uec.tar.gz
cirros-0.4.0-x86_64-disk.img
cirros-0.4.0-x86_64-uec.tar.gz
etcd-v3.1.10-linux-amd64.tar.gz
etcd-v3.2.17-linux-amd64.tar.gz
etcd-v3.3.12-linux-amd64.tar.gz
get-pip.py
stackviz-latest.tar.gz
zanata-cli-4.3.3-dist.tar.gz

I've trimmed out the vmdk, vhd, and tarball based images as we should
all be using qcow2s. Everything under etcd is provided by preexisting
static lists.

Change-Id: Iff741e8ed4c517ccabae6e6d6ba730f0aa37a272
2020-03-13 16:52:21 -07:00
James E. Blair fc792f3371 Cache dstast_graph on nodepool images
So that we can create dstat graphs at the end of devstack and other
jobs, cache the dstat_graph repository on our images.

dstat_graph is MIT licensed, but as far as I can tell, does not
have a release and is not packaged in Debian or Ubuntu.

Change-Id: I6cb0aa525dca0ee700d2d7548ccbb8d6af6ae92e
2019-07-02 07:32:11 -07:00
Andreas Jaeger 14eb58501b Fix nodepool after OpenDev changes
Fix a couple of places where orgs where renamed so that dib finds the
repos.

Change-Id: Ic52dcffcd9323c12410e05b1a3598911a28cbcaa
2019-04-21 17:05:34 +00:00
Andreas Jaeger f43766da84 Update nodepool for OpenDev
Use new locations following OpenDev changes.

Node: This changes the on-disk repo path to /opt/git/openstack/devstack

Change-Id: I7042913fefa64dcec4044779dbeb13f86daea858
2019-04-20 13:40:41 +00:00
Clark Boylan 2224100eac Update stackviz tarball location
We've patched stackviz to work under python3 properly but we are still
pulling an old tarball for stackviz that was built last year. The legacy
job that built the file at this location seems to have been removed.
Switch to the new dist/ location which appears to be correct based on
tarball file sizes.

Someone that understands stackviz better than me should confirm this new
locations is the correct one.

Change-Id: If659a6f1fb50d288afed75e3f4975f7a4d140d35
2018-10-04 10:46:08 -07:00
Ian Wienand 91b88fde0a Use https for stackviz get
We are noticing on nb03 that the curl get of this file fails; but it
starts working when you use https.  Switch to it.

Change-Id: I5fef4281547758f484cd8a93326b0cfeb790762a
2018-04-18 07:27:12 +10:00
Zuul 20244e93e4 Merge "Remove unnecessary distribution determination" 2018-03-05 09:04:43 +00:00
Dirk Mueller 6864367ad8 Remove unnecessary distribution determination
The need for the distribution was removed in I249f21a98fea3b963b7ffb8e3d0fce02cc540d46
so we can remove the code around that was determining it but not
actually making use of it anywhere anymore.

Change-Id: I572943bbc0c2665a885b7a80cdb2d1817c61bc3c
2018-03-02 01:55:11 +01:00
Andreas Jaeger 939cb6d41d Update zanata-cli to 4.3.3
The server is now running version 4.3.3, update the client as well.

Change-Id: I68727dc5fe9efef392bbb8250260b2c4524c1153
2018-03-01 09:03:48 +01:00
Ian Wienand b27d4c731e Add newline to zanata source-repository file
Without the newline this falls into the old bash "read" gotcha trick
where the last line will be skipped silently.  This means the client
isn't actualy downloaded.

dib-lint added a check in I23e293b957cd4f008611656cf9166391b1b537a2
and I tested it would have caught this.

Change-Id: Idbae2037f275fad91b87e1cf5ee80663c1f05ddc
2017-12-08 14:27:33 +11:00
Ian Wienand 3466674297 Cache zanata cli tools
Cache zanata cli tools during build, and up the prep-zanata role to
look for the cached version in preference to dowloading.

Change-Id: If70f3386d0350c94dd1548e7f3c60a1d02914824
2017-12-04 16:06:18 +11:00
Monty Taylor 36304438ee
Replace git:// with https://
git:// is not a secure protocol. It's also harder to do apache rewrite
tricks and whatnot when we eventually flatten namespaces. Change
occurances of git://git.openstack.org to https://git.openstack.org.

Change-Id: Ic54f1b84346e37a76baa87deadbb540662889bca
2017-11-17 09:06:21 -06:00
Andreas Jaeger 4adbecad99 Fix flake8
fix a "bug" in that the flake8 configuration in tox.ini was exclusively
selecting H231 as the only error it would report, so it was missing the
errors in the python modules (such as submit_log_processor_jobs). Due to
this being the case for a long time (since 2004) limit the more thorough
linting to the roles/ and playbooks/ directories where we'll be adding
ansible plugins/modules/etc. Also, lint in jenkins/script and nodepool.

Fix problems found.

We can lint everything with pep8 once the zuul v2 scripts are removed,
not worth patching them right now.

Change-Id: I479f010643cf3b67c183d763510f07a33400d38b
Co-Authored-By: Jesse Keating <omgjlk@us.ibm.com>
2017-10-21 18:37:10 +02:00
Paul Belanger 3ef635e004
Remove 99-cache-testrepository-db from cache-devstack element
It seems this optimization is no longer needed. Once we address:

  https://bugs.launchpad.net/testrepository/+bug/1416512

We could consider adding it back. However, this also means we stop
hitting logstash.openstack.org each time we build an image. So, we
should maybe consider caching that data some place over having our
image builds do it.

Change-Id: I16b56a13028c0f228d21dde423db3f70f07f4e1a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-05-25 11:45:17 -04:00
Ian Wienand 5941b67292 Python3 support for cache-devstack/openstack-repos
Add python3 support for these functions.  Tested with a python3 based
dib in a Xenial environment.  Add __future__ imports so we don't
regress on these bits.

Change-Id: Iddca88cb6c9fbff6383c3597eb81133948b3b420
2017-05-17 15:22:34 +10:00
Ian Wienand 2e308fc94a Move jenkins cache to /opt/cache
For whatever reason, the source-repositories caching element is
running in the very early extra-data phase of disk-image-builder.
cache-devstack is instructing source-repositories to put the cached
urls into /home/jenkins/cache/files (55-cache-devstack-repos makes up
the list of images to download from devstack; the other two are just
pre-configured URLS).

This causes a problem when trying to add the jenkins user later in
jenkins-slave/install.d/20-jenkins-slave.  Since source repositories
has already created /home/jenkins, useradd won't do it again and does
not copy in any of the skeleton files.

This moves the extra-data phase caching into /opt/cache, which is
probably a more logical place for it anyway.  To maintain the status
quo, after we create the jenkins user, symlink ~jenkins/cache/files to
this new location.

This was probably going wrong with puppet too.  I'm guessing it just
ignored the existing homedir when creating the user.  I have tested
this by building trusty and stopping the build before and after the
useradd calls in 20-jenkins-slave.  "/home" was empty, the useradd
creates the user, group & directory, the symlink is made correctly and
/opt/cache/files appears populated as usual.

Change-Id: Ibca6867f29b257a5110cb5522a5cca3a97fa9377
2017-04-28 16:00:23 +10:00
Paul Belanger 2a3cb6a729
Add stackviz-latest.tar.gz to ~jenkins/cache/files
In an effor to remove the stackviz element from nodepool, we can now
use the periodic job and download the package from tarballs.o.o. This
mean, we can remove npm development libraries from our base images in
a follow commit.

Move periodic-package-stackviz-element to the periodic pipeline.

Change-Id: I798c98f9ed91e0e7b2a43bd9e8d0837ce223cb8b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-16 13:13:22 -04:00
Paul Belanger fbf7661ed1
Don't force python2 for subunit2sql install on workers
We gate on python3.5 for subunit2sql, we should allow our DIB install
to use it.

Change-Id: Ie411a6317c83687db14039820845576c7838c89d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-04-10 15:47:20 -04:00
Paul Belanger 239df85cb2 Stop caching devstack packages
After talking with clarkb, we decided it might be time to also stop
caching packages for devstack. Like the cache-bindep patch before, we
have a robust mirror infra in place right now, and believe we are
ready to start using it more.

Change-Id: I249f21a98fea3b963b7ffb8e3d0fce02cc540d46
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-12-14 09:44:13 -05:00
Matthew Thode 9231f9b57d
Add Gentoo support to nodepool
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image.  The previous version of this commit had the default
section of the pkg-map in the wrong area, specifically in the following
file.

nodepool/elements/infra-package-needs/pkg-map

Change-Id: Ic686c325bc06564585a2e3ac50cadd7556612333
2016-10-16 20:37:14 -05:00
Paul Belanger 510faf0bde Revert "Add Gentoo support to nodepool"
Our images have failed to build for the last 3 days, reverting until we can debug.

This reverts commit 4c515e2073.

Change-Id: I2e653bcd8a30a85ea46a9861bdf9f95413a76f64
2016-10-13 21:39:04 +00:00
Matthew Thode 4c515e2073
Add Gentoo support to nodepool
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image.

Change-Id: I2ceeb915748a11d8e729069566e722a3fe30ba99
Signed-off-by: Matthew Thode <mthode@mthode.org>
2016-10-03 10:24:44 -05:00
Jens Rosenboom 7811850bbd Add dib-lint tags to sudo calls
In [1] some new safe_sudo checks were added, which lead to errors
in the project-config elements. We add tags to ignore these errors
for now.

[1] I161a5aea1d29dcdc7236f70d372c53246ec73749

Change-Id: If3ff10b8fbf20a8327895f439b955f9941818668
2016-06-07 09:30:52 +02:00
Clark Boylan 7f35178617 Fix package and image list comment cleanup
When cleaning up comments from lines of the form:

  foo bar # this is a comment

You want to remove remove everything after the first comment character
not after the last comment character. This is because:

  foo bar # this is # a comment

Should get everything after bar removed not everything after is.
Unfortunately when you use str.rfind() you remove everything after is
not everything after bar. Switch to regular trusty find to fix this.

Change-Id: I78aa6b51b5be03bd3b8ce7885415442171218977
2016-04-19 17:27:14 -07:00
Ian Wienand abbbf317ce Cleanup the early devstack clone script
This script is unnecessarily complex for what it is doing, which is
essentially pre-seeding the source-repositories cache for devstack.
As we can see from I41e81d6bac98875eecde2376e0865784626e11a8 it's very
confusing having large parts of the source-repositories script
copy-pasted as a separate element and has led several of us down the
wrong path.

Strip this script back to the simple thing it is doing, which is
checking out/updating the source-repositories devstack cache.

I have tested this by building an image with a warm cache and with a
cold cache, in both cases the checkout was found and the list of
images to cache in 55-cache-devstack-repos was found.

Change-Id: I6c686312de102cbe438585e26bf6986e06b6f41c
2016-01-19 11:11:28 +11:00
Ian Wienand c70cf77969 Update remote refs in early devstack clone
In another confusing addendum to
I41e81d6bac98875eecde2376e0865784626e11a8 (which was already a
confusing addendum to Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9), we
have decided that the refspec used to fetch here is not sufficient to
clear out remote branches everything when updating.

"+refs/heads/*:refs/heads/*" says to replicate everything from the
remote refs/heads into our local refs/heads, but leaves out
refs/remotes/*

The upshot of this is that I41e81d6bac98875eecde2376e0865784626e11a8
will remove the local branches (refs/heads/stable/icehouse, say) but
not remove the remote branches (refs/remotes/origin/stable/icehouse).
The devstack caching script keeps picking up these remote branches,
checking them out, and consequently trying to download old images.

*Nothing* ever removes these branches.  In the main dib cache git
update, we also have --prune, but our refspec there is even more
limited (+master:master).  This explains why these branches never seem
to die.

Note, an even better mirror would be "+refs/*:refs/*" (in fact, if you
do git clone --mirror, this is what the repo would be setup with to
fetch by default).  However, this drags in "refs/changes/*" and all
sorts of other gerrit things.  We don't really need them so we just
keep the limiting on.

Change-Id: Ia9c3ffdb2b5f72a45d629961338b415308d6dd21
2016-01-19 11:06:12 +11:00
Ian Wienand 53832d82cd Prune old branches in early devstack clone
In a rather confusing addendum to
Ieb6a6e9f55bd93f63c3d0a71828c276c2d02e1b9, we have actually mirrored a
version of the source-repo script from dib and munged it to cache
devstack early so we can use it to find the vm images to download and
cache.

However, we are not ensuring that we remove old branches in this
clone, which is leading to the problems of us picking up images from
old branches that don't exist any more.

Change-Id: I41e81d6bac98875eecde2376e0865784626e11a8
2016-01-18 17:27:31 +11:00
Andreas Jaeger d8170deb07 Remove CentOS6 from nodepool scripts
With CentOS6 usage removed, remove special handling for it from nodepool
scripts.

Change-Id: Ife5ec9ade201dcd8f5969087dd2fc88be12dbebd
2015-12-04 22:11:22 +01:00
Matthew Treinish 3ed5aa98c5 Revert "Set a temporary cap on subunit2sql before 1.0.0 release"
The release has happened, the migration worked, all's right with the
world.

This reverts commit 04eb36588d.

Change-Id: I28a3e9cf3b420a6ccca20c52c4e67adc3f5710c5
2015-11-29 17:01:45 +00:00
Matthew Treinish 04eb36588d
Set a temporary cap on subunit2sql before 1.0.0 release
This commit adds a temporary version cap on subunit2sql to be < 1.0.0.
The 1.0.0 release includes a very large database migration which will
be slow to execute. The python DB api from >=v1.0.0 will not work with
a database that doesn't have the updated schema. So while the migration
is running let's cap the version we install to prevent everything from
breaking while the migration is running. (which might take days)

Change-Id: Iab89beb5c7aba8b744a62f5063e513b72cab0ec2
2015-11-23 15:49:58 -05:00
Jeremy Stanley 23da1ea04a Revert "warm pip wheel cache during image build"
Turns out we can't do this without dirtying our images by
preinstalling all the libraries/headers our requirements want to
link against. We should revisit the wheel mirror solution which has
already been written as a saner solution.

This reverts commit 1503ef3278.

Change-Id: I69ea73b6eb4b620f1ac77467ee3784ec3c813ee2
2015-11-04 20:40:23 +00:00
Sean Dague 1503ef3278 warm pip wheel cache during image build
This installs everything in upper-constraints in a throw away venv,
which has the knock on effect of fully populating the wheel cache for
root with wheels for everything. This hopefully speeds up all devstack
runs quite a bit.

Change-Id: I429f353c33d892f76552c83d34ac1329ed18f97f
2015-11-04 12:52:22 -05:00
Ian Wienand 90c53a8ded diskimage-builder element cleanups for dib-lint
Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection.  Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)

Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script

With this, "tox -edib" passes

Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
2015-10-08 11:33:03 +11:00
Ian Wienand 32dcd6af92 Skip caching for dnf (Fedora 22) builds
We are still working on the caching story for dnf-based builds
(i.e. Fedora 22). There are a couple of options which we will work
through on the linked page.  We will sort this out before we move the
devstack job (yet to be created, because the nodes aren't there yet)
out of experimental.

In the mean time, disable the caching in these elements so we can get
Fedora 22 image builds (currently they're in a big looping failure
[1]).

[1] http://nodepool.openstack.org/image.log

Change-Id: I3a435889fc5109d7365240068047aac98abc605e
2015-10-01 15:35:16 +10:00
Jeremy Stanley 3d5b04ed19 Deal with yum --downloadonly CentOS 6.x quirks
Some packages may depend on python-setuptools, which is not
installed and cannot be reinstalled on CentOS 6.x once yum has
erased them, so use --skip-broken to avoid aborting. Also on this
platform --downloadonly causes yum to return nonzero even when it
succeeds, so ignore its exit code.

Change-Id: Iaada39ae81e1e47fe9d0bedba80fd19e4e0e6f38
2015-05-12 15:25:01 +00:00
Jenkins 47532ee5c3 Merge "Install yum-plugin-downloadonly on nodes" 2015-05-07 16:11:27 +00:00
Jeremy Stanley 0137c25b46 Install yum-plugin-downloadonly on nodes
The --downloadonly option to yum is provided by the
yum-plugin-downloadonly package. This is merely a virtual package
satisfied by yum itself in newer releases, but an optional package
in older ones such as CentOS/RHEL 6.x. Install it just to be sure it
will work, since we use this to pre-cache RPMs on nodepool images.

Change-Id: I9e2e1605f3721c410180aa46a81b7b731d08503a
2015-05-07 13:33:12 +00:00
Jeremy Stanley 572c35a827 Look for yum instead of rpm in elements too
On CentOS 6.x the rpm executable is in /bin instead of /usr/bin, but
the cache-devstack element ends up running yum if it wants to
download RPM packages anyway. Look for yum instead of rpm as an
indication of which packages to install.

Commit fa18656 fixed this in cache_devstack.py for snapshot images,
but missed that it was affecting diskimage-builder elements as well.

Change-Id: I4b76a48564cd5d703d79cfda88bbbfe97216a70a
2015-05-07 01:56:52 +00:00