Currently, we only install bindep into a virtualenv in our
cache-bindep element. We can move this into infra-package-needs as an
optimization and reduce elements we depend on.
Change-Id: I591ed9a0835609c1ff8cc6069e45231032fd9f28
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We are currently seeing the following traceback for fedora-25:
Traceback (most recent call last):
File "/usr/bindep-env/bin/bindep", line 11, in <module>
sys.exit(main())
File "/usr/bindep-env/lib/python3.5/site-packages/bindep/main.py", line 64, in main
errors = depends.check_rules(rules)
File "/usr/bindep-env/lib/python3.5/site-packages/bindep/depends.py", line 205, in check_rules
installed = self.platform.get_pkg_version(rule[0])
File "/usr/bindep-env/lib/python3.5/site-packages/bindep/depends.py", line 313, in get_pkg_version
universal_newlines=True).decode('utf-8')
AttributeError: 'str' object has no attribute 'decode'
Until fixed, pin to python2.7.
Change-Id: I5ac830b5f2f841dc50dafb58dccad6602afe448a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Today each mirror region has an AFS mirror cache of packaging, as a
result lets try having jobs use it over caching bindep.txt
dependencies on our images.
This should save us about 27mins in diskimage build times for
ubuntu-trusty and ubuntu-xenial.
Change-Id: I74ad0fdbd939948d9285dcd7ef839bdbe299319f
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image. The previous version of this commit had the default
section of the pkg-map in the wrong area, specifically in the following
file.
nodepool/elements/infra-package-needs/pkg-map
Change-Id: Ic686c325bc06564585a2e3ac50cadd7556612333
Our images have failed to build for the last 3 days, reverting until we can debug.
This reverts commit 4c515e2073.
Change-Id: I2e653bcd8a30a85ea46a9861bdf9f95413a76f64
This commit adds Gentoo support to elements needed to build a Gentoo
nodepool image.
Change-Id: I2ceeb915748a11d8e729069566e722a3fe30ba99
Signed-off-by: Matthew Thode <mthode@mthode.org>
Since support for using "-f -" to read a package list from stdin
is supported in bindep 0.1.0 and later, remove the hacky temporary
workaround which stashed it in a tempfile.
Change-Id: I02d153113a160028658e38828e113fcf02bc5ab1
Like ianw suggested, we should fallback to yum if ${YUM} is not
defined.
Change-Id: I9e2c632a3c7225d13e6245e65fa4da29ce7541df
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Currently, we don't have 100% gate coverage for
other-requirements.txt. As such, it is possible for our DIB builds to
fail on bindep. So, we only cache packages for Ubuntu Trusty ATM
until we can enable 100% coverage for centos and fedora for
other-requirements.txt per project.
Change-Id: I0f06619224ab59bc6165316226abeccb75c101cb
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Since I6c5a962260741dcf6f89da9a33b96372a719b7b0 dib has had a
standardised method for ensuring consistency of tracing and error
detection. Bring the tracing for these elements up to that standard,
but maintain the status-quo of flags such as "-e" and "pipefail" by
adding ignore flags where appropriate (we can update these separately
to avoid breakage)
Other minor changes are alphabetical-ordering in the element-deps
files and permissions on prepare-node script
With this, "tox -edib" passes
Change-Id: Ibba1dadb9e819f94294c9d583b83ff698252f93f
We are still working on the caching story for dnf-based builds
(i.e. Fedora 22). There are a couple of options which we will work
through on the linked page. We will sort this out before we move the
devstack job (yet to be created, because the nodes aren't there yet)
out of experimental.
In the mean time, disable the caching in these elements so we can get
Fedora 22 image builds (currently they're in a big looping failure
[1]).
[1] http://nodepool.openstack.org/image.log
Change-Id: I3a435889fc5109d7365240068047aac98abc605e
Invoke bindep from the bindir for its virtualenv in /usr/bindep-env,
and guard its exit code with true since its normal behavior is to
return zero only when all needed packages are already present.
While here, replace calls to tempfile with mktemp since the former
is not readily available on CentOS 6.x.
Change-Id: I926e73f305b1b54f6855cc4acf3407e91b88213a
While git show is being used to build up a list of candidate
packages to cache, it can cause the cache hook to abort if it
returns nonzero because of a nonexistent bindep list file. Guard
that with a fallback to true so that the loop can continue
normally.
Change-Id: Ic3b773ad2e8efa2c450d37a0115355c0e95da853
The version of git on CentOS 6.x is too ancient to support git
branch --list which is necessary to disambiguate -r with a name
pattern. Instead just list all remote branches and filter with grep
like a good Neanderthal.
Change-Id: I4e673e86acb70c205d159e22b5864cf60ae831df
A quirk of diskimage-builder is that no two elements you compose are
allowed to have parts with the same filename. Rename cache-bindep's
install.d/50-download-pkgs so that it doesn't conflict with the one
of the same name in the cache-devstack element.
Change-Id: I5709147fc5bd4c06fb1bba8ea0e068c0eda7d010
We intend to use bindep to tell us what distro packages should be
installed at job run-time, so need to similarly identify those in
such a way that they can be cached on our worker images.
Install the latest bindep release from PyPI into /usr/bindep-env
(consistent with the way we provide virtualenvs for access to
zuul-cloner and libraries used by our log uploads to swift).
Concatenate bindep lists from all branches of every hosted Git repo
along with our fallback list in project-config. Run bindep to render
them into a platform-specific manifest of package names and feed
that to the relevant package management tool for the platform to
download this set into the package cache.
Note that similar tooling is not applied to snapshot nodes, since we
only intend to run jobs which use bindep on diskimage-builder
created nodes.
Change-Id: I0c408fee35dfe9b4f700c51565bfaae0a3d03beb
Paul Belanger