We can not speculatively test changes to the wheel build jobs while it
is project config. These jobs are in the gate for the requirements
project, which run them when bindep.txt changes.
https://review.opendev.org/731394 moved the build-wheel-mirror-base
job/playbook/role into openstack-zuul-jobs, renamed as
build-wheel-cache-* to a) distinguish the name and b) make it a little
clearer that we're building the wheels that goes on the mirror, rather
than passing through to mirroring something existing.
This updates the job names in project-config to reference these new
jobs.
Note the publishing steps are kept here along with the AFS secret.
Depends-On: https://review.opendev.org/732085
Depends-On: https://review.opendev.org/732087
Depends-On: https://review.opendev.org/732083
Depends-On: https://review.opendev.org/732084
Depends-On: https://review.opendev.org/732086
Change-Id: Iac3a906803177e8365f4cfb611800b5ccaed4a6e
It seem that pip has updated its verbose output to say "Downloading
$filename" rather than "Downloading from url $URL". We need to update
our wheel mirror processing to handle this change so that it can
properly remove any pypi hosted wheels before publishing to our wheel
mirrors. Make it more liberal so we match pretty much any line with
"Downloading" and then eventually something which looks like a whl
file, just in case they keep fiddling with it.
Change-Id: I7965f296cdb1241c96293edd68e84e1d3d78b331
This reverts commit 8b3532c562.
The change mentioned inline has been incoporated upstream with [1]. I
don't think we've managed to replicate the corrupt wheel situation
since.
[1] 0dbab23df9
Change-Id: I58fa0a53939207427f286584e91e1c59d4863992
A recent change assumed $LOGS was a relative path, but it starts
with $(pwd) so is absolute. Correct this broken assumption which was
causing the script to exit non-zero and fail the
publish-wheel-mirror-* jobs.
Change-Id: I837a0657aa9ae11211599b019711cba8f17ee094
In the build-wheels role, parse stdout from all the build logs to
build up a unique list of all wheels downloaded from PyPI and delete
them from the wheelhouse, since this is only meant to provide built
wheels which are absent from PyPI. Those will be retrieved from our
caching proxy of PyPI served from the same mirror hosts anyway.
Change-Id: Id021ba5fd55bf6d43e99f9f3a7121aee8b0d0a6f
As pointed out by ajaeger; the wheel build jobs are failing with
Error installing 'pip from
git+https://github.com/ianw/pip.git@path-and-hash#egg=pip': editable
mode is not supported for pyproject.toml-style projects.
This appears to be an issue with pip 19.0 which would be installed on
the build hosts, interacting with projects that have a pyproject.toml
file.
The problem only seems to be related to installing in editable mode,
which we don't really need here on this ephemeral host. So just
switch to a regular install, with upgrade/reinstall to be sure.
[1] https://github.com/pypa/pip/pull/6449
Change-Id: I006500a2c98f512176fbac4ab426a8719243fc70
Follow OpenDev changes:
* Use gerrit.opendev.org instead of gerrit.openstack.org
* Use opendev.org insted of git.openstack.org
* Follow move of openstack-infra/project-config to
openstack/project-config
Change-Id: Ic27ea41407abc0bdec29768e5ee5dca33f8adeda
We are currently trying to debug persistent wheel corruption. So far,
we have been completely unable to replicate the corrupt output outside
the periodic jobs.
A first thought was that it was due to AFS corruption due to multiple
writes (I4f8a2f2c6c8164e7ea207f8e4b286e06df0b13dd), however this does
not appear to be the case.
Experimentally manually running the builds, on infra nodes, under
python2 and python3, in parallel just as done here, does not replicate
the problem.
This patched version of pip will output the sha256 hash for the final
build output of each wheel. The plan is to correlate that against any
corrupt file that appears. If the corrupt hash matches any file
produced by pip, then we know the problem is inside pip (and we will
have the exact build situation that it occured in); if the corrupt
file does not match then we must have some sort of issue copying the
files or similar.
Change-Id: I81943ed459bf4e2c77cae42e50af5fc5979682b4
I think we have a situation where a wheel build has somehow put a
corrupt wheel into our mirrors, but the next run seems to grab the
corrupt wheel from the mirror again ...
Remove the wheel mirror from the pip configuration before starting the
build so we're actually building directly from upstream, which is what
is intended.
Change-Id: Icb590e805d0d29130bff99335590d1d0f646e4e4
$16 was right until we added --exists-action=i since we want the last
field just use $NF
---
$ wget http://logs.openstack.org/41/588441/2/gate/build-wheel-mirror-ubuntu-trusty/5c9e90e/python2/master-job.log
<snip>
2018-08-30 18:16:56 (191 KB/s) - ‘master-job.log’ saved [89855/89855]
[tony@thor project-config]$ awk -e '$7!=0 {print $NF}' master-job.log
Command
systemd-python===234
---
Change-Id: If52e5655070c91f1508b815d3ec53421f55d2da5
It seems that on trusty, when two branches have the same version of a
requirement the first version builds, but then pip explodes on the
second try as it prompts you asking if you want to overwrite the
existing download. Use the "--exists-action" flag to ignore.
We have added two arguments for the upper-constraints.txt, which has
pushed out the columns for the error matching. Fix that too.
Change-Id: I53ed7d8a86e6910b00a181cc3ca37b7aa4b7bb48
The requirements project is very slow to actually tag branches as EOLd as
that can cause some testing environments and devstack to fail. For
example if tox.ini refers to a constraints file in git[1].
However building wheels for those releases isn't actually helpful as
they're not chnaging *and* can cause failed runs as pypi etc move on.
This change adds an regex for the closed branches to exclude them from
building. The regex is a little more complex than the single case
(newton) requires but allows for easier future additions.
[1] http://git.openstack.org/cgit/openstack/nova/tree/tox.ini?h=newton-eol#n12
Change-Id: Ia0c589989cb6e7e81e7b08e39d81dac78c55bf18
No need to use magic names since we now have host_vars, instead set
wheel_python explicitly.
Remove TODO comment.
Change-Id: I5051a32d843b106b013b94d57fd24b63896b741a
We only need the three wheel-X scripts from jenkins/scripts and need
them only for the playbook and its roles. Move the scripts in the
roles directory and copy them only if needed - using the scripts module.
Remove copying of complete jenkins/script for this playbook using the
role legacy-copy-project-config-scripts, it's not needed anymore.
Change-Id: I2e481fcf7ca148aac8a36ae99d8598ba26078a25
Silence remaing two ansible warnings: Since we cannot really test this
change currently, add a TODO and silence the warning.
The change to command module can be done later, let's move forward
to get the job voting.
Change-Id: I01d22702f3294d7958c8982dc7f2c31a02aa81e2
Rather than passing a string to the shell task and then using
shell.args.executable to choose an explicit executable, use
shell.cmd to set the actual script content so that shell.executable
can be used to choose the shell.
Change-Id: Ia24a21ce54746a486ba39c4eb7c32bc8470e32d4
Otherwise, we cannot write anything to the folder. Also clean up
shell task since our bash script already setups /bin/bash -ex.
Change-Id: I0a9770dfb1bf252cf092e6edc8491e6d801ab0e3
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We need sudo permissons to create /opt/wheel for build-wheel-mirror
jobs.
Change-Id: I3676a82ee8b0a7bb06896b79bdf26f61d9b75534
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Including puppet madness!
We don't need the long-lived wheel builders - we build from scratch
every time. So all we need is AFS running on a node.
AFS, however, is tricky on CentOS. Rewriting all that right now is meh.
We can do that later.
This gets us to not needing special nodes for wheel builders. Yay!
Change-Id: I6d92c47f4f7a26ca290cd400452cb0a73872b24d
NOTE: It may be inefficient to run this on our low-bandwidth nodes.
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Pass the wheel keytab across and aklog with it.
Also, split the build role into a build and a copy so that we only have
the token when we need it, and not while we're running the wheel builds
themselves. And split release from build so that we can release the
mirror once for a consistent state across python2 and python3.
Change-Id: Id30a99f5b515dd520d013fabbd356e32c82030a0
Ian Wienand