We need to run periodic jobs to build the wheel mirrors.
This removes them from the v2 config because we really should not run
these jobs from both systems at the same time and once this lands it'll
be live.
It copies the scripts from project-config to the homedir, so we can
ditch the puppetted copies of them.
Change-Id: I87a78509a97d53eb627a84c646a6a3f0a227d9a1
Even after increasing the build-time to 180 minutes with
Idc1d53d1e73b4a9eb13f258f4a4d5627ec3cf300 it's not enough to avoid
timeouts. I did a manual run and it took 214 minutes on the centos
builder.
We can parallelise builds. The "parallel" program provides us with
plenty of useful help in this regard. As described in the comments,
this will run ncpu jobs (4 on our builders) and capture and store the
logs for each individual job. A test run on the centos build host was
reduced to ~1 hour.
The job output is particularly handy because we can do a little work
to make an easy-to-parse failure log. This job is a little weird
because failures aren't hard -- we can still release everything we did
manage to build. Initially this overview log file will enable a 3rd
party monitor that can alert interested people when failing builds
occur.
We store the build logs as a .tar.bz2 bundle as described, but the
smaller files are copied directly.
Change-Id: Ifec15f71cc3530d47da4f3304d3fe094d77e0980
Depends-On: I9635721a4f8d718ad402d23600840f091267952c
As seen in [1] it can take longer than 90 minutes to run the wheel
build job.
This is probably due to I70ae19cc974a8c34ad9ff380fee5adbb17c96ff4 and
the job not running for a while. I will keep an eye on it -- if this
isn't long enough I'll take the effort to run it manually.
[1] http://logs.openstack.org/periodic/wheel-build-centos-7-amd64-python2/0cdc862/console.html
Change-Id: Idc1d53d1e73b4a9eb13f258f4a4d5627ec3cf300
We incorrectly didn't pass our python variable into the wheel-build
builder.
Change-Id: I6ef90f5fd8bce27c152c898591087e42fd57dc18
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This should allow us to break our our different python version into
different job runs.
Change-Id: I8e3ed20cf269a12fa06be657301356334953331a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Because of AFS constraints, we need to shorten our naming for centos7.
Change-Id: I7f6f389ae692c764f5e4211abe9932a9e6b6a8c7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
This patch adds a wheel-release job on our release slave,
which is triggered after a wheel is built. It runs vos release,
to push the new entries in the filesystem to our AFS masters.
Change-Id: Ic1e8a6725fc046d8240a9665f6ed1167be53cb49
Depends-on: I81c39d420d8ac70def57949ea0d4c323b8797086
In order to speed up tests where we install python packages we want to
create wheels for packages we depend on.
This patch creates four new scripts:
* afs-slug, which generates a string version of the platform architecture,
derived from the host system's parameters.
* wheel-build, which generates all wheels, from all branches, in `pwd`/wheels
* wheel-copy, which copies this wheels directory into our AFS file structure.
* wheel-index, which rebuilds the index.html file from the directories
found in the afs directory.
The above scripts are executed in sequence, with afs credentials only loaded
where necessary. Additional timeouts have been provided to prevent things from
locking.
This patch assumes that a mod_rewrite rule will be available on the serving nodes
that unmunges the URL.
Co-Authored-By: Sean Dague <sean@dague.net>
Depends-On: Ie33861dbcb413c34012820fda76f8be94fb1d151
Depends-On: I81c39d420d8ac70def57949ea0d4c323b8797086
Change-Id: I3463fba889239c565c86ffb080adffd7bfdd18f9