Commit Graph

11 Commits

Author SHA1 Message Date
Monty Taylor 7e09fc71a3
Add wheel mirror jobs
We need to run periodic jobs to build the wheel mirrors.

This removes them from the v2 config because we really should not run
these jobs from both systems at the same time and once this lands it'll
be live.

It copies the scripts from project-config to the homedir, so we can
ditch the puppetted copies of them.

Change-Id: I87a78509a97d53eb627a84c646a6a3f0a227d9a1
2017-09-06 14:44:02 -04:00
Ian Wienand 8928d5f626 Run wheel build jobs in parallel and keep logs
Even after increasing the build-time to 180 minutes with
Idc1d53d1e73b4a9eb13f258f4a4d5627ec3cf300 it's not enough to avoid
timeouts.  I did a manual run and it took 214 minutes on the centos
builder.

We can parallelise builds.  The "parallel" program provides us with
plenty of useful help in this regard.  As described in the comments,
this will run ncpu jobs (4 on our builders) and capture and store the
logs for each individual job.  A test run on the centos build host was
reduced to ~1 hour.

The job output is particularly handy because we can do a little work
to make an easy-to-parse failure log.  This job is a little weird
because failures aren't hard -- we can still release everything we did
manage to build.  Initially this overview log file will enable a 3rd
party monitor that can alert interested people when failing builds
occur.

We store the build logs as a .tar.bz2 bundle as described, but the
smaller files are copied directly.

Change-Id: Ifec15f71cc3530d47da4f3304d3fe094d77e0980
Depends-On: I9635721a4f8d718ad402d23600840f091267952c
2017-01-24 17:10:42 +11:00
Ian Wienand ab2bd0db31 Update wheel build job timeout
As seen in [1] it can take longer than 90 minutes to run the wheel
build job.

This is probably due to I70ae19cc974a8c34ad9ff380fee5adbb17c96ff4 and
the job not running for a while.  I will keep an eye on it -- if this
isn't long enough I'll take the effort to run it manually.

[1] http://logs.openstack.org/periodic/wheel-build-centos-7-amd64-python2/0cdc862/console.html

Change-Id: Idc1d53d1e73b4a9eb13f258f4a4d5627ec3cf300
2017-01-18 19:29:55 +11:00
Paul Belanger 022a55aa11
Fix wheel-build macro issue
We incorrectly didn't pass our python variable into the wheel-build
builder.

Change-Id: I6ef90f5fd8bce27c152c898591087e42fd57dc18
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-06-14 12:50:41 -04:00
Paul Belanger f9acae2df9
Add python version to wheel builders
This should allow us to break our our different python version into
different job runs.

Change-Id: I8e3ed20cf269a12fa06be657301356334953331a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-06-13 12:22:02 -04:00
Paul Belanger c149a8aed3 Update centos7 wheel mirror naming schema
Because of AFS constraints, we need to shorten our naming for centos7.

Change-Id: I7f6f389ae692c764f5e4211abe9932a9e6b6a8c7
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2016-03-23 12:58:50 -04:00
Michael Krotscheck 4583ce559b Corrected AFS volume name for wheel release
The convention is mirror.wheel.<ARCH>, not mirror.wheel-<ARCH>.

Change-Id: I08f2657d2e7c6ab1a2b25eb25ce0626d8e86b33a
2016-02-05 07:10:25 -08:00
James E. Blair 8a24e2e3e8 Fix wheel release job
Fix a typo in the wheel release job.  Also make sure that it is
not set to offline the node.

Change-Id: Ie76331068545173b863d31f1fe793f46b6578700
2016-02-04 20:01:27 -08:00
James E. Blair 053cab096a Mirror release: Add verbose flag
Because verbose is cool.

Change-Id: If351a8d9943bf21bea8d0916e0cf9b55e5dc68b4
2016-02-04 12:09:59 -08:00
Michael Krotscheck d88a460e97 Added wheel-release job
This patch adds a wheel-release job on our release slave,
which is triggered after a wheel is built. It runs vos release,
to push the new entries in the filesystem to our AFS masters.

Change-Id: Ic1e8a6725fc046d8240a9665f6ed1167be53cb49
Depends-on: I81c39d420d8ac70def57949ea0d4c323b8797086
2016-02-03 09:34:08 -08:00
Gregory Haynes 04ec88d3cb Create jobs for a wheel mirror
In order to speed up tests where we install python packages we want to
create wheels for packages we depend on.

This patch creates four new scripts:

* afs-slug, which generates a string version of the platform architecture,
  derived from the host system's parameters.
* wheel-build, which generates all wheels, from all branches, in `pwd`/wheels
* wheel-copy, which copies this wheels directory into our AFS file structure.
* wheel-index, which rebuilds the index.html file from the directories
  found in the afs directory.

The above scripts are executed in sequence, with afs credentials only loaded
where necessary. Additional timeouts have been provided to prevent things from
locking.

This patch assumes that a mod_rewrite rule will be available on the serving nodes
that unmunges the URL.

Co-Authored-By: Sean Dague <sean@dague.net>
Depends-On: Ie33861dbcb413c34012820fda76f8be94fb1d151
Depends-On: I81c39d420d8ac70def57949ea0d4c323b8797086
Change-Id: I3463fba889239c565c86ffb080adffd7bfdd18f9
2016-02-03 09:34:03 -08:00