Pip 22.0 switched[*] its HTML parser library for one which demands
strict adherence to HTML 5, and so cannot work if told to include
our wheel indices because they lack a doctype declaration on the
first line. Add one.
[*] https://github.com/pypa/pip/issues/10825
Change-Id: Ia863e313e656e67191e1ca3aea968a7f3e4059a6
Creating the index has turned out to be much more difficult than first
expected. Checksumming the files on AFS to make the index.html takes
too long and times out the jobs.
We've identified other issues such as the jobs only ever appending
wheels, making far more data than probably necessary being kept, and
the possibility of skipping those wheels already available on PyPi.
Disable this while we reconsider the approach.
Change-Id: I86a922d59b396a059b255a4d87d8c033b79f7564
I swear this should work now ... $f is the full path to the wheel
directory
Additionally ... f-strings aren't available on Python 3.5, and we
still build for Trusty which has Python 3.5 ... sigh :/ convert to
regular strings.
Change-Id: Ia03b97198f9f45914b8b99c222680d732c2cc8f8
wheel-copy.sh is the wrong place to do the index of the package
directory. Move it to wheel-index.sh where we are walking the final
wheel directories.
Change-Id: I58329650962b4dbe24f79a1f4e4ea54e709233c1
Iterating on this further; fix incorrect yum install and match for the
default Python3 install YAML.
Change-Id: Ic1f902297d869ddafaa57f5910d99461a8a004f3
Some things have become evident when generating the indexes requiring
some larger changes.
Firstly, the indexer script needs python3 on the host. Since we're
still building CentOS 7 wheels, we need to install Python 3 from EPEL
there.
Secondly, because part of the PEP503 index page is the file hash,
reading all the files back over AFS is quite slow. It's also quite
slow having ansible loop a task each time, which all adds up to job
timeouts.
Instead, make the indexes on the local disk before we copy the results
to AFS. This requires copying both scripts to the host for execution
(rather than relying on "script:" ) so the wheel-copy.sh script can
call wheel-indexer.py.
While we are there, a small refactor on the wheel-indexer.py to use
os.walk() (which makes it easier to have this as a stand-alone
recursive script later, if something changes). Also update the output
to use <ul><li> for the filenames, so it looks a little better on the
output html.
Change-Id: I85f9e132bc55fd8d33583a698e15c47665e5cf8d
From the ansible manual
"The difference between single quotes and double quotes is that in
double quotes you can use escapes:"
This turns out to be important when you're splitting based on \n ...
Change-Id: Ia033426bf05a63034fb11882e92d29e3ede8fc53
What I didn't notice is that on disk the projects are in a/ b/ c/ type
subdirectories, while from the mirror side we paper-over this with
mod_rewrite rules so the projects all look like they're at top-level.
This should make the script descend into each directory correctly to
generate the index.
Change-Id: I89d255e2bcd5b5c86a33dec1a77086c97bde4c7a
PEP503 [1] defines the simple respository API format that we should
follow for our mirrors publishing .whl files.
This loops the output directories and creates a index.html for each
project with the required tags.
We do not really have gate testing for this, so to avoid destroying
the whole mirror we write out a temporary index file first, and ignore
errors. When this works, we will revert to index.html and do a single
manual removal of the temporary indexes.
[1] https://www.python.org/dev/peps/pep-0503/
Change-Id: I0541fb58535f27589b7e3bc1462cb083e3d12f56
A recent 45.0.0 release of Setuptools dropped support for (the now
EOL) Python 2.7 interpreter. It publishes to PyPI with
options.python_requires set to >=3.5 in its setup.cfg which causes
PyPI's simple index to serve that via a data-requires-python
attribute (see PEP 503) so that pip knows not to pick that one on
unsupported interpreters. Our wheel cache is just served via Apache
with mod_autoindex for its file listing, so lacks an ability to
convey this information.
In the short term, stop copying new versions of the pip, setuptools
and virtialenv wheels into the cache we publish, so that we can
safely remove broken versions of them manually and not have to worry
about them reappearing. Longer term, it may make sense to filter out
any wheels with manylinux or none-any platform specifiers, since
those the first absolutely exist on PyPI already or and the second
are trivial to build at job runtime if they don't. We really only
care about making sure we serve platform-specific wheels we've
built, as those will save the most time in jobs.
Change-Id: I02fac6beef9bb94cdeb1d0a84774c1b93357deb1
We saw what looks very much like the index generation racing, with
"mv" unable to close the index file ...
Move to a linear strategy so that each host finishes each task before
moving on. Then add run_once to the index generation so only one of
the hosts in a python2/3 situation creates the index file.
Change-Id: I3761dd9d2ab752e66bcccdd2434405f11535b95f
We only need the three wheel-X scripts from jenkins/scripts and need
them only for the playbook and its roles. Move the scripts in the
roles directory and copy them only if needed - using the scripts module.
Remove copying of complete jenkins/script for this playbook using the
role legacy-copy-project-config-scripts, it's not needed anymore.
Change-Id: I2e481fcf7ca148aac8a36ae99d8598ba26078a25
Pass the wheel keytab across and aklog with it.
Also, split the build role into a build and a copy so that we only have
the token when we need it, and not while we're running the wheel builds
themselves. And split release from build so that we can release the
mirror once for a consistent state across python2 and python3.
Change-Id: Id30a99f5b515dd520d013fabbd356e32c82030a0
Jeremy Stanley