The tarball builds were using the wheelhouse paths
for find_links, that should have been using the
pip download cache path.
The pre-build upgrade now upgrades distribute, pbr,
and setuptools to handle the ;environment_factor
style requirements.
Change-Id: I68d3e090779d9b73613e3aab7fe9c7ee9513e45d
There's a check to make sure that we only run
mirroring against interesting branches by filtering
out any that aren't HEAD or don't start with
remotes/origin, but if we specify a branch on
the command line, we want that branch no matter what.
This is primarily so we can run mirroring tests
on proposed changes to openstack/requirements.
Change-Id: I19cd2b032b6219aed972abb95a180e3d402d529e
Wheels get built for the OS that pypi-mirror is running
on, but the non arch specific wheels weren't going into
the main tarball mirror, which they should be able to.
This change puts the -any arch wheels into the main tarball
mirror. The method rename (_write_tarball_mirror to
_write_main_mirror) is to reflect that the mirror doesn't
just contain tarballs.
It also fixes an issue with the wheelhouse location.
Previously, the same wheelhouse location was used for
all mirror targets, but this could have easily led to
unwanted versions leaking.
Change-Id: I99223bd54c4512b142753d2b97f99275db82361f
pip 1.5.1 removed options that pypi-mirror needed for
reliable downloading, and pip 1.5.0 has problems with
some proxies, so this change updates the tmp virtualenvs
to use pip>6, uses the download only option, and
parses the package names from the tarball filenames.
Also, don't use the cached downloads as an extra search
path when preparing the build virtualenvs, as older
distributions will pick up newer versions of packages
that the system can't yet install, until the pip
and virtualenv has been upgraded inside the build
virtualenv.
Change-Id: I645938aa790ba5293a8c78d3b410c3fe339f383c
The new release of pip does not allow you to specify
a dowload cache location - it is deprecated. This means
that the tarball mirror cannot be created because it
relies on the download cache being in the configured
location. This change pins the versin of pip to 1.5.6
so the download cache location still works.
Change-Id: Ib4d058f27de3c37e0c58c50516e6af2b3f8188a5
Adding a '-r' parameter which allows for specifying one or more
requirements files instead of the built-in 'global-requirements.txt'
file.
This is to enable building an internal pypi mirror using additional
internal requirements files.
Change-Id: I506a9beddf773d6290639ed4d01588339ad3f99c
* pypi_mirror/cmd/run_mirror.py: Stop whitelisting netifaces,
psutil, pysendfile and pytidylib for insecure/external hosting
off-PyPI. They all now host versions on PyPI which meet the listed
requirements in openstack/requirements:global-requirements.txt for
currently supported branches.
Change-Id: I640a85435de7bb300aaaa347cb97b1fad7543e17
When run-mirror downloads a package whose download files have a
different base name to the download directory, run-mirror ends up
writing the wrong directory name, which causes 404's on the mirror.
In this patch we use the actual upstream parent dir name exactly.
Change-Id: I90ca0e280dc59d26e5cfc70c8c02feeb734d1ae9
Closes-Bug: #1294522
Virtualenv has recently been upgraded to 1.11, which comes with pip 1.5. In
this version of pip, if 'pip install' can find a wheel, it downloads it instead
of the tar.gz. This is an issue since the version of pip used in the gates can
still be too old to work with wheels by default.
Use '--no-use-wheel' in the 'pip install' command to fix this.
Closes-Bug: #1276242
Change-Id: I00a32b3abd3b08703f66834ae30e6c4b0cf7bd7d
* pypi_mirror/cmd/run_mirror.py: Use the --allow-external and
--allow-insecure options so that pip 1.5 and later will assent to
retrieve the netifaces, psutil, pysendfile and pytidylib packages
even though they're not hosted on PyPI. The --allow-insecure option
is aliased to a clearer --allow-unverified wording in 1.5, but the
old form is being used to avoid breaking users of 1.4.x and will be
valid at least through 1.6.x according to comments in the pip
source.
Change-Id: I43d276fd6d294f74fae6d7ef36ac38d27a53f14e
OpenStack LLC isn't a thing, all of its stuff was transferred to
the OpenStack Foundation. Also, this is not an executable script,
so there is no need to have the shebang line.
Change-Id: I1b31a275e14ebc11a61b02788ffe5c0945f63bf0
Add an option to export the list of packages that got installed
to the mirror so that we can use the list to compare
the difference between the requirements (packages to be installed)
and the mirror (what actually got installed)
Change-Id: Ifda23d78ea4f324a145fc0279f4c80dc25570f73
Related-Bug: #1172417
On Centos, the lsb_release returns the text on one line. To solve that,
we need to replace not just newlines, but also spaces in the output
to match what lsb_release -i -r -s | xargs | tr ' ' '-' will do.
Change-Id: I5e4ecdb918d1f9e3c225974f89d7ceb4f1abc4f1
Clients of the mirror may need to fall back on non-wheel files,
and if an upstream project has published wheels, then the mirror
will be missing those in the non-wheel portion. Additionally,
use-wheel given to the wheel command will cause transitive depends
to be skipped in the local wheelhouse.
The only thing that should be making/injecting wheels is the wheel
creation portion of things.
Change-Id: I5e3c4754097249e52f013f6122ff86f01568243c
It turns out that upgrading setuptools and distribute in the venv
we use to build the mirror causes MySQL-python to error out. However,
as of virtualenv 1.10.1, there's a new enough version of pip (1.4)
so it's actually not necessary for us to explicitly upgrade.
Change-Id: Ib08064ad1ce674323cd5dec87b7a9911c723d12a
mysql-python is crapping the fantastic bong because of a continued
reliance on distribute. Fine. We'll just make sure we have latest
no-op package in our virtualenv so that epic fail stops killing bunnies.
Also, when we're running the wheel command in our venvs, we kinda need
wheel to be installed in them.
Change-Id: I004ca4efdbe5997b4f5f56e71a330ed7354d5cd2
There are several places in the code where out is collected but not
used. These make the code harder to reason about. Remove them and
only actually get the output if we care.
Change-Id: I0280df56538fc7a56ee2d2bb47f05908270a3045
Andreas Jaeger