Commit Graph

204 Commits

Author SHA1 Message Date
OpenDev Sysadmins 1389cea7cb OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:25:38 +00:00
Andreas Jaeger 4654d26a89 Retire pypi-mirror
This repository is unused, retire it.

Depends-On: https://review.openstack.org/597370/
Change-Id: Id38773cf2dadae6a5907b7972648393cd3a425ef
2018-08-29 09:03:02 +02:00
Nikki Heald 392a70d3b5 Update build tools and fix link paths
The tarball builds were using the wheelhouse paths
for find_links, that should have been using the
pip download cache path.

The pre-build upgrade now upgrades distribute, pbr,
and setuptools to handle the ;environment_factor
style requirements.

Change-Id: I68d3e090779d9b73613e3aab7fe9c7ee9513e45d
2015-07-15 09:27:21 +01:00
Jeremy Stanley 67e44675f6 Replace ci.o.o links with docs.o.o/infra
The http://ci.openstack.org/ documentation site has been deprecated,
replaced by redirects to corresponding paths within
http://docs.openstack.org/infra/ where other Project Infrastructure
documentation already resides.

Change-Id: I30e4029bf72f0bcafcb41e8c6a748ea7efb89d52
2015-05-14 21:38:19 +00:00
Nikki Heald 989780da2d If we specify a branch, always use it
There's a check to make sure that we only run
mirroring against interesting branches by filtering
out any that aren't HEAD or don't start with
remotes/origin, but if we specify a branch on
the command line, we want that branch no matter what.
This is primarily so we can run mirroring tests
on proposed changes to openstack/requirements.

Change-Id: I19cd2b032b6219aed972abb95a180e3d402d529e
2015-05-13 11:15:06 +01:00
Jenkins 97304ffe34 Merge "Workflow documentation is now in infra-manual" 2015-04-20 06:09:44 +00:00
Jeremy Stanley 691ed74ede Workflow documentation is now in infra-manual
Replace URLs for workflow documentation to appropriate parts of the
OpenStack Project Infrastructure Manual. Pulled in the final form used
in the gear repo.

Change-Id: Ib4d23d0db1a91d51176105ee38a30073a789374a
2015-04-17 11:57:18 -04:00
Mike Heald 4591e2abbc Mirror appropriate wheels in main mirror
Wheels get built for the OS that pypi-mirror is running
on, but the non arch specific wheels weren't going into
the main tarball mirror, which they should be able to.

This change puts the -any arch wheels into the main tarball
mirror. The method rename (_write_tarball_mirror to
_write_main_mirror) is to reflect that the mirror doesn't
just contain tarballs.

It also fixes an issue with the wheelhouse location.
Previously, the same wheelhouse location was used for
all mirror targets, but this could have easily led to
unwanted versions leaking.

Change-Id: I99223bd54c4512b142753d2b97f99275db82361f
2015-02-26 14:38:31 +00:00
Mike Heald 9aa8af0dcd Use pip 6.x for building the mirror
pip 1.5.1 removed options that pypi-mirror needed for
reliable downloading, and pip 1.5.0 has problems with
some proxies, so this change updates the tmp virtualenvs
to use pip>6, uses the download only option, and
parses the package names from the tarball filenames.

Also, don't use the cached downloads as an extra search
path when preparing the build virtualenvs, as older
distributions will pick up newer versions of packages
that the system can't yet install, until the pip
and virtualenv has been upgraded inside the build
virtualenv.

Change-Id: I645938aa790ba5293a8c78d3b410c3fe339f383c
2015-01-19 13:49:52 +00:00
Mike Heald ea821ee683 Pin pip so download cache location works
The new release of pip does not allow you to specify
a dowload cache location - it is deprecated. This means
that the tarball mirror cannot be created because it
relies on the download cache being in the configured
location. This change pins the versin of pip to 1.5.6
so the download cache location still works.

Change-Id: Ib4d058f27de3c37e0c58c50516e6af2b3f8188a5
2015-01-14 12:44:07 +00:00
Jenkins 39cae68129 Merge "Bump hacking to 0.9.x series" 2014-12-05 04:02:28 +00:00
Jenkins 707b83e6bf Merge "Replace md5 module with hashlib module" 2014-12-05 04:02:22 +00:00
Michael Krotscheck 91b9977f61 Switched documentation to point to storyboard
We've migrated this project to storyboard, I'm updating the documentation links.

Change-Id: I9ae2908f9bc993ee7238166f8f39d3821b7b1157
2014-11-12 16:40:55 +00:00
Christian Berendt 2b9fde041f Bump hacking to 0.9.x series
Change-Id: I6dfd8238c9567879f522a6f908c339ac16411e99
2014-07-22 14:35:41 +02:00
Christian Berendt dfc91b7e85 Replace md5 module with hashlib module
* H237  module md5 is removed in Python 3

Change-Id: I56ea59e0dedd99febc056095ab3e3bf15983f92b
2014-07-22 14:32:50 +02:00
Gregory Haynes fb16c5dc68 Force --no-use-mirror installaions
Some packages require this in order to get an sdist installation.

Change-Id: I8e6fca71f13835b5019366c34535c64c3c58c6ea
Closes-Bug: #1294381
2014-07-09 11:25:15 -07:00
Manuel Desbonnet 1e588a8d40 Allow for specifying requirements files on the cmd-line
Adding a '-r' parameter which allows for specifying one or more
requirements files instead of the built-in 'global-requirements.txt'
file.

This is to enable building an internal pypi mirror using additional
internal requirements files.

Change-Id: I506a9beddf773d6290639ed4d01588339ad3f99c
2014-06-20 11:48:01 +01:00
Jenkins 543d261e08 Merge "No longer whitelist externally-hosted requirements" 2014-05-28 22:05:47 +00:00
Jeremy Stanley 59a4a18150 No longer whitelist externally-hosted requirements
* pypi_mirror/cmd/run_mirror.py: Stop whitelisting netifaces,
psutil, pysendfile and pytidylib for insecure/external hosting
off-PyPI. They all now host versions on PyPI which meet the listed
requirements in openstack/requirements:global-requirements.txt for
currently supported branches.

Change-Id: I640a85435de7bb300aaaa347cb97b1fad7543e17
2014-05-20 19:03:01 +00:00
James Polley 4ea694cdc9 Add a pointer to the DIB pypi element's README
The README is very generic; every time I follow it, I end up with a repo
that doesn't work. The pip element's readme is much more useful for
setting up a mirror that works with DIB and TripleO.

Rather than replicate content from the element here, I've just added a
pointer to the element's README.

Change-Id: Id6d0f8a75d3fc91a72c4dbb9b9628d4c80dbfb9d
2014-04-11 01:47:26 +02:00
Yuriy Taraday 57992fc80f Refactor calls to Git
All pip/virtialenv calls can be refactored in the very same way.

Change-Id: I948546d12ccca0298af012c2d3dac39481049c88
2014-04-07 10:26:07 +04:00
Yuriy Taraday 4a87b5b172 Refactor if: nesting in build_mirror
Change-Id: Ia0fe43e1f44704d5d93235ceadfcf74d00aa190b
2014-04-07 10:26:07 +04:00
Jenkins 75ed21d8a4 Merge "Use openstack git urls instead of github" 2014-03-29 20:29:28 +00:00
Jenkins a4ade4b168 Merge "Add blurb on configuration and run_mirror in README" 2014-03-25 20:03:20 +00:00
Gregory Haynes aa8cbc63b0 Use openstack git urls instead of github
In the run_mirror comments we should be using the openstack git urls
rather than github's.

Change-Id: Idd22f058b5e9c00cf0a9ff388daccc753d4867e9
2014-03-24 22:46:36 -07:00
Gregory Haynes 6e1f008c41 Add blurb on configuration and run_mirror in README
Currently the configuration syntax is only documented in the run_mirror
source. Adding enough information to the RAEDME for users to get up and
running with a simple configuration is much more user friendly.

Change-Id: I54d978ac127dfe37348167f3a2b08cc09119619b
2014-03-24 22:43:32 -07:00
Clint Byrum 63dfccb6cf Fix run-mirror writing wrong project directory
When run-mirror downloads a package whose download files have a
different base name to the download directory, run-mirror ends up
writing the wrong directory name, which causes 404's on the mirror.

In this patch we use the actual upstream parent dir name exactly.

Change-Id: I90ca0e280dc59d26e5cfc70c8c02feeb734d1ae9
Closes-Bug: #1294522
2014-03-19 08:50:08 -07:00
Cyril Roelandt 0de0e10e75 Do not download wheels when running "pip install"
Virtualenv has recently been upgraded to 1.11, which comes with pip 1.5. In
this version of pip, if 'pip install' can find a wheel, it downloads it instead
of the tar.gz. This is an issue since the version of pip used in the gates can
still be too old to work with wheels by default.

Use '--no-use-wheel' in the 'pip install' command to fix this.

Closes-Bug: #1276242
Change-Id: I00a32b3abd3b08703f66834ae30e6c4b0cf7bd7d
2014-02-19 04:24:16 +01:00
Jeremy Stanley d24f19bcde Whitelist external packages
* pypi_mirror/cmd/run_mirror.py: Use the --allow-external and
--allow-insecure options so that pip 1.5 and later will assent to
retrieve the netifaces, psutil, pysendfile and pytidylib packages
even though they're not hosted on PyPI. The --allow-insecure option
is aliased to a clearer --allow-unverified wording in 1.5, but the
old form is being used to avoid breaking users of 1.4.x and will be
valid at least through 1.6.x according to comments in the pip
source.

Change-Id: I43d276fd6d294f74fae6d7ef36ac38d27a53f14e
2014-01-15 03:13:24 +00:00
Jenkins 236a6370c0 Merge "Clean up the copyright header" 2013-12-28 01:15:23 +00:00
Monty Taylor 316c6f5256 Clean up the copyright header
OpenStack LLC isn't a thing, all of its stuff was transferred to
the OpenStack Foundation. Also, this is not an executable script,
so there is no need to have the shebang line.

Change-Id: I1b31a275e14ebc11a61b02788ffe5c0945f63bf0
2013-12-10 03:48:00 +01:00
Jenkins 7a41e33157 Merge "add an export option" 2013-11-26 20:41:27 +00:00
Jenkins e638f2e516 Merge "Print help message and exit if no config file by default" 2013-11-26 11:51:29 +00:00
Khai Do 8eaf6f00e0 add an export option
Add an option to export the list of packages that got installed
to the mirror so that we can use the list to compare
the difference between the requirements (packages to be installed)
and the mirror (what actually got installed)

Change-Id: Ifda23d78ea4f324a145fc0279f4c80dc25570f73
Related-Bug: #1172417
2013-11-20 04:17:25 +00:00
Monty Taylor 46aca65aee Replace spaces with -'s
On Centos, the lsb_release returns the text on one line. To solve that,
we need to replace not just newlines, but also spaces in the output
to match what lsb_release -i -r -s | xargs | tr ' ' '-' will do.

Change-Id: I5e4ecdb918d1f9e3c225974f89d7ceb4f1abc4f1
2013-11-16 04:17:19 -05:00
Monty Taylor 7183290008 Don't use wheel when constructing the base mirror
Clients of the mirror may need to fall back on non-wheel files,
and if an upstream project has published wheels, then the mirror
will be missing those in the non-wheel portion. Additionally,
use-wheel given to the wheel command will cause transitive depends
to be skipped in the local wheelhouse.

The only thing that should be making/injecting wheels is the wheel
creation portion of things.

Change-Id: I5e3c4754097249e52f013f6122ff86f01568243c
2013-11-16 04:12:27 -05:00
Monty Taylor d68260a79a Don't upgrade setuptools or distribute
It turns out that upgrading setuptools and distribute in the venv
we use to build the mirror causes MySQL-python to error out. However,
as of virtualenv 1.10.1, there's a new enough version of pip (1.4)
so it's actually not necessary for us to explicitly upgrade.

Change-Id: Ib08064ad1ce674323cd5dec87b7a9911c723d12a
2013-11-14 13:16:38 -05:00
Monty Taylor 5dda8efa02 Add distribute to the pre-install list
mysql-python is crapping the fantastic bong because of a continued
reliance on distribute. Fine. We'll just make sure we have latest
no-op package in our virtualenv so that epic fail stops killing bunnies.

Also, when we're running the wheel command in our venvs, we kinda need
wheel to be installed in them.

Change-Id: I004ca4efdbe5997b4f5f56e71a330ed7354d5cd2
2013-11-13 14:32:28 -05:00
Qiu Yu 4ec963a675 Print help message and exit if no config file by default
Fixes bug #1204350

Change-Id: I9442bb26d3ef6374bb44e8e027b37ae37bd9cbe0
2013-10-23 19:41:38 +08:00
Jeremy Stanley 7dfc0142d7 Install some python with our package
* setup.cfg: The files section was missing, causing the sdist
package to lack the Python module being packaged. Fix it!

Change-Id: I0726708caf949f72e4fc244d746583e826a9bf9d
2013-10-16 19:13:29 +00:00
Monty Taylor 8774e55571 Remove spurious uses of output return values
There are several places in the code where out is collected but not
used. These make the code harder to reason about. Remove them and
only actually get the output if we care.

Change-Id: I0280df56538fc7a56ee2d2bb47f05908270a3045
2013-09-12 15:15:11 -05:00
Monty Taylor c28dde9d68 Strip out the things we don't need from jeepyb
Change-Id: I129b6d66840604898592de31df11b3e396cd6048
2013-09-09 13:43:53 -05:00
Monty Taylor 384c02d41f Start building wheels in the mirror
The mirror should build wheels, because they're awesome. However, because
the wheel format does not do fine-grained enough selection on
things like the high likelihood that Ubuntu precise and Ubuntu raring
might have different underlying C libraries for python 2.7, we need to
put our built wheels into a structure that allows them to be used
safely.

lsb_release -r -i is used because Centos lists lsb_release -c as "Final"
which is not particularly helpful. This will wind up with identifying
strings like Ubuntu-12.04 and Centos-6.4 - which should be good enough
for what we need. A call to a shell is used so that the same logic can
be easily duplicated in select-mirror, which is shell, to produce the
appropriate pip.conf file.

Change-Id: Icf360c921dbc357dbeb43386c6e8f51bd5f993ff
2013-09-09 13:43:47 -05:00
Monty Taylor 92a1f3a57c Rename jeepyb to pypi-mirror
Change-Id: I3f5176f59fd57c7d85211a89edd121092cfef1a9
2013-09-09 13:43:36 -05:00
Elizabeth Krumbach 95398b631c Add chown of git repo directories
When creating git repositories, we want to recursively chown
them to the cgit user so the cgit user for replication can
write to them.

Change-Id: Ib349b4af32090d0fbda933d51ef514cae7033499
2013-08-13 14:05:49 -07:00
Jenkins ce650a4fe2 Merge "Create repo when we create the cgit config file" 2013-08-13 18:04:51 +00:00
Monty Taylor eb3ab9ba30 Create repo when we create the cgit config file
Git repos need to be created locally. Ensure that we create the
necessary repos when creating the config file for them.

Change-Id: I0b0b8e183049dd9f825224f1657cb88a64edf37b
2013-08-13 12:07:48 -03:00
Emilien Macchi 3d4817c25f Update Launchpad projects for puppet modules
Since we've decided to split puppet-openstack launchpad page into
several sub-projects, this patch update the configuration to
support both blueprints & bugs for all current puppet modules
in StackForge.

Change-Id: Iaef1de681507e9de9d80d6fecd2e5e56ca960a1a
2013-08-09 18:58:18 +02:00
Jenkins 6a63c1b91a Merge "Allow hacking 0.6.0 and fix versions" 2013-08-05 22:51:25 +00:00
Monty Taylor ad88b02851 Don't double-include if global-requirements
global-requirements is a fully self-contained file, containing both
test and normal requirements. In the situation where the target repo
has a global-requirements.txt file, it should be treated as the only
file that matters- otherwise the mirror builder will hit double include
issues.

Change-Id: I01fd1e0d7f0923456f35a0f931159850cb52e3e7
2013-08-03 17:02:06 -04:00