diff --git a/diskimage_builder/elements/pip-and-virtualenv/README.rst b/diskimage_builder/elements/pip-and-virtualenv/README.rst index db8f4acbd..5217f62ff 100644 --- a/diskimage_builder/elements/pip-and-virtualenv/README.rst +++ b/diskimage_builder/elements/pip-and-virtualenv/README.rst @@ -2,14 +2,72 @@ pip-and-virtualenv ================== -This element installs pip and virtualenv in the image. If the package -installtype is used then these programs are installed from distribution -packages. If the source installtype is used these programs are installed -from get-pip.py and pip (respectively). +This element installs pip and virtualenv in the image. -To install pip and virtualenv from package: +Package install +=============== + +If the package installtype is used then these programs are installed +from distribution packages. In this case, ``pip`` and ``virtualenv`` +will be installed *only* for the python version identified by +``dib-python`` (i.e. the default python for the platform). + +Distribution packages have worked out name-spacing such that only +python2 or python3 owns common scripts like ``/usr/bin/pip`` (on most +platforms, ``pip`` refers to python2 pip, and ``pip3`` refers to +python3 pip, although some may choose the reverse). + +To install pip and virtualenv from package:: export DIB_INSTALLTYPE_pip_and_virtualenv=package +Source install +============== + +Source install is the default. If the source installtype is used, +``pip`` and ``virtualenv`` are installed from the latest upstream +releases. + +Source installs from these tools are not name-spaced. It is +inconsistent across platforms if the first or last install gets to own +common scripts like ``/usr/bin/pip`` and ``virtualenv``. + +To avoid inconsistency, we firstly install the packaged python 2 +**and** 3 versions of ``pip`` and ``virtualenv``. This prevents a +later install of these distribution packages conflicting with the +source install. We then overwrite ``pip`` and ``virtualenv`` via +``get-pip.py`` and ``pip`` respectively. + +The system will be left in the following state: + +* ``/usr/bin/pip`` : python2 pip +* ``/usr/bin/pip2`` : python2 pip (same as prior) +* ``/usr/bin/pip3`` : python3 pip +* ``/usr/bin/virtualenv`` : python2 virtualenv + +(note python3 ``virtualenv`` script is *not* installed, see below) + +Using the tools +=============== + +Due to the essentially unsolvable problem of "who owns the script", it +is recommended to *not* call ``pip`` or ``virtualenv`` directly. You +can directly call them with the ``-m`` argument to the python +interpreter you wish to install with. + +For example, to create a python3 environment do:: + + # python3 -m virtualenv myenv + # myenv/bin/pip install mytool + +To install a python2 tool from pip:: + + # python2 -m pip install mytool + +In this way, you can always know which interpreter is being used (and +affected by) the call. + +Ordering +======== Any element that uses these commands must be designated as 05-* or higher to ensure that they are first installed. diff --git a/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip b/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip index cdeb442f7..932c8031a 100755 --- a/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip +++ b/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip @@ -7,6 +7,14 @@ set -eu set -o pipefail if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then + + _do_py3=0 + packages="python-virtualenv python-pip python-setuptools" + if [[ $DISTRO_NAME =~ (fedora) ]]; then + _do_py3=1 + packages+=" python3-virtualenv python3-pip python3-setuptools" + fi + # GENERAL WARNING : mixing packaged python libraries with # pip-installed versions always creates issues. Upstream # openstack-infra uses this a lot (especially devstack) but be @@ -19,11 +27,11 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then if [[ $DISTRO_NAME = opensuse ]]; then zypper -n install python-virtualenv python-pip python-setuptools else - ${YUM:-yum} install -y python-virtualenv python-pip python-setuptools + ${YUM:-yum} install -y $packages fi - # install the latest pip; this overwrites packaged pip - ${DIB_PYTHON} /tmp/get-pip.py + # install the latest python2 pip; this overwrites packaged pip + python /tmp/get-pip.py # pip and setuptools are closely related; we want to ensure the # latest for sanity. Because distro packages don't include enough @@ -38,10 +46,38 @@ if [[ $DISTRO_NAME =~ (opensuse|fedora|centos|centos7|rhel|rhel7) ]]; then rm -rf /usr/lib/python2.7/site-packages/setuptools* pip install -U setuptools + if [[ $_do_py3 -eq 1 ]]; then + # Repeat above for python3 + # You would think that installing python3 bits first, then + # python2 would work -- alas get-pip.py doesn't seem to leave + # python3 alone: + # https://github.com/pypa/pip/issues/4435 + python3 /tmp/get-pip.py + rm -rf /usr/lib/python3.?/site-packages/setuptools* + pip3 install -U setuptools + # reclaim /usr/bin/pip back to pip2 + ln -sf /usr/bin/pip2 /usr/bin/pip + fi + # now install latest virtualenv. it vendors stuff it needs so # doesn't have issues with other system packages. + + # python[2|3]-virtualenv package has installed versioned scripts + # (/usr/bin/virtualenv-[2|3]) but upstream does not! (see [2]). + # For consistency, clear them out and then reinstall so we're just + # left with python2's version + # [2] http://pkgs.fedoraproject.org/cgit/rpms/python-virtualenv.git/tree/python-virtualenv.spec#n116) + rm /usr/bin/virtualenv* + if [[ $_do_py3 -eq 1 ]]; then + pip3 install -U virtualenv + fi pip install -U virtualenv + # at this point, we should have the latest + # pip/setuptools/virtualenv packages for python2 & 3, and + # "/usr/bin/pip" and "/usr/bin/virtualenv" should be python2 + # versions. + if [[ $DISTRO_NAME = opensuse ]]; then for pkg in virtualenv pip setuptools; do cat - >> /etc/zypp/locks <> ${conf} + echo "exclude=$packages" >> ${conf} fi else - ${DIB_PYTHON} /tmp/get-pip.py + # pre-install packages, we will overwrite with latest below + apt-get -y install python-pip python3-pip \ + python-virtualenv python3-virtualenv + + # These install into /usr/local/bin so override any packages, even + # if installed later. + python3 /tmp/get-pip.py + python2 /tmp/get-pip.py + + pip3 install virtualenv pip install virtualenv fi