In the _setup_package_with_constraints_edit name of the package was
always discovered from the setup.cfg file. But as some projects
implements PEP-621 (see [1] for the SQLAlchemy for example) it is not
enough now.
This patch adds parsing pyproject.toml file also if name is not found in
the setup.cfg file.
[1] a8dbf8763a
Closes-Bug: #2052509
Change-Id: Iee9262079d09a8bd22cd05a8f17950a41a0d1f9d
Since we are python3 only for openstack we create a single python3
virtualenv to install all the packages into. This gives us the benefits
of installing into a virtualenv while still ensuring coinstallability.
This is a major change and will likely break many things.
There are several reasons for this. The change that started this effort
was pip stopped uninstalling packages which used distutils to generate
their package installation. Many distro packages do this which meant
that pip installed packages and distro packages could not coexist in the
global install space. More recently git has made pip installing repos as
root more difficult due to file ownership concerns.
Currently the switch to the global venv is optional, but if we go down
this path we should very quickly remove the old global installation
method as it has only caused us problems.
Major hurdles we have to get over are convincing rootwrap to trust
binaries in the virtualenvs (so you'll notice we update rootwrap
configs).
Some distros still have issues, keep them using the old setup for now.
Depends-On: https://review.opendev.org/c/openstack/grenade/+/880266
Co-Authored-By: Dr. Jens Harbott <frickler@offenerstapel.de>
Change-Id: If9bc7ba45522189d03f19b86cb681bb150ee2f25
We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc
The previous change, I237f5663b0f8b060f6df130de04e17e2b1695f8a, removed
a SETUPTOOLS flag, but not the comment explaining why that flag was
previously set. Clean up that comment.
Change-Id: I32b0240fd56310d7f10596aaa8ef432679bfd66a
There are two problems with dbcounter installation on Jammy. The first
is straightforward. We have to use `py_modules` instead of `modules` to
specify the source file. I don't know how this works on other distros
but the docs [0] seem to clearly indicate py_modules does this.
The second issue is quite an issue and requires story time. When
pip/setuptools insteall editable installs (as is done for many of the
openstack projects) it creates an easy-install.pth file that tells the
python interpreter to add the source dirs of those repos to the python
path. Normally these paths are appended to your sys.path. Pip's isolated
build env relies on the assumption that these paths are appeneded to the
path when it santizes sys.path to create the isolated environemnt.
However, when SETUPTOOLS_SYS_PATH_TECHNIQUE is set to rewrite the paths
are not appended and are inserted in the middle. This breaks pip's
isolated build env which broke dbcounter installations. We fix this by
not setting SETUPTOOLS_SYS_PATH_TECHNIQUE to rewrite. Upstream indicates
the reason we set this half a decade ago has since been fixed properly.
The reason Jammy and nothing else breaks is that python3.10 is the first
python version to use pip's isolated build envs by default.
I've locally fiddled with a patch to pip [1] to try and fix this
behavior even when rewrite is set. I don't plan to push this upstream
but it helps to illustrate where the problem lies. If someone else would
like to upstream this feel free.
Finally this change makes the jammy platform job voting again and adds
it to the gate to ensure we don't regress again.
[0] https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
[1] https://paste.opendev.org/show/bqVAuhgMtVtfYupZK5J6/
Change-Id: I237f5663b0f8b060f6df130de04e17e2b1695f8a
This reverts commit 7a3a7ce876 and
bcd0acf6c0 and part of
f1ed7c77c5 which all cap our pip
installs.
Given the pip ecosystem can often incorporate major changes, tracking
upstream at least generally gives us one problem at a time to solve
rather than trying to handle version jumps when LTS distros update.
The new dependency resolver included some changes that disallow
setting URL's like "file:///path/to/project#egg=project" in
constraints. Apparently the fact it used to work was an accident of
the requires/constraints mechanism; it does make some sense as the URL
doesn't really have a version-number that the resolver can put in an
ordering graph.
The _setup_package_with_constraints_edit function comment highlights
what this is trying to do
# Updates the constraints from REQUIREMENTS_DIR to reflect the
# future installed state of this package. This ensures when we
# install this package we get the from source version.
In other words; if constraints has "foo==1.2.3" and Zuul has checked
out "foo" for testing, we have to make sure pip doesn't choose version
1.2.3 from pypi.
It seems like removing the entry from upper-requirements.txt is the
important part; adding the URL path to the on-disk version was just
something that seemed to work at the time, but isn't really necessary.
We will install the package in question which will be the latest
version (from Zuul checkout) and without the package in
upper-requirements.txt nothing will try and downgrade it.
Therefore the solution proposed here is to remove the adding of the
URL parts.
This allows us to uncap pip and restore testing with the new
dependency resolver.
Closes-Bug: #1906322
Change-Id: Ib9ba52147199a9d6d0293182d5db50c4a567d677
A new setuptools release has changed the way pip installs are done,
see [0]. With this change we switch back to using the distro
method for global pip installs.
Temporarily make grenade jobs non-voting in order to allow this
patch to be backported.
[0] http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016905.html
Change-Id: I5d8aa0e58e0409c54451b51de5eb70ba9a68d849
Recent regression spotted by Dmitry Tantsur.
DevStack dropped Py2 support but the now-unused-in-devstack
python3_enable got its result nastily inversed.
Change-Id: I4b37cc847a24705c4955cec2e6e45f0514705f1b
Focal only provides a pip3 binary, no pip3.8. Instead of working around
that with a symlink, use the module instead.
Add version information output for this variant.
Change-Id: I7af194ecc40e4d43c10ce067a661bb6ab4ca37d4
Recent change to devstack dropped installing test-requirements [1]
However, this caused gate failures due to lack of glance-store
deps for cinder and swift support.
This patch makes devstack install relevant extras depending on
enabled features.
Additionally, relevant functions are added/fixed to make this
possible.
glance-store = glance_store (for gerrit search match)
[1] https://review.opendev.org/715469
Change-Id: I0bf5792a6058b52936115b515ea8360f6264a7c9
This is a test of installing openstack and then seeing if it works.
OpenStack components do not need test-requirements to operate,
that's why they are test-requirements.
Additionally, as we look forward to depsolver pip, this is going
to screw us because we don't apply constraints to linters, which
are expressed in - you guessed it, test-requirements.
Change-Id: I8f24b839bf42e2fb9803dc7df3a30ae20cf264eb
This reverts commit 79b8e79488.
This is breaking things in various jobs, most notably because
we do not put constraints on linters - but we install
test-requirements which then can conflict with each other.
Change-Id: Ibc5603c61b38ce44db58fb27a27352f59123ad09
The new pip depsolver is coming this summer. Until it's ready,
run pip check at the end of devstack to make sure we're not
somehow installing conflicting package versions. We shouldn't
be, because of constraints, but if we are, better to know and
start figuring it out.
Change-Id: Id98f0848ff5a252d93e5f8029df2d069924d603f
After Python 2 is getting unsupported, new distros
like CentOS 8 and RHEL8 have stopped providing 'python'
package forcing user to decide which alternative to
use by installing 'python2' or 'python3.x' package
and then setting python alternative.
This change is intended to make using python3 command as
much as possible and use it as default 'python' alternative
where needed.
The final goals motivating this change are:
- stop using python2 as much as possible
- help adding support for CentOS 8 and RHEL8
Change-Id: I1e90db987c0bfa6206c211e066be03ea8738ad3f
Set default python3 version as 3 for cases python3
is not installed before running DevStack.
Implements installation of required python3x package
for RedHat family distros with package name depending on
configurable ${PYTHON3_VERSION}. Examples:
3 => python3 python3-devel (default one)
3.6 => python36 python36-devel
3.7 => python37 python37-devel
This should help in situations where there are more
than one python available for given platform and
DevStack is asked to pick one by its full 3.x version
Change-Id: I49d86bc9193165e0a41e8e8720be37d81a4e7ee0
[1] stopped installing pip for py2 when py3 is being used.
This patch makes sure we check only for py3 pip then.
Also removed some no-longer-relevant comment and
made uninstall behave the same.
Check for pip>=6 removed too.
See also [2].
[1] 279a7589b0
[2] http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012182.html
Change-Id: I36ee53e57e468d760b80a7e621b90899867a8efd
Some distros do not install python3/python3-devel with the minimal
install. F29 doesn't install -devel, and neither Centos 7 or 8
install either. This patch ensures that these packages get installed.
Ideally, PYTHON3_VERSION would be set *after* ensuring that python3
was installed, but it gets a little tricky with all of the includes.
This sets it to 3.6 as nothing uses 3.5 anymore.
Change-Id: I7bdfc408b7c18273639ec26eade475856ac43593
Everything in OpenStack *must* be Python 3 supporting now, which means
it's time to remove the functionality that allows us to blacklist
packages that didn't support Python 3.
Change-Id: I7c8cf538ec88bd4056b0109f19671e3d65f5da3a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This was added in 2012 with I89677fd54635e82b10ab674ddeb9ffb3f1a755f0,
but I can not see it being used anywhere currently.
It's use of virtualenv's has become problematic in a python2
deprecated world, but since it is not used, remove it to avoid further
confusion.
Change-Id: I65d44d24f449436ca6229928eee2c5a021793055
All the OpenStack projects should be able to run under Python 3 now so
the fallback installation of the Python 2 libraries should not be
needed any longer. This also avoids the problem of script files
installed by the libraries sometimes being overwritten by the Python 2
version leading to incorrect execution later, as discussed in
http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009226.html
This reverts commit a2eb89417f.
Change-Id: I1cdb7e4a209872f1620be556b7278879a4b86df5
This updates links going to git.openstack.org and review.openstack.org
to go to their respective opendev locations to avoid redirects.
Change-Id: I78e3bb5303718962f591117f9c0ee11f2314b128
Closes-Bug: #1833256
We know empirically that some legacy gate jobs pass and appear
to be running with python3 but actually pip was invoked with
PYTHON3_VERSION unset so that they are actually ran with python2
packages.
As a followup to this discussion [1], add a safety check in the
get_pip_command function to ensure that a python version has been
set when it is invoked.
[1] https://review.openstack.org/#/c/622415/4/inc/python@283
Change-Id: I3a08406fb7d68282c6b98abb33a625821510046a
If we are running with python3, just assume that any
package that is not blacklisted is available for py3
and just attempt to install it and let pip sort it out
whether it gets installed from a local or remote package.
Change-Id: Ic05d183e489320f6dfc721575d47e7e4d661f87c
Closes-Bug: #1820892
I7d16194d6ba1391ca31251d5b50cbb8de033fc38 added wrong behavour
on Fedora > 26 and Centos 7 when python3 disabled
pip should install packages in /usr/bin
Closes-Bug: #1820070
Change-Id: I3a8efbc8eb6e311db9c7347577c5d2047ba523a9
This makes the grep match in check_python3_support_for_package_local
the same as check_python3_support_for_package_remote.
Change I0349de2026c49279ba7f262d5e86d37018d66326 in grenade started
setting the PYTHON3_VERSION variable, and then we recently started
using bionic nodes everywhere which means we're running python 3.6.
The etcd3gw package has a python 3 and 3.5 classifier, but not 3.6:
https://pypi.org/project/etcd3gw/
The pip_install function code that is dealing with installing py3
packages is hitting a problem installing etcd3gw if the package is
local because of the more restrictive grep in the
check_python3_support_for_package_local function, and since
PYTHON3_VERSION=3.6 now, we don't install from py3 and install
etcd3gw on python 2.7 which makes services like cinder-volume and
cinder-backup, which use etcd3gw, fail when they are running under
python 3 (they get module import errors).
This simply removes the $ restriction on the grep. Looking at the
change that added those local/remote functions:
I243ea4b76f0d5ef57a03b5b0798a05468ee6de9b
There is no explanation for the difference, it just said:
Also, since not many packages are classified correctly, fallback
to looking for just "Programming Language :: Python :: 3" and
log a message for the package to highlight the problem.
So that's what this change does.
Note that alternatives would be:
1. Update the etcd3gw package to add the 3.6 classifier and do
a release (this should probably happen anyway).
2. Add etcd3gw to ENABLED_PYTHON3_PACKAGES but that would be a
short-term hack workaround.
Change-Id: Icd3768870ba0f1659bb2e6f002043d975047b73e
Closes-Bug: #1820892
when installing with python 3.6 on centos7 pip installs
packages to /usr/local/bin as it does on new versions
of fedora.
this change updates the check to include centos
Change-Id: I7d16194d6ba1391ca31251d5b50cbb8de033fc38
This adds a -bindep option to the key development library install
functions. With this option the bindep.txt file will be referenced
and the relevant packages installed.
Change-Id: I856f1f59fca49b6020920d8f859b797f3b904300
Remove the requirement that services explicitly enable python3 support
in order to be tested under python3 when running with python3
enabled. Keep the enable_python3_package() function for backwards
compatibility, for now, since it is called in some devstack plugins.
Explicitly add swift to the set of packages that should not be installed
using python3 by default until full support is available.
Change-Id: I8ab0a7c242bbf5bf3f091f5a85a98e2f4543f856
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Old-style test fails on Ubuntu when python3 enabled, with:
.../devstack/inc/python: line 52: [: 16.04: integer expression expected
Use bash-style test, which doesn't attempt to evaluate the RHS if the
LHS evaluates to false
Change-Id: If18031ab98c9060e5825c3a8d3c647bd3705cd9c
Closes-Bug: #1796174
If a project shows up in zuul's required-projects list, add it
to LIBS_FROM_GIT automatically. This way, when a user specifies
that a job requires a zuul-project, it gets used in testing, but
otherwise, it doesn't (pypi is used instead).
Also add information about what happens behind the scenes for both
LIBS_FROM_GIT and plugin dependencies.
This moves the check performed in check_libs_from_git to
a helper function which is installed for most kinds of
installations. This means that if someone sets LIBS_FROM_GIT to
"foobar", devstack won't error anymore, as nothing is going to
try to install foobar, therefore the check won't run on that.
However, as we move to automated generation of the local config,
that error is not likely to happen. This check was originally
added due to an error in the upper-constraints file (where a
constraint name did not match a package name). This location of
the check would still catch that type of error.
Change-Id: Ifcf3ad008cf42d3d4762cfb3b6c31c93cfeb40db
python-openstacksdk does not match its pip name which is openstacksdk.
So setting python-openstacksdk in LIBS_FROM_GIT leads to devstack
thinking there is a problem.
Put in a workaround for now. It would be better to either:
a) rename python-openstacksdk repo to openstacksdk
b) rename the pip name for openstacksdk back to python-openstacksdk
c) add general support in the various GIT hashes for a pip name
Change-Id: I57cf95763d54ad2060a4ce2af91c3ba18ca04db0
The 'pip list' command prints the "safe name" which converts _'s to
-'s amongst other things; e.g. glance_store becomes
"glance-store 0.21.1.dev22 /opt/stack/glance_store"
Because people may use these more familiar "file system" names in
LIBS_FROM_GIT automatically convert names when checking if libraries
are installed.
Change-Id: I30524f80a341f38dfa794a8f629d859e85a4a448
In commit f0cd9a8b08 we changed to
use column format, but it checks for zero length string and
check_libs_from_git fails.
Change-Id: I97b52b80efb33749647229a55147a08afa112dd2
The old code was strip()ing the version string instead of split()ing the
version string so we always got the first character of the version
string. This worked fine as long as the pip version was single digit but
as soon as it rolls over to '10.stuff' we will compare:
pip version 1 (instead of 10) > 6
Which fails bceause 1 is less than six. Instaed we really do want to
compare 10 > 6 so use split on '.' instead.
Change-Id: Ic7d0c04d7fa77774ab2d70fb9d11f182becec553
The pip list command supports a --format=columns option which outputs
things in space delimited columns. Switch to using that.
Change-Id: I5140a7d83bf567b1c3c67516112eb4c57074fa53
Strip the [<extras>] string from a <package_dir>[<extras>] argument
when looking for the package directory. Explain what the heck is
going on.
Change-Id: I79beb5c3e9e7c35c91cdd0d5a1d91532bebc4b6d
Closes-Bug: #1721638
As described in the change, "pip freeze" has issues with the way
zuulv3 clones repos without a remote. This is an attempt to use "pip
list" to check for local install
Change-Id: I33d25f86b6afcadb4b190a0f6c53311111c64521
lib/nova does a pip re-install of libvirt-python to rebuild the python
library incase the underlying libvirt version changed during package
installs. In offline mode, the underlying version of libvirt can't
have changed; so we have the situation that we've removed the libvirt
python bindings but can't reinstall them (because we're offline).
This fixes that particular situation, but skipping uninstalls in
offline mode seems generically OK.
Change-Id: I2b75d45d94d82f87d996c7570c125d46f5f99f6a
Closes-Bug: #1708369
Some URLs are broken, so fix them.
The others are redirect to new URLs,
so replace them with new ones.
The config options of nova serial console proxy
have been gathered in nova/conf/serial_console.py.
So the description in doc/source/guides/nova.rst
is fixed.
Change-Id: Ifd81cc09969341fbf8f135a913fc6003b94e0acc
To reduce the total number of invocations necessary for pip which isn't
the quickest thing ever (due to needing to evaluate constraints and deps
lists and what is currently installed) combine the main installation of
software with its test-requirements.txt file which should roughly halve
our pip invocations.
Change-Id: Ibcc3264136e66d34a879ad1c90a62e1bb6a84243
'pip uninstall' will hang running stack.sh if it has to
prompt the user for input, use -y.
Change-Id: Ic94639e444b87fd3538463d5a51c01a0208a2ab2
Closes-bug: #1691172
Slawek Kaplonski