Sometimes we have to require a specific library in certain python
versions only.
In oslo and a few other projects, we are now replacing pytz by built-in
zoneinfo[1] to reduce 3rd party dependency, but zoneinfo is not
available in python < 3.9 and pytz is still required in python < 3.9.
In addition we need tzdata when zoneinfo is used, which means we need
tzdata only in python >= 3.9.
However these updates are currently prohibited by requirement check
because pytz in the global requirement does not contain python version
marker.
This change allows having a local requirement with python marker along
with the corresponding global requirement without python marker to
support such usage.
[1] https://review.opendev.org/c/openstack/oslo.utils/+/897623
Change-Id: I2bd5671962b3ca1abac91c2afba6e2c566616390
In change I85501b4bff97d1c1e1873c3329ef998a8e501134, we added the
ability for stdlib backport-style libraries to have different version
specifiers (i.e. 'python_version<=3.9') from what we define in our
global requirements file. At least, that was the intention. However, we
used a regex that wouldn't actually work since it was checking for
Python versions greater than or equal to rather than the inverse. Fix
the regex, adding tests in the process (like we should have done last
time) to prove that things actually work now.
While we're here, add some whitespace and test docstrings to make the
whole thing a little more readable.
Change-Id: I6a8566d086761148d2e7a8e8f8288b2b0e447d08
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Move the library names that we are flexible with - backports from the
std.lib - from a hard coded list in code into a txt file
Change-Id: I414f7c386b838248f45fbce003a64d0c83a9919c
There are a number of backport packages in place for packages in stdlib
that are still evolving. One such package is 'importlib-metadata' which
is a backport of 'importlib.metadata'. The README [1] for this package
shows a mapping of versions of the 'importlib-metadata' third party
library to versions of stdlib 'importlib.metadata':
importlib_metadata stdlib
4.8 3.11
4.4 3.10
1.4 3.8
A project may want to use a shiny new feature only found in the Python
3.11 stdlib version while continuing to support Python 3.8, 3.9 and
3.10. If so, they could specify something like so:
importlib-metadata>=4.8;python_version<3.11
Meanwhile, other packages might require features found in the Python
3.10 stdlib version. They would want to specify something like so:
importlib-metadata>=4.4;python_version<3.10
Currently this is not possible as it is not possible to specify
arbitrary 'python_version' markers. Since 'importlib-metadata' does not
have a 'python_version' marker in 'global-requirements.txt' (not since
change I5febaed02e95ff27accd946abc32f3bcbb1a5ead anyway), both projects
would have to specify
importlib-metadata>={required_version}
Which means they'll use importlib-metadata even where they don't need
it.
Fix this issue by allowing a list of particular modules to specify a
project-specific 'python_version' marker. Currently this list only
includes 'importlib-metadata', but in the future we may wish to enhance
it to include other rolling backport libraries (e.g. mock).
[1] https://pypi.org/project/importlib-metadata/
Change-Id: I85501b4bff97d1c1e1873c3329ef998a8e501134
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
If local exclusions are not a subset of global exclusions, we probably
want to identify any item in local exclusions that are not in global
exclusions, instead of the opposite.
Change-Id: Id8e5a423288ac6db3b836ae553d766e5f04b4dec
For projects that have gone Python 3, having to keep the
'python_version' specifiers can add otherwise unnecessary noise. Special
case these until such a time as Python 2.7 is history and we can drop
all these Python version specifiers.
Change-Id: Ie6dba8bc78dcd5fa95c7ac15b0a3770cdef8ad95
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Co-Authored-By: Zane Bitter <zbitter@redhat.com>
Before doing any work on this, let's use some more meaningful names.
Change-Id: I640290359c0c15035187a62da1bc57dba5912b05
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Ussuri and later will be Python 3 only. This updates our requirements
checks to only enforce local requirements match the exact global
requirements for the py3.5 and later entries.
Python 2 requirements can still be present in the local files, but this
allows teams that want to remove all Python 2 code to clean up their
requirements files for entries that are no longer relevant.
Change-Id: Ifaa44593b08630f84bb50060871412c66315adcc
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This makes the output from checking requirements a little more verbose
when issues are found to help make it more obvious what the issue is
when there is a failure.
Change-Id: I85e3dc9525893de6be3fac4c952272bfb3474255
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Error message when package not found in the global requirements has
formatting placeholder for the missing package name, but no formatting
was being called to actually populate the name.
Change-Id: Ibf778c61f170fcb798dc93c79c9ab4cdae658c84
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The phrasing and formatting of the existing messages makes it hard for
users unfamiliar with the requirements management process to
understand what is wrong. This change tries to make the messages
clearer, removes an overly verbose error message, and adds a message
to handle the case where something is missing from the global list
entirely.
Change-Id: I40711bf53aac52d534a427f4bdd1a91c20106118
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
While we don't track the minimum version of a given requirement in
requirements.txt we still need to ensure that they exist.
Change-Id: I84bfbc3a1db4032ee1a2b5995e6930978be42828
Closes-Bug: #1776247
When a requirement setting is changed, require that all requirements
follow the rules.
Without this change, it is possible to partially update the dependency
list for a project in a way that leaves some of the dependencies out
of compliance. With this change, all dependencies must be compliant in
order to update any of them.
Change-Id: I154245339a36618ac2e9a5922bc37121d44bca29
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
We're seeing parse errors on constraints with no markers, possibly
related to a unicode issue (it only seems to happen when the empty
marker string is a unicode string). There isn't any point in doing an
expensive comparison when there are no markers for the constraint, so
skip them in that part of the search loop.
Change-Id: I2a700fb8f3e776faf6123c9924d87add12324ef2
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Requiring the comments after the version numbers to match seems
overly pedantic.
Change-Id: Ia9d94ea194128dff867cd069488773454ba121ce
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Previous versions of the loop stopped looking at exact-match
attributes when one failed. By removing the break statement we can
report all match failures in a single run of the test job.
Change-Id: I419f3a611bfc6cc17272382e1b4f00534a5eb5ae
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Without this patch we test the attributes that have to match exactly
two separate times, once to report which do not match and again to see
if they all match before testing the exclusions.
With this patch we only test the attributes one time and if we fail to
match one we do not test them again.
Change-Id: Ida4fe5d14acfbd5f72a722037270df7c88a760d2
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
When checking a value against the global requirements list, report
which property does not match.
Change-Id: I61edbe52066a00e63dd6a958d8f8daec94afeee0
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
This print statement was left in an earlier patch but is not needed.
Change-Id: I92e1f4339dcc06303eddca684c68ad7d0e1376c0
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
The requirement lower bound and lower-constraints.txt value must
match.
The lower-constraints.txt value must be compatible with the specified
range, including exclusions.
If there is no lower-constraints.txt file the additional tests are
skipped.
If no minimum is specified, the misconfiguration is left to the other
validation logic to catch.
Only files associated with test jobs are checked because other jobs
are not run with the lower constraints list.
Change-Id: Iefe3ef8a89e965537486a1c9a62ab887b4401530
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Replace the exact match check for requirements with one that ensures
that the exclusion set specified is a subset of any exclusions used in
the global requirements list. This allows the lower bounds in a
project to diverge from the value used in other projects (and the
global list) while still ensuring that the allowed set of values is
the same.
We treat a cap (<version) as an exclusion because we want to allow
caps but we don't want them to be different from what we have in
global-requirements.
Change-Id: Ic2355cf6ca15380025d76dacffcf917c3929d40c
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Refactor the validation logic so we can test one rule at a time, then
write those tests.
Change-Id: I2250f451d92414d0003679d85b38aebbdf8bd3c9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Rearrange and refactor the existing requirements check logic to better
support testing. Add a few tests for the function that determines if
a setting matches the global requirements.
Change-Id: I3ea9049cf728b11ef71ca180304ee7041303a409
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Takashi Kajinami