Commit Graph

304 Commits

Author SHA1 Message Date
Takashi Kajinami 5afd51e8cf Drop unused pymongodb from requirements
Keystone no longer depends on mongodb after cache implementation was
split to oslo.cache[1]. Also, bandit is not a runtime dependency but
a test dependency, so should live in test requirements.

[1] 4969f66fca

Change-Id: I85f376d0897dd6b4dba758f86882fae70511fb6a
2024-02-04 15:04:55 +00:00
Rafael Weingärtner 14ac08431f Keystone to honor the "domain" attribute mapping rules.
We propose to extend Keystone identity provider (IdP) attribute mapping
schema to make Keystone honor the `domain` configuration that we have
on it.

Currently, that configuration is only used to define a default domain
for groups (and then each group there, could override it). It is
interesting to expand this configuration (as long as it is in the root
of the attribute mapping) to be also applied for users and projects.

Moreover, to facilitate the development and extension concerning
attribute mappings for IdPs, we changed the way the attribute mapping
schema is handled. We introduce a new configuration
`federation_attribute_mapping_schema_version`, which defaults to "1.0".
This attribute mapping schema version will then be used to control the
validation of attribute mapping, and also the rule processors used to
process the attributes that come from the IdP. So far, with this PR,
we introduce the attribute mapping schema "2.0", which enables
operators to also define a domain for the projects they want to assign
users. If no domain is defined either in the project or in the global
domain definition for the attribute mapping, we take the IdP domain
as the default.

Change-Id: Ia9583a254336fad7b302430a38b538c84338d13d
Implements: https://bugs.launchpad.net/keystone/+bug/1887515
Closes-Bug: #1887515
2024-01-16 08:54:56 -03:00
Pete Zaitcev d2deab385b Stop pinning pep8 related packages
These practices cause conflicts periodically. Not right now:
the gate is okay with the current values, which this patch
deletes. However, like sun raising in the east it is sure
to happen again. This patch lets workarounds work that the
infra team puts in place. The downside is, we need to fix
the code once in a while as new checks get added.

Change-Id: Ia7a96fb4b6de4251862a8a96c995cefa94dbc271
2023-10-02 15:41:36 -05:00
niuke 38d9a1ebca Use py3 as the default runtime for tox
Moving on py3 as the default runtime for tox to avoid to update
this at each new cycle.

Change-Id: I05339b12b0475c80c465e1586e9990b1861ea8a0
2023-08-02 10:50:49 +08:00
Zuul 8d700900fa Merge "Remove unnecessary removal of pyc files" 2023-07-10 14:41:55 +00:00
Stephen Finucane cbe2f7f6f2 Remove unnecessary removal of pyc files
In change I8fcd9370a6adbfe8bbb2ce441a6f2efad45d089a, we started setting
the 'PYTHONDONTWRITEBYTECODE=1' flag. With this set, Python won't
generate pyc files. As these files aren't generated, there's no need to
remove them. Remove the 'find' calls that were doing this.

Change-Id: Ifd4487c4566853244c4b2c90a178b1067c17fbc6
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2023-02-28 17:27:30 +00:00
Stephen Finucane 802539ad41 tests: Enable SQLAlchemy 2.0 deprecation warnings
Well, sort of. We enable them but immediately filter out the ones we're
actually seeing, the rationale being that we can address these in a
piecemeal fashion without the risk of introducing new issues.

There's a lot more to be done here. However, the work done in oslo.db
and other projects [1] should provide a guide for how to resolve the
outstanding issues.

[1] https://review.opendev.org/q/topic:%2522sqlalchemy-20%2522

Change-Id: Id4c5e80d665fa90d07517bc58c26b44d543d34b3
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2023-02-28 17:26:06 +00:00
Dave Wilde (d34dh0r53) 3288af579d Force algo specific maximum length
The bcrypt algorithm that we use for password hashing silently
length limits the size of the password that is hashed giving the
user a false sense of security [0].  This patch adds a check
in the verify_length_and_trunc_password function for the hash in
use and updates the max_length accordingly, this will override
the configured value and log a warning if the password is truncated.

[0]: https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues

Closes-bug: #1901891
Change-Id: I8d0bb2438b23227b5a66b94af6f8e198084fcd8d
2023-02-22 14:43:35 -06:00
Kristi Nikolla 420f4ff46d Fix passenv syntax in tox and update python jobs
This updated the Python jobs and fixes the following error with tox 4:

tox.tox_env.errors.Fail: pass_env values cannot contain whitespace, use
comma to have multiple values in a single line, invalid values found
'http_proxy HTTP_PROXY https_proxy HTTPS_PROXY no_proxy NO_PROXY
PBR_VERSION'PROXY PBR_VERSION'

Change-Id: I003723766b1dba7f54c9800364207191597c6741
2023-01-03 17:57:05 -05:00
Stephen Finucane 6dfde5b48b requirements: Bump linter requirements
The pep257 dependency does not support Python 3.10 and has been
deprecated in favour of flake8-docstrings. While we're here, we bump the
other linter dependencies and remove a note regarding the order of
dependencies, which is no longer true with the new dependency resolver
introduced in pip 20.3. We also remove an import exception for six.moves
since we no longer use six.

Change-Id: I4aae75f513568126230becf27b2e07d6682d35a1
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-10-26 15:14:21 +03:00
jiaqi07 4edad6b58f Use TOX_CONSTRAINTS_FILE
UPPER_CONSTRAINTS_FILE is old name and deprecated
This allows to use upper-constraints file as more
readable way instead of UPPER_CONSTRAINTS_FILE=<lower-constraints file>.

Change-Id: Id78b5ab392ac52a52731c87a4c06cbad1516ea85
2022-07-26 08:39:13 +08:00
Stephen Finucane 7c2d0f589c tox: Don't generate byte code
Keeps directories clean.

Change-Id: I8fcd9370a6adbfe8bbb2ce441a6f2efad45d089a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-06-17 14:34:32 +01:00
wu.shiming 921f57b01b Changed minversion in tox to 3.18.0
The patch bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23

Change-Id: Iab6f7759de5cc0b5f52a6e5aa2069f5640d06e34
2022-05-22 22:39:08 +09:00
Ghanshyam Mann 5a0fbe975d Update python testing as per zed cycle teting runtime
In Zed cycle, we have dropped the python 3.6/3.7[1] testing
and its support.

[1] https://governance.openstack.org/tc/reference/runtimes/zed.html

Change-Id: I817a4d1506fb7f15e72d37015ae0ba9547e2aa52
2022-05-10 19:30:04 -05:00
Ghanshyam Mann 1561da645b Drop lower-constraints.txt and its testing
As discussed in TC PTG[1] and TC resolution[2], we are
dropping the lower-constraints.txt file and its testing.
We will keep lower bounds in the requirements.txt file but
with a note that these are not tested lower bounds and we
try our best to keep them updated.

[1] https://etherpad.opendev.org/p/tc-zed-ptg#L326
[2] https://governance.openstack.org/tc/resolutions/20220414-drop-lower-constraints.html#proposal

Change-Id: Id276f7efef3ef955b4c0b4b8d62f7c38cb535b33
2022-04-30 17:16:49 -05:00
Stephen Finucane 36295cfb4b tox: Random fixups
- Set 'ignore_basepython_conflict' so that e.g. the 'py37' environment
  actually uses Python 3.7
- De-duplicate some environments
- Use upper-constraints for functional tests

Change-Id: Ia3a43e96af0e5d42a4ad3cc9594e1cebc7e71d08
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2022-01-05 18:29:02 +00:00
Zuul 07abf2fa4d Merge "Update hacking for Python3" 2020-04-25 10:21:07 +00:00
Andreas Jaeger f36111954b Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found.

Update local hacking checks for new flake8.

Change-Id: Ic440219814ee0c2b98217e9a821f38f5baf482ec
2020-04-15 07:17:58 +02:00
Vishakha Agarwal 121ee8ce73 Stop configuring install_command in tox.
Currently, we are overriding 'install_command' to use 'pip'. This is
considered poor behavior and 'python -m pip' should be used instead:

https://snarky.ca/why-you-should-use-python-m-pip/

It turns out that this is the the default value provided by tox:

https://tox.readthedocs.io/en/latest/config.html#conf-install_command

So we can remove the line and simply use the default value.

Change-Id: I2bf8b8509788fdc2f23c9ddf4211b19de7b3fb87
2020-04-08 11:11:16 +00:00
Colleen Murphy 13410383cf Constraint dependencies for docs build
Without this patch, new library releases (even if not specifically
related to sphinx or docs) can break the docs build. For example, the
latest Werkzeug release is breaking the docs for the stable/stein branch
due to an import error that was fixed in later branches. We shouldn't be
letting new library releases affect the docs environment for the same
reason we want to control the other tox environments. This change adds
-c to the docs tox environment so ensure that all libraries installed in
this environment are known good versions.

Change-Id: I7cec5478bd276594485ed3ce434ba8dda1975ce4
2020-02-19 18:52:59 +00:00
Zuul af1c1a822a Merge "Stop testing Python 2" 2019-11-14 10:28:52 +00:00
Arthur Dayne a92885a98b Stop testing Python 2
In Ussuri, Drop support for Python 2 according to [1] and [2]

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010356.html
[2] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation-timeline.html

Change-Id: I35e871992dd21e96e7e812e440e6cafdb61d26ac
2019-11-01 08:57:08 +08:00
Colleen Murphy e2d83ae95d Re-enable line-length linter
In 09088690 we mistakenly added E501 to the flake8 ignore list. Since
then, many new violations have been introduced. This patch re-enables
the check and corrects all violations, except in some cases like unit
test names where the subunit output would suffer if we attempted to
shorten the function name.

This may appear to be a pointless no-op that messes with
git-blameability, and it is, but the reason to do this is that if PEP8
violations are introduced in master and then backported to a stable
branch, most stable branches will fail the pep8 job since the flake8
ignore list is correct for those branches. Rather than loosening the
check in older branches or requiring those backports to fix the linter
errors independently of what's been merged in master, we should fix it
now so that we don't introduce more errors in the future and patches can
more easily be backported.

Change-Id: I9f71926105eb448bb0200201d1838b67d4963cd6
2019-10-21 08:48:47 -07:00
Colleen Murphy 5e35efd55f Split protection unit tests into its own job
There are so many protection tests now, and for the moment they are so
inefficient, that running them all as part of our main unit test suite
for py27, py36, py37, and cover jobs yields a high rate of timeouts
which reduces our own development velocity and negatively impacts every
project that co-gates with keystone. This change splits the protection
tests into their own level of tests outside of the configured stestr
test_path and adds a separate tox environment and zuul job to run just
the protection tests on their own. Parallelizing these tests should help
alleviate the timeout issue while we work on making these tests more
efficient.

Change-Id: Ibb12053bd6864a153f7e3998dbd008b6eec4295b
2019-09-16 10:56:42 -07:00
Vishakha Agarwal ce4d065f23 Generate PDF documentation
This patch adds a new tox job/command for building the pdf
version of documentation.

tox -epdf-docs

In addition to adjusting the infra requirement for PDF build
support, the following changes are made to build a PDF doc

- Download link of config_options, sample config and policy
  files are disabled for PDF doc as relative links do not
  make sense.
- Pre-create the doc/build/pdf/_static directory as a
  workaround because sphinx_feature_classification.support_matrix extension
  is operating under the assumption that the _static directory already exist
  and trying to copy support-matrix.css into it. We need to remove
  this workaround once the problem in the support_matrix extension is fixed.
- Use the toctree_only=True workaround to avoid TOC duplicate links.

Change-Id: Ief8df5f6a5a22b8d0530458ac45f344a72cc3e49
2019-09-10 16:36:48 +09:00
Andreas Jaeger f10f95b455 Docs: Make robust with using real links
Our tools noticed that keystone links to
https://docs.openstack.org/keystone/latest/admin/identity-domain-specific-config.html
which does not exist anymore.

The page was removed but the link to it was not changed. Replace this
and similar links with internal links that will work even if files are
moved - and can be verified, thus sphinx will error in case of broken
targets.

These changes include a few other fixes for broken keystone links, e.g.
to renamed anchors.

For the include files in admin/configuration.rst and
admin/federation/configure_federation.rst: Rename them to *inc.
The files were
published twice (as separate files and on this page) and thus
referencing failed. Renaming avoids this.

Also, put doctree outside of html tree so that it does not get
published.

Change-Id: I3d07637b0046cc88a66bcb51a0a4fe7c146c1549
2019-08-09 20:15:14 +02:00
Zuul 41f812fd59 Merge "Add Python 3 Train unit tests" 2019-07-02 02:30:02 +00:00
Corey Bryant 956bcd735f Add Python 3 Train unit tests
This is a mechanically generated patch to ensure unit testing is in place
for all of the Tested Runtimes for Train.

See the Train python3-updates goal document for details:
https://governance.openstack.org/tc/goals/train/python3-updates.html

Change-Id: I867fc8783fbb8dca8b6500e6e3944bd56062a6bd
Story: #2005924
Task: #34215
2019-06-26 18:43:53 -04:00
Colleen Murphy 374f56580a Switch order of precedence for unit test deps
Without this patch, oslo.db is listed both in requirements.txt and
test-requirements.txt. Since it is listed without extras in
requirements.txt, those extras (like the mysql driver) aren't installed,
which means opportunistic database tests are not run. This change fixes
the issue by ensuring the oslo.db listed in test-requirements.txt takes
precedence.

Change-Id: I0ad6a847ad89a7b5ae7d12b6dd90433e4a617778
2019-06-25 08:52:55 -07:00
Zuul d81225376a Merge "Fix E731 flake8" 2019-06-21 12:53:03 +00:00
Vishakha Agarwal 1925aa9c24 Fix E731 flake8
This patch fixes E731 for flake8/pep8 which recommends preferring
functions over lambda expressions.

Change-Id: Ied905e1d55ae73b8d390240d3320a8ba0dcca17c
2019-06-19 17:40:01 +05:30
Vishakha Agarwal ce614bf003 Drop use opendev.org for tox deps
The recommended url for upper contraints is updated
in this patch as per [1]

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006647.html

Change-Id: I8732051c4097abe1e526426e80fd832be1e4dc3f
2019-06-19 06:52:00 +00:00
Zuul 53f97dc58e Merge "Pep8 environment to run on delta code only" 2019-05-23 07:09:43 +00:00
Vishakha Agarwal e054b368dc Pep8 environment to run on delta code only
Currently tox -epep8 will run flake8 on whole code.
To make this fast, flake8 support is added for only
updated(delta) code. Same can be run by "tox -efast8".

It also determines how many commits to check based on the
$FAST8_NUM_COMMITS env variable. If set to "smart",
it uses git to try to run against all unsubmitted commits.
This allows fast8 to be more useful when actively
developing a series of patches.

Closes-Bug: #1829455
Change-Id: Ic02d7a91c6f6b227abf14bc6f7cb85815591c0d3
2019-05-21 10:38:51 +05:30
Jim Rollenhagen dc3175afb1 Revert "Blacklist bandit 1.6.0"
This reverts commit ebac8330d8.

Using the glob that I had not yet had enough coffee to do correctly is a
better solution, and allows us to fix new failures in 1.6.0 which would
break us again when 1.6.1 was released.

Change-Id: Ica473ba71b224cdc0acf815f82d534b6c70a7f54
2019-05-14 21:09:32 +00:00
ZhongShengping 5f90447b40 Replace git.openstack.org URLs with opendev.org URLs
Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.

Change-Id: I7cd39dacaa1f2cdc2d74da39ae3971246e922907
2019-04-24 11:51:00 +08:00
Vishakha Agarwal 068e1f51c0 Update the min version of tox
In Train, we will use python3.6 and 3.7 for which
the minimum tox version required is 2.5[1]

[1]https://tox.readthedocs.io/en/latest/changelog.html#v2-6-0-2017-02-04

Change-Id: I07c39d6503ae2c667e5967ef2c22bb3a6350d37d
2019-04-09 14:27:25 +05:30
Vishakha Agarwal a022e27307 Drop py35 jobs
Python 3.5 was the target runtime for the Rocky release.
The current target py3 runtime for Stein is Python 3.6,
so there is no reason to keep testing against the older
version.

https://governance.openstack.org/tc/reference/runtimes/stein.html#python-runtime-for-stein

Change-Id: I618c0d263d4c0f497aef59f24215b60169948ea9
2019-03-05 10:56:57 +05:30
Colleen Murphy 76165b3f56 Add py37 tox env
With python3.7 jobs added to the Zuul jobs, it makes sense to add this
to the tox config too so that developers can run python3.7 tests as
well.

Change-Id: I87ba933d57de746651427386d76d5b70b7cf45b8
2019-02-25 12:07:00 +01:00
wangxiyuan 090886902c Use pycodestyle in place of pep8
The lower version of pycodestyle lib (aliased to pep8) doesn't work
for py36 env. This commit unblocks the py36 gate by adding a
dependency on pycodestyle and using that for style checks.

Bump the "hacking" lib version to v1.1.0 which depends
on a higher verion of pycodestyle.

Change-Id: I3fc591e09c1e25a3bd2a3922880772ea9617f1e3
2018-11-20 17:16:01 +00:00
Colleen Murphy fb67da8016 Clean up python3.5 usage in tox.ini
We already have a debug env that uses python3 as the base python, it's
not that useful to have another one specific to 3.5. Also relax the
functional test environment to allow any python3 so that we have
flexibility in the future if we want to update it.

Change-Id: Ib43b41e0a1ce427d161b51227d0ffe0d4dec0a98
2018-11-06 12:54:32 +01:00
Colleen Murphy 605da1015f Add py36 tox environment
We already run python3.6 unit tests in CI. Add the py36 environment to
the tox file so that developers with python3.6 available locally can opt
into running that version too.

Change-Id: I207db150716139bd55d31e4a4c6eb7e7f48916b0
2018-11-06 12:52:15 +01:00
Nguyen Van Trung 104956c65e Don't quote {posargs} in tox.ini
Quotes around {posargs} cause the entire string to be combined into one
arg that gets passed to stestr. This prevents passing multiple args
(e.g. '--concurrency=16 some-regex')

Change-Id: Ia3353073bbdf9d18bbd5bd97425b3519f3d5aad0
2018-10-09 10:14:09 +07:00
Tony Breeds 114cdeb7eb Move use of constraints out of install_cmd
If you have a -c in the install_cmd it gets used with all the deps
supplied this means that the lower-constraints job actually install from
upper-constraints :(

You can see what I mean in [1]
Note both lower-constraints.txt and upper-constraints.txt are used ; and
---
Collecting oslo.log===3.39.0 (from -c /home/zuul/src/git.openstack.org/openstack/requirements/upper-constraints.txt (line 247))
---

With this fixed we find a few minimums that needs to be bumped:

 * oslo.policy >= 1.33.0
   keystone uses the scope_types[2] kwarg to RuleDefault which was
   introduced in 52c82ff9ab04dd78ff7045cb30d2f5de535dd7da which is
   contained in 1.32.0  ; also we need the 'policy-in-code' feature
   which is in 1.33.0
 * oslo.log >= 0.38.0
   keystone used the ROCKY[3] constant for deprecations which was
   introduced in d68a895ee8e61b5c9d4ef368e7f04252e84649e9 which is
   contained in 3.38.0
 * msgpack >= 0.5.0
   the 0.4.x versions have been removed from pypi so we have to bump the
   minimum :(
 * SQLAlchemy >= 1.0.13
   identity_provider_id in token payload is byte in python3 which
   triggers a sqlalchemy bug[4]. The bug has been fixed in 1.0.13
 * keystonemiddleware >= 5.1.0
   unified limit feature uses system scope feature which is supported
   in keystonemiddleware after 5.1.0
   

We also add correct some errors in bindep.txt related to use on Fedora

[1] http://logs.openstack.org/47/599447/2/check/openstack-tox-lower-constraints/bbc912b/tox/lower-constraints-1.log
[2] http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/policies/access_token.py#n24
[3] http://git.openstack.org/cgit/openstack/keystone/tree/keystone/conf/default.py#n50
[4] http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-a14dd2e73d889d065acc07a77b1ee7cb

Change-Id: Ic0de6799fddd86a70abae2c87c92d565072ebdb9
2018-09-12 03:54:38 +00:00
Doug Hellmann f5017d9ccd fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.

Change-Id: I4660a35ff63f3f0212978dda668ba2cb08774e3a
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-06 16:06:00 -04:00
Nguyen Hai a04a1dc782 Follow the new PTI for document build
- Follow new PTI for docs build
- Add sphinxcontrib.apidoc to replace pbr autodoc

REF:
https://governance.openstack.org/tc/reference/project-testing-interface.html
http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html

Depends-On: https://review.openstack.org/#/c/559334/
Change-Id: I1d5f01cf49f36567f5a891c69c7269d3e966b9c5
2018-04-09 01:13:58 +09:00
Doug Hellmann 6cb71c36c4 add lower-constraints job
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.

Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.

Add openstack-tox-lower-constraints job to the zuul configuration.

See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.

Change-Id: Ide4d192e390ac78f860634014f2437dd6ea7b919
Depends-On: https://review.openstack.org/555034
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-03-23 08:40:54 -04:00
Zuul ccc9893fa5 Merge "Migrate functional tests to stestr" 2017-12-22 16:47:26 +00:00
qinglin.cheng 07bba320ae Create doc/requirements.txt
For compliance with the Project Testing Interface as described in:

https://governance.openstack.org/tc/reference/project-testing-interface.html

Refer to:

http://lists.openstack.org/pipermail/openstack-dev/2017-November/124815.html

Change-Id: Ic05a55fb14741bb0802c217c81b0035b6e1f7de2
2017-12-19 16:17:26 +01:00
Gage Hugo 3b209ed938 Migrate functional tests to stestr
The functional tests in tox.ini were still set to use testr,
this change makes it use the new stestr.

Change-Id: I2ca4e2ae5d8395ae27f8d14cce8038c737aa65f2
2017-12-14 09:28:48 -06:00