This commit makes keystone tests for oslo policy
warning to check policy name explicitly. oslo policy
3.10 modified the warning text to include the policy
name as well as the check-str so new warning message
verified in keystone test will not work on old
oslo policy (<3.10), so bumping the oslo.policy
in requirements.txt
Change-Id: I14132dfced48ddc93c29ce2b4c20ed2cabc1dbd6
Since 3.7.0, oslo policy started the DeprecationWarning[1] if
deprecated_reason and deprecated_since param are not passed
in DeprecatedRule or they are passed in RuleDefault object.
These warnings are logged for every test which increase the
log size and sometime can full the log buffer and fail the
job.
[1] https://github.com/openstack/oslo.policy/blob/3.7.0/oslo_policy/policy.py#L1538
Change-Id: Id9d89a04b480cbdcefead93ce55a1f174f948f5d
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: Ic65d2fd6ce7215b4a47a6fb41b9cbf991f27773b
The newest vine release (5.0.0) is incompatible with the version of amqp
used on stable branches and results in test errors[1]:
"AttributeError: False does not have the attribute 'info'"
Ensure the lower-constraints tox environment uses the same version of
vine as defined in upper-constraints.txt.
[1] https://zuul.opendev.org/t/openstack/build/78dfcdd4272241459fb904aa70600e5b
Change-Id: I4b9b11b89e9f4120ae4b7fb8e213628dd9e8121c
l-c job template moved the l-c jobs running on Focal
and currently fails on many constraints.
Let's keep running l-c job on bionic as it was before and we
can move it to Focal once issues are identified and fixed.
- Fixing the hacking tests which are behaving differently between
< 3.8.0 (until Ubuntu Bionic) and 3.8.2 (Ubuntu Focal).
Squashing below review also
- https://review.opendev.org/#/c/750786/
Co-Author: Lance Bragstad <lbragstad@gmail.com>
Change-Id: If733e9824d87d8c73797f753e4daf95489bed9c2
We migrated to os-testr some time ago. There's no reason to keep this
around as a dependency.
Change-Id: Iedde135b9de03229c27ed57638d0c404169f43ab
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* parallelizing building of documents
Update Sphinx version as well.
openstackdocstheme renames some variables, so follow the renames. A
couple of variables are also not needed anymore, remove them.
Set openstackdocs_auto_name to use project as name.
Set openstackdocs_pdf_link to link to PDF file.
Remove docs requirements from lower-constraints, they are not installed.
Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.
See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html
Change-Id: I320a69816b4101bb76b88448881f3177c892ea92
Previously this call to the conf object couldn't parse cli args
because the oslo.policy tool was registering its cli opts on a
private conf object, so attempting to parse them on the global
object would fail. The dependency makes oslo.policy use the global
object instead so cli arg parsing works correctly.
This is important because ignoring cli args as this was previously
doing caused things like --config-file to be dropped, which meant
that running the tool with that option specified did not work as
expected.
Depends-On: https://review.opendev.org/690628
Change-Id: Id553743277a35660a40d6b3b02847d7a35abbfb9
Closes-Bug: 1849518
pip doesn't have a dependency resolver. Instead, it "simply uses the
first specification it finds for a project." [1] In Train, keystone
switched from hacking 0.12.x/0.13.x to hacking 1.1.x [2]. That change
explicitly added a pycodestyle dependency for reasons that aren't
entirely clear to me, but pip's broken dependency resolution leads to
the below funkiness when trying to install the dependencies.
ERROR: flake8 2.6.2 has requirement pycodestyle<2.1,>=2.0, but you'll have pycodestyle 2.5.0 which is incompatible.
As seen below, this can be easily reproduced and seems to happen because
pip doesn't go further than one level of dependencies, meaning it knows
about the dependency on flake8<2.7.0,>=2.6.0 from hacking, but not the
dependency on pycodestyle<2.1,>=2.0 that this in-turn introduces.
$ virtualenv venv
$ source venv/bin/activate
$ (venv) cat requirements.txt
hacking>=1.1.0,<1.2.0 # Apache-2.0
pycodestyle>=2.0.0 # MIT License
$ pip install -r requirements-new.txt
Collecting hacking<1.2.0,>=1.1.0
Using cached ...
Collecting pycodestyle>=2.0.0
Using cached ...
Collecting six>=1.10.0
Using cached ...
Collecting flake8<2.7.0,>=2.6.0
Using cached ...
Collecting pbr!=2.1.0,>=2.0.0
Using cached ...
Collecting mccabe<0.6,>=0.2.1
Using cached ...
Collecting pyflakes!=1.2.0,!=1.2.1,!=1.2.2,<1.3,>=0.8.1
Using cached ...
ERROR: flake8 2.6.2 has requirement pycodestyle<2.1,>=2.0, but you'll have pycodestyle 2.5.0 which is incompatible.
Installing collected packages: six, pycodestyle, mccabe, pyflakes, flake8, pbr, hacking
Successfully installed flake8-2.6.2 hacking-1.1.0 mccabe-0.5.3 pbr-5.4.3 pycodestyle-2.5.0 pyflakes-1.2.3 six-1.12.0
The solution is simple: stop explicitly requiring this dependency and
instead rely on flake8 bringing it in.
[1] https://pip.pypa.io/en/stable/user_guide/#requirements-files
[2] I3fc591e09c1e25a3bd2a3922880772ea9617f1e3
Change-Id: Ic0991d3eeae018609be0ecbd43fa0b0b9f13d6ba
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This change adds application credential access rules to the token model
and ensures that only clients (that is, keystonemiddleware) that support
access rule enforcement are allowed to validate tokens containing
access rules.
Depends-on: https://review.openstack.org/633369
bp whitelist-extension-for-app-creds
Change-Id: I301651369cf03e06550bc29eb534506674e56a1f
Since the WSGI app is reinitialized pretty much for every unit test, and
we have deprecated so many policies, we get hundreds of megabytes of
deprecation warnings in the unit test logs. This is unnecessary for unit
tests, a noisy hindrance to developers, and causes a high failure rate
in our CI due to the huge logs. This change fixes the issue for the unit
tests by adding warnings filters for DeprecationWarnings and
UserWarnings from oslo.policy and oslo.context.
This does not solve the issue that deployers see a lot of noise in their
logs. However, since production servers aren't reinitializing the WSGI
app quite so often, it's a less severe issue.
Related-bug: #1836568
Change-Id: Iaa7eae03bde7ab43a8c5a5886686f834cf7ec620
Depends-on: https://review.opendev.org/673932
This commit adds protection testing for the token API along with
changes to default policies to properly consume system-scope and
default roles.
Originally, this work was going to include the ability for project and
domain administrator to validate, check, or revoke tokens within the
context of their authorization (e.g., a domain administrator could
revoke tokens on projects within their domain). This seems like extra
work for not much benefit since we're using bearer tokens. The holder
of the token can do anything with that token, which means they can
validate it or revoke it without using their own token. Adding
project and domain administrator support seems unnecessary given the
existing functionality. If someone comes forward asking for this
functionality, we can re-evaluate the effort. For now, this patch is
limited to system user support, allowing them to validate, check, and
revoke any token in the system. Service users can still validate
tokens on behalf of users. Users can do anything they wish with their
own tokens.
This commit also bumps the minimum version of oslo.log so that we can
use the official TRAIN deprecated release marker.
Change-Id: Ia8b35258b43213bd117df4275c907aac223342b3
Closes-Bug: 1818844
Closes-Bug: 1750676
upper-constraints pins Werkzeug to a known working version. The
lower-constraints job uses a different constraints file, the
locally-maintained lower-constraints.txt, which didn't pin Werkzeug and
resulted in the job breaking on the most recent 0.15.0 Werkzeug release.
This change pins Werkzeug to match the known working version in
upper-constraints. If and when upper-constraints raises its cap, we can
address the breakage on the regular unit test jobs.
Change-Id: I926e79e34072505f9edc4879b1d9237a3b85accd
A subsequent patch will be using this library to create JWS tokens.
Here, we are requiring a minimum version of 1.6.1 since that version
includes an exception exposed from PyJWT that we need in keystone:
9d980786c9
bp json-web-tokens
Change-Id: I6b4b565fc7160fffe5e445673ccea9b3bba584d6
In Python 3, python-ldap no longer allows bytes for some fields (DNs,
RDNs, attribute names, queries). Instead, text values are represented
as str, the Unicode text type. Compatibility support is provided for
Python 2 by setting bytes_mode=False [1].
Update the keystone LDAP backend to adhere to this behavior by using
bytes_mode=False for Python 2 and dropping UTF-8 encoding and decoding
fields that are now represented as text in python-ldap.
[1] More details about byte/str usage in python-ldap can be found at:
http://www.python-ldap.org/en/latest/bytes_mode.html#bytes-mode
Note that at a minimum python-ldappool 2.3.1 is required. For more
details see Depends-On's below.
Change-Id: Ifdd0644cd7042407a008c85c0b2c40a971c90bc3
Closes-Bug: #1798184
Depends-On: https://review.openstack.org/611401
Depends-On: https://review.openstack.org/613632
Depends-On: https://review.openstack.org/614052
oslo.policy 1.43.1 includes support for domain scope types, which we
will need moving forward as we implement basic default role support.
oslo.context 2.22.0 includes support for domain-scoped tokens which
allows for better integration with oslo.policy.
We aren't going to consume oslo.policy 1.43.0 because it can possibly
log passwords for users when performing user operations with fully
logged RBAC enforcement data.
Change-Id: I44fd26d73fc5a331355542751eeb640ea394eb6e
The oslo.policy library actually accepts context objects as a first
class citizen, instead of a hand-built `creds` dictionary. This is a
perferred approach because it's easier for services to use
oslo.context to generate a context object that they can automatically
pass to oslo.policy for enforcement instead of inspecting the context
object and building a dictionary manually to pass to oslo.policy.
This commit makes allows keystone to partake in this by pulling the
keystone request object, which is a subclass of oslo.context's
RequestContext object, and uses it in enforcement. Additionally,
we're overriding the to_policy_values() method of oslo.context
in order to make sure we port keystone-specific values to the policy
dict representation of a context object. This ensures we have values
present that we rely on with our default policies.
This commit also bumps the lower requirement for oslo.policy to
make sure we're always using a version that understands context
objects.
Change-Id: I63e713f4aebf3e8cf5189a6060569d2828bc364d
The lower version of pycodestyle lib (aliased to pep8) doesn't work
for py36 env. This commit unblocks the py36 gate by adding a
dependency on pycodestyle and using that for style checks.
Bump the "hacking" lib version to v1.1.0 which depends
on a higher verion of pycodestyle.
Change-Id: I3fc591e09c1e25a3bd2a3922880772ea9617f1e3
One of the community goals for Stein is to implement a command-line
tool for operators that runs programmable checks that might impact
upgradability.
This commit lays down the basic structure for the upgrade checks and
ties it up to `keystone-status` command.
Story: 2003657
Task: 26135
Change-Id: I6586827104156ac549217967a1b9171f1a3b32e4
If you have a -c in the install_cmd it gets used with all the deps
supplied this means that the lower-constraints job actually install from
upper-constraints :(
You can see what I mean in [1]
Note both lower-constraints.txt and upper-constraints.txt are used ; and
---
Collecting oslo.log===3.39.0 (from -c /home/zuul/src/git.openstack.org/openstack/requirements/upper-constraints.txt (line 247))
---
With this fixed we find a few minimums that needs to be bumped:
* oslo.policy >= 1.33.0
keystone uses the scope_types[2] kwarg to RuleDefault which was
introduced in 52c82ff9ab04dd78ff7045cb30d2f5de535dd7da which is
contained in 1.32.0 ; also we need the 'policy-in-code' feature
which is in 1.33.0
* oslo.log >= 0.38.0
keystone used the ROCKY[3] constant for deprecations which was
introduced in d68a895ee8e61b5c9d4ef368e7f04252e84649e9 which is
contained in 3.38.0
* msgpack >= 0.5.0
the 0.4.x versions have been removed from pypi so we have to bump the
minimum :(
* SQLAlchemy >= 1.0.13
identity_provider_id in token payload is byte in python3 which
triggers a sqlalchemy bug[4]. The bug has been fixed in 1.0.13
* keystonemiddleware >= 5.1.0
unified limit feature uses system scope feature which is supported
in keystonemiddleware after 5.1.0
We also add correct some errors in bindep.txt related to use on Fedora
[1] http://logs.openstack.org/47/599447/2/check/openstack-tox-lower-constraints/bbc912b/tox/lower-constraints-1.log
[2] http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/policies/access_token.py#n24
[3] http://git.openstack.org/cgit/openstack/keystone/tree/keystone/conf/default.py#n50
[4] http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-a14dd2e73d889d065acc07a77b1ee7cb
Change-Id: Ic0de6799fddd86a70abae2c87c92d565072ebdb9
Known moderate severity security vulnerability detected in pysaml2
<=4.4.0. Let's bump our requirements pass the versions with known
vulnerabilities.
Change-Id: I235062eace1fa9f581018f2eec519f3cbea11ef7
Add project_id filter for listing limit. This filter
can be only used by system-scoped request to fetch the
specified project's limits.
bp: strict-two-level-model
Change-Id: I1b8cc227ed0710702aa099f09821f6eb897bb32c
Keystone's move to flask requires more than the general Flask library
as Keystone is a RESTful (ish) API. We will be using Flask-RESTful for
the easier mechanism to implment a REST API than standard flask
blueprints.
This also increases the base flask minimum requirements to unblock
requirements updates (flask has been updated in g-r to minimum of
1.0.2)
Partial-Bug: #1776504
Change-Id: I398acad439f4e525df3ca4e17fdd3e3ba90d58cc
Basic conversion of Keystone's core application to flask framework.
This doesn't add much in the way of flask-specific-isms but should
get keystone running directly under flask. This implementation does
not use paste-deploy.
Change-Id: Ib4c1ed3f645dd55fbfb76395263ecdaf605caae7
Create a tox environment for running the unit tests against the lower
bounds of the dependencies.
Create a lower-constraints.txt to be used to enforce the lower bounds
in those tests.
Add openstack-tox-lower-constraints job to the zuul configuration.
See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
for more details.
Change-Id: Ide4d192e390ac78f860634014f2437dd6ea7b919
Depends-On: https://review.openstack.org/555034
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Ghanshyam Mann