We propose to extend Keystone identity provider (IdP) attribute mapping
schema to make Keystone honor the `domain` configuration that we have
on it.
Currently, that configuration is only used to define a default domain
for groups (and then each group there, could override it). It is
interesting to expand this configuration (as long as it is in the root
of the attribute mapping) to be also applied for users and projects.
Moreover, to facilitate the development and extension concerning
attribute mappings for IdPs, we changed the way the attribute mapping
schema is handled. We introduce a new configuration
`federation_attribute_mapping_schema_version`, which defaults to "1.0".
This attribute mapping schema version will then be used to control the
validation of attribute mapping, and also the rule processors used to
process the attributes that come from the IdP. So far, with this PR,
we introduce the attribute mapping schema "2.0", which enables
operators to also define a domain for the projects they want to assign
users. If no domain is defined either in the project or in the global
domain definition for the attribute mapping, we take the IdP domain
as the default.
Change-Id: Ia9583a254336fad7b302430a38b538c84338d13d
Implements: https://bugs.launchpad.net/keystone/+bug/1887515
Closes-Bug: #1887515
Resolve the following LegacyAPIWarning warning:
The Query.get() method is considered legacy as of the 1.x series of
SQLAlchemy and becomes a legacy construct in 2.0. The method is now
available as Session.get()
Change-Id: I30d0bccaddff6a1d91fcd5660f490f904e7c8965
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warnings:
The autoload parameter is deprecated and will be removed in version
2.0. Please use the autoload_with parameter, passing an engine or
connection.
Change-Id: I2e523a31e322c19512e3666aec1597fd088eb53f
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
The legacy calling style of select() is deprecated and will be removed
in SQLAlchemy 2.0. Please use the new calling style described at
select().
Change-Id: I3c4c8fb63ab13bb10bcad6cc460f0f2c588c4ab9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Keystone's update_user() method in the SQL driver processes a lot of
information about how to update users. This includes evaluating password
logic and authentication attempts for PSI-DSS. This logic is evaluated
after keystone pulls the user record from SQL and before it exits the
context manager, which performs the write.
When multiple clients are all updating the same user reference, it's
more likely they will see an HTTP 500 because of race conditions exiting
the context manager. The HTTP 500 is due to stale data when updating
password expiration for old passwords, which happens when setting a new
password for a user.
This commit attempts to handle that case more gracefully than throwing a
500 by detecting StaleDataErrors from sqlalchemy and retrying. The
identity sql backend will retry the request for clients that have
stale data change from underneath them.
Change-Id: I75590c20e90170ed862f46f0de7d61c7810b5c90
Closes-Bug: 1885753
assertItemsEqual was removed from Python's unittest.TestCase in
Python 3.3 [1][2]. We have been able to use them since then, because
testtools required unittest2, which still included it. With testtools
removing Python 2.7 support [3][4], we will lose support for
assertItemsEqual, so we should switch to use assertCountEqual.
[1] - https://bugs.python.org/issue17866
[2] - https://hg.python.org/cpython/rev/d9921cb6e3cd
[3] - testing-cabal/testtools#286
[4] - testing-cabal/testtools#277
Change-Id: I7725cead76c0c7349af9a8c8e8a54290caebce9c
When a federated user authenticates, they are added to their
mapped groups during shadowing.
Closes-Bug: 1809116
Change-Id: I19dc400b2a7aa46709b242cdeef82beaca975ff3
Modify the base driver and SQL driver to support expiring group
memberships.
Additions to the SQL Driver to support listing expiring groups
for user.
Change-Id: I7d52cd2003f511483619a429de57201df4990209
Partial-Bug: 1809116
Depends-On: I4294a879071dde07e5eb1da4df133de8032e1059
The mock library is a third party lib that attempted to bridge the gap
between Python 2 and Python 3 mocking. Now that we have moved to py3
only, there is no need to use a third party lib and we can use the
standard built-in mocking support.
Change-Id: I8bbcedb7ad3f0bc2e06dfa13878a97411ee1dc6d
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This repo does not support Python 2 anymore, so we don't need
six for compatibility between Python2 and 3, convert six usage to Python
3 code.
Change-Id: Icba56808f38277b27af2ae5aac4b8507dee71b3b
SQLAlchemy passes a SQL NULL result value, which in Python comes
back as None, to the TypeDecorator.process_result_value method
for a datatype that is handling that column. Python's json.loads()
function does not accommodate Python None directly. A NULL
value may be returned for a JsonBlob expression either if SELECTing
from a table where the value is NULL, as there are several
NULLable columns in Keystone's model which use this type, or
if the column is linked to a SQL result such as from an outer join
that does not contain the value.
The reason this bug has not been seen in Keystone is because the only
occurrence of SQL NULL for this column type has been in an ORM
"joinedload()" LEFT OUTER JOIN expression, and the ORM does not
fetch the actual columns when the identity is NULL. However, upcoming
changes in SQLAlchemy include that the lower level row object
will be firing off these result handlers up front, rather than
deferring them for fetch, which has illustrated this issue.
The patch includes a test suite exercising the JsonBlob and
DateTimeInt types completely, as there do not seem to be tests that
target these constructs individually.
Change-Id: Ib1448fa08d7106fd55e18a0f92a705011bc3ce2b
The .c attribute on a SQLAlchemy select() object implicitly
creates an unnamed subquery when invoked, and the columns
returned are in reference to this subquery, not the SELECT
statement itself. The usefulness of
this subquery is close to zero, whereas it confuses users
with its misleading behaviors, complicates the internals,
and is rejected by the two leading databases (PostgreSQL and
MySQL) at the SQL level in any case, and as such it's going
away most likely in SQLAlchemy 1.4, after
a short deprecation period that has yet to be announced in the 1.3
series.
Keystone's schema-oriented unit tests appear to be applying
the select() construct to a Table object that's been reflected with
autoload for the usually immediate purpose of iterating through
the columns of this Table and comparing them to a set of
comparison tuples, however the ".c." collection
is available on all "from clause" objects of which Table is a
member so there is no need to apply a select() first. In particular,
there is a reference to an internal
SQLAlchemy API "_proxies" which is attempting to reconcile the
fact that the implicitly created subquery columns don't refer
to the original Table column's "default" parameter (there is in fact
a public API for this system as well, but it is not needed
once the select() is removed).
This patch removes the "select()" applied to the Table returned
by select_table() and renames the method to load_table() to more
accurately reflect what this method does. The use of SQLAlchemy's
private "_proxies" API can now be removed, and the two occurrences
where the select() were actually used to invoke a SELECT
statement are replaced with explicit
production of a select() construct as well as making use of the
already-present Session to invoke the query rather than using
an out-of-band ad-hoc transaction (connectionless execution is
also going away by SQLAlchemy 2.0 in any case).
Change-Id: Ica7037541a1305308ed743d8e04a3c2f02a2b811
With this, list_users takes only 3 call to db, where before it took 11.
With the change it takes ~0.352s on my PC to list users
Without the change it takes ~1.5s on my PC to list users
Change-Id: I35ad7b02cd1f695634f5feba6d924fd0301e7640
Now `domain` is considered as the first level of project depth.
Update the check function to make sure the depth check is correct.
Change-Id: I38411001271518b9c2f58d53b2f654c361e952e8
bp: domain-level-limit
The password column in Password table which stores less secure
password can be dropped safely now.
Change-Id: I7bfe67e0bdb085b66b86fe83a4ff31bbaab49599
The request input format validation now is handled by jsonschema
check. The clean file is useless now.
Change-Id: I9806722cca8bcd1d4c73618cf1b36107929d37b0
The unit test uses sqlite for test which closes db foreign keys
function by default. This patch enabled the sqlite foreign keys
function for unit test by default.
The "project" table is a self referencing FK table(id <-> domain_id
column). So when the FK is enabled, there must exists a root record
before insert data to this table. It's <<keystone.domain.root>>.
Usually, the <<keystone.domain.root>> recored is inserted into the
table once operators run "keystone-manage db_sync" command when
deploy Keystone. But the unit test code doesn't run this command,
it initialise the db schema by reading sqlalchemy object model, so
the <<keystone.domain.root>> record is missed. Then we can't create
any project record, it'll raise FK error.
So in this patch, before creating any projects in the test, we must
ensure the <<keystone.domain.root>> record exists first.
Change-Id: I565d12395ca39a58ba90faf8641a9e02d986aeb9
Closes-Bug: #1744195
The method dict.get(key) will return None
in the following two cases:
A. if there is no key in the dictionary.
B. if the real value of the key is just None.
The above two cases will results in passing
the assertion which is not the expected
behaviour of the test.
Change-Id: Ib4a7f2d5beb1e4e9b4d8d9dd90de3f52a5c29e74
Enable the sqlite foreign keys function for unit test.
This patch is the first part to solve sql backend test issues.
Change-Id: I5d29d05e64b76ff6530c9af5ee39a2df1b26aa03
Partial-Bug: #1744195
Add a check function for project hierarchical tree check
when starting Keystone. If the tree depth exceed the
enforcement model's depth, fail to start keystone process
Change-Id: I4ce6a48505b8b9688bbdd18ee46ce035ee0938ed
bp: strict-two-level-model
What this patch did and why:
1. added an auto increment primary column `interenal_id` for both
registered_limit and limit tables. Removed the primary key but
added unique index for `id` column. This change can improve
the db performance.
2. dropped the forieign keys in limit table. The `project_id`
column has a foreign key. Using foreign key between different
backends can lead some unexpected error.
3. dropped the unique constraints and foreign key constraint in both
tables. Because `region_id` can be null, in this case, both
constraints can't work well in all kinds of DBs. Instead, we'll
check the unique and foreign reference in code.
NOTE: For MySQL and PostgreSQL, we did the change inner tables. But
for SQLite, it doesn't support adding a primary column into an existed
table, so that we recreated the tables instead..
Closes-bug: #1777893
Change-Id: Ibb408758466ff367f57bafbd4b8c9213499f8dc3
Both of these drivers were staged for removal in Rocky. Now that
Rocky is open for development we can remove them. This commit removes
just the bare-bones aspects of each. Subsequent patches will do the
following:
- Remove test class that were only meant for sql or uuid scenarios
- Refactor the notification framework to not hint at token storage
- Refactor the token provider API interfaces to be simpler and
cleaner
- Remove the needs_persistence property from the token provider API
and document the ability to push that logic into individual
providers that require it
- Return 403 Forbidden for all requests to fetch a revocation list
- Remove the signing directory configuration options
These changes will result in simpler interfaces which will be
important for people implementing their own token providers and
storage layers.
bp removed-as-of-rocky
Change-Id: I76d5c29f6b1572ee3ec7f2b1af63ff31572de2ce
Just like the APIs in keystone, we should be using the
`keystone.common.provider_api.ProviderAPIs` to load managers instead
of self.
This makes those changes for a few of the test modules. Finding
occurrences can be done with `grep -R 'self.*_api' keystone/tests/`.
Change-Id: Ic2094dca56158d8e4cd843eadff837f3a17ea38f
We've already converted Password objects to use the DateTimeInt format
for its datetime attributes[1]. This was necessary to cope with
differences in date storage formats between different DBMSs that was
causing intermittent test failures. While we're not experiencing those
CI problems any more, the DateTimeInt format is the way forward for
consistent datetime storage. This patch converts the trust table and
model to use the new format.
[1] https://review.openstack.org/#/c/493259/
Related-bug: #1702211
Change-Id: If524c743170924e5b8cfdafa862ed31b06db018c
This change implements all the manager APIs to grant roles to users
at a system level. A subsequent patch will do the same thing for
groups and hook everything up to the controllers.
bp system-scope
Change-Id: I49b43693410273ae8c7fa275ce0d827708ece535
"add_user_to_project" and "remove_user_from_project" are only
used for V2. This patch removed these two useless functions.
Change-Id: I94f7573997035c4395ec77eabe0d6e45ff9c3bf7
This change adds in tags as a hybrid property for projects.
Also adds in relevant tests for backend logic.
Change-Id: Ie153130b794e3541f197f8fa1e73bf30a64f12dc
Partially-Implements: bp project-tags
Co-Authored-By: Jaewoo Park <jp655p@att.com>
Co-Authored-By: Nicolas Helgeson <nh202b@att.com>
Depends-On: I00f094a5584be40ab477cbf680a5f6d1afb4d21b
Depends-On: Ib3db74cb08e7fdec01d55d8988e0906948d80a32
Due to MySQL (in some versions) not storing datetime resolution below
one second, keystone occasionally ends up with weird behavior such as
a New password not being valid. The password created at and expires at
columns now store both datetime (for rolling upgrades) and integers.
Keystone from Pike and beyond leans on the new created_at_int column
and expires_at_int column.
Change-Id: I2c219b4b9b353f1e2cce6088849a773196f0e443
Closes-Bug: #1702211
Support bcrypt, pbkdf2_sha512, or scrypt in password hashing for
passwords managed within keystone. sha512_crypt is insufficient to
hash passwords in a secure way for storage in the DB. Keystone defaults
now to using bcrypt but can handle scrypt and pbkdf2_sha512 with a number
of tuning options if desired.
Closes-bug: #1543048
Closes-bug: #1668503
Change-Id: Id05026720839d94de26d0e44631deb34bcc0e610
[0][1] made it so that every write operation to the LDAP backend by
users returns a 403 Forbidden. However, since all our unit tests
depend on being able to write to the LDAP backend, the code to
perform writes is still in the codebase and it's toggled by the
unit tests.
This patch cleans up delete operations from unit tests. And since
those unit tests don't depend anymore on being able to do deletes
to LDAP, the associated delete logic in the driver is removed.
None of this is user facing or changes anything in the Keystone
behavior.
[0] I13eada3d5c3a166223c3e3ce70b7054eaed1003a
[1] I225daf0e00742c54d5d009e456d4a3ad864356a0
Change-Id: I9f183492fd318c755ba026e9e402615fe66f100c
The main reason for this change is to create real Identity Provider data
for the shadow users (backend and core) tests related to federated
users. This is needed in a subsequent patch where we set the domain for
federated users.
I ended up moving the shadow user tests out of the identity tests, as my
changes were conflicting with some of those tests and it provided a
simpler implementation.
Partial-Bug: #1642687
Partially-Implements: bp support-federated-attr
Depends-On: I08a8f3cb59150c8e9a2f90c5ea6b0aa197a03572
Change-Id: If8c8ad39c4c55a2d800bf4432411db59799e84e6
All users (including federated) should belong to a domain. Currently,
the domain_id is being populated in the local_user and nonlocal_user
(ldap) tables. However, it isn't being set for federated users. This
patch moves the domain_id up to the user table, and creates composite
foreign key (fk) relationships to the local_user and nonlocal_user
tables, as the domain_id is still needed in those tables to enforce
user name uniqueness:
+-----------+
| user |
| (pk) id |
| domain_id |
+-----------+
1:1
+----------------+
| local_user | (and nonlocal_user)
| (pk) id |
| (fk) user_id |
| (fk) domain_id |
+----------------+
Likewise, creating a unique constraint on user (id, domain_id) to
support the new composite fk.
This will allow us to set the domain_id in the user table and ensure
that it is in sync with the local_user and nonlocal_user tables, such
that a user belongs to a domain.
Partial-Bug: #1642687
Partially-Implements: bp support-federated-attr
Change-Id: I08a8f3cb59150c8e9a2f90c5ea6b0aa197a03572
The following file(s) added utf-8 encoding but never used. So we can
remove them at all.
keystone/tests/unit/test_backend_ldap_pool.py
keystone/tests/unit/test_backend_sql.py
keystone/tests/unit/test_v2_validation.py
Change-Id: I18c0bae5bff2108add7ae52e4e678c904ba53f48
A service user from auth_token middleware should be able to fetch a
token that has expired within a certain window so that long running
operations can finish.
Implements bp: allow-expired
Change-Id: I784f719be88481048f5aa7a79d34a54907438cf3
This is the first step of several to remove PKI token support in
keystone. A large issue in removing PKI support is support for the
revocation list must be maintained.
This patch removes support for the token format, it's surrounding tests
and examples that are generated. Additionally, some wording has been
changed around the CLI and config options to make the distinction
between keys and certs used for PKI tokens and those used for getting
the revocation list (a list of tokens that are revoked, which is signed).
Future patches will:
- Remove the keystone-manage commands for generating certs
- Modify the revocation list (at /auth/tokens/OS-PKI/revoked) to return
a 403 if pki is not configured (instead of raising a 500). We cannot
remove the API as that would break an API contract.
- Options to configure PKI will be marked as deprecated
- If PKI is configured a normal signed list will be returned (same
behavior as today)
- Follow up patch to keystonemiddleware will make sure auth_token does
not rely on the revocation api at all.
Related-Bug: 1626778
Related-Bug: 1626779
Co-Authored-By: Boris Bobrov <bbobrov@mirantis.com>
bp removed-as-of-ocata
Change-Id: Icf1ebced44a675c88fb66a6c0431208ff5181574
`endpoint_filter.sql` backend is the only left-over from
endpoint filter extension, all others has been moved into
keystone catalog dir.
This patch deprecate `endpoint_filter.sql` backend and
consolidate the backend with SQL backend.
This patch also update some related testcases to make sure
project id exists instead of some random uuids since original
logic from endpoint filter extension has the constraint and
this is make sense to inherent into SQL backend as well.
Partially implements: bp deprecated-as-of-ocata
Change-Id: I28b37fc98cf63da11c0dd200b5f657507c0bca6a
Now that we no longer support driver versioning, we don't need to
specify specific driver versioning in our tests. Unwinding this code.
Change-Id: I25d1202ada6b9e2806e102aa22e22ccc77429a53
Migration 002 sets the password created_at column to a TIMESTAMP type
with a server_default=sql.func.now(). There are a couple problems
that have been uncovered with this change:
* We cannot guarantee that func.now() will generate a UTC timestamp.
* For some older versions of MySQL, the default TIMESTAMP column will
automatically be updated when other columns are updated:
https://dev.mysql.com/doc/refman/5.5/en/timestamp-initialization.html
This patch fixes the problem by recreating the password created_at
column back to a DateTime type without a server_default:
1) Drop and recreate the created_at column
2) Update the created_at value
3) Set the created_at column as not nullable
Closes-Bug: #1621200
Change-Id: Id5c607a777afb6565d66a336028eba796e3846b2
This commit implements credential encryption through the following changes:
- additive schema change to store key hashes for credentials
- database migration to encrypt all pre-existing credentials
- contractive schema change to remove unencrypted credential column
- added code to the credential Manager to handle credential encryption
All credentials will be encrypted by default. There will not be a way to store
unencrypted credentials in keystone from this point forward.
Note that this implementation uses database triggers in the migration process.
If operators use the traditional offline migration method, it would be more
reliable if we didn't try to setup and tear down triggers, as they'll never be
used anyway. This makes it so that expand and contract migrations can skip
anything related to triggers.
Co-Authored-By: Werner Mendizabal <nonameentername@gmail.com>
bp credential-encryption
Depends-On: I433da9a257daa21ec3b5996b2bca571211f1fbba
Depends-On: Id3e8922adc154cfec5f7a36613e22eb0b49eeffe
Change-Id: I31b7539db436ad270462cfaa3b14213e0ed1fc04
The 105 migration added a password created_at column that was left as
nullable. This patch sets a default value for password created_at and
makes this column not nullable.
Closes-Bug: #1596500
Change-Id: I394467d554c786ecd9bf55367435c856c6723042
The minimum password age is the period at which a password must be used
before it can be changed. This prevents users from immediately wiping
out their password history in order to use an old password.
Partially-implements: blueprint pci-dss
Change-Id: Ib1367bc69b791ef35de8f18704437e8fc233afdf
This patch satisfies the following PCI-DSS lockout requirements:
* PCI-DSS 8.1.6: Limit repeated access attempts by locking out the user
ID after not more than 6 attempts.
Co-Authored-By: Alexander Makarov amakarov@mirantis.com
Partially-implements: blueprint pci-dss
Change-Id: I8e1bdd0d6febcd5f51a4de3249e2eb3ae16dcee5
Rafael Weingärtner