Keystone provided two in-tree catalog drivers, sql and templated.
However the templated driver hasn't been properly maintained.
The default template had not been updated for 8 years until it was
recently updated by [1].
This deprecates the driver assuming it's not widely used and sql driver
meets usual requirements.
This also restores the image service endpoints which were wrongly
removed by [1].
[1] c32bedb654
Related-Bug: #2013473
Change-Id: Iadb7bd5d7c4cf82aea2a7dbc1d8c4dbe53b9f763
When calling the s3tokens or ec2tokens API with a
HTTP GET we should get a 405 Method Not Allowed but
we get a 500 Internal Server Error because we enforce
that method.
Closes-Bug: #2052916
Change-Id: I5f60d10dc25551175cc73ca8f3f28b0b95ec9f99
Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>
Because these were removed by [1]. Also update the previous release
note to document the upgrade impact on catalog information (like
endpoint urls) including string interpolations requiring these removed
options.
[1] 2a3c73c49b
Change-Id: If78d0b93665410b86754ea35653ca9d4c15c81c5
The eventlet server implementation was removed during Newton, and have
not been used by any other implementations for a while.
Change-Id: I01f9adfc3e610d820c1834209d36c10568cccf41
These options have had no effect and were formally deprecated during
Yoga cycle[1].
[1] 9a8686aee0
Related-Bug: #1941020
Change-Id: I9ac00109bd278bc4813a45358aeda848ab7318de
This patch adds new hashing alhorythm bcrypt_sha256, which is based on
the bcrypt but does not have limitations on the leght of the passwords,
since passwords are passed through HMAC-SHA2-256 first.
At accepts exactly same parameters as bcrypt does.
However, it prefix the hash using `prefix` attribute rather then
`indent_values` which are same as for bcrypt.
Change-Id: I5430ebf5a20142c1a9caab960ced9b3ee2e782c1
bcrypt hashing algorythm has a limitation on length of passwords it
can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
implemented, which resulted in password being invalidated after the
keystone upgrade, since passwords are trimmed differently by bcrypt
itself, as well as len(str()) is not always equal to
len(str().encode()) as trimming should be done based on bytes and not
string itself.
With the change we return a byte object from
`verify_length_and_trunc_password`, so it does not need to
be encoded afterwards, since we need to strip based on bytes
rather then on length of the string.
[1] https://review.opendev.org/c/openstack/keystone/+/828595
Closes-Bug: #2028809
Related-Bug: #1901891
Change-Id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1
The OAuth2.0 Access Token API is modified, support to get an OAuth2.0
certificate-bound access token from the keystone identity server with
OAuth 2.0 credentials and Mutual-TLS certificates.
Co-Authored-By: Hiromu Asahina <hiromu.asahina.az@hco.ntt.co.jp>
Change-Id: I885527bec61429b1437a046097a16491848b5a0a
Implements: blueprint support-oauth2-mtls
sqlalchemy-migrate does not (and will not) support sqlalchemy 2.0. We
need to drop these migrations to ensure we can upgrade our sqlalchemy
version.
Change-Id: I31ba9e4f129a7cc28744e814b5fd28eb284ae3de
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The bcrypt algorithm that we use for password hashing silently
length limits the size of the password that is hashed giving the
user a false sense of security [0]. This patch adds a check
in the verify_length_and_trunc_password function for the hash in
use and updates the max_length accordingly, this will override
the configured value and log a warning if the password is truncated.
[0]: https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues
Closes-bug: #1901891
Change-Id: I8d0bb2438b23227b5a66b94af6f8e198084fcd8d
If a token is issued with an application credential we need to check
the expiration of the application credential to ensure that the token
does not outlive the application credential. This ensures that if the
token expiration is greaten than that of the application credential it
is reset to the expiration of the application credential and a warning
is logged. Please see CVE-2022-2447 for more information.
Closes-Bug: 1992183
Change-Id: If6f9f72cf25769d022a970fac36cead17b2030f2
Since LDAP is now readonly, the current behavior might be
unexpected. By randomizing the list, we assure a more gradual
failure scenario if the first server on the list (as specified
by the user) fails.
Change-Id: I23f31bd85443784013a6aa158d80c7aeeb343993
Closes-Bug: #1953622
Resolves: rhbz#2024602
Switch to alembic for real by integrating it into the 'db sync' command
flow. From a user-facing perspective, things should remain pretty much
the same as before, with the key difference being that version
information (i.e. what's shown by 'keystone-manage db_sync --check' or
'keystone-manage db_version') will now take the form of a hash rather
than an integer. There are a few differences for contributors however.
The changes are described in the included release note and
documentation.
Note that there are a couple of important design decisions here that are
worth examining:
- We drop the idea of the 'data_migration' branch entirely and the
'keystone-manage db_sync --migrate' command is now a no-op. Neutron
doesn't do data migrations like we do and yet they manage just fine.
Dropping this gets us closer to neutron's behavior, which is a good
thing for users.
- We haven't re-added the ability to specify a version when doing
'db_sync'. Neutron has this, but the logic needed to get this working
is complex and of questionable value. We've managed without the
ability to sync to a version since Newton and can continue to do so
until someone asks for it (and does the work).
- sqlalchemy-migrate is not removed entirely. Instead, upon doing a
'db_sync' we will apply all sqlalchemy-migrate migrations up to the
final '079_expand_update_local_id_limit' migration and dummy apply the
initial alembic migration, after which we will switch over to alembic.
In a future release we can remove the sqlalchemy-migrate migrations
and rely entirely on alembic. Until then, keeping this allows fast
forward upgrades to continue as a thing.
- Related to the above, we always apply *all* sqlalchemy-migrate
migrations when calling 'db_sync', even if this command is called with
e.g. '--expand' (meaning only apply the expand branch). This is
because there is at most one "real" migration to apply, the Xena-era
'079_expand_update_local_id_limit' migration, which is an expand-only
migration. There is no risk to applying the empty "data_migration" and
"contract" parts of this migration, and applying everything in one go
results in *much* simpler logic.
Future changes will update documentation and add developer tooling for
(auto-)generating new migrations, a la 'neutron-db-manage revision'.
Change-Id: Ia376cb87f5159a4e79e2cfbab8442b6bcead708f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The OAuth2.0 Access Token API is added, support to get an OAuth2.0
access token from the keystone identity server with application
credentials.
Change-Id: I4c54649a51534637be831450afc32d3ef8644ee5
Remove option from 'db sync' and 'db version' commands of
'keystone-manage'. We also remove handling code for providing '--expand'
and '--migrate' in the same invocation, which isn't possible since
they're in a mutually exclusive group. Finally, we clean some formatting
things up in preparation for the alembic integration.
Change-Id: I198822d6f55353f1adeea9024db67abb24be54c7
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Value of 0 causes the pool to fail before it attempts to connect
to ldap, raising MaxConnectionReachedError.
Change-Id: Ia8450dc45dad5ceb4661807f51de66b5d70a6207
This is now folded into the initial migration of the 'expand_repo'
repository. Previously, this was a dummy migration. We simply move
things across and remove any code that was trying to work with the older
repo since it's no longer necessary.
A release note is added, even though it's not really necessary since
nothing will change for users. It's more of a heads up that things are
afoot.
Change-Id: I59882d88fe593ec1ae37415b2157584f7f3c85f8
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
These options were used by the memcache_pool backend for token
persistence, which was removed during Pike cycle.
Closes-Bug: #1941020
Change-Id: I2a0c2d46ebe81728f4ba0ff6d3072348e70f92dd
Ceph Object Gateway can use keystone for authenticating user requests
to its S3-compatible API, but recent versions also provide two other
AWS-compatible APIs for managing user access: Security Token Service
(STS) and Identity and Access Management (IAM). These attempt to
authenticate requests with Keystone but always receive 403 Access
Denied because _calculate_signature_v4() in api/s3tokens.py only
accepts "s3" as the service name. This patch accepts any of "s3" or
"sts" or "iam" as valid service names.
Change-Id: I69f16ed55dd9852859307b701a8391ba1e71c042
Closes-Bug: #1897280
This avoids the "String length exceeded." error, when using LDAP
domain specific backend in case the user uses a user id
attribute, which can exceed the previous constraint of 64 chars.
Change-Id: I923a2a2a5e79c8f265ff436e96258288dddb867b
Closes-Bug: #1929066
Resolves: rhbz#1959345
Previously, the fernet token provider would log warnings when a fernet
token exceeded 255 characters, which is common for LDAP-backed
deployments. The warning is always issued, even when operators configure
keystone's max_token_size to a higher value, causing confusion because
it appears the configuration value is silently ignored.
This commit fixes that issue by using the max_token_size configuration
parameter consistently in the fernet token provider.
Closes-Bug: 1926483
Change-Id: I4bb54aac9b950d59082a4468203a3249790839d7
This change hides the AccountLocked exception from being returned
to the end user to hide sensitive information that a potential
malicious person could gain insight from.
The notification handler catches the AccountLocked exception as
before, but after sending the audit notification, it instead
bubbles up Unauthorized rather than AccountLocked.
Co-Authored-By: Samuel de Medeiros Queiroz <samueldmq@gmail.com>
Change-Id: Id51241989b22c52810391f3e8e1cadbf8613d873
Related-Bug: #1688137
Keystone's update_user() method in the SQL driver processes a lot of
information about how to update users. This includes evaluating password
logic and authentication attempts for PSI-DSS. This logic is evaluated
after keystone pulls the user record from SQL and before it exits the
context manager, which performs the write.
When multiple clients are all updating the same user reference, it's
more likely they will see an HTTP 500 because of race conditions exiting
the context manager. The HTTP 500 is due to stale data when updating
password expiration for old passwords, which happens when setting a new
password for a user.
This commit attempts to handle that case more gracefully than throwing a
500 by detecting StaleDataErrors from sqlalchemy and retrying. The
identity sql backend will retry the request for clients that have
stale data change from underneath them.
Change-Id: I75590c20e90170ed862f46f0de7d61c7810b5c90
Closes-Bug: 1885753
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: Ic65d2fd6ce7215b4a47a6fb41b9cbf991f27773b
python-ldap3.0 or later running on python3 uses str or bytes
data type according to what fields are returned.
local_id may be a bytes data type.
To handle it properly, mapping[key] needs to be examined for
identifying its data type and what python version is used.
Closes-Bug: #1901654
Change-Id: Iac097235fd31e166028c169d14ec0937c663c21c
The application credential policies use the `rule:owner` policy to allow
users to manage their own credentials. The policy engine pulled the
user_id attribute from the request path instead of the actual
application credential. This allowed for users to exploit the
enforcement and view or delete application credentials they don't own.
This commit attempts to resolve the issue by updating the flask
parameters before they're translated to policy arguments and target
data, prior to policy enforcement.
Change-Id: I903d20fa41270499ca1c39d296120dd97cef5405
Closes-Bug: 1901207
Keystone's paging implementation contains a memory leak. The issue is
noticeable if you integrate keystone with an LDAP server that supports
paging and set `keystone.conf [ldap] page_size` to a low integer
(e.g., 5).
Keystone's LDAP backend uses `python-ldap` to interact with LDAP
servers. For paged requests, it uses `search_ext()`, which is an
asynchronous API [0]. The server responds with a message ID, which the
client uses to retrieve all data for the request. In keystone's case,
the `search_ext()` method is invoked with a page control that tells
the server to deliver responses in increments according to the page
size configured with `keystone.conf [ldap] page_size`. So long as the
client has the original connection used to fetch the message ID, it
can request the rest of the information associated to the request.
Keystone's paging implementation loops continuously for paged
requests. It takes the message ID it gets from `search_ext()` and
calls `result3()`, asking the server for the data associated with that
specific message. Keystone continues to do this until the server sends
an indicator that it has no more data relevant to the query (via a
cookie). The `search_ext()` and `result3()` methods must use the same
LDAP connection.
Given the above information, keystone uses context managers to provide
connections. This is relevant when deploying connection pools, where
certain connections are re-used from a pool. Keystone relies on Python
context managers to handle connections, which is pretty typical
use-case for context managers. Connection managers allow us to do the
following (assuming pseudocode):
with self.get_connection as conn:
response = conn.search_s()
return format(response)
The above snippet assumes the `get_connection` method provides a
connection object and a callable that implements `search_s`. Upon
exiting the `with` statement, the connection is disconnected, or put
back into the pool, or whatever the implementation of the context
manager decides to do. Most connections in the LDAP backend are
handled in this fashion.
Unfortunately, the LDAP driver is somewhat oblivious to paging, it's
control implementation, or the fact that it uses an asynchronous API.
Instead, the driver leaves it up to the handler objects it uses for
connections to determine if the request should be controlled via
paging. This is an anti-pattern since the backend establishes the
connection for the request but doesn't ensure that connection is
safely handled for asynchronous APIs.
This forces the `search_ext()` and `result3()` implementations in the
PooledLDAPHandler to know how to handle connections and context
managers, since it needs to ensure the same connection is used for
paged requests. The current code tried to clean up the context
manager responsible for connections after the results are collected
from the server using the message ID. I believe it does this because
it needs to get a new connection for each message in the paged
results, even though it already operates from within a connection
established via a context manager and the PooledLDAPHandler almost
always returns the same connection object from the pool. The code
tries to use a weak reference to create a callback that tears down the
context manager when nothing else references it. At a high-level, the
idea is similar to the following pseudocode:
with self.get_connection as conn:
while True:
ldap_data = []
context_manager = self.get_connection()
connection = context_manager.__enter__()
message_id = connection.search_ext()
results = connection.result3(message_id)
ldap_data.append(results)
context_manager.__exit__()
I wasn't able to see the callback get invoked or work as described in
comments, resulting in memory bloat, especially with low page sizes
which results in more requests. A weak reference invokes the callback
when the weak reference is called, but there are no other references
to the original object [1]. In our case, I don't think we invoke that
path because we don't actually do anything with the weak reference. We
assume it's going to run the callback when the object is garbage
collected.
This commit attempts to address this issue by using the concept of a
finalizer [2], which was designed for similar cases. It also attempts
to hide the cleanup implementation in the AsynchronousMessage object,
so that callers don't have to worry about making sure they invoke the
finalizer.
An alternative approach would be to push more of the paging logic and
implementation up into the LDAP driver. This would make it easier to
put the entire asynchronous API flow for paging into a `with`
statement and relying on the normal behavior of context managers to
clean up accordingly. This approach would remove the manual cleanup
invocation, regardless of using weak references or finalizer objects.
However, this approach would likely require a non-trivial amount of
design work to refactor the entire LDAP backend. The LDAP backend has
other issues that would complicate the re-design process:
- Handlers and connection are generalized to mean the same thing
- Method names don't follow a convention
- Domain-specific language from python-ldap bleeds into keystone's
implementation (e.g., get_all, _ldap_get_all, add_member) at
different points in the backend (e.g., UserApi (BaseLdap), GroupApi
(BaseLdap), KeystoneLDAPHandler, PooledLDAPHandler,
PythonLDAPHandler)
- Backend contains dead code from when keystone supported writeable
LDAP backends
- Responsibility for connections and connection handling is spread
across objects (BaseLdap, LDAPHandler)
- Handlers will invoke methods differently based on configuration at
runtime, which is a sign that the relationship between the driver,
handlers, and connection objects isn't truely polymorphic
While keeping the logic for properly handling context managers and
connections in the Handlers might not be ideal, it is a relatively
minimal fix in comparison to a re-design or backend refactor. These
issues can be considered during a refactor of the LDAP backend if or
when the community decides to re-design the LDAP backend.
[0] https://www.python-ldap.org/en/python-ldap-3.3.0/reference/ldap.html#ldap.LDAPObject.search_ext
[1] https://docs.python.org/3/library/weakref.html#weakref.ref
[2] https://docs.python.org/3/library/weakref.html#finalizer-objects
Closes-Bug: 1896125
Change-Id: Ia45a45ff852d0d4e3a713dae07a46d4ff8d370f3
This makes it easier for operators to troubleshoot connection issues to
Memcached.
Related-Bug: 1332058
Change-Id: I6e67363822480314b93608bb1eae3514f1480f6d
If LDAP returns a UUID as an octet string the LDAP driver will fail to
convert it to something meaningful. The error usually looks something
like:
ID attribute objectGUID not found in LDAP object
Microsoft AD's `objectGUID` parameter is stored and transmitted as an
octet string [0]. If you attempt to use the `objectGUID` to generate
user or group IDs, you'll get an HTTP 404 because keystone can't decode
it properly. This is unfortunate because `objectGUID` are a fixed
length, UUID format, and ideal for generating IDs in keystone. As
opposed to using the object's CN, which is variable length, and can
generate hashes that are larger than keystone's database table limit for
user IDs.
[0] https://docs.microsoft.com/en-us/windows/win32/ad/reading-an-objectampaposs-objectguid-and-creating-a-string-representation-of-the-guid
Change-Id: Id80b17bdff015e10340e636102576b7435bd564f
Closes-Bug: 1889936
Takashi Kajinami