In past days there were discussions about various issues
with memcached connections [1][2][3].
After investigation it looks like common root cause for above
problems is keystonemiddleware. More precisely said the way
how keystonemiddleware is caching tokens.
Currently it's using some home-made CachePool with direct
usage of memcached library, moreover it looks like its
approach is not eventlet-safe.
Discussion can be mainly found in [4].
Fortunately keystonemiddleware can use "advanced cache pool",
which is oslo.cache's implementation and was added long time ago [5],
but it is turned on only if memcache_use_advanced_pool=True.
This patch is switching to more elaborated oslo.cache CachePool
and adding deprecation warning about eventlet-unsafe variant
of keystonemiddleware's memcache pool.
How to reproduce ?
with memcache_use_advanced_pool=False
1. Build clean ENV of openstack
2. Deploy core projects (keystone,glance,nova,placement...)
3. Run while true; do COMMAND FOR SERVICE; done
- several bashes, in parallel (5-7)
COMMAND FOR SERVICE:
- openstack network list
- openstack volume list
- openstack server list
- openstack image list
4. Check memcached connections (which will grow up):
- ss | grep 11211 | wc -l every second
How to fix and test it ?
Repeat above, to fix:
- with memcache_use_advanced_pool=True
OR
- apply this patch
Compare measurements in graph.
[1] https://bugs.launchpad.net/keystonemiddleware/+bug/1892852
[2] https://bugs.launchpad.net/oslo.cache/+bug/1888394
[3] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659
[4] https://review.opendev.org/c/openstack/oslo.cache/+/742193
[5] https://review.opendev.org/c/openstack/keystonemiddleware/+/268664
Closes-Bug: #1883659
Closes-Bug: #1892852
Closes-Bug: #1888394
Change-Id: I0e96334b65a0bf369ebf1d88651d13feb8d2ecac
In [0] the ``interface``option was added in order to allow the Identity
endpoint that is being used when validating tokens to be
configured by the deployer. Change the default to using the internal
endpoint, as that should be what most deployments will end up using.
[0] https://review.opendev.org/651790
Depends-On: https://review.opendev.org/651492
Closes-Bug: 1830002
Change-Id: I0ce8b6d8cd408c7fac8107972e7be70839e337fb
This commit adds a validation step in the auth_token middleware to check
for the presence of an access_rules attribute in an application
credential token and to validate the request against the permissions
granted for that token. During token validation it sends a header to
keystone to indicate that it is capable of validating these access
rules, and not providing this header for a token like this would result
in the token failing validation. This disregards access rules for a
service request made by a service on behalf of a user, such as nova
making a request to glance, because such a request is not under the
control of the user and is not expected to be explicitly allowed in the
access rules.
bp whitelist-extension-for-app-creds
Depends-On: https://review.opendev.org/670377
Change-Id: I185e0541d5df538d74edadf9976b3034a2470c88
Keystone server no longer supports PKI/PKIZ. This change removes
keystonemiddleware's support of PKI/PKIZ and associated code.
Change-Id: I9a6639a2aa3774be61972d57f38220f66fd5c0e8
closes-bug: #1649735
partial-bug: #1736985
Previously the admin Identity endpoint was hardcoded to be used. Now
that keystone has dropped v2 support, deploying an admin Identity
endpoint is no longer useful, so allow this to be changed by the
deployer. Keep the default as using the `admin` endpoint, but create
a deprecation message so that we can change the default in the future.
Partial-Bug: 1830002
Change-Id: I993a45ccb1109d67e65bf32d1e134cc9bec2d88e
With keystone's move to eliminating pki, pkiz, and uuid tokens the
revocation list is no longer generated. Keystonemiddleware no longer
needs to attempt to retrieve it and reference it.
Change-Id: Ief3bf1941e62f9136dbed11877bca81c4102041b
closes-bug: #1361743
partial-bug: #1649735
partial-bug: #1736985
The [keystone_authtoken]/auth_uri middleware parameter has been causing
extreme confusion amongst operators and developers ever since the
keystonemiddleware started accepting keystoneauth plugin parameters
including auth_url. The two parameters look identical and yet have
completely different meanings and are both required. This patch
deprecates auth_uri and renames it to www_authenticate_uri, which more
accurately describes the WWW-Authenticate header it is configuring and
is dissimilar to any other keystone_authtoken middleware parameter. This
also renames the internal variable names for consistency with the config
option.
Change-Id: I0cf11da3d395749df28077427689fdafc8a6b981
When user generate config using oslo genconfig, the comments in the
keystone middleware section contains double spaces between some words.
This patch set removes the extra spaces in the help comment for
auth_uri.
TrivialFix
Change-Id: I4ac514df6d2d229de1b04557771ddb0253c5baa0
When a service token is present we should bypass the expiry checks and
pass the allow_expired flag to the server. This will let the server
return expired tokens.
This has a very basic policy enforcement that is not backwards
compatible with the current (sensible) default. We will need to discuss
how we can make this work.
Implements bp: allow-expired
Change-Id: If3583ac08e33380f1c52ad50d7d5c74194393480
The PKI token option is in the process of removal from keystone,
therefore the options in keystonemiddleware can be deprecated.
Change-Id: I4668476b03c8978ce6468552fb3a7a3695a0b2ae
By default oslo.cfg sets the default value as None. There is no
need to explicitly do this.
TrivialFix
Change-Id: Icba893180617ab03e42b477706ce089cc5c5b5d1
The _conf_get ugliness in auth_token middleware has been around for a
long time now to handle the abstraction from different oslo.config
options and the paste overrides. This logic is now also being needed in
other middlewares. Extract this into a common config object that has a
better interface and is easier to work with.
Change-Id: I8b8a1427bc527e43bb1baec25a881d93df3f93cc
The intent of providing the list_auth_token_opts function was to provide
the oslo_config sample config file generator a list of options to
include in its sample files. However, services like zaqar have come to
rely on the list_auth_token_opts to list all the options that may be
consumed by auth_token middleware so that they can register them against
a non-global oslo_config object.
By removing deprecated options from the list_auth_token_opts we remove
these options from the config objects that the services use, however by
keeping them we will forever have deprecated options in sample config
files.
To split these two functionalities create a new function that lists the
options available for sample config files and update the entrypoint to
reflect this. This function is currently private because it should only
need to be accessed via entrypoint. The old deprecated options are then
added back to the original list_auth_token_opts function.
Closes-Bug: #1533932
Change-Id: I2aae5483c9309ab75985298c8de5b6f24cbc0f0d
Michal Arbet