Previously token cache was not correctly handling the case when data
in memcached is un-decryptable.
The cache process was returning a null value that was not considered
resulting a python exception raised
The commit fixes the issue by adding a condition to validate the value
returned.
Closes-bug: #2023015
Change-Id: Ic48d20569980781febc194083651736bed446953
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@industrialdiscipline.com>
In past days there were discussions about various issues
with memcached connections [1][2][3].
After investigation it looks like common root cause for above
problems is keystonemiddleware. More precisely said the way
how keystonemiddleware is caching tokens.
Currently it's using some home-made CachePool with direct
usage of memcached library, moreover it looks like its
approach is not eventlet-safe.
Discussion can be mainly found in [4].
Fortunately keystonemiddleware can use "advanced cache pool",
which is oslo.cache's implementation and was added long time ago [5],
but it is turned on only if memcache_use_advanced_pool=True.
This patch is switching to more elaborated oslo.cache CachePool
and adding deprecation warning about eventlet-unsafe variant
of keystonemiddleware's memcache pool.
How to reproduce ?
with memcache_use_advanced_pool=False
1. Build clean ENV of openstack
2. Deploy core projects (keystone,glance,nova,placement...)
3. Run while true; do COMMAND FOR SERVICE; done
- several bashes, in parallel (5-7)
COMMAND FOR SERVICE:
- openstack network list
- openstack volume list
- openstack server list
- openstack image list
4. Check memcached connections (which will grow up):
- ss | grep 11211 | wc -l every second
How to fix and test it ?
Repeat above, to fix:
- with memcache_use_advanced_pool=True
OR
- apply this patch
Compare measurements in graph.
[1] https://bugs.launchpad.net/keystonemiddleware/+bug/1892852
[2] https://bugs.launchpad.net/oslo.cache/+bug/1888394
[3] https://bugs.launchpad.net/keystonemiddleware/+bug/1883659
[4] https://review.opendev.org/c/openstack/oslo.cache/+/742193
[5] https://review.opendev.org/c/openstack/keystonemiddleware/+/268664
Closes-Bug: #1883659
Closes-Bug: #1892852
Closes-Bug: #1888394
Change-Id: I0e96334b65a0bf369ebf1d88651d13feb8d2ecac
Keystonemiddleware's abstraction for the memcache pool was broken
when converting to use a queue.Queue. The logic that placed the
connection back into the pool was moved to .acquire and the reserve
method was not using acquire.
Change-Id: I0eda5981cbb661f63790258cf8e70c7340615159
Closes-Bug: #1782404
The memcache client class actually has no __exit__ function.
Remove the "with" usage to avoid the __exit__ error.
Change-Id: I15b3d08f4afae289e7eb0848ff1db08141196d3c
Closes-Bug: #1747565
Now keystonemiddleware use oslo.cache to init the
MemcacheClientPool. The MemcacheClientPool in
Olso.cache needs (urls, arguments, **kwargs) parameter
to init, but keystonemiddleware passed only
(urls, **kwargs). Then it leads the error:
__init__() takes exactly 3 arguments (2 given)
This patch fixed this issue.
Please note that even this error is fixed, set
"memcache_use_advanced_pool = True" will lead another
error, see bug #1747565 for the detail. It will be
fixed in the following patch.
Closes-bug: #1748160
Change-Id: I642f959ab8b010207314312a6b6a06a6de23e92c
In continuation of I00e953abb3e835a94353fe458100c96e8e9c095a,
this change adds the release note and documentation.
Related-bug #1737115
Change-Id: I456239842d139074cc38cfd620bb88561bb4d0d7
Now, we depend on oslo.cache [1], and use the private/internal
memcache_pool code of the lib, making oslo.cache failing to import
instead of just log an error about missing requirement for selected
drivers at runtime.
This change restores the previous behavior by lazy loading the module.
[1] 9d8e2836fe
Change-Id: I00e953abb3e835a94353fe458100c96e8e9c095a
Closes-bug: #1737115
Use the new oslo.cache library instead of using memcached directly.
This keeps the old options around and will continue to use those in
preference to the oslo.config library as there is no way to test whether
oslo.cache was explicitly configured to use that in preference.
Currently there are no messages or anything to deprecate the old options
until we've had a chance to test it in production environments.
Closes-Bug: #1523375
Change-Id: Ifccacc5db311ad538ce60191cbe221644d1a5807
Co-Authored-By: Nicolas Helgeson <nh202b@att.com>
Oslo-incubator has been deprecated. The only thing that we use from it
is the memorycache which is hopefully going away soon. Copy this
memorycache code into the _cache module so that we can refactor it as
necessary without worrying about oslo-incubator.
Involves some minor cleanups for pep8 fixes and making functions
private.
Change-Id: I7a19d4ded8b538b6ea02e4a08068c863705194a3
Currently tox ignores D204, D209 and D301:
D204: 1 blank line required after class docstring.
D209: Multi-line docstring closing quotes should be on a separate line.
D301: Use r""" if any backslashes in a docstring.
This change makes keystonemiddleware docstrings compliant with D204.
D209 and D301 is already passing, so this commit also enables them.
Change-Id: I11e02ef5af7fc793f1a2438e091bbfb18618a7f5
Move the logic that handles invalid cache entries out of the cache
object itself. This makes the cache interface more normal and will make
it easier to convert to oslo.cache later.
Rename the store method to set because this is the interface that
oslo.cache uses.
Related-Bug: #1523375
Change-Id: Ia7b11a69f14e17a9d7a36c009d9a965be16509d0
For a long time now if you don't configure memcache then
auth_token middleware would cache the tokens in process
memory.
This is not the job of auth_token middleware. If you need to
cache you should configure memcache otherwise auth_token will
authenticate with keystone for every token request.
As such, this feature is deprecated and may be removed in the
5.0.0 release or the "O" development cycle (whichever is later).
Change-Id: Ied2b88c8cefe5655a88d0c2f334de04e588fa75a
This reverts commit f27d7f776e.
This change broke the gate due to causing timeouts. The
functionality needs to go through the normal deprecation before
being removed.
Change-Id: I4ab07f6e2bd5bd084bd16707126728929a4ba0f7
For a long time now if you don't configure memcache then auth_token
middleware would cache the tokens in process memory.
This is not the job of auth_token middleware. If you need to cache you
should configure memcache otherwise auth_token will authenticate with
keystone for every token request.
Change-Id: Idf7d864fe8b054738d8a240bc3da377a95eb7e62
The oslo_config library added support for a choices keyword argument in
version 1.2.0a3. This commit leverages the use of choices for StrOpts of
keystonemiddleware configuration.
Change-Id: I8d9ee833263560caaffe083487abc5eda862f8ea
Closes-Bug: 1423973
Previously, there were a string of commits to keystone that addresed ignored
hacking checks. This commit does the same for H405 in keystonemiddleware. This
also modifies our tox.ini so that we no longer ignore H405 violations.
This is a non-functional change.
Change-Id: I7bbe99719feb39e96634c903991294c18c33112b
Closes-Bug: 1482773
_memcache_pool is imported within a function so that we don't have a hard
dependency on memcache. This is a good idea but _memcache_pool doesn't
have a hard dependency on memcache either so there's no reason to
protect against this here.
Change-Id: I81418fc9fd41e40b9d508ad937d157b2f9ec44c1
We have three distinct ways that a cache pool can be created, however
only two cache pool classes with the regular _CachePool doubling up.
There's no reason for this and it's harder to read.
Create a very simple _EnvCachePool for the situation where a cache
object was passed down through environment variables.
Change-Id: I86147e887efd6ae302caa12ce996be697d36aa17
Tracing the usage of memcache pool arguments goes through 3 different
functions and involves slightly tweaking the parameter names each time.
This was making it impossible to reason with so collapse down the
memcache arguments into a dictionary that is passed around.
Change-Id: I8bd5cb50183795d9b8fd953dabd935f15f03a385
Do not assume a token_id will result in a sane length for a memcache
key length. In cases such as Fernet, these ids can easily exceed the
limit on memcache key size. This change ensures we always use a SHA256
of the token id passed in, resulting in a fixed length cache key.
Change-Id: I550e0a1b190047438756bbf40490815a5f177ea7
Closes-Bug: #1460225
There are two times related to storing token values in memcache. The
memcache timeout and the expiry that is saved alongside the token. We
purposefully set the timeout to be longer than the expiry so that we can
cache that we have seen an expired token.
The expiry that is saved with the token is taken from within the token
body itself and tested for expiry before being returned from the cache.
This expiry is still present in the token body though and it means that
we have two different but exactly the same paths for where we test
expiry from.
Change the cache so that it always returns the full token body from the
cache. This way it is a drop in replacement for fetching the body from
keystone server or from within PKI and the token data is validated in
one place.
Change-Id: Ibe9816826e7d78b7ae25085f07ff9a2f18db9db0
We hash a token under multiple configurable algorithms for revocations
and caching and so must check all of them. This hash generation was
being done and returned by the cache check and reused in revocation
check. This is an unusual pattern and requires the cache object to have
knowledge of token types and how to hash them.
Change validate so that we generate the hash values in the main function
and pass that to the cache and revocation functions. This moves the
function to get the first available token id from the list to the main
file. This means the cache interface is much more normal with get and
set id functions and encapsulates the hash list function in the one
file.
Change-Id: I9194dbd052674f64122ff74329ce292a342512d3
On Python 3, memcache returns data as Unicode, but the hmac module
expects bytes. Encode data to UTF-8.
Fix also DisableModuleFixture.clear_module() on Python 3. dict.keys() is
now an iterator on Python 3: create a list because sys.modules is
modified in the loop body.
Closes-Bug: #1449423
Change-Id: Id4805c01b0127a3a86235a7d1a4bb3863b51a6b6
Extract the two types of TokenCache and its helper classes to their
own file.
Change-Id: If8d703597b4dd1c578eaf5adc24e97962d5f08a2
Implements: bp refactor-extract-module
Sahid Orentino Ferdjaoui