The fasteners lib in version 0.15.0 removed the
threading.current_thread workaround for eventlet[1] because eventlet
seemed to fixed the current_thread issues tracked in [2]. However the
fix for [2] only fixed half of the problem. The threading.current_thread
call works if it is called from thread created by eventlet.spawn.
However if the thread is created with eventlet.spawn_n then
threading.current_thread is still broken and returns the ID of the
python native thread.
The fasteners' ReaderWriterLock depends heavily on
threading.current_thread to decide which thread holds a lock and to
allow re-entry of that thread. This leads to the situation that
multiple threads created from spawn_n could take the same
ReaderWriterLock at the same time.
The fair internal lock in oslo.concurrency uses ReaderWriterLock and
as a result such lock is broken for threads created with spawn_n.
Note that this issue was raised with eventlet in [3] when the nova team
detected it via a direct usage of ReaderWriterLock in the nova test
code. As [3] did not lead to a solution in eventlet nova implemented a
nova local fix for the test code in [4].
However now we detected that oslo.concurrency is affected by this issue
as well.
This patch restores the workaround that was removed by [1].
Note that a fasteners issue [5] also opened to restore the
workaround[1].
[1] 467ed75ee1
[2] https://github.com/eventlet/eventlet/issues/172
[3] https://github.com/eventlet/eventlet/issues/731
[4] https://review.opendev.org/c/openstack/nova/+/813114
[5] https://github.com/harlowja/fasteners/issues/96
Closes-Bug: #1988311
Change-Id: Ia873bcc6b07121c9bd0b94c593567d537b4c1112
We currently have no log entry when we start trying to acquire a lock.
In general this is ok, but there are cases where it can be problematic,
for example if we have a deadlock situation or if a lock takes a very
long time to be acquired.
In those scenarios looking at the logs we would see the operation
proceed normally and suddenly go completely silent without knowing that
it's waiting for a lock to be freed.
This patch adds a debug log message right before trying to acquire the
lock so we can detect those situations.
Change-Id: I1354dfb98b0927ae167802ecc4ab1d34f6b4d720
When lock_file is not present on the [oslo_concurrency] section of the
config file, we are getting a misleading error saying that it is not
preset on the DEFAULT group. This commit fixes it by providing the group
arg to the RequiredOptError constructor.
Closes-Bug: #1966320
Signed-off-by: Thiago Brito <thiago.brito@windriver.com>
Change-Id: Idee6987739917c7eae4faee15a66085a9ef7f6d2
This adds support for a non-blocking behavior of the lock : the context-based
lock now raises an AcquireLockFailedException if the lock can't be acquired on
the first try.
At a higher layer, we just caught this exception inside the 'synchronized'
decorator to prevent calling the wrapped function. In which case, we then
trace this acquisition failure.
For now at least, disabling blocking is not supported when using fair locks
because the ReaderWriterLock.write_lock() provided by the fasteners module
doesn't implements this behavior.
Change-Id: I409da79007c9ba4fb8585da881e3d56998b0b98b
There may be cases were library users want to request the removal of an
external lock that may not have been created and don't want to pollute
the logs when the file is not present.
We raise the log level from info to warning, but we don't log it if the
file didn't exist, since the end result the caller wanted is there.
Change-Id: I5ce8be34c9f2c4c59ea99dabc6760c3300f743a3
Following up on comments in [1], clarify and unify the docs for the
following lockutils methods:
- lock_with_prefix
- synchronized_with_prefix
- remove_external_lock_file_with_prefix
[1] I4e723ee3be1e57c543684390b607c84388c6e930
Change-Id: I4179e8732dba7367bd0c835cbd11939ca7b8cc42
There's a convenience wrapper around the lockutils.synchronized
decorator that lets oslo.concurrency consumers set up a prefix to use
for all lock files (presumably to obviate collisions with other
consumers when using jejune lock names, like 'lock'). This commit adds
an equivalent wrapper around lockutils.lock, the context manager
counterpart to the lockutils.synchronized decorator.
Note that the unit test added for lock_with_prefix is pretty bare; but
it follows the precedent set by the existing tests. Future commits
should make all these tests more thorough/robust.
Change-Id: I4e723ee3be1e57c543684390b607c84388c6e930
This adds support for a "fair" variant of the lock. When there are
multiple entities within a single process that are blocked waiting
for the lock the fair lock will ensure that they acquire the lock
in FIFO order.
For now at least, when fair locks are in use we don't support
the "semaphores" argument.
If external locks are enabled, the inter-process ordering will be
determined by the underlying OS lock ordering and process scheduling.
Change-Id: I37577becff4978bf643c65fa9bc2d78d342ea35a
According to Openstack summit session [1],
stestr is maintained project to which all Openstack projects should migrate.
Let's switch to stestr as other projects have already moved to it.
[1] https://etherpad.openstack.org/p/YVR-python-pti
Change-Id: I753f32ecf3275cf49d8c93bf648a6a26bc6da8e7
In a case when a number of threads are concurrently accessing a
resource that has a external lock there is no indication when
one of the threads has taken the external lock.
This patch adds in a debug log to try and add some clarity.
Change-Id: Ibd7cfd3b39370b1a54f2ab50c75c6b92a876ae1a
This reverts commit 5021ef82fd.
That commit set default location for lock files to a world-writable
directory (like /tmp) which violates security precaution stated right in
the help string of lock_path config variable: lock_path should be
writable by the user of the process using it only.
There is some history behind this:
- when we decided to not do tempfile.mkdtemp if lock_path were not set
(because it was broken), it broke upgrades for all components
(starting with Cinder;
- it was suggested to provide the same default value:
https://review.openstack.org/60274
but it was not accepted because it weakens security;
- there was a thread about this in ML, please read it for more history:
http://lists.openstack.org/pipermail/openstack-dev/2013-December/021055.html
it ended up with understanding that only a project using the library
can provide sane default value for lock_path, not the library itself.
Change-Id: I801004ad3c83862849696f00c6e7ca9877fd496b
It seems we provide (and projects use) the provided decorator
'synchronized_with_prefix' but we don't provide an equivalent function
to complement that function to clean up its created lock files.
Providing a complementary method seems pretty useful for projects
that actually clean up these lock files (if any actually do in the
first place).
Change-Id: I601ce3992411e6a2ddded13aba4ac068cf8f14e2
Document in which version new types and functions were added using
".. versionadded:: x.y". Document changes using
".. versionchanged:: x.y."
For watchdog module, add the versionadded tag in the module top
docstring, not on each type/function.
Add docstring to ssh_execute().
I used "git blame" + "git tag --contains=SHA1" to find these version,
and then I checked manually each version.
Change-Id: I56a7d8a4335833e9b49eedab026d20adfeedf942
Using the utility function gets a better name.
For example:
$ python
>>> from oslo_utils import reflection
>>> class A(object):
... def m(self):
... pass
...
>>> z = A()
>>> reflection.get_callable_name(z.m)
'__main__.A.m'
Versus:
>>> z.m.__name__
'm'
Change-Id: I2daadd969383aaf49ad87876ba108dd80dd56f08
To ensure we show accurate timing information about
lock acqusition and release times use a timing mechanism
which can not/should not move backwards.
Change-Id: I9559b20cf7de67fc474e6e17eda23791ecc4122e
The fasteners library (extracted from this library and a couple other
variations) provides the interprocess lock logic and the reader writer
lock logic so we can remove the local version and we can just use it
from that library instead.
The tests that were ensuring the internals of this file lock have
now moved to the repo where that library is (for the time being),
currently travis is testing that repo against py2.6, py2.7 and py3.4.
https://github.com/harlowja/fasteners/tree/master/fasteners/tests
Docs also exist at:
http://fasteners.readthedocs.org/en/latest/
Change-Id: I98565b22e68358efe28fea62f74f8ebfcc438ff7
Provide an API to discover the path being used to store external lock
files. Tempest will use this to set up some additional locking and to
test its own behavior. Other projects might use it for error reporting
or other purposes.
Change-Id: Iad40c67072333cc25a6d3e39d7535ff14b573504
Taskflow has a reader/writer lock that is likely useful
to other projects; and it seems better at home in this
module.
The class provides a way to create reader/writer locks
where there may be many readers at the same time (but
only one writer). It does not allow (currently) for privilege
escalation (but this could be added with limited support
in the future).
Change-Id: Ie763ef92f31c34869d83a533bc8761b0fbd77217
In Neutron, we usually specify lock_path as:
lock_path = $state_path/lock
It turned out that ConfigFilter does not work correctly with
substitution feature, so its usage blocks Neutron adoption of the
library.
The issue is also not specific to Neutron: any deployer should be able
to use substitutions ($smth) in their configuration files without
determining whether those configuration options he's interested in are
defined as ConfigOpts or ConfigFilter.
So the best short term approach to push forward the adoption of the
library is to refrain from using ConfigFilter at least until there is
consensus on how to support substitution feature with the class.
Once oslo.config sorts the issue out, we may get back to using
ConfigFilter for lockutils.
Change-Id: I7b0b5c840381b2d9afc00ff116fee1d970743183
Closes-Bug: #1399897
Sometime in a future retrying version there will
likely/hopefully exist a ability to more easily
filter on specific exceptions; this commit adds a
note to the upstream pull request that may add this
feature.
Change-Id: I44f1c9fe4fbbd5f77032ca7bfff4e2e6b7ff7622
When a lock can't be acquired there is currently a hard coded
delay (0.01) that is used before trying again, instead of having
a hard coded delay we should allow this delay to be configured
since having it set at a hard coded value can limit concurrency (if
the delay is actually way to high) or cause to much contention (if
the delay is actually way to low).
This review adds on that logic and also uses the retrying library
to perform the acquisition attempts (and associated failures when/if
they occur); as well as shows logs after a given amount of time has
elapsed with the logs being output at a given periodicity.
Change-Id: Ideeefba1439ddd677c608d01becb4f6a0d4bc83d
It can be undesirable to have a globally shared sempahore
container, especially since oslo.concurrency can now be shared
among many disjoint applications and libraries.
When a single container is used it is now possible to have those
disjoint applications/libraries collide on the same sempahore names.
This is not a good pattern to continue with, so in order to move away
from it allow a custom container to be provided (which defaults to the
existing global one) so that users of oslo.concurrency may provide there
own container if they so desire.
Change-Id: I9aab42e21ba0f52997de3e7c9b0fea51db5c7289
Move the public API out of oslo.concurrency to oslo_concurrency. Retain
the ability to import from the old namespace package for backwards
compatibility for this release cycle.
bp/drop-namespace-packages
Change-Id: I20d1647b1c3ef8cab3b69eccfe168eeb01703b72
Takashi Kajinami