openstack
/
heat-tempest-plugin
Code Issues Proposed changes

Avoid race in OSWaitCondition test 

While sending a signal to a WaitCondition is synchronous, the actual update
of the WaitConditionHandle metadata happens asynchronously since the fix
for bug 1394095. As a result, it's not guaranteed that even the first 6
signals (which are sent in serially, as opposed to the later ones which are
deliberately sent in parallel) will be stored in the same order that they
are sent.

Crucially, that means that one or more of the signals explicitly sent with
id 5 may arrive when there have been only three previous signals stored.
This means that the next signal to arrive with an implicit ID will be the
fifth signal stored, and therefore also get id 5. Of course we have a log
message to indicate when an existing signal is overwritten by another with
the same ID, and we are not seeing it except in the intended case where we
explicitly send in the same ID twice. That's because the keys have
different types in the data dict - the explicitly specified ID is the
string "5", but the implicitly calculated one is the integer 5. But - get
this - when we serialise the data to JSON both keys are serialised to the
string "5", and upon deserialisation they collide and one is silently
dropped on the floor.

So if the signal with the explicit ID "5" is stored just before the one
with reason "signal 4", then "signal 4" will effectively be silently
ignored as the 5th signal to arrive - a slot already filled. And since that
signal is ignored, the next signal will also be treated as the 5th to
arrive and ignored, and so on. This leads inexorably to the dreaded
"WaitConditionTimeout: resources.wait_condition: 4 of 25 received" error.

For this reason, it's a bad idea to mix explicit IDs that are also integers
with implicitly assigned IDs. Use an ID that won't collide instead.

Change-Id: I507b43dba8dcea87d3e0c179f7ca6b34b2b31a12
Closes-Bug: #1738653
This commit is contained in:
Zane Bitter 2018-02-09 19:57:44 -05:00
parent 9b02d1e662
commit 2cff12bceb
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
heat_tempest_plugin/tests/functional/test_os_wait_condition.py
View File

@ -57,11 +57,11 @@ resources:
wc_notify --data-binary ''{"status": "SUCCESS", "reason":
"signal4", "data": "data4"}''
# check signals with the same number
# check signals with the same ID
wc_notify --data-binary ''{"status": "SUCCESS", "id": "5"}''
wc_notify --data-binary ''{"status": "SUCCESS", "id": "test5"}''
wc_notify --data-binary ''{"status": "SUCCESS", "id": "5"}''
wc_notify --data-binary ''{"status": "SUCCESS", "id": "test5"}''
# loop for 20 signals without reasons and data