openstack
/
nova
Code Issues Proposed changes

XenAPI: resolve VBD unplug failure with VM_MISSING_PV_DRIVERS error 

The virtual block device(VBD) hot unplugging should be done when VM has
paravirtualization(PV) driver installed. But we observed failures with
VM_MISSING_PV_DRIVERS error on a real PV VM sometime.

Actually it's a race condition. At VM booting, it needs some time for
the PV driver to be connected. If the VBD.unplug operation goes before
the PV driver connected to xen, it will failed with VM_MISSING_PV_DRIVERS.

The fix is to retry unplug. If really there is no PV driver installed in
VM, failed after the attempts.

Change-Id: I6b788ba24aa3e7c061dac06c52570ff3818bc91a
Closes-Bug: #1727134
This commit is contained in:
Jianghua Wang 2017-10-25 03:04:16 +00:00
parent 8352d555a5
commit e18608d579
2 changed files with 28 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
nova/tests/unit/virt/xenapi/test_vm_utils.py
View File

@ -913,6 +913,11 @@ class UnplugVbdTestCase(VMUtilsTestBase):
self._test_uplug_vbd_retries(mock_sleep,
"INTERNAL_ERROR")
@mock.patch.object(greenthread, 'sleep')
def test_uplug_vbd_retries_on_missing_pv_drivers_error(self, mock_sleep):
self._test_uplug_vbd_retries(mock_sleep,
"VM_MISSING_PV_DRIVERS")
class VDIOtherConfigTestCase(VMUtilsTestBase):
"""Tests to ensure that the code is populating VDI's `other_config`

32
nova/virt/xenapi/vm_utils.py
View File

@ -316,15 +316,29 @@ def is_enough_free_mem(session, instance):
def _should_retry_unplug_vbd(err):
# Retry if unplug failed with DEVICE_DETACH_REJECTED
# For reasons which we don't understand,
# we're seeing the device still in use, even when all processes
# using the device should be dead.
# Since XenServer 6.2, we also need to retry if we get
# INTERNAL_ERROR, as that error goes away when you retry.
return (err == 'DEVICE_DETACH_REJECTED'
or
err == 'INTERNAL_ERROR')
"""Retry if failed with some specific errors.
The retrable errors include:
1. DEVICE_DETACH_REJECTED
For reasons which we don't understand, we're seeing the device
still in use, even when all processes using the device should
be dead.
2. INTERNAL_ERROR
Since XenServer 6.2, we also need to retry if we get INTERNAL_ERROR,
as that error goes away when you retry.
3. VM_MISSING_PV_DRIVERS
NOTE(jianghuaw): It requires some time for PV(Paravirtualization)
driver to be connected at VM booting, so retry if unplug failed
with VM_MISSING_PV_DRIVERS.
"""
can_retry_errs = (
'DEVICE_DETACH_REJECTED',
'INTERNAL_ERROR',
'VM_MISSING_PV_DRIVERS',
)
return err in can_retry_errs
def unplug_vbd(session, vbd_ref, this_vm_ref):