Parameterize iDRAC is ready retries at class level
Web Services Management (WS-Management and WS-Man) requests/commands can fail or return invalid results when issued to an integrated Dell Remote Access Controller (iDRAC) whose Lifecycle Controller remote service is not "ready". Specifically, that applies to the WS-Man Enumerate and Invoke operations. A Dell technical white paper [0], "Lifecycle Controller Integration -- Best Practices Guide", states that for Lifecycle Controller firmware 1.5.0 and later "The Lifecycle Controller remote service must be in a 'ready' state before running any other WSMAN commands." That applies to almost all of the workflows and use cases documented by that paper and supported by this project, openstack/python-dracclient. That document describes how to determine the readiness of the Lifecycle Controller remote service. This patch parameterizes the iDRAC is ready retry behavior at the class level. That makes it possible for consumers of this project, such as project openstack/ironic, to configure it library API-wide. Additionally, this patch improves the names of the parameters to class __init__() methods that control the retry behavior on SSL errors, so that they are not confused with those added by this patch. Finally, it defines constants for the default values of the retry behavior on SSL errors and iDRAC is ready retry parameters, and utilizes those new constants. [0] http://en.community.dell.com/techcenter/extras/m/white_papers/20442332 Change-Id: Ie866466a8ddf587a24c6d25ab903ec7b24022ffd Partial-Bug: #1697558 Related-Bug: #1691272 Related-Bug: #1691808
This commit is contained in:
Richard Pioso
parent
bb3313de14
commit
c75969dd8d
|
|
@ -18,6 +18,7 @@ Wrapper for pywsman.Client
|
|||
import logging
|
||||
import time
|
||||
|
||||
from dracclient import constants
|
||||
from dracclient import exceptions
|
||||
from dracclient.resources import bios
|
||||
from dracclient.resources import idrac_card
|
||||
|
|
@ -40,8 +41,14 @@ class DRACClient(object):
|
|||
|
||||
BIOS_DEVICE_FQDD = 'BIOS.Setup.1-1'
|
||||
|
||||
def __init__(self, host, username, password, port=443, path='/wsman',
|
||||
protocol='https', retries=3, retry_delay=0):
|
||||
def __init__(
|
||||
self, host, username, password, port=443, path='/wsman',
|
||||
protocol='https',
|
||||
ssl_retries=constants.DEFAULT_WSMAN_SSL_ERROR_RETRIES,
|
||||
ssl_retry_delay=constants.DEFAULT_WSMAN_SSL_ERROR_RETRY_DELAY_SEC,
|
||||
ready_retries=constants.DEFAULT_IDRAC_IS_READY_RETRIES,
|
||||
ready_retry_delay=(
|
||||
constants.DEFAULT_IDRAC_IS_READY_RETRY_DELAY_SEC)):
|
||||
"""Creates client object
|
||||
|
||||
:param host: hostname or IP of the DRAC interface
|
||||
|
|
@ -50,11 +57,17 @@ class DRACClient(object):
|
|||
:param port: port for accessing the DRAC interface
|
||||
:param path: path for accessing the DRAC interface
|
||||
:param protocol: protocol for accessing the DRAC interface
|
||||
:param retries: number of resends to attempt on failure
|
||||
:param retry_delay: number of seconds to wait between retries
|
||||
:param ssl_retries: number of resends to attempt on SSL failures
|
||||
:param ssl_retry_delay: number of seconds to wait between
|
||||
retries on SSL failures
|
||||
:param ready_retries: number of times to check if the iDRAC is
|
||||
ready
|
||||
:param ready_retry_delay: number of seconds to wait between
|
||||
checks if the iDRAC is ready
|
||||
"""
|
||||
self.client = WSManClient(host, username, password, port, path,
|
||||
protocol, retries, retry_delay)
|
||||
protocol, ssl_retries, ssl_retry_delay,
|
||||
ready_retries, ready_retry_delay)
|
||||
self._job_mgmt = job.JobManagement(self.client)
|
||||
self._power_mgmt = bios.PowerManagement(self.client)
|
||||
self._boot_mgmt = bios.BootManagement(self.client)
|
||||
|
|
@ -532,12 +545,17 @@ class DRACClient(object):
|
|||
|
||||
return self.client.is_idrac_ready()
|
||||
|
||||
def wait_until_idrac_is_ready(self, retries=24, retry_delay=10):
|
||||
def wait_until_idrac_is_ready(self, retries=None, retry_delay=None):
|
||||
"""Waits until the iDRAC is in a ready state
|
||||
|
||||
:param retries: The number of times to check if the iDRAC is ready
|
||||
:param retry_delay: The number of seconds to wait between retries
|
||||
|
||||
:param retries: The number of times to check if the iDRAC is
|
||||
ready. If None, the value of ready_retries that
|
||||
was provided when the object was created is
|
||||
used.
|
||||
:param retry_delay: The number of seconds to wait between
|
||||
retries. If None, the value of
|
||||
ready_retry_delay that was provided
|
||||
when the object was created is used.
|
||||
:raises: WSManRequestFailure on request failures
|
||||
:raises: WSManInvalidResponse when receiving invalid response
|
||||
:raises: DRACOperationFailed on error reported back by the DRAC
|
||||
|
|
@ -551,6 +569,37 @@ class DRACClient(object):
|
|||
class WSManClient(wsman.Client):
|
||||
"""Wrapper for wsman.Client with return value checking"""
|
||||
|
||||
def __init__(
|
||||
self, host, username, password, port=443, path='/wsman',
|
||||
protocol='https',
|
||||
ssl_retries=constants.DEFAULT_WSMAN_SSL_ERROR_RETRIES,
|
||||
ssl_retry_delay=constants.DEFAULT_WSMAN_SSL_ERROR_RETRY_DELAY_SEC,
|
||||
ready_retries=constants.DEFAULT_IDRAC_IS_READY_RETRIES,
|
||||
ready_retry_delay=(
|
||||
constants.DEFAULT_IDRAC_IS_READY_RETRY_DELAY_SEC)):
|
||||
"""Creates client object
|
||||
|
||||
:param host: hostname or IP of the DRAC interface
|
||||
:param username: username for accessing the DRAC interface
|
||||
:param password: password for accessing the DRAC interface
|
||||
:param port: port for accessing the DRAC interface
|
||||
:param path: path for accessing the DRAC interface
|
||||
:param protocol: protocol for accessing the DRAC interface
|
||||
:param ssl_retries: number of resends to attempt on SSL failures
|
||||
:param ssl_retry_delay: number of seconds to wait between
|
||||
retries on SSL failures
|
||||
:param ready_retries: number of times to check if the iDRAC is
|
||||
ready
|
||||
:param ready_retry_delay: number of seconds to wait between
|
||||
checks if the iDRAC is ready
|
||||
"""
|
||||
super(WSManClient, self).__init__(host, username, password,
|
||||
port, path, protocol, ssl_retries,
|
||||
ssl_retry_delay)
|
||||
|
||||
self._ready_retries = ready_retries
|
||||
self._ready_retry_delay = ready_retry_delay
|
||||
|
||||
def invoke(self, resource_uri, method, selectors=None, properties=None,
|
||||
expected_return_value=None):
|
||||
"""Invokes a remote WS-Man method
|
||||
|
|
@ -624,12 +673,17 @@ class WSManClient(wsman.Client):
|
|||
|
||||
return message_id == IDRAC_IS_READY
|
||||
|
||||
def wait_until_idrac_is_ready(self, retries=24, retry_delay=10):
|
||||
def wait_until_idrac_is_ready(self, retries=None, retry_delay=None):
|
||||
"""Waits until the iDRAC is in a ready state
|
||||
|
||||
:param retries: The number of times to check if the iDRAC is ready
|
||||
:param retry_delay: The number of seconds to wait between retries
|
||||
|
||||
:param retries: The number of times to check if the iDRAC is
|
||||
ready. If None, the value of ready_retries that
|
||||
was provided when the object was created is
|
||||
used.
|
||||
:param retry_delay: The number of seconds to wait between
|
||||
retries. If None, the value of
|
||||
ready_retry_delay that was provided when the
|
||||
object was created is used.
|
||||
:raises: WSManRequestFailure on request failures
|
||||
:raises: WSManInvalidResponse when receiving invalid response
|
||||
:raises: DRACOperationFailed on error reported back by the DRAC
|
||||
|
|
@ -637,6 +691,12 @@ class WSManClient(wsman.Client):
|
|||
:raises: DRACUnexpectedReturnValue on return value mismatch
|
||||
"""
|
||||
|
||||
if retries is None:
|
||||
retries = self._ready_retries
|
||||
|
||||
if retry_delay is None:
|
||||
retry_delay = self._ready_retry_delay
|
||||
|
||||
# Try every 10 seconds over 4 minutes for the iDRAC to become ready
|
||||
while retries > 0:
|
||||
LOG.debug("Checking to see if the iDRAC is ready")
|
||||
|
|
|
|||
|
|
@ -11,6 +11,15 @@
|
|||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
# iDRAC is ready retry constants
|
||||
DEFAULT_IDRAC_IS_READY_RETRIES = 24
|
||||
DEFAULT_IDRAC_IS_READY_RETRY_DELAY_SEC = 10
|
||||
|
||||
# Web Services Management (WS-Management and WS-Man) SSL retry on error
|
||||
# behavior constants
|
||||
DEFAULT_WSMAN_SSL_ERROR_RETRIES = 3
|
||||
DEFAULT_WSMAN_SSL_ERROR_RETRY_DELAY_SEC = 0
|
||||
|
||||
# power states
|
||||
POWER_ON = 'POWER_ON'
|
||||
POWER_OFF = 'POWER_OFF'
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ import mock
|
|||
import requests_mock
|
||||
|
||||
import dracclient.client
|
||||
from dracclient import constants
|
||||
from dracclient import exceptions
|
||||
from dracclient.resources import uris
|
||||
from dracclient.tests import base
|
||||
|
|
@ -104,6 +105,58 @@ class WSManClientTestCase(base.BaseTest):
|
|||
client = dracclient.client.WSManClient(**test_utils.FAKE_ENDPOINT)
|
||||
self.assertFalse(client.is_idrac_ready())
|
||||
|
||||
@mock.patch.object(dracclient.client.WSManClient, 'is_idrac_ready',
|
||||
autospec=True)
|
||||
@mock.patch('time.sleep', autospec=True)
|
||||
def test_wait_until_idrac_is_ready_with_none_arguments(
|
||||
self, mock_requests, mock_ts, mock_is_idrac_ready):
|
||||
ready_retries = 2
|
||||
ready_retry_delay = 1
|
||||
|
||||
side_effect = (ready_retries - 1) * [False]
|
||||
side_effect.append(True)
|
||||
mock_is_idrac_ready.side_effect = side_effect
|
||||
|
||||
fake_endpoint = test_utils.FAKE_ENDPOINT.copy()
|
||||
fake_endpoint['ready_retries'] = ready_retries
|
||||
fake_endpoint['ready_retry_delay'] = ready_retry_delay
|
||||
|
||||
client = dracclient.client.WSManClient(**fake_endpoint)
|
||||
client.wait_until_idrac_is_ready(retries=None, retry_delay=None)
|
||||
|
||||
self.assertEqual(mock_is_idrac_ready.call_count, ready_retries)
|
||||
self.assertEqual(mock_ts.call_count, ready_retries - 1)
|
||||
mock_ts.assert_called_with(ready_retry_delay)
|
||||
|
||||
@mock.patch.object(dracclient.client.WSManClient, 'is_idrac_ready',
|
||||
autospec=True)
|
||||
@mock.patch('time.sleep', autospec=True)
|
||||
def test_wait_until_idrac_is_ready_with_non_none_arguments(
|
||||
self, mock_requests, mock_ts, mock_is_idrac_ready):
|
||||
retries = 2
|
||||
self.assertNotEqual(retries, constants.DEFAULT_IDRAC_IS_READY_RETRIES)
|
||||
|
||||
retry_delay = 1
|
||||
self.assertNotEqual(
|
||||
retry_delay, constants.DEFAULT_IDRAC_IS_READY_RETRY_DELAY_SEC)
|
||||
|
||||
side_effect = (retries - 1) * [False]
|
||||
side_effect.append(True)
|
||||
mock_is_idrac_ready.side_effect = side_effect
|
||||
|
||||
fake_endpoint = test_utils.FAKE_ENDPOINT.copy()
|
||||
fake_endpoint['ready_retries'] = (
|
||||
constants.DEFAULT_IDRAC_IS_READY_RETRIES)
|
||||
fake_endpoint['ready_retry_delay'] = (
|
||||
constants.DEFAULT_IDRAC_IS_READY_RETRY_DELAY_SEC)
|
||||
|
||||
client = dracclient.client.WSManClient(**fake_endpoint)
|
||||
client.wait_until_idrac_is_ready(retries, retry_delay)
|
||||
|
||||
self.assertEqual(mock_is_idrac_ready.call_count, retries)
|
||||
self.assertEqual(mock_ts.call_count, retries - 1)
|
||||
mock_ts.assert_called_with(retry_delay)
|
||||
|
||||
def test_wait_until_idrac_is_ready_ready(self, mock_requests):
|
||||
expected_text = test_utils.LifecycleControllerInvocations[
|
||||
uris.DCIM_LCService]['GetRemoteServicesAPIStatus']['is_ready']
|
||||
|
|
|
|||
|
|
@ -162,9 +162,9 @@ class ClientTestCase(base.BaseTest):
|
|||
@requests_mock.Mocker()
|
||||
@mock.patch('time.sleep', autospec=True)
|
||||
def test_client_retry_delay(self, mock_requests, mock_ts):
|
||||
retry_delay = 5
|
||||
ssl_retry_delay = 5
|
||||
fake_endpoint = test_utils.FAKE_ENDPOINT.copy()
|
||||
fake_endpoint['retry_delay'] = retry_delay
|
||||
fake_endpoint['ssl_retry_delay'] = ssl_retry_delay
|
||||
client = dracclient.wsman.Client(**fake_endpoint)
|
||||
expected_resp = '<result>yay!</result>'
|
||||
mock_requests.post('https://1.2.3.4:443/wsman',
|
||||
|
|
@ -175,7 +175,7 @@ class ClientTestCase(base.BaseTest):
|
|||
{'selector': 'foo'}, {'property': 'bar'})
|
||||
|
||||
self.assertEqual('yay!', resp.text)
|
||||
mock_ts.assert_called_once_with(retry_delay)
|
||||
mock_ts.assert_called_once_with(ssl_retry_delay)
|
||||
|
||||
|
||||
class PayloadTestCase(base.BaseTest):
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import uuid
|
|||
from lxml import etree as ElementTree
|
||||
import requests.exceptions
|
||||
|
||||
from dracclient import constants
|
||||
from dracclient import exceptions
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
|
@ -41,7 +42,10 @@ class Client(object):
|
|||
"""Simple client for talking over WSMan protocol."""
|
||||
|
||||
def __init__(self, host, username, password, port=443, path='/wsman',
|
||||
protocol='https', retries=3, retry_delay=0):
|
||||
protocol='https',
|
||||
ssl_retries=constants.DEFAULT_WSMAN_SSL_ERROR_RETRIES,
|
||||
ssl_retry_delay=(
|
||||
constants.DEFAULT_WSMAN_SSL_ERROR_RETRY_DELAY_SEC)):
|
||||
"""Creates client object
|
||||
|
||||
:param host: hostname or IP of the DRAC interface
|
||||
|
|
@ -50,8 +54,9 @@ class Client(object):
|
|||
:param port: port for accessing the DRAC interface
|
||||
:param path: path for accessing the DRAC interface
|
||||
:param protocol: protocol for accessing the DRAC interface
|
||||
:param retries: number of resends to attempt on failure
|
||||
:param retry_delay: number of seconds to wait between retries
|
||||
:param ssl_retries: number of resends to attempt on SSL failures
|
||||
:param ssl_retry_delay: number of seconds to wait between
|
||||
retries on SSL failures
|
||||
"""
|
||||
|
||||
self.host = host
|
||||
|
|
@ -60,8 +65,8 @@ class Client(object):
|
|||
self.port = port
|
||||
self.path = path
|
||||
self.protocol = protocol
|
||||
self.retries = retries
|
||||
self.retry_delay = retry_delay
|
||||
self.ssl_retries = ssl_retries
|
||||
self.ssl_retry_delay = ssl_retry_delay
|
||||
self.endpoint = ('%(protocol)s://%(host)s:%(port)s%(path)s' % {
|
||||
'protocol': self.protocol,
|
||||
'host': self.host,
|
||||
|
|
@ -74,7 +79,7 @@ class Client(object):
|
|||
{'endpoint': self.endpoint, 'payload': payload})
|
||||
|
||||
num_tries = 1
|
||||
while num_tries <= self.retries:
|
||||
while num_tries <= self.ssl_retries:
|
||||
try:
|
||||
resp = requests.post(
|
||||
self.endpoint,
|
||||
|
|
@ -93,9 +98,9 @@ class Client(object):
|
|||
error_type=type(ex).__name__,
|
||||
host=self.host,
|
||||
num_tries=num_tries,
|
||||
retries=self.retries)
|
||||
retries=self.ssl_retries)
|
||||
|
||||
if num_tries == self.retries:
|
||||
if num_tries == self.ssl_retries:
|
||||
LOG.error(error_msg)
|
||||
raise exceptions.WSManRequestFailure(
|
||||
"A {error_type} error occurred while communicating "
|
||||
|
|
@ -107,8 +112,8 @@ class Client(object):
|
|||
LOG.warning(error_msg)
|
||||
|
||||
num_tries += 1
|
||||
if self.retry_delay > 0 and num_tries <= self.retries:
|
||||
time.sleep(self.retry_delay)
|
||||
if self.ssl_retry_delay > 0 and num_tries <= self.ssl_retries:
|
||||
time.sleep(self.ssl_retry_delay)
|
||||
|
||||
except requests.exceptions.RequestException as ex:
|
||||
error_msg = "A {error_type} error occurred while " \
|
||||
|
|
|
|||
Loading…
Reference in New Issue