Deal with tgt already exists errors

So there's a major problem in the iscsi code (and has been
for quite some time).  The tgt show command sometimes seems
to be corrupt and the issued command is actually a tgt create.
The result is that we already have a tgt and the call raises
which then causes the operation to fail.

An example of the issue:
  Stdout: Unexpected error while running command.Command:
  sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgt-admin
  --update iqn.2010-10.org.openstack:volume-f055d3c5-db7a-
  484e-9d0d-b98495439413
  Exit code: 22
  Stdout:
  Command:tgtadm -C 0 --lld iscsi --op new --mode target
  --tid 1 -T iqn.2010-10.org.openstack:volume-f055d3c5-db7a-
  484e-9d0d-b98495439413
  exited with code: 22.
  Stderr: u'tgtadm: this target already exists

What's disturbing however is that in that section of code
we're sending a --op show!!  Could be something we're
doing with our member executor?  Or maybe something to
do with the new oslo concurrency code?

Regardless, his patch intends to provide a clear marker for
ER in the case that create export fails due to the
target entry already existing.

Also this patch will enable us to go ahead and just use
the existing target rather than bomb out and fail everything.
Root cause of why we're getting a second create is still
unknown and needs addressed, but this might help in getting
more info as well as keeping things stable until we address
the root issue.

Change-Id: I0faf4d49b2d3e631b08ec1dff4361ff2376e3308
Partial-Bug: #1398078

cinder/tests/targets/test_tgt_driver.py

@@ -14,6 +14,7 @@ import tempfile
 
 import mock
 from oslo.utils import timeutils
+from oslo_concurrency import processutils as putils
 
 from cinder import context
 from cinder import test
@@ -197,6 +198,36 @@ class TestTgtAdmDriver(test.TestCase):
                 0,
                 self.fake_volumes_dir))
 
+    def test_create_iscsi_target_already_exists(self):
+        def _fake_execute(*args, **kwargs):
+            raise putils.ProcessExecutionError(
+                exit_code=1,
+                stdout='',
+                stderr='target already exists',
+                cmd='tgtad --lld iscsi --op show --mode target')
+
+        self.stubs.Set(self.target,
+                       '_execute',
+                       _fake_execute)
+
+        self.stubs.Set(self.target,
+                       '_get_target',
+                       lambda x: 1)
+
+        self.stubs.Set(self.target,
+                       '_verify_backing_lun',
+                       lambda x, y: True)
+
+        test_vol = 'iqn.2010-10.org.openstack:'\
+                   'volume-83c2e877-feed-46be-8435-77884fe55b45'
+        self.assertEqual(
+            1,
+            self.target.create_iscsi_target(
+                test_vol,
+                1,
+                0,
+                self.fake_volumes_dir))
+
     def test_create_create_export(self):
 
         def _fake_execute(*args, **kwargs):

cinder/volume/targets/tgt.py

@@ -232,13 +232,25 @@ class TgtAdm(iscsi.ISCSITarget):
                                        run_as_root=True)
             LOG.debug("Targets after update: %s" % out)
         except putils.ProcessExecutionError as e:
-            LOG.warning(_LW("Failed to create iscsi target for volume "
-                        "id:%(vol_id)s: %(e)s")
-                        % {'vol_id': vol_id, 'e': e})
+            if "target already exists" in e.stderr:
+                LOG.warning(_LW('Could not create target because '
+                                'it already exists for volume: %s'), vol_id)
+                # NOTE(jdg): We've run into issues where the command being sent
+                # was not correct. This may be related to using the executor
+                # directly? Even though the above call specified is a show
+                # we see a new being called instead...
 
-            # Don't forget to remove the persistent file we created
-            os.unlink(volume_path)
-            raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
+                # Adding the additional Warning message above for a clear
+                # ER marker (Ref bug: #1398078).
+                pass
+            else:
+                LOG.warning(_LW("Failed to create iscsi target for volume "
+                            "id:%(vol_id)s: %(e)s")
+                            % {'vol_id': vol_id, 'e': e})
+
+                # Don't forget to remove the persistent file we created
+                os.unlink(volume_path)
+                raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
 
         iqn = '%s%s' % (self.iscsi_target_prefix, vol_id)
         tid = self._get_target(iqn)