fix race with syslog while renaming remote log dir

When a node gets new name nailgun renames log directory which had name
in form of <IP> to <node name> and then creates symlink <IP> ->
<node name>. The problem is that node is still active at this moment
and sends some logs to syslog on master node. So when we renamed <IP>
to <node name> and haven't created symlink name syslog may find
missing log dir and it creates it again in this case. Then nailgun
tries to create symlink and fails.

To avoid this situation this patch adds attempts to remove a directory
and create symlink until symlink will being successfully created.

Approach used in master does not work in <=8.0. In MOS=<8.0 nailgun
and rsyslog work in different PID namespaces and can not interact
with each other by signals.

Change-Id: Ia402508dac427909aa1593787f9aba54f2df05e4
Closes-bug: #1506112
(cherry picked from commit 11ce99ab4e)

nailgun/nailgun/utils/logs.py

@@ -80,7 +80,16 @@ def prepare_syslog_dir(node, prefix=settings.SYSLOG_DIR):
                          "Trying to remove", l)
             shutil.rmtree(l)
         logger.debug("Creating symlink %s -> %s", l, new)
-        os.symlink(objects.Node.get_node_fqdn(node), l)
+        while not os.path.islink(l):
+            try:
+                os.symlink(objects.Node.get_node_fqdn(node), l)
+            except OSError as exc:
+                if exc.errno == os.errno.EEXIST:
+                    logger.debug("%s was created again. "
+                                 "Trying to remove", l)
+                    shutil.rmtree(l)
+                else:
+                    raise
 
     os.system("/usr/bin/pkill -HUP rsyslog")