Defer radosgw key provision until OSD's detected

The RADOS gateway on startup will try to initialize a number
of pools in the Ceph cluster; if no OSD's are present at the
point of startup, this operation is re-tried for a period of
5 minutes.

As ceph-radosgw and ceph-mon are typically deployed in LXC
containers, they are up and running before ceph-osd units
are fully operational, resulting in ceph-radosgw units with
no running radosgw process.

By deferring the provision of keys to related ceph-radosgw
units until ceph-osd units are detected, the chance of this
race happening is greatly reduced.

Also add trailing space to ceph.conf as ceph requires that
configuration files end with a newline.

Change-Id: I2a21f021502bde5c688bd3ac4b84ef24a8bdef68
Closes-Bug: 1577519
Closes-Bug: 1578403

hooks/ceph_hooks.py

@@ -504,10 +504,27 @@ def osd_relation(relid=None):
         }
         relation_set(relation_id=relid,
                      relation_settings=data)
+        # NOTE: radosgw key provision is gated on presence of OSD
+        #       units so ensure that any deferred hooks are processed
+        notify_radosgws()
     else:
         log('mon cluster not in quorum - deferring fsid provision')
 
 
+def related_osds(num_units=3):
+    '''
+    Determine whether there are OSD units currently related
+
+    @param num_units: The minimum number of units required
+    @return: boolean indicating whether the required number of
+             units where detected.
+    '''
+    for r_id in relation_ids('osd'):
+        if len(related_units(r_id)) >= num_units:
+            return True
+    return False
+
+
 @hooks.hook('radosgw-relation-changed')
 @hooks.hook('radosgw-relation-joined')
 def radosgw_relation(relid=None, unit=None):
@@ -516,8 +533,11 @@ def radosgw_relation(relid=None, unit=None):
     if not unit:
         unit = remote_unit()
 
-    if ceph.is_quorum():
-        log('mon cluster in quorum - providing radosgw with keys')
+    # NOTE: radosgw needs some usage OSD storage, so defer key
+    #       provision until OSD units are detected.
+    if ceph.is_quorum() and related_osds():
+        log('mon cluster in quorum and osds related '
+            '- providing radosgw with keys')
         public_addr = get_public_addr()
         data = {
             'fsid': leader_get('fsid'),
@@ -539,7 +559,7 @@ def radosgw_relation(relid=None, unit=None):
 
         relation_set(relation_id=relid, relation_settings=data)
     else:
-        log('mon cluster not in quorum - deferring key provision')
+        log('mon cluster not in quorum or no osds - deferring key provision')
 
 
 @hooks.hook('client-relation-joined')

templates/ceph.conf

@@ -36,3 +36,4 @@ keyring = /var/lib/ceph/mon/$cluster-$id/keyring
 
 [mds]
 keyring = /var/lib/ceph/mds/$cluster-$id/keyring
+