Wait for compute service to check in

With cell v2, on initial bring up, discover hosts can't run unless all
the compute nodes have checked in. The documentation says that you
should run ``nova service-list --binary nova-compute`` and see all
your hosts before running discover hosts. This isn't really viable in
a multinode devstack because of how things are brought up in parts.

We can however know that stack.sh will not complete before the compute
node is up by waiting for the compute node to check in before
completing. This happens quite late in the stack.sh run, so shouldn't
add any extra time in most runs.

Cells v1 and Xenserver don't use real hostnames in the service table
(they encode complex data that is hostname like to provide more
topology information than just hostnames). They are exempted from this
check.

Related-Bug: #1708039

Change-Id: I32eb59b9d6c225a3e93992be3a3b9f4b251d7189
(cherry picked from commit c2fe916fc7)
This commit is contained in:
Sean Dague 2017-07-28 11:29:18 +00:00
parent 0ba247385c
commit cfe0077981
3 changed files with 49 additions and 0 deletions

View File

@ -407,6 +407,26 @@ EOF
return $rval
}
function wait_for_compute {
local timeout=$1
local rval=0
time_start "wait_for_service"
timeout $timeout bash -x <<EOF || rval=$?
ID=""
while [[ "\$ID" == "" ]]; do
sleep 1
ID=\$(openstack --os-cloud devstack-admin --os-region "$REGION_NAME" compute service list --host `hostname` --service nova-compute -c ID -f value)
done
EOF
time_stop "wait_for_service"
# Figure out what's happening on platforms where this doesn't work
if [[ "$rval" != 0 ]]; then
echo "Didn't find service registered by hostname after $timeout seconds"
openstack --os-cloud devstack-admin --os-region "$REGION_NAME" compute service list
fi
return $rval
}
# ping check
# Uses globals ``ENABLED_SERVICES``, ``TOP_DIR``, ``MULTI_HOST``, ``PRIVATE_NETWORK``

View File

@ -944,6 +944,28 @@ function start_nova_conductor {
done
}
function is_nova_ready {
# NOTE(sdague): with cells v2 all the compute services must be up
# and checked into the database before discover_hosts is run. This
# happens in all in one installs by accident, because > 30 seconds
# happen between here and the script ending. However, in multinode
# tests this can very often not be the case. So ensure that the
# compute is up before we move on.
if is_service_enabled n-cell; then
# cells v1 can't complete the check below because it munges
# hostnames with cell information (grumble grumble).
return
fi
# TODO(sdague): honestly, this probably should be a plug point for
# an external system.
if [[ "$VIRT_DRIVER" == 'xenserver' ]]; then
# xenserver encodes information in the hostname of the compute
# because of the dom0/domU split. Just ignore for now.
return
fi
wait_for_compute 60
}
function start_nova {
# this catches the cells v1 case early
_set_singleconductor

View File

@ -1431,6 +1431,13 @@ fi
# Sanity checks
# =============
# Check that computes are all ready
#
# TODO(sdague): there should be some generic phase here.
if is_service_enabled n-cpu; then
is_nova_ready
fi
# Check the status of running services
service_check