Wait for cluster to settle in yum_update.sh
Occasionally we hit "Error: unable to push cib" during update. This is
probably due to the fact that when we try to replace cib in
yum_update.sh, services on the previous updated controller are still
coming up and changing cib, and racing/conflicting with the cib push
from yum_update.sh.
This commit adds waiting for the cluster to settle before exiting from
yum_update.sh, to avoid this kind of conflict.
Also a check for cib-push success is added, to make the update fail
properly instead of hanging indefinitely as we've observed with this
issue.
Change-Id: I953087e0e565474ac553fd57bea2459d2e3a6081
Closes-Bug: #1527644
(cherry picked from commit ac7467bacd
)
This commit is contained in:
parent
67aae3b26a
commit
3ec6ef46e8
|
@ -24,6 +24,7 @@ update_identifier=${update_identifier//[^a-zA-Z0-9-_]/}
|
|||
# seconds to wait for this node to rejoin the cluster after update
|
||||
cluster_start_timeout=600
|
||||
galera_sync_timeout=360
|
||||
cluster_settle_timeout=1800
|
||||
|
||||
timestamp_file="$timestamp_dir/$update_identifier"
|
||||
if [[ -a "$timestamp_file" ]]; then
|
||||
|
@ -128,7 +129,10 @@ openstack-nova-scheduler"
|
|||
pcs -f $pacemaker_dumpfile resource update mongod op stop timeout=100s
|
||||
|
||||
echo "Applying new Pacemaker config"
|
||||
pcs cluster cib-push $pacemaker_dumpfile
|
||||
if ! pcs cluster cib-push $pacemaker_dumpfile; then
|
||||
echo "ERROR failed to apply new pacemaker config"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Pacemaker running, stopping cluster node and doing full package update"
|
||||
node_count=$(pcs status xml | grep -o "<nodes_configured.*/>" | grep -o 'number="[0-9]*"' | grep -o "[0-9]*")
|
||||
|
@ -188,6 +192,12 @@ if [[ "$pacemaker_status" == "active" ]] ; then
|
|||
fi
|
||||
done
|
||||
|
||||
echo "Waiting for pacemaker cluster to settle"
|
||||
if ! timeout -k 10 $cluster_settle_timeout crm_resource --wait; then
|
||||
echo "ERROR timed out while waiting for the cluster to settle"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
pcs status
|
||||
|
||||
else
|
||||
|
|
Loading…
Reference in New Issue