Currently, the private-address for the local unit is queried using
unit_get, which can cause it to return an address from a different
binding. This patch changes it to always query from the relation.
Closes-bug: #2020669
Change-Id: I128420c572d5491b9af4cf34614f4534c787d02c
hacluster uses the command "crm configure get-property <CMD>" to obtain
a property of the cluster, although "get-property" has been deprecated
in favor of "get_property", since crmsh-4.2.1 a warning is printed to
stdout[0] breaking the parsing.
# crm configure get-property maintenance-mode 2>/dev/null
WARNING: This command 'get-property' is deprecated, please use 'get_property'
INFO: "get-property" is accepted as "get_property"
true
[0] 86282af8e5
Change-Id: Id0ee9ab1873d14dcd1c960001cdeb8318f599ef5
Closes-Bug: #2008704
list_nodes() recently had some changes to run 'crm node show'
in jammy+ instead of 'crm node status'. The difference is that
'crm node show' returns the pacemaker-remote nodes in addition
to the hacluster nodes. This change limits the nodes returned
by list_nodes() to the hacluster nodes (ie. the nodes that
have a node ID).
Closes-Bug: #1995295
Change-Id: Ia405d4270f56c949f79167f8b75c1304b598b918
The command 'crm node show' is used on jammy to retrieve the list of
nodes defined in a cluster. The output for nodes includes ': member'
which breaks ensuing commands that are using list_nodes() output.
For example:
juju-3f6cb6-zaza-4135aa8b2509-8.project.serverstack: member
This change trims everything including and after the ':' from the
output.
Closes-Bug: #1994160
Change-Id: I54a4f854f3e293503ec97d99a49b6dc51ee50c87
The command 'crm node show' is used on jammy to retrieve the list of
nodes defined in a cluster, although this command also includes the
properties set on a node (e.g. standby=off) which breaks the current
logic parsing.
This change uses a regular expresion to filter out all the lines from
the output that don't start with a non-white character (^\S+).
Change-Id: I3e00daa1b877a7faae1370f08b2d9c5bd7795c5f
Closes-Bug: #1987685
Related-Bug: #1972022
- add contrib/templating
- using render instead of render_template
- remove render_template function
Change-Id: I395d7dc06618998b9e6023ff649f4aa8c5930cc0
The version of crmsh available on jammy doesn't have the 'crm node
status' subcommand available since it was removed[0], this change uses
the command 'crm node attribute' to figure out if the node is in standby
mode when running on ubuntu>=jammy, and 'crm node show' to get the list
of nodes.
[0] https://github.com/ClusterLabs/crmsh/pull/753
Change-Id: Iafb711be220573cb701527ec84de285edd8942cf
Closes-Bug: #1972022
The install hooks rsync a set of scripts and one of the destinations
is /usr/lib/stonith/plugins/external, this directory is created by the
installation of the package cluster-glue which is a pulled as an
indirect dependency of pacemaker, this changed on >=jammy where an
intermediate package named resource-agents was split into
resource-agents-base and resource-agents-extra wher the latter doesn't
get installed and it's the one that depends on cluster-glue.
The specific chain of dependencies are:
focal:
pacemaker -> pacemaker-resource-agents -> resource-agents -> cluster-glue
jammy
pacemaker -> pacemaker-resource-agents -> resource-agents-base
Change-Id: Ia00061bff2ebe16d35d52b256c61243935edabba
Closes-Bug: #1971841
Hooks are expected to be idempotent, if the install hook for whatever
reason needs to be re-run and the /etc/corosync directory already exists,
because for example it was created in a previous run, the exception
FileExistsError will be raised, this change captures the exception and
moves on.
Change-Id: If43a5c95bb59c9cca7f1a975214a9f013ad6f4d6
Closes-Bug: #1971762
Whenever a peer returns None as its IP, it results in
misconfiguration in corosync.conf, which results in
a series of cascading hook errors that are difficult to
sort out.
More specifically, this usually happens when network-get
does not work for the current binding. The main problem
is that when changing bindings, a hook fires before the
network-get data is updated. This hook fails and prevents
the network-get from being re-read.
This patch changes the code behavior to ignore None IP
entries, therefore gracefully exiting and deferring further
configuration due to insufficient number of peers when that
happens, so that a later hook can successfully read the IP
from the relation and set the IPs correctly in corosync.
Closes-bug: #1961448
Change-Id: I5ed140a17e184fcf6954d0f66e25f74564bd281c
Starting in focal, the ubuntu version of corosync package synced in from
debian includes node1 as the default name for the local node with a nodeid
of 1. This causes the cluster to have knowledge of this extra node1 node,
which affects quorum, etc. Installing the charm's corosync.conf file
before package installation prevents this conditioning from happening.
Additionally this change removes some Xenial bits in the charm and always
includes a nodelist in corosync.conf as it is compulsory in focal and
newer. It is optional in the bionic packages, so we'll always just
render the nodelist.
Change-Id: I06b9c23eb57274f0c99a3a05979c0cabf87c8118
Closes-Bug: #1874719
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.
Note that https://github.com/openstack/charms.openstack is used during tests
and he need `mock`, unfortunatelly it doesn't declare `mock` in its
requirements so it retrieve mock from other charm project (cross dependency).
So we depend on charms.openstack first and when
Ib1ed5b598a52375e29e247db9ab4786df5b6d142 will be merged then CI
will pass without errors.
Depends-On: Ib1ed5b598a52375e29e247db9ab4786df5b6d142
Change-Id: I631d32e1a330bcd17b53ee873833e8434023958f
Provide the delete-node-from-ring action to safely remove a known node
from the corosync ring.
Update the less safe update-ring action to avoid LP Bug #1933223 and
provide warnings in actions.yaml on its use.
Change-Id: I56cf2360ac41b12fc0a508881897ba63a5e89dbd
Closes-Bug: #1933223
Adds a config item for what to do when the cluster does not have quorum.
This is useful with stateless services where, e.g., we only need a VIP
and that can be up on a single host with no problem.
Though this would be a good relation data setting, many sites would
prefer to stop the resources rather than have a VIP on multiple hosts,
causing arp issues with the switch.
Closes-bug: #1850829
Change-Id: I961b6b32e7ed23f967b047dd0ecb45b0c0dff49a
This was happening because
trigger_corosync_update_from_leader() was being called
not only in `ha` relation hooks but also in
`pacemaker-remote` relation hooks after the implementation
for the related bug landed.
Closes-Bug: #1920124
Related-Bug: #1400481
Change-Id: I4952ef694589de6b72f04b387e30ca2333bc4cbc
The `state` action will provide details about the health of the cluster.
This action has one parameter to display the history of the cluster status,
which is false by default.
Closes-Bug: #1717831
Change-Id: Iaf6e4a75a36491eab8e6802a6f437e5f410ed29e
Add an `update-ring` action for that purpose.
Also print more on various pacemaker failures.
Also removed some dead code.
Func-Test-PR: https://github.com/openstack-charmers/zaza-openstack-tests/pull/369
Change-Id: I35c0c9ce67fd459b9c3099346705d43d76bbdfe4
Closes-Bug: #1400481
Related-Bug: #1874719
Co-Authored-By: Aurelien Lourot <aurelien.lourot@canonical.com>
Co-Authored-By: Felipe Reyes <felipe.reyes@canonical.com>
At bionic crmsh was dropped as a dependency of corosync so it
becomes a candidate for removal for older charm deployments
upgrading from xenial->bionic.
Mark the package as manually installed in the pre-upgrade hook
to ensure that it never becomes a candidate for removal.
Change-Id: I675684fee5410f86aace2e42515d3e325d8d12f8
Closes-Bug: 1900206
This patch replaces the sqlite3 kvstore implementation provided by charmhelpers. Patches the kvstore used in pcmk for both the test_pcmk and the test_hacluster_hooks tests.
closes-bug: #1908282
Change-Id: I3320735314f0b03aecec6635ef82ddd44eecaff1
This value is hard-coded to 60 seconds into the charm code.
This change adds a charm config option (with 60 secs as the
default value) in order to make the `cluster-recheck-interval`
property configurable.
Change-Id: I58f8d4831cf8de0b25d4d026f865e9b8075efe8b
Corosync 2.99 altered the status output for udp/udpu rings to
be hardcoded to 'OK'. This breaks the check_corosync_rings nrpe
check which is looking for 'ring $number active with no faults'.
Since the value has been hardcoded to show 'OK', the check itself
does not provide any real meaningful value.
Change-Id: I642ecf11946b1ea791a27c54f0bec54adbfecb83
Closes-Bug: #1902919
Previously, paused hacluster units showed up CRITICAL error
in nagios even though they were only in the 'standby' mode
in corosync.
The hacluster charm now uses the '-s' option of the check_crm
nrpe script to ignore alerts of the standby units.
Change-Id: I976d5ff01d0156fbaa91f9028ac81b44c96881af
Closes-Bug: #1880576
The old check_crm script had separate checks for failcounts and failed
actions, but since failed actions cause failcounts, the two will always be
present together, and expire together.
Furthermore, the previous defaults effectively caused the failed actions
check to shadow the failcount one, because the former used to cause
CRITICALs, while the latter was only causing WARNINGs.
This version of check_crm deprecates failed actions detection in favor of
only failcount alerting, but adds support for separate warn/crit
thresholds.
Default thresholds are set at 3 and 10 for warn and crit, respectively.
Although sending criticals for high fail counter entries may seem
redundant when we already do that for stopped resources, some resources
are configured with infinite migration thresholds and will therefore
never show up as failed in crm_mon. Having separate fail counter
thresholds can therefore still be valuable, even if for most resources
migration-threshold will be set lower than the critical fail-counter threshold.
Closes-Bug: #1864040
Change-Id: I417416e20593160ddc7eb2e7f8460ab5f9465c00
* charm-helpers sync for classic charms
* charms.ceph sync for ceph charms
* rebuild for reactive charms
* sync tox.ini files as needed
* sync requirements.txt files to sync to standard
Change-Id: I7c643447959cfd82234653fdbd2bab1c0594469c
There appears to be a window between a pacemaker remote resource
being added and the location properties for that resource being
added. In this window the resource is down and pacemaker may fence
the node.
The window is present because the charm charm currently does:
1) Set stonith-enabled=true cluster property
2) Add maas stonith device that controls pacemaker remote node that
has not yet been added.
3) Add pacemaker remote node
4) Add pacemaker location rules.
I think the following two fixes are needed:
1) For initial deploys update the charm so it does not enable stonith
until stonith resources and pacemaker remotes have been added.
2) For scale-out do not add the new pacemaker remote stonith resource
until the corresponding pacemaker resource has been added along
with its location rules.
Depends-On: Ib8a667d0d82ef3dcd4da27e62460b4f0ce32ee43
Change-Id: I7e2f568d829f6d0bfc7859a7d0ea239203bbc490
Closes-Bug: #1884284
Use location directives to spread pacemaker remote resources across
cluster. This is to prevent multiple resources being taken down in
the event of a single node failure. This would usually not be a
problem but if the node is being queried by masakari host
monitors at the time the node goes down then the query can hang.
Change-Id: Ib8a667d0d82ef3dcd4da27e62460b4f0ce32ee43
Partial-Bug: #1889094
Depends-On: Ic45dbdd9d8581f25549580c7e98a8d6e0bf8c3e7
If stonith is enabled then when a compute node is detected as failed
it is powered down. This can include a lxd container which is also
part of the cluster. In this case because stonith is enabled at a
global level, pacemaker will try and power off the lxd container
too. But the container does not have a stonith device and this causes
the container to be marked as unclean (but not down). This running
unclean state prevents resources being moved and causes any
pacemaker-remotes that are associated with the lost container from
losing their connection which prevents masakari hostmonitor from
ascertaining the cluster health.
The way to work around this is to create a dummy stonith device for
the lxd containers. This allows the cluster to properly mark the lost
container as down and resources are relocated.
Change-Id: Ic45dbdd9d8581f25549580c7e98a8d6e0bf8c3e7
Partial-Bug: #1889094
The old configure_stonith last worked on xenial. This patch bypassies
it given it is broken and it is interfering with the new pacemaker
remote stonith resource setup. The code will be removed by a follow
up patch. This patch is designed to be small for easy cherry picking.
Change-Id: I2a704c1221bda242caaa5e87849d9984db3c6b71
Partial-Bug: #1881114
The recent maas stonith resources, introduced to support stonith
with pacemaker-remotes, included a hash of the combined url and
api key in the resource name. But the charm only supports one
stonith resource (single maas_url api key config options). Having
the hash makes managing the resources more complicated espically
when the url or api key change. So remove any existing resource
(there is very unlikely to be one as the feature is only just
out of preview state) and replace with a single resource called
'st-maas'.
Change-Id: I053f1dc882eebfdef384cbbbfa7cabc82bce5f8b
Check whether peers have sent series upgrade notifications before
pausing a unit. If notifications have been sent then HA services
will have been shutdown and pausing will fail.
Similarly, if series upgrade notifications have been sent then
do not try and issue crm commands when assessing status.
Change-Id: I4de0ffe5d5e24578db614c2e8640ebd32b8cd469
Closes-Bug: #1877937
Now that nova-compute uses an fqdn to identify the hypervisor *1
switch to registering remote nodes using their fqdn to keep
everything consistent. This is a requirement for masakari host
monitoring.
*1 1869bfbc97
Change-Id: I6cd55fdf21482de908920bcc4d399ac03f903d58
Stop HA services accross all units of an application when doing a
series upgrade to avoid the situation where the cluster has some
nodes on LTS N-1 and some on LTS N.
1) In the 'pre-series-upgrade' send a notification to peers informing
them that the unit is doing a series upgrade and to which Ubuntu
version.
2) Peers receive notification. If they are on a later Ubuntu version
than the one in the notification then they do nothing. Otherwise
they shutdown corosync and pacemaker and add an entry to the local
kv store with waiting-unit-upgrade=True.
3) In the 'post-series-upgrade' hook the notification is removed from
the peer relation. waiting-unit-upgrade is set to False and
corosync and pacemaker are started.
The result of this is that when the first unit in the cluster starts
a series upgrade all cluster services are shutdown across all units.
They then rejoin the cluster one at a time when they have been
upgraded to the new version.
I added the waiting-unit-upgrade key to deal with the situation where
the first node clears the notification after it has successfully
upgraded, with out the waiting-unit-upgrade the peers would not know
they were in a mixed Ubuntu version cluster.
Change-Id: Id9167534e8933312c561a6acba40399bca437706
Closes-Bug: 1859150
Pacemaker will refuse to delete a resource that it's running, so it needs
to be stopped always before deleting it.
Change-Id: I3c6acdef401e9ec18fedc65e9c77db4719fe60ec
Closes-Bug: #1838528
The current charm does not indicated to the end user when a specific
resource is not running. Neither does it indicate when a node is offline
or stopped.
Validate that configured resources are actually running and let the end
user know if they are not.
Closes-Bug: #1834263
Change-Id: I1171e71ae3b015b4b838b7ecf0de18eb10d7c8f2
This commit adds two new options, failed_actions_alert_type and
failed_actions_threshold, which map onto the check_crm options
--failedactions and --failcounts, respectively.
The default option values make check_crm generate critical alerts if
actions failed once.
The actions check can be entirely bypassed if failed_actions_alert_type
is set to 'ignore'.
Closes-Bug: #1796400
Change-Id: I72f65bacba8bf17a13db19d2a3472f760776019a
As explained here[0], setting failure-timeout means that the cib will 'forget'
that a resource agent action failed by setting failcount to 0:
- if $failure-timeout seconds have elapsed from the last failure
- if an event wakes up the policy engine (i.e. at the global resource
recheck in an idle cluster)
By default the failure-timeout will be set to 0, which disables the feature,
however this change allows for tuning.
[0] https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/#_failure_response
Change-Id: Ia958a8c5472547c7cf0cb4ecd7e70cb226074b88
Closes-Bug: #1802310
Stonith is being disabled at the global cluster level despite it being needed
for pacemaker-remote nodes.
The legacy hacluster charm option 'stonith_enable' covers the main 'member'
nodes and if it is set to false then stonith resources are not created for
them and the stonith-enabled cluster parameter is set to false. However, in a
masakari deploy stonith is not required for the member nodes but is for the
remote nodes. In this case stonith-enabled cluster option should be set to
true.
Change-Id: Ie1affa17dd3cfcd677aa866b6e3d1c1004bb13c9
Closes-Bug: #1824828
When setting up resources for pacemaker remote nodes use the IP
address supplied by the remote node for communication. This
ensures that communication happens over the desired network
space.
Depends-On: I5aa6993ec702f97403d1a659e09a3fb2f5af4202
Change-Id: I9bb20b5f0b0d780fbf4cc0ac0e5f86fe277c4715
Closes-Bug: #1824514
The libmaas stonith plugin uses libmaas which is supplied via
python3-libmaas. python3-libmaas is available from bionic onwards
so install it where possible.
Also, drive by fix to precreate /etc/pacemaker dir if needed to fix
trusty installs.
Closes-Bug: #1823300
Closes-Bug: #1823302
Change-Id: Ib14146f7f667f9c52e11d222f950efcb7cb47a7f
This change adds support for pacmaker-remots joining the cluster
via the pacemaher-remote relation. The pacemaker-remotes can
advertise whether they should host resources. If the
pacemaker-remotes are only being used for failure detection
(as is the case with masakari host monitors) then they
will not host resources.
Pacemaker remotes are managed in the cluster as resources
which nominally run on a member in the main cluster. The
resource that corresponds to the pacemaker-remote is managed
via configure_pacemaker_remotes and configure_pacemaker_remote
functions.
If the pacemaker-remotes should not run resources then the
cluster needs to be switched to an opt-in cluster. In an
opt-in cluster location rules are needed to explicitly
allow a resource to run on a specific node. This behaviour
is controlled via the global cluster parameter 'symmetric-cluster'
this is set via the new method set_cluster_symmetry. The method
add_location_rules_for_local_nodes is used for creating these
explicit rules.
Change-Id: I0a66cfb1ecad02c2b185c5e6402e77f713d25f8b
This change adds a new 'member_ready' state to the peer relation.
This purpose of this is to indicate that a unit has rendered its
configuration and started its services. This is distinct from the
existing 'ready' indicator. 'ready' indicates that the unit can
render its config and start services and ensures that all units are
present prior to starting services. This seems to safe guard
against units starting to early and forming single node clusters or
small clusters with a sub-set of units. The 'member_ready' flag
is later in the process and if it is set it shows that this unit
has joined the cluster and can be referenceed explicitly in any
resource configuration.
Change-Id: I80d42a628a3fe51bc6f8d02610031afd9386d7a4
The change adds a stonith plugin for maas and method for creating
stonith resources that use the plugin.
Change-Id: I825d211d68facce94bee9c6b4b34debaa359e836
The current list_nodes command tries to parse the output of
"crm node list" to get a list of nodes. But the output is messy if
remote nodes are present so use "crm node status" to get properly
formatted XML which can be reliably parsed.
Change-Id: Iea7ef3dca194e7440dc2cde2624d07e990006685
Tiago Pasqualini