The additional check is based on cluster being bootstrapped and the last
backup being a SST.
The change includes new function for checking the last backup was SST and unittests to verify said function as well as the main charm_check_func where the check is used and seeded file is replaced.
Closes-Bug: #2000107
Signed-off-by: Rafael Lopez <rafael.lopez@canonical.com>
Change-Id: I8e516059da5299cc0e0ce8ef0802d3a46abb1a54
`get_wresp_value` can fail in `cursor.execute()` and no log
message would be written.
This commit fix and ensure the right message is logged.
Closes-Bug: #1942936
Change-Id: Idbb1170bbf43fdecec233137d6581bf8f799baa9
This patch provides one of the alternative solution to address the
problem that the error log might grow too large.
Closes-bug: #1812950
Change-Id: If7c0c71492eb30f24cbcc03ca05a67e6ea571f4e
This change adds the config option max_connect_errors for MySQL to this
charm and sets a default of 100.
Commit also includes the inclusion of this (default) config setting in the
unit tests.
Closes-Bug: #1776908
Change-Id: I33b9e29bd64ad8a1fec0edc3dfd657a87648b537
Check that that the asynchronous replication relations exist prior
to providing the databases_to_replicate to the templating engine
code. Specifying the databases-to-replicate config option is not
supposed to apply when the asynchronous relation is not established.
Closes-Bug: #1934680
Change-Id: Iedd0532268f0bf533305412105590c5afdd7c0ec
The associated bug is due to a change introduced in commit d55dcde
which was to ensure that the correct password update is used for
different versions of mysql (pre and post 5.7.5). However, this change
has broken the nagios user creation due to not setting the password.
This patch creates the nagios user and passord at the same time. The
updating of the password is only done if the account already exists.
The change also corrects the nagios password store in leader-settings to
the 'mysql-nagios.passwd' key instead of 'nagios-password'. This was an
unfortuante error when the nagios change password was introduced. The
charm detects if the 'nagios-password' key is used on charm-upgrade and
moves it to 'mysql-nagios.passwd'. This enables the key to work with
the standard MySQLHelper functions.
Finally, the ALTER command (on percona) doesn't update non-InnoDB tables
and thus needs to be run for each unit when the nagios password is
changed via the action. The changes in percona_utils.py enable this to
happen.
Whilst the change looks large it ONLY affects the nagios password parts
of the charm.
The related bug is a tracking bug to serve as a reminder to fix this in
charm-helpers and this charm (i.e. make the charm-helpers code work to
change a password for any user other than root, and then enable this
charm to use that code).
Change-Id: Ibc751bef7b4654ebffdf843c556b193373e6e80c
Related-Bug: #1925377
Closes-Bug: #1925042
On mysql pre 5.7.5 and older (e.g. xenial) a different password update
SQL statement is needed. Detect the ubuntu version (assuming the
packages are used) and use the version for that password.
Change-Id: Ifd73f20c2523de164fb23e8f5a5e757393d489e3
Closes-Bug: #1916672
Install a systemd unit override with LimitNOFILE raised to the required
number based on the same calculation the server performs to ensure both
the requested max_connections and table_open_cache values can be met.
The MySQL server does make some attempt to do this itself when started
as root which worked under Xenial, however in Bionic the systemd service
is started as the mysql user with LimitNOFILE=5000. As a result, the
server cannot raise the limit itself and caps max-connections to 4190,
table-open-cache to 200 and logs a warning to that effect:
[Warning] Changed limits: max_connections: 4190 (requested 12000)
More background:
https://www.percona.com/blog/2017/10/12/open_files_limit-mystery/
Change-Id: I25b429cc9b4970e3d7ef39bb9e6d738fe943686f
Closes-Bug: #1905366
Add a NRPE check to monitor the number of threads connected
to the MySQL database, in proportion to the maximum number of connections.
For this check, a nagios user will be created. This user does not have any
permissions set, does not have access to any database and can only connect
from localhost.
Warning and Critical thresholds (in percentage) can be configured.
Add an action to reset nagios's password. This action could only be run
on the leader unit.
Closes-Bug: #1816759
Change-Id: Id35b0331322c2744a9f839b3eb153eed1bc53aac
After changing the `source` or `key` in the configuration,
the `config-changed` hook was activated, but did not call `add_source`
and `apt_update`.
This logic has been added to the `config-changed` hook as well as
the state changed to `blocked` if the addition of a new source
or key fails.
Closes-Bug: #1893034
Change-Id: I0e7afd5a06c4945341329dd96e8eef3016367386
* charm-helpers sync for classic charms
* charms.ceph sync for ceph charms
* rebuild for reactive charms
* sync tox.ini files as needed
* sync requirements.txt files to sync to standard
Change-Id: I1bf6a834b3678f631f6335af9f0e8b779f863c66
During resume on a non-leader unit, the 'seeded' file can go missing
when the package syncs over to the /var/lib/mysql or
/var/lib/percona-xtradb-cluster directories (vivid+). It's not really
clear why it doesn't do this every time (i.e. not every non-leader unit
fails), but this fix ensures that if the unit *is* seeded prior to the
series-upgrade, then it stays seeded after the series upgrade.
The related zaza-openstack-tests change [1] is about fixing the
series-upgrade test.
Note there is no trusty-mitaka test here as the charm doesn't support
trusty. The last version of percona-cluster in the charm store that
supports trusty is rev. 276
[1]: https://github.com/openstack-charmers/zaza-openstack-tests/pull/406
Change-Id: I628be1c24081d7e0e150e5064c5fa4ab694632e9
Closes-bug: #1868326
Fix mysqldump action by checking distro version and removing
unsupported flags from the mysqldump command.
DocImpact
Closes-Bug: #1878272
Implements: MySQL dump test
Change-Id: Iddf1ff408e5657e96ced6f853dfc42acf47072c1
When a node is bootstrapped and the other join, they will receive a copy of
the database via SST with the help of XtraBackup, during this process the
/var/lib/percona-xtradb-cluster will be emptied so after this the seeded
file will be missing. Since the action bootstrap-pxc will trigger the
cluster-relation-changed hook it's a good place to call mark_seeded().
Change-Id: I8510bb81435a3096d6a005610fce88ff2b7ffeab
Closes-Bug: #1868326
Func-Test-Pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/203
When the percona cluster is undergoing a series upgrade, clients
should suspend db activity in their hooks (like db migrations).
This change sents a notification of upgrade down the shared-db
relation which clients can then react to.
Change-Id: I5d8ed7d3935db5568c50f8d585e37a4d0cc6914f
This patch ensures that the PROCESS permission is added to the sstuser
user during a series upgrade from xenial to bionic. Also patches the
README to correct some of the commands in terms of order of parameters.
Change-Id: I8ddc21e3914e3da0f64ffcedef7a6a4bb18ba4cb
Closes-Bug: #1838044
Occasionally after cold boot yaml load of grastate will be throw an
exception. Do not error out in this instance.
Update percona to use TEST_ variables
Fix HA overlays
func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/116
Change-Id: I6e40970423acb6f70dcc3b91f8b5109de6f46bfc
After a cold boot, the leader has likely changed. Do not add this node
as a former leader to the hosts list. This may cause changes to
mysqld.cnf and an unwanted attempted restart of mysql.
Change-Id: I5fc4b7822a4550e53e97655771938a903f92fcb1
Close-Bug: #1838648
Ensure ocf file is installed early so that mysql monitor resource can
start as soon as it is defined. Otherwise it will be down and may
not come backup without manual intervention.
Change-Id: I5da6da4ddde25219e06ef47bba5f3004ccf678ee
Closes-Bug: #1837401
Using DDL for the root password update triggers the cluster to
replicate the password rather than relying on the charm to apply it
to each unit. This means that only the leader should update the
password.
To avoid a password update via config being missed the implicit save
of config data is turned off on update-status hooks.
Change-Id: I00fe3c7c32a000c6b697263a9ad32273515952a7
Closes-Bug: #1838124
Closes-Bug: #1838125
It turns out a subsequent required step after a cold boot bootstrap is
notifying the cluster of the new bootstrap UUID.
The notify-bootstrapped action should be run on a different node than
the one which ran the bootstrap-pxc action.
This action will ensure the cluster converges on the correct bootstrap
UUID.
A subsequent patch stacked on this one will include tests for the new
cold boot actions.
Change-Id: Idee12d5f7e28498c5ab6ccb9605f751c6427ac30
Partial-Bug: #1744393
To assist in DB cluster recovery after a cold boot, display the
grstate sequence number and the safe to boot status of the instance
in workload status.
Change-Id: Ia4e0e86e7d10b2b22148237688ff77cac4ebee7d
This improves performance significantly for environments constrained by
calls to sync() such as HDDs or lower-end SSDs (or just very busy
environments running many queries)
By default the the queries from other nodes are only processed with
1 thread, which means they will always run slower than on the master and
any long running query will hold up all other queries behind it.
Additionally, when multiple queries commit at once the server can
combine them together into a single on-disk sync ('group commit') which
is not possible otherwise. This optimisation appears to only occur on
Bionic (Percona 5.7) and not Xenial (Percona 5.6).
On Bionic, default to 48 threads which experimentally is a good number
for OpenStack environments without being too crazy high. Galera ensures
that queries that are dependent on each other are still executed
sequentially and generally it is not expected to cause replication
inconsistencies.
However Percona Cluster 5.6 on Xenial appears to have a bug handling
foreign key constraints that causes them to be violated (LP #1823850).
The result is that the slave node crashes out and has to do a full SST
to recover. The same issue is not present on the master. Thus we leave
the default wsrep_slave_threads=1 on Xenial to avoid this issue for now
particularly since Xenial does not appear to be able to use Group Commit
to optimise the number of sync requests generated by the queries - so
this option does not really improve performance there anyway.
Partial-Bug: #1822903
Change-Id: Ic9cdd6562f30a3e52aa3d26fea53ba7c2bbdc771
It's broken at trusty and needs to be re-worked due to a lack of python3-mysqldb at trusty
This reverts commit 03f93dbc76.
Change-Id: I2b722014fc1ed5823635a6b45b3307326fd901af
The pre-install operations may fail, yet that failure is not
elevated to the user. This masks the failure and makes early
package install issues difficult to troubleshoot.
If the basic pre-install script fails, the charm should not
proceed to later hooks as the requirements may not be met.
Hashbangs for bash should specify -e (errexit) on all of the
pre-install bash scripts.
Change-Id: I9f6990cc8601abb703e27a56c4cb243245031234
Closes-bug: #1815243
Partial-bug: #1815231
These options allow tuning the Galera replication to avoid flow control from
slowing down the primary.
Change-Id: Ib275cae0db02e4c8c0a85fcc8cb138b26eb26982
Closes-Bug: 1799622
This patchset implements new relations: "master" and "slave" based
on a common "mysql-async-replication" interface which are used for
the purpose on enabling MySQL asynchronous replication between
multiple Percona XtraDB Clusters.
Change-Id: I94710bff17091516875c81ca769d8078ef5efd10
Closes-Bug: 1776171
Use helpers from charmhelpers to generate the data to send down the
relation to the hacluster charm.
This results in a few changes in behaviour:
1) The charm will no longer specify a nic name to bind the vip. This
is because Pacemaker VIP resources are able to automatically
detect and configure correct iface and netmask parameters based
on local configuration of the unit.
2) The original iface named VIP resource will be stopped and deleted
prior to the creation of the new short hash named VIP resource.
Change-Id: Id3804fb7913662b8c573f59d84e663561a687b1f
The percona-cluster packages have been fixed in the package
repositories and no longer need this option to work around the
duplicate primary key error that this worked around. This patch
removes this option.
Change-Id: I493c64c3167b51399484f6fec75f674bfe53e2fb
Related-Bug: #1366997
Use local heler for now due to the transitional state of this charm
from py2 to py3, where cmp does not exist.
Change-Id: I86161b8886da69833c4cc074c0f7ede15f456c9d
There may be situations where leader-elected hook is executed without
being the leader and making leader_set() calls fail, this patch verifies
if it's the leader before attempting to use leader_set()
Change-Id: Ia435069f5df80bfdf0236d14664317990bd06407
Closes-Bug: 1784584
Use a for loop to handle relation_ids for the case when the
relationship has not yet been created.
Change-Id: Ia9cf980591a1bf707f3a1a009229c5f5c222206f
Closes-Bug: #1777508
The percona-cluster charm was rendering, and therefore, restarting
after each peer joined. This causes race conditions with the leader
handing out client credentials.
This is due to the parallel but conflicting goals of building the
cluster as quickly as possible but also delaying client relations until
the full cluster is completely ready.
The charm relied on the bootstrap_uuid being set among peers. However,
the bootstrap_uuid is set intentionally early in pursuit of the first
goal building the cluster quickly. It did not signify true readiness.
This change adds another peer relation setting, "ready", that each node
sets when it has bootstrapped, its hacluster is complete, and it has a
sufficient number of peers.
The cluster is considered ready when (min-cluster-size) peers have
indicated readiness. This fulfils goal number two delaying client
relations until the full cluster is ready.
Change-Id: I0998407fcb5efbdb0f7734ac39363e8d41088c79
Closes-Bug: #1775682
Percona-cluster was failing to start when enable-binlogs was set to
True due to missing server_id in the configuration.
Set the server_id to a unique, non-zero, positive number from 1 to
2**32 - 1.
Closes-Bug: #1772947
Change-Id: Id7da6dd8f27e364e0d5244e4ec33ae4645e4ef96
PXC5.7, which is default one on bionic, introduces the new config
option, pxc_stric_mode, which is set to 'enforcing' by default.
This, however, prevents installing older versions of Openstack
with PXC5.7 on bionic (use case is, of course, testing).
This change makes pxc-strict-mode configurable via the charm, and
it defaults it to 'enforcing', but user can change it at hers or
his desire.
This option is going to be ignored on PXC installed on series less
than bionic.
Change-Id: I7d82e8bea2e75da6159b5baf2e88f2bd44ce84b3
Rafael Lopez