一、机器环境准备
MHA：192.168.247.150 2vcpu 2G centos7
master: 192.168.247.151 2vcpu 4G rocky8.6 mysql8.0.26
slave-01: 192.168.247.152 2vcpu 4G rocky8.6 mysql8.0.26
slave-02: 192.168.247.153 2vcpu 4G rocky8.6 mysql8.0.26
二、mha配置管理
软件包下载 wget https://github.com/yoshinorim/mha4mysql-manager/releases/download/v0.58/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm wget https://github.com/yoshinorim/mha4mysql-node/releases/download/v0.58/mha4mysql-node-0.58-0.el7.centos.noarch.rpm yum localinstall mha4mysql-node-0.58-0.el7.centos.noarch.rpm -y yum localinstall mha4mysql-manager-0.58-0.el7.centos.noarch.rpm -y yum install -y rsync 在所有mysql 节点安装mha4mysql-node yum install -y rsync scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm [email protected]:/root scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm [email protected]:/root scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm [email protected]:/root yum localinstall mha4mysql-node-0.58-0.el7.centos.noarch.rpm -y
创建密钥互信
[root@mha-manager ~]# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:hNohAruUsgQWaRoW7QKQCX+R/LaNng1LuHuzBjFOjHA root@mha The key's randomart image is: +---[RSA 2048]----+ |B*=... | |OBE.o. . | |BB+oo.o . | |=+.+=+oo | |o .o.=.+S | | + = . | | = = | | . B . | | .+.o | +----[SHA256]-----+ [root@mha ~]# [root@mha ~]# ssh-copy-id 127.0.0.1 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established. ECDSA key fingerprint is SHA256:XgCQNiZS5ZJTc9ejJyeI6j2vKF+JaHyBoSwK0Bq9P6w. ECDSA key fingerprint is MD5:db:b0:b1:41:79:0a:b9:94:69:f2:a7:49:d8:78:fb:be. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys [email protected]'s password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '127.0.0.1'" and check to make sure that only the key(s) you wanted were added. [root@mha ~]# [root@mha ~]# rsync -av .ssh 192.168.247.151:/root [email protected]'s password: bash: rsync: command not found rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: remote command not found (code 127) at io.c(226) [sender=3.1.2] [root@mha ~]# rsy rsync rsyslogd rsyslog-recover-qi.pl [root@mha ~]# rsync -av .ssh 192.168.247.151:/root [email protected]'s password: bash: rsync: command not found rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: remote command not found (code 127) at io.c(226) [sender=3.1.2] [root@mha ~]# rsync -av .ssh 192.168.247.151:/root [email protected]'s password: sending incremental file list .ssh/ .ssh/authorized_keys .ssh/id_rsa .ssh/id_rsa.pub .ssh/known_hosts sent 3,497 bytes received 102 bytes 1,439.60 bytes/sec total size is 3,157 speedup is 0.88 [root@mha ~]# rsync -av .ssh 192.168.247.152:/root [email protected]'s password: Permission denied, please try again. [email protected]'s password: sending incremental file list .ssh/ .ssh/authorized_keys .ssh/id_rsa .ssh/id_rsa.pub .ssh/known_hosts sent 3,497 bytes received 102 bytes 423.41 bytes/sec total size is 3,157 speedup is 0.88 [root@mha ~]# rsync -av .ssh 192.168.247.153:/root [email protected]'s password: sending incremental file list .ssh/ .ssh/authorized_keys .ssh/id_rsa .ssh/id_rsa.pub .ssh/known_hosts sent 3,497 bytes received 102 bytes 1,439.60 bytes/sec total size is 3,157 speedup is 0.88 [root@mha ~]#
配置mha
[root@mha-manager ~]# mkdir /etc/mastermha [root@mha-manager ~]# cd /etc/mastermha/ [root@mha-manager mastermha]# ls [root@mha-manager mastermha]# vi app1.cnf [root@mha-manager mastermha]# cat app1.cnf [server default] check_repl_delay=0 manager_log=/data/mastermha/app1/manager.log manager_workdir=/data/mastermha/app1/ master_binlog_dir=/data/mysql/ master_ip_failover_script=/usr/local/bin/master_ip_failover password=magedu ping_interval=1 remote_workdir=/data/mastermha/app1/ repl_password=magedu repl_user=repluser report_script=/usr/local/bin/sendmail.sh ssh_user=root user=mhauser [server1] candidate_master=1 hostname=192.168.247.151 [server2] hostname=192.168.247.152 [server3] candidate_master=1 hostname=192.168.247.153 [root@mha-manager mastermha]#
[root@mha-manager ~]# cat /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
# Copyright (C) 2011 DeNA Co.,Ltd.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
## Note: This is a sample script and is not complete. Modify the script based on your environment.
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
use MHA::DBHelper;
my (
$command, $ssh_user, $orig_master_host,
$orig_master_ip, $orig_master_port, $new_master_host,
$new_master_ip, $new_master_port, $new_master_user,
$new_master_password
);
#执行时必须删除下面三行注释
my $vip = '192.168.247.100/24'; #设置Virtual IP
my $key = "1"; #指定VIP所在网卡的别名
my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip"; #指定VIP所在网卡
my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
'new_master_password=s' => \$new_master_password,
);
exit &main();
sub main {
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
eval {
# updating global catalog, etc
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc)here.
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host\n";
&start_vip();
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --
orig_master_host=host --orig_master_ip=ip --orig_master_port=port --
new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
[root@mha-manager ~]#
[root@mha-manager ~]# chmod +x /usr/local/bin/master_ip_failover
三、设置邮件报警
安装mailx包
yum install mailx -y
配置邮件发送内容 [root@mha-manager ~]# cat /usr/local/bin/sendmail.sh #!/bin/bash echo "MHA is failover!" | mail -s "MHA Warning" [email protected] [root@mha-manager ~]#
增加发送端邮件授权码 cat >> /etc/mail.rc <<EOF set [email protected] set smtp=smtp.163.com set [email protected] set smtp-auth-password=xxxxxxzzzzzyyyyy #发送端邮箱授权码 EOF
四、mysql主从配置
mysql配置
yum install mysql-server -y
[root@mysql-master ~]# cat /etc/my.cnf # # This group is read both both by the client and the server # use it for options that affect everything # [client-server] # # include all files from the config directory # !includedir /etc/my.cnf.d [mysqld] server_id=151 log-bin=/data/mysql/mysql-bin skip_name_resolve=1 general_log [root@mysql-master ~]# systemctl enable --now mysqld
创建mysql bin-log日志目录并指定用户组权限为mysql
mkdir /data/mysql chown -R mysql:mysql /data/mysql systemctl enable --now mysqld 在master创建账号并授权 create user repluser@'192.168.247.%' identified by 'magedu'; grant replication slave on *.* to repluser@'192.168.247.%'; create user mhauser@'192.168.247.%' identified by 'magedu'; grant all on *.* to mhauser@'192.168.247.%';
[root@mysql-master mysql]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.26 Source distribution Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> show master logs; +------------------+-----------+-----------+ | Log_name | File_size | Encrypted | +------------------+-----------+-----------+ | mysql-bin.000001 | 156 | No | +------------------+-----------+-----------+ 1 row in set (0.00 sec) mysql> create user repluser@'192.168.247.%' identified by 'magedu'; Query OK, 0 rows affected (0.01 sec) mysql> grant replication slave on *.* to repluser@'192.168.247.%'; Query OK, 0 rows affected (0.00 sec) mysql> create user mhauser@'192.168.247.%' identified by 'magedu'; Query OK, 0 rows affected (0.00 sec) mysql> grant all on *.* to mhauser@'192.168.247.%'; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql>
配置vip
[root@mysql-master ~]# ifconfig eth0:1 192.168.247.100/24 配置slave mkdir /data/mysql chown -R mysql:mysql /data/mysql vi /etc/my.cnf [mysqld] server_id=152 log-bin=/data/mysql/mysql-bin read_only relay_log_purge=0 skip_name_resolve=1 general_log
systemctl enable --now mysqld
[root@mysql-slave-01 ~]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.26 Source distribution Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CHANGE MASTER TO -> MASTER_HOST='192.168.247.151', -> MASTER_USER='repluser', -> MASTER_PASSWORD='magedu', -> MASTER_LOG_FILE='mysql-bin.000001', -> MASTER_LOG_POS=156; Query OK, 0 rows affected, 8 warnings (0.01 sec) mysql> start slave; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: 192.168.247.151 Master_User: repluser Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 1388 Relay_Log_File: mysql-slave-01-relay-bin.000002 Relay_Log_Pos: 1556 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1388 Relay_Log_Space: 1774 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 151 Master_UUID: 7bf8f5a4-2bff-11ed-824b-000c29aae8d7 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: 1 row in set, 1 warning (0.01 sec) mysql>
slave-02配置
[root@mysql-slave-02 ~]# vi /etc/my.cnf [root@mysql-slave-02 ~]# [root@mysql-slave-02 ~]# cat /etc/my.cnf # # This group is read both both by the client and the server # use it for options that affect everything # [client-server] # # include all files from the config directory # !includedir /etc/my.cnf.d [mysqld] server_id=153 log-bin=/data/mysql/mysql-bin read_only relay_log_purge=0 skip_name_resolve=1 general_log [root@mysql-slave-02 ~]# mkdir -p /data/mysql/ [root@mysql-slave-02 ~]# chown -R mysql.mysql /data/mysql/ [root@mysql-slave-02 ~]# [root@mysql-slave-02 ~]# [root@mysql-slave-02 ~]# systemctl enable --now mysqld Created symlink /etc/systemd/system/multi-user.target.wants/mysqld.service → /usr/lib/systemd/system/mysqld.service. [root@mysql-slave-02 ~]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.26 Source distribution Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CHANGE MASTER TO -> MASTER_HOST='192.168.247.151', -> MASTER_USER='repluser', -> MASTER_PASSWORD='magedu', -> MASTER_LOG_FILE='mysql-bin.000001', -> MASTER_LOG_POS=156; Query OK, 0 rows affected, 8 warnings (0.01 sec) mysql> start slave; Query OK, 0 rows affected, 1 warning (0.01 sec) mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: 192.168.247.151 Master_User: repluser Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 1388 Relay_Log_File: mysql-slave-02-relay-bin.000002 Relay_Log_Pos: 1556 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1388 Relay_Log_Space: 1774 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 151 Master_UUID: 7bf8f5a4-2bff-11ed-824b-000c29aae8d7 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: 1 row in set, 1 warning (0.00 sec) mysql>
五、启动MHA服务
检查mha环境
masterha_check_ssh --conf=/etc/mastermha/app1.cnf masterha_check_repl --conf=/etc/mastermha/app1.cnf masterha_check_status --conf=/etc/mastermha/app1.cnf
[root@mha-manager ~]# masterha_check_ssh --conf=/etc/mastermha/app1.cnf Sun Sep 4 03:14:53 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 03:14:53 2022 - [info] Reading application default configuration from /etc/mastermha/app1.conf.. Sun Sep 4 03:14:53 2022 - [info] Reading server configuration from /etc/mastermha/app1.conf.. Sun Sep 4 03:14:53 2022 - [info] Starting SSH connection tests.. Sun Sep 4 03:14:54 2022 - [debug] Sun Sep 4 03:14:53 2022 - [debug] Connecting via SSH from [email protected](192.168.247.151:22) to [email protected](192.168.247.152:22).. Sun Sep 4 03:14:54 2022 - [debug] ok. Sun Sep 4 03:14:54 2022 - [debug] Connecting via SSH from [email protected](192.168.247.151:22) to [email protected](192.168.247.153:22).. Sun Sep 4 03:14:54 2022 - [debug] ok. Sun Sep 4 03:14:55 2022 - [debug] Sun Sep 4 03:14:54 2022 - [debug] Connecting via SSH from [email protected](192.168.247.152:22) to [email protected](192.168.247.151:22).. Sun Sep 4 03:14:54 2022 - [debug] ok. Sun Sep 4 03:14:54 2022 - [debug] Connecting via SSH from [email protected](192.168.247.152:22) to [email protected](192.168.247.153:22).. Sun Sep 4 03:14:55 2022 - [debug] ok. Sun Sep 4 03:14:56 2022 - [debug] Sun Sep 4 03:14:54 2022 - [debug] Connecting via SSH from [email protected](192.168.247.153:22) to [email protected](192.168.247.151:22).. Sun Sep 4 03:14:55 2022 - [debug] ok. Sun Sep 4 03:14:55 2022 - [debug] Connecting via SSH from [email protected](192.168.247.153:22) to [email protected](192.168.247.152:22).. Sun Sep 4 03:14:55 2022 - [debug] ok. Sun Sep 4 03:14:56 2022 - [info] All SSH connection tests passed successfully. [root@mha-manager ~]# [root@mha-manager ~]# [root@mha-manager ~]# masterha_check_repl --conf=/etc/mastermha/app1.cnf Sun Sep 4 03:15:00 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 03:15:00 2022 - [info] Reading application default configuration from /etc/mastermha/app1.conf.. Sun Sep 4 03:15:00 2022 - [info] Reading server configuration from /etc/mastermha/app1.conf.. Sun Sep 4 03:15:00 2022 - [info] MHA::MasterMonitor version 0.58. Sun Sep 4 03:15:01 2022 - [info] GTID failover mode = 0 Sun Sep 4 03:15:01 2022 - [info] Dead Servers: Sun Sep 4 03:15:01 2022 - [info] Alive Servers: Sun Sep 4 03:15:01 2022 - [info] 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:15:01 2022 - [info] 192.168.247.152(192.168.247.152:3306) Sun Sep 4 03:15:01 2022 - [info] 192.168.247.153(192.168.247.153:3306) Sun Sep 4 03:15:01 2022 - [info] Alive Slaves: Sun Sep 4 03:15:01 2022 - [info] 192.168.247.152(192.168.247.152:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 03:15:01 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:15:01 2022 - [info] 192.168.247.153(192.168.247.153:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 03:15:01 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:15:01 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 03:15:01 2022 - [info] Current Alive Master: 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:15:01 2022 - [info] Checking slave configurations.. Sun Sep 4 03:15:01 2022 - [info] read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 03:15:01 2022 - [warning] relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 03:15:01 2022 - [info] Checking replication filtering settings.. Sun Sep 4 03:15:01 2022 - [info] binlog_do_db= , binlog_ignore_db= Sun Sep 4 03:15:01 2022 - [info] Replication filtering check ok. Sun Sep 4 03:15:01 2022 - [info] GTID (with auto-pos) is not supported Sun Sep 4 03:15:01 2022 - [info] Starting SSH connection tests.. Sun Sep 4 03:15:04 2022 - [info] All SSH connection tests passed successfully. Sun Sep 4 03:15:04 2022 - [info] Checking MHA Node version.. Sun Sep 4 03:15:05 2022 - [info] Version check ok. Sun Sep 4 03:15:05 2022 - [info] Checking SSH publickey authentication settings on the current master.. Sun Sep 4 03:15:05 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable. Sun Sep 4 03:15:05 2022 - [info] Master MHA Node version is 0.58. Sun Sep 4 03:15:05 2022 - [info] Checking recovery script configurations on 192.168.247.151(192.168.247.151:3306).. Sun Sep 4 03:15:05 2022 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000001 Sun Sep 4 03:15:05 2022 - [info] Connecting to [email protected](192.168.247.151:22).. Creating /data/mastermha/app1 if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql/, up to mysql-bin.000001 Sun Sep 4 03:15:06 2022 - [info] Binlog setting check done. Sun Sep 4 03:15:06 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sun Sep 4 03:15:06 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.152 --slave_ip=192.168.247.152 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-01-relay-bin.000002 --slave_pass=xxx Sun Sep 4 03:15:06 2022 - [info] Connecting to [email protected](192.168.247.152:22).. Checking slave recovery environment settings.. Relay log found at /var/lib/mysql, up to mysql-slave-01-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mysql-slave-01-relay-bin.000002 Checking if super_read_only is defined and turned on.. not present or turned off, ignoring. Testing mysql connection and privileges.. mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Sep 4 03:15:06 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.153 --slave_ip=192.168.247.153 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-02-relay-bin.000002 --slave_pass=xxx Sun Sep 4 03:15:06 2022 - [info] Connecting to [email protected](192.168.247.153:22).. Checking slave recovery environment settings.. Relay log found at /var/lib/mysql, up to mysql-slave-02-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mysql-slave-02-relay-bin.000002 Checking if super_read_only is defined and turned on.. not present or turned off, ignoring. Testing mysql connection and privileges.. mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Sep 4 03:15:06 2022 - [info] Slaves settings check done. Sun Sep 4 03:15:06 2022 - [info] 192.168.247.151(192.168.247.151:3306) (current master) +--192.168.247.152(192.168.247.152:3306) +--192.168.247.153(192.168.247.153:3306) Sun Sep 4 03:15:06 2022 - [info] Checking replication health on 192.168.247.152.. Sun Sep 4 03:15:06 2022 - [info] ok. Sun Sep 4 03:15:06 2022 - [info] Checking replication health on 192.168.247.153.. Sun Sep 4 03:15:06 2022 - [info] ok. Sun Sep 4 03:15:06 2022 - [info] Checking master_ip_failover_script status: Sun Sep 4 03:15:06 2022 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 Checking the Status of the script.. OK Sun Sep 4 03:15:07 2022 - [info] OK. Sun Sep 4 03:15:07 2022 - [warning] shutdown_script is not defined. Sun Sep 4 03:15:07 2022 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK. [root@mha-manager ~]# [root@mha-manager ~]# [root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.cnf app1 is stopped(2:NOT_RUNNING). [root@mha-manager ~]#
#开启MHA,默认是前台运行,生产环境一般为后台执行 nohup masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover &> /dev/null #测试环境： #masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover #如果想停止后台执行的MHA,可以执行下面命令 [root@mha-master ~]#masterha_stop --conf=/etc/mastermha/app1.cnf #查看状态 masterha_check_status --conf=/etc/mastermha/app1.cnf
查看到健康性检查
六、故障模拟与恢复
停掉主库
[root@mysql-master ~]# systemctl stop mysqld
mha机器守护进程自动跳出
[root@mha-manager ~]# masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover Sun Sep 4 03:23:52 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 03:23:52 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 03:23:52 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf.. Creating /data/mastermha/app1 if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql/, up to mysql-bin.000001 Creating /data/mastermha/app1 if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql/, up to mysql-bin.000001 Sun Sep 4 05:42:25 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 05:42:25 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 05:42:25 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf.. [root@mha-manager ~]# [root@mha-manager ~]# [root@mha-manager ~]# cat /data/mastermha/app1/manager.log Sun Sep 4 02:49:41 2022 - [info] MHA::MasterMonitor version 0.58. Sun Sep 4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/Server.pm, ln180] Got MySQL error when connecting 192.168.247.151(192.168.247.151:3306) :1045:Access denied for user 'mhauser'@'192.168.247.150' (using password: YES), but this is not a MySQL crash. Check MySQL server settings. Sun Sep 4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297. Sun Sep 4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/Server.pm, ln180] Got MySQL error when connecting 192.168.247.152(192.168.247.152:3306) :1045:Access denied for user 'mhauser'@'192.168.247.150' (using password: YES), but this is not a MySQL crash. Check MySQL server settings. Sun Sep 4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297. Sun Sep 4 02:49:42 2022 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln309] Got fatal error, stopping operations Sun Sep 4 02:49:42 2022 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line 329. Sun Sep 4 02:49:42 2022 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers. Sun Sep 4 02:49:42 2022 - [info] Got exit code 1 (Not master dead). Sun Sep 4 03:23:52 2022 - [info] MHA::MasterMonitor version 0.58. Sun Sep 4 03:23:53 2022 - [info] GTID failover mode = 0 Sun Sep 4 03:23:53 2022 - [info] Dead Servers: Sun Sep 4 03:23:53 2022 - [info] Alive Servers: Sun Sep 4 03:23:53 2022 - [info] 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:23:53 2022 - [info] 192.168.247.152(192.168.247.152:3306) Sun Sep 4 03:23:53 2022 - [info] 192.168.247.153(192.168.247.153:3306) Sun Sep 4 03:23:53 2022 - [info] Alive Slaves: Sun Sep 4 03:23:53 2022 - [info] 192.168.247.152(192.168.247.152:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 03:23:53 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:23:53 2022 - [info] 192.168.247.153(192.168.247.153:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 03:23:53 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:23:53 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 03:23:53 2022 - [info] Current Alive Master: 192.168.247.151(192.168.247.151:3306) Sun Sep 4 03:23:53 2022 - [info] Checking slave configurations.. Sun Sep 4 03:23:53 2022 - [info] read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 03:23:53 2022 - [warning] relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 03:23:53 2022 - [info] Checking replication filtering settings.. Sun Sep 4 03:23:53 2022 - [info] binlog_do_db= , binlog_ignore_db= Sun Sep 4 03:23:53 2022 - [info] Replication filtering check ok. Sun Sep 4 03:23:53 2022 - [info] GTID (with auto-pos) is not supported Sun Sep 4 03:23:53 2022 - [info] Starting SSH connection tests.. Sun Sep 4 03:23:55 2022 - [info] All SSH connection tests passed successfully. Sun Sep 4 03:23:55 2022 - [info] Checking MHA Node version.. Sun Sep 4 03:23:56 2022 - [info] Version check ok. Sun Sep 4 03:23:56 2022 - [info] Checking SSH publickey authentication settings on the current master.. Sun Sep 4 03:23:56 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable. Sun Sep 4 03:23:57 2022 - [info] Master MHA Node version is 0.58. Sun Sep 4 03:23:57 2022 - [info] Checking recovery script configurations on 192.168.247.151(192.168.247.151:3306).. Sun Sep 4 03:23:57 2022 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000001 Sun Sep 4 03:23:57 2022 - [info] Connecting to [email protected](192.168.247.151:22).. Creating /data/mastermha/app1 if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql/, up to mysql-bin.000001 Sun Sep 4 03:23:57 2022 - [info] Binlog setting check done. Sun Sep 4 03:23:57 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sun Sep 4 03:23:57 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.152 --slave_ip=192.168.247.152 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-01-relay-bin.000002 --slave_pass=xxx Sun Sep 4 03:23:57 2022 - [info] Connecting to [email protected](192.168.247.152:22).. Checking slave recovery environment settings.. Relay log found at /var/lib/mysql, up to mysql-slave-01-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mysql-slave-01-relay-bin.000002 Checking if super_read_only is defined and turned on.. not present or turned off, ignoring. Testing mysql connection and privileges.. mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Sep 4 03:23:57 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.153 --slave_ip=192.168.247.153 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-02-relay-bin.000002 --slave_pass=xxx Sun Sep 4 03:23:57 2022 - [info] Connecting to [email protected](192.168.247.153:22).. Checking slave recovery environment settings.. Relay log found at /var/lib/mysql, up to mysql-slave-02-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mysql-slave-02-relay-bin.000002 Checking if super_read_only is defined and turned on.. not present or turned off, ignoring. Testing mysql connection and privileges.. mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Sep 4 03:23:58 2022 - [info] Slaves settings check done. Sun Sep 4 03:23:58 2022 - [info] 192.168.247.151(192.168.247.151:3306) (current master) +--192.168.247.152(192.168.247.152:3306) +--192.168.247.153(192.168.247.153:3306) Sun Sep 4 03:23:58 2022 - [info] Checking master_ip_failover_script status: Sun Sep 4 03:23:58 2022 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 Checking the Status of the script.. OK Sun Sep 4 03:23:58 2022 - [info] OK. Sun Sep 4 03:23:58 2022 - [warning] shutdown_script is not defined. Sun Sep 4 03:23:58 2022 - [info] Set master ping interval 1 seconds. Sun Sep 4 03:23:58 2022 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes. Sun Sep 4 03:23:58 2022 - [info] Starting ping health check on 192.168.247.151(192.168.247.151:3306).. Sun Sep 4 03:23:58 2022 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sun Sep 4 05:37:45 2022 - [warning] Got timeout on MySQL Ping(SELECT) child process and killed it! at /usr/share/perl5/vendor_perl/MHA/HealthCheck.pm line 432. Sun Sep 4 05:37:47 2022 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin Sun Sep 4 05:37:47 2022 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sun Sep 4 05:37:48 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable. Sun Sep 4 05:42:21 2022 - [warning] Got error on MySQL select ping: 1053 (Server shutdown in progress) Sun Sep 4 05:42:21 2022 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin Sun Sep 4 05:42:22 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable. Sun Sep 4 05:42:22 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.247.151' (111)) Sun Sep 4 05:42:22 2022 - [warning] Connection failed 2 time(s).. Sun Sep 4 05:42:23 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.247.151' (111)) Sun Sep 4 05:42:23 2022 - [warning] Connection failed 3 time(s).. Sun Sep 4 05:42:24 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.247.151' (111)) Sun Sep 4 05:42:24 2022 - [warning] Connection failed 4 time(s).. Sun Sep 4 05:42:24 2022 - [warning] Master is not reachable from health checker! Sun Sep 4 05:42:24 2022 - [warning] Master 192.168.247.151(192.168.247.151:3306) is not reachable! Sun Sep 4 05:42:24 2022 - [warning] SSH is reachable. Sun Sep 4 05:42:24 2022 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mastermha/app1.cnf again, and trying to connect to all servers to check server status.. Sun Sep 4 05:42:24 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 05:42:24 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 05:42:24 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 05:42:25 2022 - [info] GTID failover mode = 0 Sun Sep 4 05:42:25 2022 - [info] Dead Servers: Sun Sep 4 05:42:25 2022 - [info] 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:25 2022 - [info] Alive Servers: Sun Sep 4 05:42:25 2022 - [info] 192.168.247.152(192.168.247.152:3306) Sun Sep 4 05:42:25 2022 - [info] 192.168.247.153(192.168.247.153:3306) Sun Sep 4 05:42:25 2022 - [info] Alive Slaves: Sun Sep 4 05:42:25 2022 - [info] 192.168.247.152(192.168.247.152:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:25 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:25 2022 - [info] 192.168.247.153(192.168.247.153:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:25 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:25 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 05:42:25 2022 - [info] Checking slave configurations.. Sun Sep 4 05:42:25 2022 - [info] read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 05:42:25 2022 - [warning] relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 05:42:25 2022 - [info] Checking replication filtering settings.. Sun Sep 4 05:42:25 2022 - [info] Replication filtering check ok. Sun Sep 4 05:42:25 2022 - [info] Master is down! Sun Sep 4 05:42:25 2022 - [info] Terminating monitoring script. Sun Sep 4 05:42:25 2022 - [info] Got exit code 20 (Master dead). Sun Sep 4 05:42:26 2022 - [info] MHA::MasterFailover version 0.58. Sun Sep 4 05:42:26 2022 - [info] Starting master failover. Sun Sep 4 05:42:26 2022 - [info] Sun Sep 4 05:42:26 2022 - [info] * Phase 1: Configuration Check Phase.. Sun Sep 4 05:42:26 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] GTID failover mode = 0 Sun Sep 4 05:42:27 2022 - [info] Dead Servers: Sun Sep 4 05:42:27 2022 - [info] 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:27 2022 - [info] Checking master reachability via MySQL(double check)... Sun Sep 4 05:42:27 2022 - [info] ok. Sun Sep 4 05:42:27 2022 - [info] Alive Servers: Sun Sep 4 05:42:27 2022 - [info] 192.168.247.152(192.168.247.152:3306) Sun Sep 4 05:42:27 2022 - [info] 192.168.247.153(192.168.247.153:3306) Sun Sep 4 05:42:27 2022 - [info] Alive Slaves: Sun Sep 4 05:42:27 2022 - [info] 192.168.247.152(192.168.247.152:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:27 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:27 2022 - [info] 192.168.247.153(192.168.247.153:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:27 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:27 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 05:42:27 2022 - [info] Starting Non-GTID based failover. Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] ** Phase 1: Configuration Check Phase completed. Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] * Phase 2: Dead Master Shutdown Phase.. Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] Forcing shutdown so that applications never connect to the current master.. Sun Sep 4 05:42:27 2022 - [info] Executing master IP deactivation script: Sun Sep 4 05:42:27 2022 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 --command=stopssh --ssh_user=root Sun Sep 4 05:42:27 2022 - [info] done. Sun Sep 4 05:42:27 2022 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Sun Sep 4 05:42:27 2022 - [info] * Phase 2: Dead Master Shutdown Phase completed. Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] * Phase 3: Master Recovery Phase.. Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] The latest binary log file/position on all slaves is mysql-bin.000001:1388 Sun Sep 4 05:42:27 2022 - [info] Latest slaves (Slaves that received relay log files to the latest): Sun Sep 4 05:42:27 2022 - [info] 192.168.247.152(192.168.247.152:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:27 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:27 2022 - [info] 192.168.247.153(192.168.247.153:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:27 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:27 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 05:42:27 2022 - [info] The oldest binary log file/position on all slaves is mysql-bin.000001:1388 Sun Sep 4 05:42:27 2022 - [info] Oldest slaves: Sun Sep 4 05:42:27 2022 - [info] 192.168.247.152(192.168.247.152:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:27 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:27 2022 - [info] 192.168.247.153(192.168.247.153:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:27 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:27 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:27 2022 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. Sun Sep 4 05:42:27 2022 - [info] Sun Sep 4 05:42:28 2022 - [info] Fetching dead master's binary logs.. Sun Sep 4 05:42:28 2022 - [info] Executing command on the dead master 192.168.247.151(192.168.247.151:3306): save_binary_logs --command=save --start_file=mysql-bin.000001 --start_pos=1388 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58 Creating /data/mastermha/app1 if not exists.. ok. Concat binary/relay logs from mysql-bin.000001 pos 1388 to mysql-bin.000001 EOF into /data/mastermha/app1//saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog .. Binlog Checksum enabled Dumping binlog format description event, from position 0 to 156.. ok. No need to dump effective binlog data from /data/mysql//mysql-bin.000001 (pos starts 1388, filesize 1388). Skipping. Binlog Checksum enabled /data/mastermha/app1//saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog has no effective data events. Event not exists. Sun Sep 4 05:42:29 2022 - [info] Additional events were not found from the orig master. No need to save. Sun Sep 4 05:42:29 2022 - [info] Sun Sep 4 05:42:29 2022 - [info] * Phase 3.3: Determining New Master Phase.. Sun Sep 4 05:42:29 2022 - [info] Sun Sep 4 05:42:29 2022 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sun Sep 4 05:42:29 2022 - [info] All slaves received relay logs to the same position. No need to resync each other. Sun Sep 4 05:42:29 2022 - [info] Searching new master from slaves.. Sun Sep 4 05:42:29 2022 - [info] Candidate masters from the configuration file: Sun Sep 4 05:42:29 2022 - [info] 192.168.247.153(192.168.247.153:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 05:42:29 2022 - [info] Replicating from 192.168.247.151(192.168.247.151:3306) Sun Sep 4 05:42:29 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 05:42:29 2022 - [info] Non-candidate masters: Sun Sep 4 05:42:29 2022 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sun Sep 4 05:42:29 2022 - [info] New master is 192.168.247.153(192.168.247.153:3306) Sun Sep 4 05:42:29 2022 - [info] Starting master failover.. Sun Sep 4 05:42:29 2022 - [info] From: 192.168.247.151(192.168.247.151:3306) (current master) +--192.168.247.152(192.168.247.152:3306) +--192.168.247.153(192.168.247.153:3306) To: 192.168.247.153(192.168.247.153:3306) (new master) +--192.168.247.152(192.168.247.152:3306) Sun Sep 4 05:42:29 2022 - [info] Sun Sep 4 05:42:29 2022 - [info] * Phase 3.4: New Master Diff Log Generation Phase.. Sun Sep 4 05:42:29 2022 - [info] Sun Sep 4 05:42:29 2022 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Sep 4 05:42:29 2022 - [info] Sun Sep 4 05:42:29 2022 - [info] * Phase 3.5: Master Log Apply Phase.. Sun Sep 4 05:42:29 2022 - [info] Sun Sep 4 05:42:29 2022 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. Sun Sep 4 05:42:29 2022 - [info] Starting recovery on 192.168.247.153(192.168.247.153:3306).. Sun Sep 4 05:42:29 2022 - [info] This server has all relay logs. Waiting all logs to be applied.. Sun Sep 4 05:42:29 2022 - [info] done. Sun Sep 4 05:42:29 2022 - [info] All relay logs were successfully applied. Sun Sep 4 05:42:29 2022 - [info] Getting new master's binlog name and position.. Sun Sep 4 05:42:29 2022 - [info] mysql-bin.000002:1423 Sun Sep 4 05:42:29 2022 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.247.153', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=1423, MASTER_USER='repluser', MASTER_PASSWORD='xxx'; Sun Sep 4 05:42:29 2022 - [info] Executing master IP activate script: Sun Sep 4 05:42:29 2022 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 --new_master_host=192.168.247.153 --new_master_ip=192.168.247.153 --new_master_port=3306 --new_master_user='mhauser' --new_master_password=xxx Enabling the VIP - 192.168.247.100/24 on the new master - 192.168.247.153 Sun Sep 4 05:42:30 2022 - [info] OK. Sun Sep 4 05:42:30 2022 - [info] Setting read_only=0 on 192.168.247.153(192.168.247.153:3306).. Sun Sep 4 05:42:30 2022 - [info] ok. Sun Sep 4 05:42:30 2022 - [info] ** Finished master recovery successfully. Sun Sep 4 05:42:30 2022 - [info] * Phase 3: Master Recovery Phase completed. Sun Sep 4 05:42:30 2022 - [info] Sun Sep 4 05:42:30 2022 - [info] * Phase 4: Slaves Recovery Phase.. Sun Sep 4 05:42:30 2022 - [info] Sun Sep 4 05:42:30 2022 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Sun Sep 4 05:42:30 2022 - [info] Sun Sep 4 05:42:30 2022 - [info] -- Slave diff file generation on host 192.168.247.152(192.168.247.152:3306) started, pid: 20985. Check tmp log /data/mastermha/app1//192.168.247.152_3306_20220904054225.log if it takes time.. Sun Sep 4 05:42:31 2022 - [info] Sun Sep 4 05:42:31 2022 - [info] Log messages from 192.168.247.152 ... Sun Sep 4 05:42:31 2022 - [info] Sun Sep 4 05:42:30 2022 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sun Sep 4 05:42:31 2022 - [info] End of log messages from 192.168.247.152. Sun Sep 4 05:42:31 2022 - [info] -- 192.168.247.152(192.168.247.152:3306) has the latest relay log events. Sun Sep 4 05:42:31 2022 - [info] Generating relay diff files from the latest slave succeeded. Sun Sep 4 05:42:31 2022 - [info] Sun Sep 4 05:42:31 2022 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. Sun Sep 4 05:42:31 2022 - [info] Sun Sep 4 05:42:31 2022 - [info] -- Slave recovery on host 192.168.247.152(192.168.247.152:3306) started, pid: 20987. Check tmp log /data/mastermha/app1//192.168.247.152_3306_20220904054225.log if it takes time.. Sun Sep 4 05:42:32 2022 - [info] Sun Sep 4 05:42:32 2022 - [info] Log messages from 192.168.247.152 ... Sun Sep 4 05:42:32 2022 - [info] Sun Sep 4 05:42:31 2022 - [info] Starting recovery on 192.168.247.152(192.168.247.152:3306).. Sun Sep 4 05:42:31 2022 - [info] This server has all relay logs. Waiting all logs to be applied.. Sun Sep 4 05:42:31 2022 - [info] done. Sun Sep 4 05:42:31 2022 - [info] All relay logs were successfully applied. Sun Sep 4 05:42:31 2022 - [info] Resetting slave 192.168.247.152(192.168.247.152:3306) and starting replication from the new master 192.168.247.153(192.168.247.153:3306).. Sun Sep 4 05:42:31 2022 - [info] Executed CHANGE MASTER. Sun Sep 4 05:42:31 2022 - [info] Slave started. Sun Sep 4 05:42:32 2022 - [info] End of log messages from 192.168.247.152. Sun Sep 4 05:42:32 2022 - [info] -- Slave recovery on host 192.168.247.152(192.168.247.152:3306) succeeded. Sun Sep 4 05:42:32 2022 - [info] All new slave servers recovered successfully. Sun Sep 4 05:42:32 2022 - [info] Sun Sep 4 05:42:32 2022 - [info] * Phase 5: New master cleanup phase.. Sun Sep 4 05:42:32 2022 - [info] Sun Sep 4 05:42:32 2022 - [info] Resetting slave info on the new master.. Sun Sep 4 05:42:32 2022 - [info] 192.168.247.153: Resetting slave info succeeded. Sun Sep 4 05:42:32 2022 - [info] Master failover to 192.168.247.153(192.168.247.153:3306) completed successfully. Sun Sep 4 05:42:32 2022 - [info] Deleted server1 entry from /etc/mastermha/app1.cnf . Sun Sep 4 05:42:32 2022 - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.247.151(192.168.247.151:3306) to 192.168.247.153(192.168.247.153:3306) succeeded Master 192.168.247.151(192.168.247.151:3306) is down! Check MHA Manager logs at mha-manager:/data/mastermha/app1/manager.log for details. Started automated(non-interactive) failover. Invalidated master IP address on 192.168.247.151(192.168.247.151:3306) The latest slave 192.168.247.152(192.168.247.152:3306) has all relay logs for recovery. Selected 192.168.247.153(192.168.247.153:3306) as a new master. 192.168.247.153(192.168.247.153:3306): OK: Applying all logs succeeded. 192.168.247.153(192.168.247.153:3306): OK: Activated master IP address. 192.168.247.152(192.168.247.152:3306): This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 192.168.247.152(192.168.247.152:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.247.153(192.168.247.153:3306) 192.168.247.153(192.168.247.153:3306): Resetting slave info succeeded. Master failover to 192.168.247.153(192.168.247.153:3306) completed successfully. Sun Sep 4 05:42:32 2022 - [info] Sending mail.. [root@mha-manager ~]#
此时也收到邮件告警提醒
mha节点服务状态节点检查异常
vip也自动漂到指定的从节点
app1.cnf配置文件自动剔除manager 151节点信息
如果要再次运行MHA,需要先删除下面文件
因为MHA只能漂移一次，如果多次使用必须删除以下文件，要不MHA不可重用
[root@mha-manager ~]# rm -fr /data/mastermha/app1/* [root@mysql-master ~]# ls /data/mastermha/app1/ saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog [root@mysql-master ~]# [root@mysql-master ~]# rm -fr /data/mastermha/app1/* [root@mysql-master ~]#
将原有的master当slave加入
[root@mysql-slave-02 ~]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 65 Server version: 8.0.26 Source distribution Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show master logs; +------------------+-----------+-----------+ | Log_name | File_size | Encrypted | +------------------+-----------+-----------+ | mysql-bin.000001 | 179 | No | | mysql-bin.000002 | 12487 | No | +------------------+-----------+-----------+ 2 rows in set (0.00 sec) mysql>
将旧的master当成slave加入集群
[root@mysql-master ~]# ls /data/mastermha/app1/
saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog
[root@mysql-master ~]#
[root@mysql-master ~]# rm -fr /data/mastermha/app1/*
[root@mysql-master ~]#
[root@mysql-master ~]# systemctl start mysqld
[root@mysql-master ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.26 Source distribution
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CHANGE MASTER TO
-> MASTER_HOST='192.168.247.153',
-> MASTER_USER='repluser',
-> MASTER_PASSWORD='magedu',
-> MASTER_LOG_FILE='mysql-bin.000002',
-> MASTER_LOG_POS=12487;
Query OK, 0 rows affected, 8 warnings (0.02 sec)
mysql> start slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for source to send event
Master_Host: 192.168.247.153
Master_User: repluser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000002
Read_Master_Log_Pos: 12487
Relay_Log_File: mysql-master-relay-bin.000002
Relay_Log_Pos: 324
Relay_Master_Log_File: mysql-bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 12487
Relay_Log_Space: 540
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 153
Master_UUID: f8c35181-2c01-11ed-b28f-000c29d095cd
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
Master_public_key_path:
Get_master_public_key: 0
Network_Namespace:
1 row in set, 1 warning (0.00 sec)
mysql>
在次检查并启动mha
[root@mha-manager ~]# masterha_check_ssh --conf=/etc/mastermha/app1.cnf Sun Sep 4 07:49:06 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 07:49:06 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 07:49:06 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 07:49:06 2022 - [info] Starting SSH connection tests.. Sun Sep 4 07:49:07 2022 - [debug] Sun Sep 4 07:49:06 2022 - [debug] Connecting via SSH from [email protected](192.168.247.153:22) to [email protected](192.168.247.152:22).. Sun Sep 4 07:49:07 2022 - [debug] ok. Sun Sep 4 07:49:07 2022 - [debug] Connecting via SSH from [email protected](192.168.247.153:22) to [email protected](192.168.247.151:22).. Sun Sep 4 07:49:07 2022 - [debug] ok. Sun Sep 4 07:49:08 2022 - [debug] Sun Sep 4 07:49:07 2022 - [debug] Connecting via SSH from [email protected](192.168.247.152:22) to [email protected](192.168.247.153:22).. Sun Sep 4 07:49:07 2022 - [debug] ok. Sun Sep 4 07:49:07 2022 - [debug] Connecting via SSH from [email protected](192.168.247.152:22) to [email protected](192.168.247.151:22).. Sun Sep 4 07:49:07 2022 - [debug] ok. Sun Sep 4 07:49:09 2022 - [debug] Sun Sep 4 07:49:07 2022 - [debug] Connecting via SSH from [email protected](192.168.247.151:22) to [email protected](192.168.247.153:22).. Sun Sep 4 07:49:08 2022 - [debug] ok. Sun Sep 4 07:49:08 2022 - [debug] Connecting via SSH from [email protected](192.168.247.151:22) to [email protected](192.168.247.152:22).. Sun Sep 4 07:49:08 2022 - [debug] ok. Sun Sep 4 07:49:09 2022 - [info] All SSH connection tests passed successfully. [root@mha-manager ~]# [root@mha-manager ~]# [root@mha-manager ~]# masterha_check_repl --conf=/etc/mastermha/app1.cnf Sun Sep 4 07:49:12 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 07:49:12 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 07:49:12 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 07:49:12 2022 - [info] MHA::MasterMonitor version 0.58. Sun Sep 4 07:49:13 2022 - [info] GTID failover mode = 0 Sun Sep 4 07:49:13 2022 - [info] Dead Servers: Sun Sep 4 07:49:13 2022 - [info] Alive Servers: Sun Sep 4 07:49:13 2022 - [info] 192.168.247.153(192.168.247.153:3306) Sun Sep 4 07:49:13 2022 - [info] 192.168.247.152(192.168.247.152:3306) Sun Sep 4 07:49:13 2022 - [info] 192.168.247.151(192.168.247.151:3306) Sun Sep 4 07:49:13 2022 - [info] Alive Slaves: Sun Sep 4 07:49:13 2022 - [info] 192.168.247.152(192.168.247.152:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 07:49:13 2022 - [info] Replicating from 192.168.247.153(192.168.247.153:3306) Sun Sep 4 07:49:13 2022 - [info] 192.168.247.151(192.168.247.151:3306) Version=8.0.26 (oldest major version between slaves) log-bin:enabled Sun Sep 4 07:49:13 2022 - [info] Replicating from 192.168.247.153(192.168.247.153:3306) Sun Sep 4 07:49:13 2022 - [info] Primary candidate for the new Master (candidate_master is set) Sun Sep 4 07:49:13 2022 - [info] Current Alive Master: 192.168.247.153(192.168.247.153:3306) Sun Sep 4 07:49:13 2022 - [info] Checking slave configurations.. Sun Sep 4 07:49:13 2022 - [info] read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 07:49:13 2022 - [warning] relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306). Sun Sep 4 07:49:13 2022 - [info] Checking replication filtering settings.. Sun Sep 4 07:49:13 2022 - [info] binlog_do_db= , binlog_ignore_db= Sun Sep 4 07:49:13 2022 - [info] Replication filtering check ok. Sun Sep 4 07:49:13 2022 - [info] GTID (with auto-pos) is not supported Sun Sep 4 07:49:13 2022 - [info] Starting SSH connection tests.. Sun Sep 4 07:49:15 2022 - [info] All SSH connection tests passed successfully. Sun Sep 4 07:49:15 2022 - [info] Checking MHA Node version.. Sun Sep 4 07:49:16 2022 - [info] Version check ok. Sun Sep 4 07:49:16 2022 - [info] Checking SSH publickey authentication settings on the current master.. Sun Sep 4 07:49:16 2022 - [info] HealthCheck: SSH to 192.168.247.153 is reachable. Sun Sep 4 07:49:17 2022 - [info] Master MHA Node version is 0.58. Sun Sep 4 07:49:17 2022 - [info] Checking recovery script configurations on 192.168.247.153(192.168.247.153:3306).. Sun Sep 4 07:49:17 2022 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000002 Sun Sep 4 07:49:17 2022 - [info] Connecting to [email protected](192.168.247.153:22).. Creating /data/mastermha/app1 if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /data/mysql/, up to mysql-bin.000002 Sun Sep 4 07:49:17 2022 - [info] Binlog setting check done. Sun Sep 4 07:49:17 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sun Sep 4 07:49:17 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.152 --slave_ip=192.168.247.152 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-01-relay-bin.000002 --slave_pass=xxx Sun Sep 4 07:49:17 2022 - [info] Connecting to [email protected](192.168.247.152:22).. Checking slave recovery environment settings.. Relay log found at /var/lib/mysql, up to mysql-slave-01-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mysql-slave-01-relay-bin.000002 Checking if super_read_only is defined and turned on.. not present or turned off, ignoring. Testing mysql connection and privileges.. mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Sep 4 07:49:17 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.151 --slave_ip=192.168.247.151 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-master-relay-bin.000002 --slave_pass=xxx Sun Sep 4 07:49:17 2022 - [info] Connecting to [email protected](192.168.247.151:22).. Checking slave recovery environment settings.. Relay log found at /var/lib/mysql, up to mysql-master-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mysql-master-relay-bin.000002 Checking if super_read_only is defined and turned on.. not present or turned off, ignoring. Testing mysql connection and privileges.. mysql: [Warning] Using a password on the command line interface can be insecure. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sun Sep 4 07:49:18 2022 - [info] Slaves settings check done. Sun Sep 4 07:49:18 2022 - [info] 192.168.247.153(192.168.247.153:3306) (current master) +--192.168.247.152(192.168.247.152:3306) +--192.168.247.151(192.168.247.151:3306) Sun Sep 4 07:49:18 2022 - [info] Checking replication health on 192.168.247.152.. Sun Sep 4 07:49:18 2022 - [info] ok. Sun Sep 4 07:49:18 2022 - [info] Checking replication health on 192.168.247.151.. Sun Sep 4 07:49:18 2022 - [info] ok. Sun Sep 4 07:49:18 2022 - [info] Checking master_ip_failover_script status: Sun Sep 4 07:49:18 2022 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.247.153 --orig_master_ip=192.168.247.153 --orig_master_port=3306 Checking the Status of the script.. OK Sun Sep 4 07:49:18 2022 - [info] OK. Sun Sep 4 07:49:18 2022 - [warning] shutdown_script is not defined. Sun Sep 4 07:49:18 2022 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK. [root@mha-manager ~]# [root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.cnf app1 is stopped(2:NOT_RUNNING). [root@mha-manager ~]# [root@mha-manager ~]# [root@mha-manager ~]# masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover Sun Sep 4 07:49:54 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Sep 4 07:49:54 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf.. Sun Sep 4 07:49:54 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf..
此时监控启动正常，故障模拟恢复成功
标签：实战,info,Sep,Sun,05,192.168,MHA,案例,2022 From： https://www.cnblogs.com/cyh00001/p/16654088.html