官网:https://code.google.com/p/mysql-master-ha/
参考:http://blog.csdn.net/wulantian/article/details/13287975
参考:http://www.cnblogs.com/wingsless/p/4033093.html
参考:http://www.cnblogs.com/xuanzhi201111/p/4231412.html#jtss-tsina
参考:http://ylw6006.blog.51cto.com/470441/890360/
参考:http://os.51cto.com/art/201307/401702.htm
参考:http://www.it165.net/database/html/201508/13780.html
下载:http://pan.baidu.com/s/1pJ0VkSz
部署:
[root@dns packages]# tar -zxvf mha4mysql-manager-0.53.tar.gz[root@dns mha4mysql-manager-0.53]# perl Makefile.PL[root@dns mha4mysql-manager-0.53]# make && make install配置有很多参数,可以看看libMHA下面的Config.pm代码[root@dns mha4mysql-manager-0.53]# vim /etc/3307.cnf[server default]manager_log=/data1/masterha/3307_leju/manager.logmanager_workdir=/data1/masterha/3307_lejumulti_tier_slave=3password=passwdping_interval=1repl_password=x8uf7nbv5x64repl_user=replshutdown_script="/samples/scripts/power_manager" //设置故障发生后关闭故障主机脚本(该脚本的主要作用是关闭主机放在发生脑裂
master_pid_file=/data1/mysql/3307_leju/mysql.pid //shutdown_script脚本中需要使用的pid文件位置,通过pid文件中的pid进程号强制杀掉主库实例ssh_user=rootuser=ha[server1]hostname=10.207.0.125master_binlog_dir="/data1/mysql/3307_leju"port=3307[server2]hostname=10.207.0.126master_binlog_dir="/data1/mysql/3307_leju"port=3307[server3]hostname=10.207.0.127master_binlog_dir="/data1/mysql/3307_leju"port=3307[server4]hostname=10.207.0.128master_binlog_dir="/data1/mysql/3307_leju"port=3307启动前检查复制关系[root@dns bin]# masterha_check_repl --conf=/etc/masterha/3307.cnf启动managermasterha_manager启动有很多参数,参数说明不列举说明
[root@dns bin]# masterha_manager --conf=/etc/masterha/3307.cnf --remove_dead_master_conf --ignore_last_failover
功能分析:
MHA的故障切换是由lib/MHA/MasterFailover.pm代码完成。
编译后位置移动到/usr/local/share/perl5/MHA/MasterFailover.pm。
代码主体为do_master_failover函数完成,代码片段:
sub do_master_failover { my $error_code = 1; my ( $dead_master, $new_master ); eval { my @servers_config = init_config(); $log->info("Starting master failover."); $log->info("* Phase 1: Configuration Check Phase..n");
$dead_master = check_settings( @servers_config ); $log->info("** Phase 1: Configuration Check Phase completed.n"); $log->info("* Phase 2: Dead Master Shutdown Phase..n");
force_shutdown($dead_master); $log->info("* Phase 2: Dead Master Shutdown Phase completed.n"); $log->info("* Phase 3: Master Recovery Phase..n");$log->info("* Phase 3.1: Getting Latest Slaves Phase..n");
check_set_latest_slaves(); $log->info("* Phase 3.2: Saving Dead Master's Binlog Phase..n");
save_master_binlog($dead_master); $log->info("* Phase 3.3: Determining New Master Phase..n");
my $latest_base_slave = find_latest_base_slave($dead_master); $new_master = select_new_master( $dead_master, $latest_base_slave ); my ( $master_log_file, $master_log_pos ) = recover_master( $dead_master, $new_master, $latest_base_slave ); $new_master->{activated} = 1; $log->info("* Phase 3: Master Recovery Phase completed.n"); $log->info("* Phase 4: Slaves Recovery Phase..n"); $error_code = recover_slaves( $dead_master, $new_master, $latest_base_slave, $master_log_file, $master_log_pos); if ( $g_remove_dead_master_conf && $error_code == 0 ) { MHA::Config::delete_block_and_save( $g_config_file, $dead_master->{id}, $log );} cleanup(); }; ...}
可以通过增加几行代码实现MHA切换通知DNS达到前端请求的切换。
# 建立连接my $dbh = DBI->connect("DBI:mysql:database=dblayer;mysql_socket=/tmp/mysql_3307.sock","dblayer","dblayer");# 下线主库域名,IsLive状态修改(Bind_DLZ的MySQL驱动查询增加IsLive字段判断)my $rows = $dbh->do("UPDATE dns_records SET IsLive=0 WHERE data='$dead_master->{ip}' and port='$dead_master->{port}'");# 获取主库host(Bind的dns_records表中使用IP和PORT对应一个唯一HOST,多实例的支持)my $sth = $dbh->prepare("select host from dns_records where data='$dead_master->{ip}' and port='$dead_master->{port}'");$sth->execute();my $ref = $sth->fetchrow();# 将新主库更新hostmy $rows = $dbh->do("UPDATE dns_records SET HOST='$ref' WHERE data='$new_master->{ip}' and port='$dead_master->{port}'");# 将切换记录到库my $rows = $dbh->do("INSERT INTO masterfailover_log(old_master_ip,new_master_ip,port,type) values ('$dead_master->{ip}','$new_master->{ip}','$dead_master->{port}',1)");# 断开连接
$sth->finish;
$dbh->disconnect();
在加入shutdown_script后,出现报错:
Thu Sep 24 14:21:42 2015 - [info] /usr/local/mha_manager/samples/scripts/power_manager --command=status --ssh_user=root --host=10.207.0.125 --ip=10.207.0.125 Undefined subroutine &main::FIXME_xxx called at /usr/local/mha_manager/samples/scripts/power_manager line 387.Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln235] Failed to get power status with return code 1:0.Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. at /usr/local/bin/masterha_check_repl line 48Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers.Thu Sep 24 14:21:42 2015 - [info] Got exit code 1 (Not master dead).MySQL Replication Health is NOT OK!
打开power_manager可以看到,报错的部分是一个未定义函数FIXME_xxx,代码是想通过服务器管理地址进行操作,尝试将未定义函数部分去掉会出现更多问题。
在power_manager代码下面有一些注释说明,说明中有:
# killing mysqld with specified pid file. This is useful when you run multiple MySQL instances and want to stop only specified instancepower_manager --command=stopssh --host=master_server --ssh_user=root --pid_file=/var/lib/mysql/mysqld.pid
所以不在配置中添加shutdown_script,直接在MasterFailover.pm中force_shutdown_internal函数最后添加power_manager的调用。
system("/usr/local/mha_manager/samples/scripts/power_manager --command=stopssh --host='$dead_master->{ip}' --ssh_user=root --pid_file='$dead_master->{master_pid_file}'");
因为power_manager有未定义的函数,如果直接调用,在异常主库可以ssh通但是pid已经不存在或者根本ssh不通的情况下,因为stopssh函数返回值异常仍会报错未命名函数,所以改一下power_manager,注释掉一部分代码,防止killall -9 mysql mysqld。
sub stopssh { ...if ($pid_file) { $command =""if [ ! -e $pid_file ]; then exit 1; fi; pid=\`cat $pid_file\`; rm -f $pid_file; kill -9 \$pid; a=\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \$1'}\`; if [ "a\$a" = "a0" ]; then exit 10; fi; sleep 1; a=\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \$1'}\`; if [ "a\$a" = "a0" ]; then exit 10; else exit 1; fi""; ( $high_ret, $low_ret ) = MHA::ManagerUtil::exec_system( "ssh $ssh_user_host $MHA::ManagerConst::SSH_OPT_CHECK $command"); if ( $high_ret == $SSH_STOP_OK && $low_ret == 0 ) { print "ssh reachable. mysqld stopped. power off not needed.n"; return $high_ret; } elsif ( $high_ret == 1 && $low_ret == 0 ) { print "ssh reachable. not found $pid_file,mysqld stopped. power off not needed.n"; return 10; } else { print "Killing mysqld instance based on $pid_file failed.n";return 10; } } # 后面killall -9 mysql mysqld 注释}
在配置里面加report_script=/usr/local/bin/send_report后
修改mha_manager/samples/scripts/send_report的代码可以通知ZABBIX和邮件通知。ZABBIX在mha_manager主机加上一个监控项,把Item设置为Zabbix trapper,Type of information设置为字符
my @array=split(/ /,$subject);my $sender=join('',$array[0],$array[7]);system("/data1/scripts/zabbix_sender --zabbix-server serverip --port 10051 --host 'mhaserver' --key mysql.MHA --value '$sender'");system("/bin/echo '$body' | /usr/bin/mutt -s '$subject' 'jiangxu@test.com'");