爱程序网

【MySQL】MHA部署与MasterFailover代码分析

来源: 阅读:

官网:https://code.google.com/p/mysql-master-ha/

参考:http://blog.csdn.net/wulantian/article/details/13287975

参考:http://www.cnblogs.com/wingsless/p/4033093.html

参考:http://www.cnblogs.com/xuanzhi201111/p/4231412.html#jtss-tsina

参考:http://ylw6006.blog.51cto.com/470441/890360/

参考:http://os.51cto.com/art/201307/401702.htm

参考:http://www.it165.net/database/html/201508/13780.html

下载:http://pan.baidu.com/s/1pJ0VkSz

部署:

[root@dns packages]# tar -zxvf mha4mysql-manager-0.53.tar.gz[root@dns mha4mysql-manager-0.53]# perl Makefile.PL[root@dns mha4mysql-manager-0.53]# make && make install配置有很多参数,可以看看libMHA下面的Config.pm代码[root@dns mha4mysql-manager-0.53]# vim /etc/3307.cnf[server default]manager_log=/data1/masterha/3307_leju/manager.logmanager_workdir=/data1/masterha/3307_lejumulti_tier_slave=3password=passwdping_interval=1repl_password=x8uf7nbv5x64repl_user=replshutdown_script="/samples/scripts/power_manager" //设置故障发生后关闭故障主机脚本(该脚本的主要作用是关闭主机放在发生脑裂
master_pid_file=/data1/mysql/3307_leju/mysql.pid //shutdown_script脚本中需要使用的pid文件位置,通过pid文件中的pid进程号强制杀掉主库实例
ssh_user=rootuser=ha[server1]hostname=10.207.0.125master_binlog_dir="/data1/mysql/3307_leju"port=3307[server2]hostname=10.207.0.126master_binlog_dir="/data1/mysql/3307_leju"port=3307[server3]hostname=10.207.0.127master_binlog_dir="/data1/mysql/3307_leju"port=3307[server4]hostname=10.207.0.128master_binlog_dir="/data1/mysql/3307_leju"port=3307启动前检查复制关系[root@dns bin]# masterha_check_repl --conf=/etc/masterha/3307.cnf启动managermasterha_manager启动有很多参数,参数说明不列举说明
[root@dns bin]# masterha_manager
--conf=/etc/masterha/3307.cnf --remove_dead_master_conf --ignore_last_failover

功能分析:

MHA的故障切换是由lib/MHA/MasterFailover.pm代码完成。

编译后位置移动到/usr/local/share/perl5/MHA/MasterFailover.pm

代码主体为do_master_failover函数完成,代码片段:

sub do_master_failover {  my $error_code = 1;  my ( $dead_master, $new_master );  eval {    my @servers_config = init_config();    $log->info("Starting master failover.");    $log->info("* Phase 1: Configuration Check Phase..n");
$dead_master = check_settings( @servers_config ); $log->info("** Phase 1: Configuration Check Phase completed.n"); $log->info("* Phase 2: Dead Master Shutdown Phase..n");
force_shutdown(
$dead_master); $log->info("* Phase 2: Dead Master Shutdown Phase completed.n"); $log->info("* Phase 3: Master Recovery Phase..n");$log->info("* Phase 3.1: Getting Latest Slaves Phase..n");
check_set_latest_slaves();
$log->info("* Phase 3.2: Saving Dead Master's Binlog Phase..n");
save_master_binlog(
$dead_master); $log->info("* Phase 3.3: Determining New Master Phase..n");
my $latest_base_slave = find_latest_base_slave($dead_master); $new_master = select_new_master( $dead_master, $latest_base_slave ); my ( $master_log_file, $master_log_pos ) = recover_master( $dead_master, $new_master, $latest_base_slave ); $new_master->{activated} = 1; $log->info("* Phase 3: Master Recovery Phase completed.n"); $log->info("* Phase 4: Slaves Recovery Phase..n"); $error_code = recover_slaves( $dead_master, $new_master, $latest_base_slave, $master_log_file, $master_log_pos); if ( $g_remove_dead_master_conf && $error_code == 0 ) { MHA::Config::delete_block_and_save( $g_config_file, $dead_master->{id}, $log );} cleanup(); }; ...}

可以通过增加几行代码实现MHA切换通知DNS达到前端请求的切换。

# 建立连接my $dbh = DBI->connect("DBI:mysql:database=dblayer;mysql_socket=/tmp/mysql_3307.sock","dblayer","dblayer");# 下线主库域名,IsLive状态修改(Bind_DLZ的MySQL驱动查询增加IsLive字段判断)my $rows = $dbh->do("UPDATE dns_records SET IsLive=0 WHERE data='$dead_master->{ip}' and port='$dead_master->{port}'");# 获取主库host(Bind的dns_records表中使用IP和PORT对应一个唯一HOST,多实例的支持)my $sth = $dbh->prepare("select host from dns_records where data='$dead_master->{ip}' and port='$dead_master->{port}'");$sth->execute();my $ref = $sth->fetchrow();# 将新主库更新hostmy $rows = $dbh->do("UPDATE dns_records SET HOST='$ref' WHERE data='$new_master->{ip}' and port='$dead_master->{port}'");# 将切换记录到库my $rows = $dbh->do("INSERT INTO masterfailover_log(old_master_ip,new_master_ip,port,type) values ('$dead_master->{ip}','$new_master->{ip}','$dead_master->{port}',1)");# 断开连接
$sth->finish;
$dbh->disconnect();

在加入shutdown_script后,出现报错:

Thu Sep 24 14:21:42 2015 - [info]   /usr/local/mha_manager/samples/scripts/power_manager --command=status --ssh_user=root --host=10.207.0.125 --ip=10.207.0.125 Undefined subroutine &main::FIXME_xxx called at /usr/local/mha_manager/samples/scripts/power_manager line 387.Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln235]  Failed to get power status with return code 1:0.Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations.  at /usr/local/bin/masterha_check_repl line 48Thu Sep 24 14:21:42 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers.Thu Sep 24 14:21:42 2015 - [info] Got exit code 1 (Not master dead).MySQL Replication Health is NOT OK!

打开power_manager可以看到,报错的部分是一个未定义函数FIXME_xxx,代码是想通过服务器管理地址进行操作,尝试将未定义函数部分去掉会出现更多问题。

在power_manager代码下面有一些注释说明,说明中有:

# killing mysqld with specified pid file. This is useful when you run multiple MySQL instances and want to stop only specified instancepower_manager --command=stopssh --host=master_server --ssh_user=root --pid_file=/var/lib/mysql/mysqld.pid

所以不在配置中添加shutdown_script,直接在MasterFailover.pm中force_shutdown_internal函数最后添加power_manager的调用。

system("/usr/local/mha_manager/samples/scripts/power_manager --command=stopssh --host='$dead_master->{ip}' --ssh_user=root --pid_file='$dead_master->{master_pid_file}'");

因为power_manager有未定义的函数,如果直接调用,在异常主库可以ssh通但是pid已经不存在或者根本ssh不通的情况下,因为stopssh函数返回值异常仍会报错未命名函数,所以改一下power_manager,注释掉一部分代码,防止killall -9 mysql mysqld。

sub stopssh {  ...if ($pid_file) {    $command =""if [ ! -e $pid_file ]; then exit 1; fi; pid=\`cat $pid_file\`; rm -f $pid_file; kill -9 \$pid; a=\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \$1'}\`; if [ "a\$a" = "a0" ]; then exit 10; fi; sleep 1; a=\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \$1'}\`; if [ "a\$a" = "a0" ]; then exit 10; else exit 1; fi"";    ( $high_ret, $low_ret ) = MHA::ManagerUtil::exec_system(      "ssh $ssh_user_host $MHA::ManagerConst::SSH_OPT_CHECK $command");    if ( $high_ret == $SSH_STOP_OK && $low_ret == 0 ) {      print "ssh reachable. mysqld stopped. power off not needed.n";      return $high_ret;    }    elsif ( $high_ret == 1 && $low_ret == 0 ) {      print "ssh reachable. not found $pid_file,mysqld stopped. power off not needed.n";      return 10;    }    else {      print "Killing mysqld instance based on $pid_file failed.n";return 10;    }  }  # 后面killall -9 mysql mysqld 注释}

在配置里面加report_script=/usr/local/bin/send_report后

修改mha_manager/samples/scripts/send_report的代码可以通知ZABBIX和邮件通知。ZABBIX在mha_manager主机加上一个监控项,把Item设置为Zabbix trapper,Type of information设置为字符

my @array=split(/ /,$subject);my $sender=join('',$array[0],$array[7]);system("/data1/scripts/zabbix_sender --zabbix-server serverip --port 10051 --host 'mhaserver' --key mysql.MHA --value '$sender'");system("/bin/echo '$body' | /usr/bin/mutt -s '$subject' 'jiangxu@test.com'");

 

相关文章列表:
关于爱程序网 - 联系我们 - 广告服务 - 友情链接 - 网站地图 - 版权声明 - 人才招聘 - 帮助