MySQL主主複製+MMM實現高可用(一)

jx_yu發表於2014-12-15
架構圖:

環境規劃

角色

IP

VIP

用途

DB1

192.168.56.101

192.168.56.111/113

Master提供wirte+read

DB2

192.168.56.102

192.168.56.112

Master提供read

Monitor Server

192.168.56.103

NULL

負責所有的監控工作的監控守護程式,決定節點的移除等等

準備工作

所有節點上關閉iptablesselinux

~]# iptables -F

~]# iptables -t nat -F

~]# /etc/init.d/iptables stop

~]# chkconfig iptables off

~]# chkconfig --list iptables

iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off

~]# grep SELINUX= /etc/selinux/config |grep ^[^#]

SELINUX=disabled

軟體環境準備

OS VersionRed Hat Enterprise Linux Server release 6.3 (Santiago)

MySQL Version: MySQL-server-5.6.21-1.el6.x86_64

MMM Version:mysql-mmm-2.2.1

MySQL DownLoad Path: http://dev.mysql.com/downloads/mysql/

MMM DownLoad Path:http://mysql-mmm.org/_media/:mmm2:mysql-mmm-2.2.1.tar.gz

前提:MySQL主主同步已經配置完成

DB1>show processlist\G

*************************** 2. row ***************************

     Id: 6

   User: slave

   Host: 192.168.56.102:39971

     db: NULL

Command: Binlog Dump

   Time: 658

  State: Master has sent all binlog to slave; waiting for binlog to be updated

   Info: NULL

*************************** 3. row ***************************

     Id: 7

   User: system user

   Host:

     db: NULL

Command: Connect

   Time: 135

  State: Waiting for master to send event

   Info: NULL

*************************** 4. row ***************************

     Id: 8

   User: system user

   Host:

     db: NULL

Command: Connect

   Time: 135

  State: Slave has read all relay log; waiting for the slave I/O thread to update it

   Info: NULL

4 rows in set (0.00 sec)

 

DB2>show processlist\G

*************************** 1. row ***************************

     Id: 1

   User: system user

   Host:

     db: NULL

Command: Connect

   Time: 778

  State: Slave has read all relay log; waiting for the slave I/O thread to update it

   Info: NULL

*************************** 2. row ***************************

     Id: 2

   User: system user

   Host:

     db: NULL

Command: Connect

   Time: 778

  State: Waiting for master to send event

   Info: NULL

*************************** 4. row ***************************

     Id: 20

   User: slave

   Host: 192.168.56.101:32870

     db: NULL

Command: Binlog Dump

   Time: 255

  State: Master has sent all binlog to slave; waiting for binlog to be updated

   Info: NULL

4 rows in set (0.00 sec)

注意

One should set read-only=1 in the confiuration of all MySQL servers, MMM will change
that to read-only=0 on the host with the active master role.

應該把所有節點設定為read_only模式,MMM將會修改active master為read-only=0提供wirte功能

MMM搭建步驟

1.安裝Mysql-mmm

#所有節點都安裝

~]# tar -zxvf mysql-mmm-2.2.1.tar.gz

~]# cd mysql-mmm-2.2.1

~]# make && make install

2.建立MMM代理和監控賬號

#在DB1和DB2上分別執行

由於AB互為主從,故在一邊執行、兩邊檢查即可

#建立監控賬號

DB1> GRANT REPLICATION CLIENT ON *.* TO 'mmm_monitor'@'192.168.56.%' IDENTIFIED BY 'monitor_password';

Query OK, 0 rows affected (0.02 sec)

#建立代理賬號

DB1>GRANT SUPER,REPLICATION CLIENT,PROCESS ON *.* TO 'mmm_agent'@'192.168.56.%'  IDENTIFIED BY 'agent_password';

Query OK, 0 rows affected (0.00 sec)

DB1>flush privileges;

Query OK, 0 rows affected (0.03 sec)

#檢查A和B兩db上是否都存在監控和代理賬號

DB1>select user,host,password from mysql.user where user in ('mmm_monitor','mmm_agent');

+-------------+--------------+-------------------------------------------+

| user        | host         | password                                  |

+-------------+--------------+-------------------------------------------+

| mmm_monitor | 192.168.56.% | *C1F414D9BAF378B656A849B31F9F8AF3125F558B |

| mmm_agent   | 192.168.56.% | *9404D503D0FAE1825CFEF873D0E7A38A26BBAB0F |

+-------------+--------------+-------------------------------------------+

2 rows in set (0.02 sec)

DB2>select user,host,password from mysql.user where user in ('mmm_monitor','mmm_agent');

+-------------+--------------+-------------------------------------------+

| user        | host         | password                                  |

+-------------+--------------+-------------------------------------------+

| mmm_monitor | 192.168.56.% | *C1F414D9BAF378B656A849B31F9F8AF3125F558B |

| mmm_agent   | 192.168.56.% | *9404D503D0FAE1825CFEF873D0E7A38A26BBAB0F |

+-------------+--------------+-------------------------------------------+

2 rows in set (0.00 sec)

3.配置mysql-mmm

所有的配置選項都集合在了一個叫/etc/mysql-mmm/mmm_common.conf的單獨檔案中,系統中所有主機的該檔案內容都是一樣的配置完後不要忘記了拷貝這個檔案到所有的主機(包括監控主機)!

a)      mmm_common.conf

#在DB1上配置mmm_common.conf,並且拷貝給其他所有主機(包括監控主機)

~]# vim /etc/mysql-mmm/mmm_common.conf

active_master_role      writer

 

        cluster_interface               eth2

 

        pid_path                                /var/run/mmm_agentd.pid

        bin_path                                /usr/lib/mysql-mmm/

 

    replication_user       slave        #MySQL主主同步的帳號(這些要和前面設定的保持一致!)

    replication_password   mysqlab              #同步的密碼

 

        agent_user      mmm_agent           #mmm-agent使用者名稱

        agent_password  agent_password  #mmm-agent密碼

 

        ip              192.168.56.101      #DB1的IP

        mode            master

        peer            db1

 

        ip              192.168.56.102      #DB2的IP

        mode            master

        peer            db2

 

        hosts                                   db1, db2

        ips                                     192.168.56.113              #write VIP

        mode                                    exclusive

 

        hosts                                   db1, db2

        ips                                     192.168.56.111,192.168.56.112           #Read VIP

        mode                                    balanced

#拷貝給所有主機(包括監控主機)

~]# scp /etc/mysql-mmm/mmm_agent.conf 192.168.56.102:/etc/mysql-mmm/

~]# scp /etc/mysql-mmm/mmm_agent.conf 192.168.56.103:/etc/mysql-mmm/

b)     mmm_agent.conf

MySQL DB主機(DB1DB2)上,需要編輯/etc/mysql-mmm/mmm_agent.conf檔案

#根據其他主機的不同更改db1的值(db2就將db1更改成db2)

~]# cat /etc/mysql-mmm/mmm_agent.conf

include mmm_common.conf

this db1             #DB2上此處為db2

c)      mmm_mon.conf

監控主機上我們需要編輯/etc/mysql-mmm/mmm_mon.conf檔案

~]# vim /etc/mysql-mmm/mmm_mon.conf

include mmm_common.conf

 

        ip                                      127.0.0.1

        pid_path                                /var/run/mmm_mond.pid

        bin_path                                /usr/lib/mysql-mmm/

        status_path                             /var/lib/misc/mmm_mond.status

        ping_ips                                192.168.56.101, 192.168.56.102         #監控伺服器ip

auto_set_online                                             60

 

        monitor_user                    mmm_monitor                #mmm_monitor使用者名稱

        monitor_password                RepMonitor          #mmm_monitor使用者名稱

 

debug 0

4.啟動MMM代理

#(在DB1DB2主機上)編輯/etc/default/mysql-mmm-agent來開啟

~]#  cat /etc/default/mysql-mmm-agent

ENABLED=1

#然後啟動它:

~]# /etc/init.d/mysql-mmm-agent start

Daemon bin: '/usr/sbin/mmm_agentd'

Daemon pid: '/var/run/mmm_agentd.pid'

Starting MMM Agent daemon... Ok

5.啟動MMM監控

#在監控主機上

~]# /etc/init.d/mysql-mmm-monitor start

Daemon bin: '/usr/sbin/mmm_mond'

Daemon pid: '/var/run/mmm_mond.pid'

Starting MMM Monitor daemon: Ok

6.設定開機自啟動MMM服務

#DB節點

~]# chkconfig --add mysql-mmm-agent

~]# chkconfig mysql-mmm-agent on

~]# chkconfig --list mysql-mmm-agent

mysql-mmm-agent 0:off   1:off   2:on    3:on    4:on    5:on    6:off

#Monitor節點

~]# chkconfig --add mysql-mmm-monitor

~]# chkconfig  mysql-mmm-monitor on

~]# chkconfig --list mysql-mmm-monitor

mysql-mmm-monitor       0:off   1:off   2:on    3:on    4:on    5:on    6:off

7.檢視監控狀態

~]# mmm_control  show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), writer(192.168.56.113)

  db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.112)

~]# mmm_control  checks

db2  ping         [last change: 2014/12/02 18:18:41]  OK

db2  mysql        [last change: 2014/12/02 18:18:41]  OK

db2  rep_threads  [last change: 2014/12/02 18:18:41]  OK

db2  rep_backlog  [last change: 2014/12/02 18:18:41]  OK: Backlog is null

db1  ping         [last change: 2014/12/02 18:18:41]  OK

db1  mysql        [last change: 2014/12/02 18:18:41]  OK

db1  rep_threads  [last change: 2014/12/02 18:18:41]  OK

db1  rep_backlog  [last change: 2014/12/02 18:18:41]  OK: Backlog is null

8.測試切換

#以下所有測試都是基於如下狀態

~]# mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.112)

  db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.111), writer(192.168.56.113)

a)      停止DB1MySQL服務

~]# tail -f /var/log/mysql-mmm/mmm_mond.log

tem error: 111

2014/12/03 10:37:50 ERROR Check 'mysql' on 'db1' has failed for 10 seconds! Message: ERROR: Connect error (host = 192.168.56.101:3306, user = mmm_monitor)! Lost connection to MySQL server at 'reading initial communication packet', system error: 111

2014/12/03 10:37:52 FATAL State of host 'db1' changed from ONLINE to HARD_OFFLINE (ping: OK, mysql: not OK)

2014/12/03 10:37:52  INFO Removing all roles from host 'db1':

2014/12/03 10:37:52  INFO     Removed role 'reader(192.168.56.112)' from host 'db1'

2014/12/03 10:37:52  INFO Orphaned role 'reader(192.168.56.112)' has been assigned to 'db2'

結果

~]# mmm_control show

  db1(192.168.56.101) master/HARD_OFFLINE. Roles:

  db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.111), reader(192.168.56.112), writer(192.168.56.113)

DB1上的VIP【reader(192.168.56.112)】會自動遷移DB2

b)     停止DB2MySQL服務

~]# tail -f /var/log/mysql-mmm/mmm_mond.log

2014/12/03 10:54:10 ERROR Check 'mysql' on 'db2' has failed for 10 seconds! Message: ERROR: Connect error (host = 192.168.56.102:3306, user = mmm_monitor)! Lost connection to MySQL server at 'reading initial communication packet', system error: 111

2014/12/03 10:54:10  INFO Check 'rep_backlog' on 'db1' is ok!

2014/12/03 10:54:11 FATAL State of host 'db2' changed from ONLINE to HARD_OFFLINE (ping: OK, mysql: not OK)

2014/12/03 10:54:11  INFO Removing all roles from host 'db2':

2014/12/03 10:54:11  INFO     Removed role 'reader(192.168.56.111)' from host 'db2'

2014/12/03 10:54:11  INFO     Removed role 'writer(192.168.56.113)' from host 'db2'

2014/12/03 10:54:11  INFO Orphaned role 'writer(192.168.56.113)' has been assigned to 'db1'

2014/12/03 10:54:11  INFO Orphaned role 'reader(192.168.56.111)' has been assigned to 'db1'

2014/12/03 10:54:29  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 10:54:32  INFO Check 'rep_backlog' on 'db1' is ok!

~]# tail -f /var/log/mysql-mmm/mmm_agentd.log

2014/12/03 10:53:59 FATAL Couldn't allow writes: ERROR: Can't connect to MySQL (host = 192.168.56.102:3306, user = mmm_agent)! Lost connection to MySQL server at 'reading initial communication packet', system error: 111

2014/12/03 10:54:02 FATAL Couldn't allow writes: ERROR: Can't connect to MySQL (host = 192.168.56.102:3306, user = mmm_agent)! Lost connection to MySQL server at 'reading initial communication packet', system error: 111

2014/12/03 10:54:05 FATAL Couldn't allow writes: ERROR: Can't connect to MySQL (host = 192.168.56.102:3306, user = mmm_agent)! Lost connection to MySQL server at 'reading initial communication packet', system error: 111

2014/12/03 10:54:08 FATAL Couldn't allow writes: ERROR: Can't connect to MySQL (host = 192.168.56.102:3306, user = mmm_agent)! Lost connection to MySQL server at 'reading initial communication packet', system error: 111

2014/12/03 10:54:11  INFO We have some new roles added or old rules deleted!

2014/12/03 10:54:11  INFO Deleted: reader(192.168.56.111), writer(192.168.56.113)

2014/12/03 10:54:11 FATAL Couldn't deny writes: ERROR: Can't connect to MySQL (host = 192.168.56.102:3306, user = mmm_agent)! Lost connection to MySQL server at 'reading initial communication packet', system error: 111

結果

~]# mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), reader(192.168.56.112), writer(192.168.56.113)

  db2(192.168.56.102) master/HARD_OFFLINE. Roles:

DB2上負責讀的VIP(192.168.56.111) 以及負責寫的VIP(192.168.56.113) 會自動遷移到 DB1

啟動DB2上的MySQL服務

Monitor~]# tail -f /var/log/mysql-mmm/mmm_mond.log

2014/12/03 11:02:13  INFO Check 'mysql' on 'db2' is ok!

2014/12/03 11:02:14 FATAL State of host 'db2' changed from HARD_OFFLINE to AWAITING_RECOVERY

2014/12/03 11:02:14  WARN Check 'rep_backlog' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.56.101:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.56.101' (4)

2014/12/03 11:02:19  INFO Check 'rep_backlog' on 'db1' is ok!

2014/12/03 11:02:24  WARN Check 'rep_threads' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.56.101:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.56.101' (4)

2014/12/03 11:02:26  WARN Check 'rep_backlog' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.56.101:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.56.101' (4)

2014/12/03 11:02:29  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 11:02:31  INFO Check 'rep_backlog' on 'db1' is ok!

2014/12/03 11:02:41  WARN Check 'rep_threads' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.56.101:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.56.101' (4)

2014/12/03 11:02:46  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 11:02:54  WARN Check 'rep_backlog' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.56.101:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.56.101' (4)

2014/12/03 11:02:59  INFO Check 'rep_backlog' on 'db1' is ok!

2014/12/03 11:03:04  WARN Check 'rep_threads' on 'db1' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.56.101:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.56.101' (4)

2014/12/03 11:03:09  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 11:03:14 FATAL State of host 'db2' changed from AWAITING_RECOVERY to ONLINE because of auto_set_online(60 seconds). It was in state AWAITING_RECOVERY for 60 seconds

2014/12/03 11:03:14  INFO Moving role 'reader(192.168.56.112)' from host 'db1' to host 'db2'

DB1~]# tail -f /var/log/mysql-mmm/mmm_agentd.log

2014/12/03 11:03:15  INFO We have some new roles added or old rules deleted!

2014/12/03 11:03:15  INFO Deleted: reader(192.168.56.112)

DB2~]# tail -f /var/log/mysql-mmm/mmm_agentd.log

2014/12/03 11:03:14  INFO We have some new roles added or old rules deleted!

2014/12/03 11:03:14  INFO Added:   reader(192.168.56.112)

2014/12/03 11:03:15 FATAL Couldn't configure IP '192.168.56.112' on interface 'eth1': undef

結果

~]# mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), writer(192.168.56.113)

  db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.112)

DB1上負責讀的VIP(192.168.56.112)自動遷移到 DB2上,但是負責寫的VIP,仍在DB1上。

c)      只讀節點上stop slave

~]#  mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), writer(192.168.56.113)

  db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.112)

#DB2 stop slave

mysql> stop slave;

Query OK, 0 rows affected (0.07 sec)

Monitor~]# tail -f /var/log/mysql-mmm/mmm_mond.log

2014/12/03 12:10:14 FATAL State of host 'db2' changed from ONLINE to REPLICATION_FAIL

2014/12/03 12:10:14  INFO Removing all roles from host 'db2':

2014/12/03 12:10:14  INFO     Removed role 'reader(192.168.56.112)' from host 'db2'

2014/12/03 12:10:14  INFO Orphaned role 'reader(192.168.56.112)' has been assigned to 'db1'

2014/12/03 12:10:17 FATAL State of host 'db2' changed from REPLICATION_FAIL to ONLINE

2014/12/03 12:10:17  INFO Moving role 'reader(192.168.56.112)' from host 'db1' to host 'db2'

2014/12/03 12:10:24  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 12:10:51  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 12:11:15  INFO Check 'rep_backlog' on 'db1' is ok!

2014/12/03 12:11:20 FATAL State of host 'db2' changed from ONLINE to REPLICATION_FAIL

2014/12/03 12:11:20  INFO Removing all roles from host 'db2':

2014/12/03 12:11:20  INFO     Removed role 'reader(192.168.56.112)' from host 'db2'

2014/12/03 12:11:20  INFO Orphaned role 'reader(192.168.56.112)' has been assigned to 'db1'

2014/12/03 12:11:23 FATAL State of host 'db2' changed from REPLICATION_FAIL to ONLINE

2014/12/03 12:11:23  INFO Moving role 'reader(192.168.56.112)' from host 'db1' to host 'db2'

2014/12/03 12:11:54  INFO Check 'rep_backlog' on 'db1' is ok!

2014/12/03 12:12:22  INFO Check 'rep_backlog' on 'db1' is ok!

2014/12/03 12:12:26 FATAL State of host 'db2' changed from ONLINE to REPLICATION_FAIL

2014/12/03 12:12:26  INFO Removing all roles from host 'db2':

2014/12/03 12:12:26  INFO     Removed role 'reader(192.168.56.112)' from host 'db2'

2014/12/03 12:12:26  INFO Orphaned role 'reader(192.168.56.112)' has been assigned to 'db1'

2014/12/03 12:12:29 FATAL State of host 'db2' changed from REPLICATION_FAIL to ONLINE

2014/12/03 12:12:29  INFO Moving role 'reader(192.168.56.112)' from host 'db1' to host 'db2'

2014/12/03 12:12:30  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 12:12:53  INFO Check 'rep_threads' on 'db1' is ok!

2014/12/03 12:13:30 FATAL State of host 'db2' changed from ONLINE to REPLICATION_FAIL

2014/12/03 12:13:30 FATAL Host db2 is flapping!

2014/12/03 12:13:30  INFO Removing all roles from host 'db2':

2014/12/03 12:13:30  INFO     Removed role 'reader(192.168.56.112)' from host 'db2'

2014/12/03 12:13:31  INFO Orphaned role 'reader(192.168.56.112)' has been assigned to 'db1'

2014/12/03 12:13:33 FATAL State of host 'db2' changed from REPLICATION_FAIL to AWAITING_RECOVERY (because it's flapping)

2014/12/03 12:13:45  INFO Check 'rep_threads' on 'db1' is ok!

結果

~]#  mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), reader(192.168.56.112), writer(192.168.56.113)

  db2(192.168.56.102) master/AWAITING_RECOVERY. Roles:

DB2上的VIP192.168.56.112)會自動遷移到DB1

d)     讀寫節點上stop slave

#正常

~]#  mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), writer(192.168.56.113)

  db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.112)

#db1stop slave

結果

~]#  mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), writer(192.168.56.113)

  db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.112)

無任何變化,理論上也應該是對現有的環境無任何影響

e)     停止MMM監控主機上的monitor服務

~]# /etc/init.d/mysql-mmm-monitor stop

Daemon bin: '/usr/sbin/mmm_mond'

Daemon pid: '/var/run/mmm_mond.pid'

Shutting down MMM Monitor daemon: .. Ok

Monitor~]# tail -f /var/log/mysql-mmm/mmm_mond.log

2014/12/03 14:09:28  INFO Signal received: exiting...

2014/12/03 14:09:28  INFO Shutting down checker 'ping_ip'...

2014/12/03 14:09:28  INFO Shutting down checker 'mysql'...

2014/12/03 14:09:29  INFO Shutting down checker 'rep_backlog'...

2014/12/03 14:09:29  INFO Shutting down checker 'rep_threads'...

2014/12/03 14:09:29  INFO Shutting down checker 'ping'...

2014/12/03 14:09:29  INFO END

2014/12/03 14:09:29  INFO Child exited normally (with exitcode 0), shutting down

結果

DB1 ~]# ip a|grep eth1

3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    inet 192.168.56.101/24 brd 192.168.56.255 scope global eth1

    inet 192.168.56.111/32 scope global eth1

    inet 192.168.56.113/32 scope global eth1

DB2 ~]# ip a|grep eth1

3: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    inet 192.168.56.102/24 brd 192.168.56.255 scope global eth1

inet 192.168.56.112/32 scope global eth1

VIP都還在之前的節點上

啟動MMM監控服務

~]#  mmm_control show

  db1(192.168.56.101) master/ONLINE. Roles: reader(192.168.56.111), writer(192.168.56.113)

   db2(192.168.56.102) master/ONLINE. Roles: reader(192.168.56.112)

結果

 DB1DB2上的相關服務無影響。

 


來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/14710393/viewspace-1651913/,如需轉載,請註明出處,否則將追究法律責任。

相關文章