haporxy+keepalived實現負載均衡+高可用

zld_555發表於2020-12-26
負載

環境準備

主機規劃

伺服器IP主機名服務
centos7x2(haproxy01)192.168.138.132centos7x2Haproxy、Nginx、keepalived
centos7x3(haproxy02)192.168.138.133centos7x3Haproxy、Nginx、keepalived
192.168.138.200（vip）

其中haproxy負責負債均衡，nginx單純作為web服務

hosts檔案解析

[root@centos7x2 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.138.132 centos7x2
192.168.138.133 centos7x3

作業系統

centos7.7

[root@centos7x2 ~]# cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)
[root@centos7x2 ~]# uname -a
Linux centos7x2 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

系統環境

關閉selinux

sed -i '7s#enforcing#disabled#g' /etc/selinux/config
setenforce 0

關閉iptables

systemctl stop firewalld.service
systemctl disable firewalld.service

安裝基本包

yum -y install net-tools vim lrzsz tree screen lsof tcpdump nc mtr nmap gcc glibc gcc-c++

軟體版本

haproxy：1.5.18
nginx：1.18.0
keepalived：1.3.5

安裝HAProxy

安裝haproxy和其他軟體類似，基本上也分為原始碼安裝和yum安裝，採用yu’m安裝得版本可能稍微比較舊，在CentOS7系統下一般為1.5版本。這兩個方式本身區別並不太大，但是安裝目錄會有一定得出入，在配置程式得時候需要稍微注意。

這裡的安裝均採用yum方式，編譯安裝就不細說，兩者任選其一均可

yum安裝

[root@centos7x3 ~]# yum install -y haproxy

[root@centos7x2 ~]# haproxy -v
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

配置rsyslog

[root@centos7x3 ~]# mkdir /var/log/haproxy
[root@centos7x3 ~]# chmod a+w /var/log/haproxy/
[root@centos7x3 ~]# vim /etc/rsyslog.conf 
#將如下兩行得註釋取消
$ModLoad imudp
$UDPServerRun 514
#在該檔案新增如下內容：
# Save haproxy log
local3.*          /var/log/haproxy/haproxy.log
[root@centos7x3 ~]# vim /etc/sysconfig/rsyslog
[root@centos7x3 ~]# cat /etc/sysconfig/rsyslog
# Options for rsyslogd
# Syslogd options are deprecated since rsyslog v3.
# If you want to use them, switch to compatibility mode 2 by "-c 2"
# See rsyslogd(8) for more details
#SYSLOGD_OPTIONS=""
SYSLOGD_OPTIONS="-r -m 0 -c 2"

#-c 2 使用相容模式，預設是-c 5； -r 開啟遠端日誌； -m 0 標記時間戳。單位是分鐘，為0時，表示禁用該功能。
[root@centos7x3 ~]# systemctl restart rsyslog

配置haproxy

  • 第一臺機器的haproxy配置，/etc/haproxy/haproxy.cfg
global
    maxconn 10000
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    nbproc 1 
    pidfile /var/lib/haproxy/haproxy.pid 
    log 127.0.0.1 local3 info

defaults
    mode http
    log global
    option http-keep-alive
    maxconn 10000
    timeout connect 5000ms
    timeout client  50000ms
    timeout server 50000ms

listen stats
    mode http
    bind 0.0.0.0:8888
    stats refresh 30s
    stats enable
    stats uri     /stats 
    stats auth    haproxy:123456

frontend frontend_www_example_com
    # bind 192.168.138.132:80
    bind 0.0.0.0:80
    mode http
    option httplog
    log global
    default_backend backend_www_example_com

backend backend_www_example_com
    option forwardfor header X-REAL-IP
    option httpchk HEAD / HTTP/1.0
    # balance source #hash方法
    balance roundrobin #輪詢方法
    server web-node1  192.168.138.132:8080 check inter 2000 rise 30 fall 15
    server web-node2  192.168.138.133:8080 check inter 2000 rise 30 fall 15
  • haproxy2上的配置，做同樣的配置與上邊沒有區別
global
    maxconn 10000
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    nbproc 1 
    pidfile /var/lib/haproxy/haproxy.pid 
    log 127.0.0.1 local3 info

defaults
    mode http
    log global
    option http-keep-alive
    maxconn 10000
    timeout connect 5000ms
    timeout client  50000ms
    timeout server 50000ms

listen stats
    mode http
    bind 0.0.0.0:8888
    stats refresh 30s
    stats enable
    stats uri     /stats 
    stats auth    haproxy:123456

frontend frontend_www_example_com
    # bind 192.168.138.133:80
    bind 0.0.0.0:80
    mode http
    option httplog
    log global
    default_backend backend_www_example_com

backend backend_www_example_com
    option forwardfor header X-REAL-IP
    option httpchk HEAD / HTTP/1.0
    balance source
    server web-node1  192.168.138.132:8080 check inter 2000 rise 30 fall 15
    server web-node2  192.168.138.133:8080 check inter 2000 rise 30 fall 15

重啟haproxy檢視結果

[root@centos7x3 ~]# systemctl start haproxy
[root@centos7x3 ~]# tailf /var/log/haproxy/haproxy.log 
2020-12-24T23:34:46+08:00 localhost haproxy[2085]: Proxy stats started.
2020-12-24T23:34:46+08:00 localhost haproxy[2085]: Proxy backend_www_example_com started.

可以通過http://ip:8888/stats 來檢視狀態，其實此時還有backend存在，所以還沒有效果。

接下來把web啟動起來。

安裝nginx

這裡只yum安裝，僅僅是為了web服務

yum安裝

rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
yum -y install nginx

#檢視版本資訊
[root@centos7x2 haproxy]# nginx -v
nginx version: nginx/1.18.0

配置web服務

  • 修改監聽埠為8080，預設的80已經給給haproxy的frontend給用了

    sed -i 's#80#8080#g' /etc/nginx/conf.d/default.conf

  • 修改主頁資訊，便於分辨是哪臺機器的web頁面

    • 第一臺
echo haproxy01 > /usr/share/nginx/html/index.html
  • 第二臺
echo haproxy02 > /usr/share/nginx/html/index.html

啟動nginx

systemctl start nginx.service

驗證HAproxy的負載均衡效果

通過訪問http://192.168.138.132 和http://192.168.138.133 可以看到web頁面在來回切換

安裝keepalived

YUM直接安裝

yum -y install keepalived

# 檢視keepalived版本
[root@centos7x3 ~]# keepalived -v
Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

Copyright(C) 2001-2017 Alexandre Cassen, <acassen@gmail.com>

配置keepalived

配置檔案位於/etc/keepalived/keepalived.conf

  • haproxy01的keepalived配置

    ! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id haproxy_ha  #標識資訊，一個名字而已；
}

vrrp_instance haproxy_ha {
    state MASTER          #角色是master(不同點)
    interface ens33       #vip 繫結埠
    virtual_router_id 51  #讓master 和backup在同一個虛擬路由裡，id 號必須相同；
    priority 150          #優先順序,誰的優先順序高誰就是master(不同點)
    advert_int 1          #心跳間隔時間
    authentication {
        auth_type PASS    #認證
        auth_pass 1111    #密碼 
    }
    virtual_ipaddress {
        192.168.138.200         #虛擬ip
    }
}

  • haproxy02的配置

    ! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id haproxy_ha  #標識資訊，一個名字而已；
}

vrrp_instance haproxy_ha {
    state BACKUP          #角色是backup(不同點)
    interface ens33       #vip 繫結埠
    virtual_router_id 51  #讓master 和backup在同一個虛擬路由裡，id 號必須相同；
    priority 100          #優先順序,誰的優先順序高誰就是master(不同點)
    advert_int 1          #心跳間隔時間
    authentication {
        auth_type PASS    #認證
        auth_pass 1111    #密碼 
    }
    virtual_ipaddress {
        192.168.138.200         #虛擬ip
    }
}

啟動keepalived

systemctl start keepalived
systemctl enable keepalived

檢視效果

haproxy01為master，可以看到有虛擬ip存在

[root@centos7x2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:78:b1:3b brd ff:ff:ff:ff:ff:ff
    inet 192.168.138.132/24 brd 192.168.138.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.138.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::ad9e:50b:adc3:4c6e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

haproxy02為backup,所以沒有vip

[root@centos7x3 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:d6:35:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.138.133/24 brd 192.168.138.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::5fa9:1a15:67e2:75e0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@centos7x3 ~]#

狀態切換

關閉keepalived或者關閉haproxy01，來模擬故障。可以看到haproxy02獲得了vip

[root@centos7x3 ~]# ip a
...
...
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:d6:35:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.138.133/24 brd 192.168.138.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.138.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::5fa9:1a15:67e2:75e0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

恢復haproxy01，然後haproxy01獲得vip，二haproxy02丟失vip。

keepalived監控haproxy

前面的keepalived並不能監控haproxy的狀態，而是通過keepalived本身是不是正常的來進行切換，與haproxy無關。

繼續增加配置，來實現haproxy服務的關聯。

編寫haproxy狀態檢測指令碼

[root@centos7x2 ~]# cat /etc/keepalived/check_haproxy.sh 
#!/bin/bash
if [ $(ps -C haproxy --no-header |wc -l) -eq 0 ];then
    echo "start haproxy">>/tmp/keepalived.log
    systemctl start haproxy
    sleep 2
    if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ];then
        echo "stop keepalived">>/tmp/keepalived.log
        systemctl stop keepalived
    fi
fi

以上，如果haproxy出現問題，嘗試啟動haproxy，如果啟動不成功，則關閉keepalived，從而觸發vip切換。

注意，經過驗證，此處的sleep時間應該小於keepalived 設定中interval的間隔時間，否則指令碼走不完就退出了。

修改keepalived配置

[root@centos7x2 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id haproxy_ha  #標識資訊，一個名字而已；
}

vrrp_script chk_haproxy {                        #Haproxy服務啟動
   script "/etc/keepalived/check_haproxy.sh"     #監控haproxy程式的指令碼
   interval 3                                    #監控的時間間隔
   weight   5                                    #對優先順序的影響
}

vrrp_instance haproxy_ha {
    state MASTER          #角色是master
    interface ens33       #vip 繫結埠
    virtual_router_id 51  #讓master 和backup在同一個虛擬路由裡，id 號必須相同；
    priority 150          #優先順序,誰的優先順序高誰就是master ;
    advert_int 1          #心跳間隔時間
    authentication {
        auth_type PASS    #認證
        auth_pass 1111    #密碼 
    }
    track_script {        # 呼叫監測指令碼
        chk_haproxy
    }
    virtual_ipaddress {
        192.168.138.200         #虛擬ip
    }
}

驗證效果

因為chk_haproxy.sh會嘗試啟動haproxy，所以單純關閉haproxy沒有什麼用，此處故意將haproxy的配置檔案改錯，使其啟動失敗。

例如

[root@centos7x2 ~]# cat /etc/haproxy/haproxy.cfg
global
    maxconn 10000
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    nbproc 1 
    pidfile /var/lib/haproxy/haproxy.pid 
    log 127.0.0.1 local3 info

defaults
    mode httpno     # 故意改錯的地方
	...
	...
  • 重啟keepalived使配置生效
[root@centos7x2 ~]# sytemctl restart keepalived
  • 關閉haproxy，觸發chk_haproxy中的stop keeaplived動作
[root@centos7x2 ~]# sytemctl stop haproxy

可以通過日誌看到stop keepalived了

[root@centos7x2 ~]# tailf /tmp/keepalived.log 
start haproxy
stop keepalived

隨著haproxy01的keepalived的結束，vip地址漂移到haproxy02上。

相關文章