nginx內建高可用配置與第三方高可用模組nginx_ustream_check_mudule配置

1. nginx 第三方高可用模組

IP	備註
10.0.0.63	proxy
10.0.0.64	web1
10.0.0.65	web2

這裡會講解一些nignx常用高可用方案,以及引入第三方高可用模組來了解nginx作為高可用服務它是如何執行的

2. 基礎環境安裝

10.0.0.63  nginx 增加 nginx_upstream_check_module模組
10.0.0.64  nginx
10.0.0.65  nginx 
-------------------------------------------------------------
#3臺都安裝上nginxx:
useradd  www -u 1200 -M -s /sbin/nologin
mkdir -p /var/log/nginx
yum install -y cmake pcre pcre-devel openssl openssl-devel gd-devel \
    zlib-devel gcc gcc-c++ net-tools iproute telnet wget curl &&\
    yum clean all && \
    rm -rf /var/cache/yum/*
mkdir -p /server/tools
cd /server/tools
wget https://www.chenleilei.net/soft/nginx-1.16.1.tar.gz
tar xf nginx-1.16.1.tar.gz
cd nginx-1.16.1
./configure --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www \
    --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module \
    --pid-path=/var/run/nginx/nginx.pid
make -j 4 && make install && \
    rm -rf /usr/local/nginx/html/*  && \
    echo "nginx daemo hello" >/usr/local/nginx/html/index.html  && \
echo "export PATH=$PATH:/usr/local/nginx/sbin" >>/etc/profile
chown -R www.www /var/log/nginx /usr/local/nginx
source /etc/profile
nginx -s reload
ps -ef|grep nginx

3. nginx的高可用配置

預設的nginx高可用是一個後端連線不可用時且沒有健康檢查,他會一臺臺輪詢.在 如: 金錢交易的期間都會關閉這個預設高可用'proxy_next_upstream',原因就是,敏感交易有可能在預設時間未結束前得到了響應,而另一方面有其他伺服器響應了該請求,可能會出現重複交易情況,所以建議關閉預設的高可用策略,從而改用更簡單易用的其他高可用模組來實現,它就是 "ngx_http_proxy_module"

3.1 預設高可用策略介紹

server 10.0.0.10:80 max_fails=2 fail_timeout=30;
weight        權重
max_fails     失敗次數後停止該伺服器
fail_timeout  踢出後重新檢測時間. 一般max_fails和fail_timeout一起啟用的,也就是失敗n次,等待N秒
backup        備用伺服器
max_conns     允許最大連線數
slow_start    節點恢復後不立即加入到叢集.

3.2 安裝 ngx_upstream_check_module [10.0.0.63]

github: https://github.com/yaoweibin/nginx_upstream_check_module

[root@master tools]# git clone https://github.com/yaoweibin/nginx_upstream_check_module.git
[root@master tools]# ll
drwxr-xr-x 7 root root      4096 Apr 16 21:53 nginx_upstream_check_module
[root@master tools]# cd nginx_upstream_check_module/
[root@master nginx_upstream_check_module]# pwd
/server/tools/nginx_upstream_check_module   #這個路徑用於編譯nginx新增模組使用.


# nginx_upstream_check_module 模組目錄:
[root@master nginx_upstream_check_module]# ll
total 268
-rw-r--r-- 1 root root      0 Apr 16 21:53 CHANGES
-rw-r--r-- 1 root root   7921 Apr 16 21:53 check_1.11.1+.patch
-rw-r--r-- 1 root root   8330 Apr 16 21:53 check_1.11.5+.patch
-rw-r--r-- 1 root root   8060 Apr 16 21:53 check_1.12.1+.patch
-rw-r--r-- 1 root root   8054 Apr 16 21:53 check_1.14.0+.patch
-rw-r--r-- 1 root root   8409 Apr 16 21:53 check_1.16.1+.patch
-rw-r--r-- 1 root root   5483 Apr 16 21:53 check_1.2.1.patch
-rw-r--r-- 1 root root   7130 Apr 16 21:53 check_1.2.2+.patch
-rw-r--r-- 1 root root   7094 Apr 16 21:53 check_1.2.6+.patch
-rw-r--r-- 1 root root   6791 Apr 16 21:53 check_1.5.12+.patch
-rw-r--r-- 1 root root   8295 Apr 16 21:53 check_1.7.2+.patch
-rw-r--r-- 1 root root   8346 Apr 16 21:53 check_1.7.5+.patch
-rw-r--r-- 1 root root   8509 Apr 16 21:53 check_1.9.2+.patch
-rw-r--r-- 1 root root   6943 Apr 16 21:53 check.patch
-rw-r--r-- 1 root root    749 Apr 16 21:53 config
drwxr-xr-x 2 root root     43 Apr 16 21:53 doc
-rw-r--r-- 1 root root   1709 Apr 16 21:53 nginx-sticky-module.patch
drwxr-xr-x 2 root root     29 Apr 16 21:53 nginx-tests
-rw-r--r-- 1 root root 112085 Apr 16 21:53 ngx_http_upstream_check_module.c
-rw-r--r-- 1 root root    529 Apr 16 21:53 ngx_http_upstream_check_module.h
-rw-r--r-- 1 root root   2848 Apr 16 21:53 ngx_http_upstream_jvm_route_module.patch
-rw-r--r-- 1 root root  11509 Apr 16 21:53 README
drwxr-xr-x 6 root root     79 Apr 16 21:53 test
-rw-r--r-- 1 root root   3342 Apr 16 21:53 upstream_fair.patch
drwxr-xr-x 2 root root     81 Apr 16 21:53 util


#編譯nginx:
1. 獲取nginx原有編譯引數: 
configure arguments: --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid

2. 同步模組程式碼:
cd /server/tools/nginx-1.16.1
 patch -p1 < /server/tools/nginx_upstream_check_module/check_1.16.1+.patch
輸出:
[root@master nginx-1.16.1]# patch -p1 < /server/tools/nginx_upstream_check_module/check_1.16.1+.patch
patching file src/http/modules/ngx_http_upstream_hash_module.c
patching file src/http/modules/ngx_http_upstream_ip_hash_module.c
patching file src/http/modules/ngx_http_upstream_least_conn_module.c
patching file src/http/ngx_http_upstream_round_robin.c
patching file src/http/ngx_http_upstream_round_robin.h

3. 重新編譯nginx來新增模組:
[root@master ~ ]# cd /server/tools/nginx-1.16.1
[root@master nginx-1.16.1]# ./configure --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid --add-module=/server/tools/nginx_upstream_check_module/

[root@master nginx-1.16.1]# make 
make結束後拷貝新的nginx二進位制檔案替換老版本,這和版本升級流程一樣

#拷貝二進位制檔案替換老的nginx二進位制檔案
[root@master nginx-1.16.1]# \cp -a objs/nginx /usr/local/nginx/sbin/

#檢視程式:
[root@master nginx-1.16.1]# ps -ef|grep nginx
root      43754      1  0 22:17 ?        00:00:00 nginx: master process nginx
www       43755  43754  0 22:17 ?        00:00:00 nginx: worker process
root      43774  37915  0 22:17 pts/0    00:00:00 grep --color=auto nginx

#關閉nginx後重新啟動nginx:
[root@master nginx-1.16.1]# pkill nginx
[root@master nginx-1.16.1]# nginx

#檢查模組安裝是否成功:
[root@master nginx-1.16.1]# nginx -V
nginx version: nginx/1.16.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid --add-module=/server/tools/nginx_upstream_check_module/

#可以看到已經安裝了

順便做一下語法高亮顯示的小優化:
mkdir ~/.vim -p
cp -r contrib/vim/* ~/.vim/

4.第三方高可用模組nginx_ustream_check_mudule的配置使用

安裝完了模組後,需要對nginx進行配置才能使得新模組得到應用

egrep -v "#|^$" /usr/local/nginx/conf/nginx.conf.default > /usr/local/nginx/conf/nginx.conf
vim /usr/local/nginx/conf/nginx.conf

配置為如下:
#-----------------------------------------------------------------------------#
worker_processes  1;
events {
    worker_connections  1024;
}
http {
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
access_log  logs/access.log  main;
upstream web {
        server 10.0.0.64;
        server 10.0.0.65;
        }
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  locaalhost;
        location / {
                proxy_connect_timeout 2s;
                proxy_read_timeout 2s;
                proxy_next_upstream off;
                proxy_pass http://web;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}
#-----------------------------------------------------------------------------#


2. 修改新增模組配置項:
worker_processes  1;
events {
    worker_connections  1024;
}
http {
log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';
access_log  logs/access.log  main;
upstream web {
        server 10.0.0.64;
        server 10.0.0.65;
        #模組配置項 用於檢查後端伺服器是否正常
        check interval=5000 rise=1 fall=3 timeout=4000;
        check interval=3000 rise=2 fall=5 timeout=1000 type=ssl_hello;
        check interval=3000 rise=2 fall=5 timeout=1000 type=http;
        check_http_send "HEAD / HTTP/1.0\r\n\r\n";
        check_http_expect_alive http_2xx http_3xx http_4xx http_5xx;
        #模組配置項 用於檢查後端伺服器是否正常
        }
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  locaalhost;
        location / {
                proxy_connect_timeout 2s;
                proxy_read_timeout 2s;
                proxy_next_upstream off;
                proxy_pass http://web;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

驗證模組可用性:

1. 為後端nginx新增時別內容:

10.0.0.64伺服器新增:
echo  "10.0.0.64 nginx daemon" >index.html 

10.0.0.65伺服器新增:
echo  "10.0.0.65 nginx daemon" >index.html 

可以看出節點分配正常

4.1 後端健康檢查測試

前端負載均衡 開啟日誌記錄,獲取訪問的後端節點IP,便於檢視配置是否正確,這個配置在企業中也是必備,務必掌握

http模組下修改預設日誌記錄為如下:

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time $upstream_addr $upstream_status';

access_log  logs/access.log  main;

配置完畢後,繼續測試負載均衡模組的可用性

預設埠都開 檢查日誌訪問情況:
10.0.0.1 - - [16/Apr/2020:23:11:29 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.002 10.0.0.64:80 200
10.0.0.1 - - [16/Apr/2020:23:11:29 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.001 10.0.0.65:80 200
10.0.0.1 - - [16/Apr/2020:23:11:30 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.002 10.0.0.64:80 200
10.0.0.1 - - [16/Apr/2020:23:11:32 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.001 10.0.0.65:80 200

#從這裡可以看出,我們訪問10.0.0.63負載均衡的時候,他會一次去訪問10.0.0.64,一次去訪問10.0.0.65, 照此依次輪詢.
#現在關閉後端某個節點,用於測試模組是否正常檢測:

2020/04/16 23:18:38 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:41 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:44 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:47 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:50 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:50 [error] 46559#0: disable check peer: 10.0.0.64:80   #<---- 移除了該節點

他會進行5次檢測,發現後端節點無法訪問,直接將故障節點移除叢集.
同時當這個叢集角色恢復後,該模組依然能夠檢測出並將其重新加入叢集.
如下:
2020/04/16 23:20:53 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:20:56 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:21:03 [error] 46559#0: enable check peer: 10.0.0.64:80    #<---- 新增了該節點