高階k8s HA 叢集搭建（一）

前言

嚐到k8s甜頭以後，我們就想著應用到生產環境裡去，以提高業務迭代效率，可是部署在生產環境裡有一個要求，就是k8s叢集不可以存在單點故障。。。誒唷我的乖乖，這不就要求k8s叢集高可用嗎，好，下面就是介紹兩種目前比較火的k8s叢集master高可用方式。

---

介紹

首先介紹的第一種k8sHA叢集我覺得更應該叫做主從結構k8s叢集，它由三臺master組成，有三個keepalived提供一個vip 來作為apiserver的ip入口，keepalived設定權重，使得vip落在權重大的master節點上，node節點透過訪問這個vip從而訪問到這一臺master，另外兩臺master則透過etcd叢集，來完成資料同步。

缺點：這樣的叢集是透過keepalived來實現高可用的，也就是說在權重較大的節點沒有故障之前，keepalived所指向的流量永遠都是經過主master，只有當主master出現故障或者當機的情況下，才有可能轉移到另外兩臺從master節點上。這樣會導致主master節點壓力過大，而另外兩臺從master可能永遠不會被呼叫，導致資源浪費等等情況。

不過，這也是排除單點故障的一種方式。

下面是理想的高可用架構圖。

k8s 理想HA高可用

本文中要部署高可用的架構圖：

本文高可用架構

上圖摘抄至

好了，到此我們整理一下本文中需要使用的技術棧

keepalived+etcd+k8s master

其中keepalived提供vip供node做apiserver入口，etcd必須是高可用叢集，實現資料同步；以及基本的k8s master節點部署。

---

安裝準備

節點部署相關情況

軟體版本：

docker17.03.2-ce

socat-1.7.3.2-2.el7.x86_64

kubelet-1.10.0-0.x86_64

kubernetes-cni-0.6.0-0.x86_64

kubectl-1.10.0-0.x86_64

kubeadm-1.10.0-0.x86_64

以上軟體在上一篇裡已經介紹並附有下載地址。

環境配置

systemctl stop firewalldsystemctl disable firewalld

修改每個節點hostname

cat < /etc/hosts > /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.1 master1

192.168.100.2 master2

192.168.100.3 master3

EOF

swapoff -a

sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce  0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536"  >> /etc/security/limits.conf

echo "* hard nproc 65536"  >> /etc/security/limits.conf

echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf

echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

---

keepalived安裝

libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

wget 

yum install -y libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

yum -y install libnl libnl-devel

tar -xzvf keepalived-1.4.3.tar.gz

cd keepalived-1.4.3

./configure --prefix=/usr/local/keepalived #檢查環境配置

出現上圖即為正確環境，如果出現錯誤

checking openssl/ssl.h usability... no

checking openssl/ssl.h presence... no

checking foropenssl/ssl.h... no

configure: error: 

  !!! OpenSSL is not properly installed on your system. !!!

  !!! Can not include OpenSSL headers files.            !!!

則：安裝openssl和openssl-devel包，然後從新編譯配置檔案。

yum install openssl openssl-devel

./configure --prefix=/usr/local/keepalived

make && make install

cp keepalived/etc/init.d/keepalived /etc/init.d/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ 

ps -aux |grep keepalived

chkconfig keepalived on

透過systemctl status keepalived檢視keepalived狀態

三臺master重複以上步驟，直到完成keepalived的安裝。

安裝完成後編寫配置檔案：

master1的keepalived.conf

cat >/etc/keepalived/keepalived.conf  <<EOF

global_defs {

    router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

        script "curl -k https://192.168.100.4:6443"

        interval 3

        timeout 9

        fall 2

        rise 2    

}

vrrp_instance VI_1 {

    state MASTER

    interface ens33 #本機物理網路卡名字,可透過ip a來檢視

    virtual_router_id 61

    priority 120  # 主節點權重最高 依次減少

    advert_int 1

    mcast_src_ip 192.168.100.1  #修改為本地IP

    nopreempt

    authentication {

        auth_type PASS

        auth_pass awzhXylxy.T

    }

    unicast_peer{

        #註釋掉本地IP 

        #192.168.100.1

        192.168.100.2

        192.168.100.3

    }

    virtual_ipaddress {

        192.168.100.4/22 #VIP

    }

    track_script {

        #CheckK8sMaster#這個方法在沒部署k8s之前最好註釋掉，因為很可能因為這個報錯

    }

}

EOF

master2的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

    router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

        script "curl -k https://192.168.100.4:6443"

        interval 3

        timeout 9

        fall 2

        rise 2    

}

vrrp_instance VI_1 {

    state BACKUP

    interface ens33 #本機物理網路卡名字,可透過ip a來檢視

    virtual_router_id 61

    priority 110  # 主節點權重最高 依次減少

    advert_int 1

    mcast_src_ip 192.168.100.2  #修改為本地IP

    nopreempt

    authentication {

        auth_type PASS

        auth_pass awzhXylxy.T

    }

    unicast_peer{

        #註釋掉本地IP 

        192.168.100.1

        #192.168.100.2

        192.168.100.3

    }

    virtual_ipaddress {

        192.168.100.4/22 #VIP

    }

    track_script {

        #CheckK8sMaster#這個方法在沒部署k8s之前最好註釋掉，因為很可能因為這個報錯

    }

}

EOF

master3的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

    router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

        script "curl -k https://192.168.100.4:6443"

        interval 3

        timeout 9

        fall 2

        rise 2    

}

vrrp_instance VI_1 {

    state BACKUP

    interface ens33 #本機物理網路卡名字,可透過ip a來檢視

    virtual_router_id 61

    priority 100  # 主節點權重最高 依次減少

    advert_int 1

    mcast_src_ip 192.168.100.3  #修改為本地IP

    nopreempt

    authentication {

        auth_type PASS

        auth_pass awzhXylxy.T

    }

    unicast_peer{

        #註釋掉本地IP 

        192.168.100.1

        192.168.100.2

        #192.168.100.3

    }

    virtual_ipaddress {

        192.168.100.4/22 #VIP

    }

    track_script {

        #CheckK8sMaster#這個方法在沒部署k8s之前最好註釋掉，因為很可能因為這個報錯

    }

}

EOF

啟動keepalived

systemctl restart keepalived

透過ip a可以檢視

除了本機ip還多了一個虛擬ip

---

也可以透過ping ip去驗證vip是否生效。

安裝ETCD

1：設定cfssl環境

wget

wget

wget

chmod +x cfssl_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

chmod +x cfssljson_linux-amd64

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

chmod +x cfssl-certinfo_linux-amd64

mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfoexport PATH=/usr/local/bin:$PATH

2：建立 CA 配置檔案（下面配置的IP為etc節點的IP

mkdir /root/ssl

cd /root/ssl

cat >  ca-config.json <<EOF

{"signing": {"default": {  "expiry": "8760h"},"profiles": {  "kubernetes-Soulmate": {    "usages": [        "signing",        "key encipherment",        "server auth",        "client auth"    ],    "expiry": "8760h"  }}}}EOF

cat >  ca-csr.json <<EOF

{"CN": "kubernetes-Soulmate","key": {"algo": "rsa","size": 2048},"names": [{  "C": "CN",  "ST": "shanghai",  "L": "shanghai",  "O": "k8s",  "OU": "System"}]}EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF

{  "CN": "etcd",  "hosts": [    "127.0.0.1",    "192.168.100.1",    "192.168.100.2",    "192.168.100.3"  ],  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "shanghai",      "L": "shanghai",      "O": "k8s",      "OU": "System"    }  ]}EOF

cfssl gencert -ca=ca.pem

  -ca-key=ca-key.pem

  -config=ca-config.json

  -profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

3:master1分發etcd證照到master2、master3上面

mkdir -p /etc/etcd/ssl

cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/

ssh -n master2 "mkdir -p /etc/etcd/ssl && exit"

ssh -n master3 "mkdir -p /etc/etcd/ssl && exit"

scp -r /etc/etcd/ssl#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

然後執行在master上留下的kubeadm join 192.168.100.1:6443 --token hpobow.vw1g1ya5dre7sq06 --discovery-token-ca-cert-hash sha256:f79b68fb698c92b9336474eb3bf184e847fgerbc58a6296911892662b98b1315即可。

作者：我的橙子很甜
連結：