前言

嚐到k8s甜頭以後，我們就想著應用到生產環境裡去，以提高業務迭代效率，可是部署在生產環境裡有一個要求，就是k8s叢集不可以存在單點故障。。。誒唷我的乖乖，這不就要求k8s叢集高可用嗎，好，下面就是介紹兩種目前比較火的k8s叢集master高可用方式。

介紹

首先介紹的第一種k8sHA叢集我覺得更應該叫做主從結構k8s叢集，它由三臺master組成，有三個keepalived提供一個vip 來作為apiserver的ip入口，keepalived設定權重，使得vip落在權重大的master節點上，node節點透過訪問這個vip從而訪問到這一臺master，另外兩臺master則透過etcd叢集，來完成資料同步。

缺點：這樣的叢集是透過keepalived來實現高可用的，也就是說在權重較大的節點沒有故障之前，keepalived所指向的流量永遠都是經過主master，只有當主master出現故障或者當機的情況下，才有可能轉移到另外兩臺從master節點上。這樣會導致主master節點壓力過大，而另外兩臺從master可能永遠不會被呼叫，導致資源浪費等等情況。

不過，這也是排除單點故障的一種方式。

下面是理想的高可用架構圖。

k8s 理想HA高可用

本文中要部署高可用的架構圖：

本文高可用架構

上圖摘抄至

好了，到此我們整理一下本文中需要使用的技術棧

keepalived+etcd+k8s master

其中keepalived提供vip供node做apiserver入口，etcd必須是高可用叢集，實現資料同步；以及基本的k8s master節點部署。

安裝準備

節點部署相關情況

軟體版本：

docker17.03.2-ce

socat-1.7.3.2-2.el7.x86_64

kubelet-1.10.0-0.x86_64

kubernetes-cni-0.6.0-0.x86_64

kubectl-1.10.0-0.x86_64

kubeadm-1.10.0-0.x86_64

以上軟體在上一篇裡已經介紹並附有下載地址。

環境配置

systemctl stop firewalldsystemctl disable firewalld

修改每個節點hostname

cat < /etc/hosts > /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.1 master1

192.168.100.2 master2

192.168.100.3 master3

EOF

swapoff -a

sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

keepalived安裝

libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

wget

yum install -y libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

yum -y install libnl libnl-devel

tar -xzvf keepalived-1.4.3.tar.gz

cd keepalived-1.4.3

./configure --prefix=/usr/local/keepalived #檢查環境配置

出現上圖即為正確環境，如果出現錯誤

checking openssl/ssl.h usability... no

checking openssl/ssl.h presence... no

checking foropenssl/ssl.h... no

configure: error:

!!! OpenSSL is not properly installed on your system. !!!

!!! Can not include OpenSSL headers files. !!!

則：安裝openssl和openssl-devel包，然後從新編譯配置檔案。

yum install openssl openssl-devel

./configure --prefix=/usr/local/keepalived

make && make install

cp keepalived/etc/init.d/keepalived /etc/init.d/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

ps -aux |grep keepalived

chkconfig keepalived on

透過systemctl status keepalived檢視keepalived狀態

三臺master重複以上步驟，直到完成keepalived的安裝。

安裝完成後編寫配置檔案：

master1的keepalived.conf

cat >/etc/keepalived/keepalived.conf <<EOF

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state MASTER

interface ens33 #本機物理網路卡名字,可透過ip a來檢視

virtual_router_id 61

priority 120 # 主節點權重最高依次減少

advert_int 1

mcast_src_ip 192.168.100.1 #修改為本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#註釋掉本地IP

#192.168.100.1

192.168.100.2

192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#這個方法在沒部署k8s之前最好註釋掉，因為很可能因為這個報錯

}

EOF

master2的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state BACKUP

interface ens33 #本機物理網路卡名字,可透過ip a來檢視

virtual_router_id 61

priority 110 # 主節點權重最高依次減少

advert_int 1

mcast_src_ip 192.168.100.2 #修改為本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#註釋掉本地IP

192.168.100.1

#192.168.100.2

192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#這個方法在沒部署k8s之前最好註釋掉，因為很可能因為這個報錯

}

EOF

master3的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state BACKUP

interface ens33 #本機物理網路卡名字,可透過ip a來檢視

virtual_router_id 61

priority 100 # 主節點權重最高依次減少

advert_int 1

mcast_src_ip 192.168.100.3 #修改為本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#註釋掉本地IP

192.168.100.1

192.168.100.2

#192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#這個方法在沒部署k8s之前最好註釋掉，因為很可能因為這個報錯

}

EOF

啟動keepalived

systemctl restart keepalived

透過ip a可以檢視

除了本機ip還多了一個虛擬ip

也可以透過ping ip去驗證vip是否生效。

安裝ETCD

1：設定cfssl環境

wget

chmod +x cfssl_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

chmod +x cfssljson_linux-amd64

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

chmod +x cfssl-certinfo_linux-amd64

mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfoexport PATH=/usr/local/bin:$PATH

2：建立 CA 配置檔案（下面配置的IP為etc節點的IP

mkdir /root/ssl

cd /root/ssl

cat > ca-config.json <<EOF

{"signing": {"default": { "expiry": "8760h"},"profiles": { "kubernetes-Soulmate": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" }}}}EOF

cat > ca-csr.json <<EOF

{"CN": "kubernetes-Soulmate","key": {"algo": "rsa","size": 2048},"names": [{ "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System"}]}EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF

{ "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.100.1", "192.168.100.2", "192.168.100.3" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System" } ]}EOF

cfssl gencert -ca=ca.pem

-ca-key=ca-key.pem

-config=ca-config.json

-profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

3:master1分發etcd證照到master2、master3上面

mkdir -p /etc/etcd/ssl

cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/

ssh -n master2 "mkdir -p /etc/etcd/ssl && exit"

ssh -n master3 "mkdir -p /etc/etcd/ssl && exit"

scp -r /etc/etcd/ssl#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

然後執行在master上留下的kubeadm join 192.168.100.1:6443 --token hpobow.vw1g1ya5dre7sq06 --discovery-token-ca-cert-hash sha256:f79b68fb698c92b9336474eb3bf184e847fgerbc58a6296911892662b98b1315即可。

作者：我的橙子很甜
連結：

高階k8s HA 叢集搭建（一）

前言

介紹

安裝準備

keepalived安裝

master1的keepalived.conf

master2的keepalived.conf

master3的keepalived.conf

安裝ETCD

相關文章