使用Kubeadm搭建高可用Kubernetes叢集

RidingWind發表於2022-04-14

1.概述

Kubenetes叢集的控制平面節點（即Master節點）由資料庫服務（Etcd）+其他元件服務（Apiserver、Controller-manager、Scheduler...）組成。
整個叢集系統執行的互動資料都將儲存到資料庫服務（Etcd）中，所以Kubernetes叢集的高可用性取決於資料庫服務（Etcd）在多個控制平面（Master）節點構建的資料同步複製關係。
由此搭建Kubernetes的高可用叢集可以選擇以下兩種部署方式：
使用堆疊的控制平面（Master）節點，其中etcd與組成控制平面的其他元件在同臺機器上；
使用外部Etcd節點，其中Etcd與控制平臺的其他元件在不同的機器上。
參考文件：https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

1.1.堆疊Etcd拓撲（推薦）

Etcd與其他元件共同執行在多臺控制平面（Master）機器上，構建Etcd叢集關係以形成高可用的Kubernetes叢集。
先決條件：

  • 最少三個或更多奇數Master節點；
  • 最少三個或更多Node節點；
  • 叢集中所有機器之間的完整網路連線（公共或專用網路）；
  • 使用超級使用者許可權；
  • 在叢集中的任何一個節點上都可以使用SSH遠端訪問；
  • Kubeadm和Kubelet已經安裝到機器上。

使用這種方案可以減少要使用機器的數量，降低成本，降低部署複雜度；多元件服務之間競爭主機資源，可能導致效能瓶頸，以及當Master主機發生故障時影響到所有元件正常工作。
在實際應用中，你可以選擇部署更多數量>3的Master主機，則該拓撲的劣勢將會減弱！
這是kubeadm中的預設拓撲，kubeadm會在Master節點上自動建立本地etcd成員。

1.2.外部Etcd拓撲

控制平面的Etcd元件執行在外部主機上，其他元件連線到外部的Etcd叢集以形成高可用的Kubernetes叢集。
先決條件：

  • 最少三個或更多奇數Master主機；
  • 最少三個或更多Node主機；
  • 還需要三臺或更多奇數Etcd主機。
  • 叢集中所有主機之間的完整網路連線（公共或專用網路）；
  • 使用超級使用者許可權；
  • 在叢集中的任何一個節點主機上都可以使用SSH遠端訪問；
  • Kubeadm和Kubelet已經安裝到機器上。

使用外部主機搭建起來的Etcd叢集，擁有更多的主機資源和可擴充套件性，以及故障影響範圍縮小，但更多的機器將導致增加部署成本。

2.搭建Kubernetes叢集（堆疊Etcd拓撲）

2.1.執行環境與主機規劃

主機系統：CentOS Linux release 7.6.1810 (Core)
Kubernetes版本：Kubernetes-1.23.0
Kubernetes與Docker相容性：v20.10.7+不相容 -> v20.10.12+不相容
本文使用Docker版本：Docker-ce-19.03.0
硬體條件：叢集中的機器最少需要2GB或者以上的記憶體，最少需要2核或者以上更多的CPU

主機名 主機地址 主機角色 執行服務
k8s-master01 192.168.124.128
VIP：192.168.124.100		 control plane node（master） kube-apiserver
etcd
kube-scheduler
kube-controller-manager
docker
kubelet
keepalived
ipvs
k8s-master02 192.168.124.130
VIP：192.168.124.100		 control plane node（master） kube-apiserver
etcd
kube-scheduler
kube-controller-manager
docker
kubelet
keepalived
ipvs
k8s-master03 192.168.124.131
VIP：192.168.124.100		 control plane node（master） kube-apiserver
etcd
kube-scheduler
kube-controller-manager
docker
kubelet
keepalived
ipvs
k8s-node01 192.168.124.133 worker node（node） kubelet
kube-proxy
docker

2.2.檢查和配置主機環境

1、驗證每個主機上的MAC地址和Product_id的唯一性

所有主機上：

[root@localhost ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:40:e3:9f brd ff:ff:ff:ff:ff:ff
[root@localhost ~]# cat /sys/class/dmi/id/product_uuid
B70F4D56-1F69-3997-AD55-83725A40E39F

2、檢查執行Kubernetes所需的埠是否未被佔用

角色 協議 方向 服務:埠範圍
Master（Control Plane） TCP Inbound Kubernetes API server:6443
etcd server client API:2379-2380
Kubelet API:10250
kube-scheduler:10259
kube-controller-manager:10257
Node（Worker Node） TCP Inbound Kubelet API:10250
NodePort Services+:30000-32767

所有master主機上：

[root@localhost ~]# ss -alnupt |grep -E '6443|10250|10259|10257|2379|2380'

所有node主機上：

[root@localhost ~]# ss -alnupt |grep -E '10250|3[0-2][0-7][0-6][0-7]'

3、配置主機名稱

k8s-master01:

[root@localhost ~]# echo "k8s-master01" >/etc/hostname
[root@localhost ~]# cat /etc/hostname | xargs hostname
[root@localhost ~]# bash
[root@k8s-master01 ~]#

k8s-master02：

[root@localhost ~]# echo "k8s-master02" >/etc/hostname
[root@localhost ~]# cat /etc/hostname | xargs hostname
[root@localhost ~]# bash
[root@k8s-master02 ~]#

k8s-master03：

[root@localhost ~]# echo "k8s-master03" >/etc/hostname
[root@localhost ~]# cat /etc/hostname | xargs hostname
[root@localhost ~]# bash
[root@k8s-master03 ~]#

k8s-node01：

[root@localhost ~]# echo "k8s-node01" >/etc/hostname
[root@localhost ~]# cat /etc/hostname | xargs hostname
[root@localhost ~]# bash
[root@k8s-node01 ~]#

4、新增hosts名稱解析

所有主機上：

[root@k8s-master01 ~]# cat >> /etc/hosts << EOF
192.168.124.128 k8s-master01
192.168.124.130 k8s-master02
192.168.124.131 k8s-master03
192.168.124.132 k8s-node01
EOF

5、主機間時間同步

k8s-master01:
設定優先從cn.ntp.org.cn公共時間伺服器上同步時間。

# 安裝NTP時間服務和NTP客戶端
[root@k8s-master01 ~]# yum -y install epel-release.noarch
[root@k8s-master01 ~]# yum -y install ntp ntpdate
# 使用NTP客戶端從外部公共NTP時間伺服器同步本機時間
[root@k8s-master01 ~]# ntpdate cn.ntp.org.cn
# 配置NTP時間服務
[root@k8s-master01 ~]# vim /etc/ntp.conf
# 訪問控制
# 允許外部客戶端從本機同步時間，但不允許外部客戶端修改本機時間
restrict default nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict ::1

# 從外部伺服器主動同步時間
# 如果外部伺服器連線失敗時則以本機時間為準
server 127.127.1.0
Fudge 127.127.1.0 stratum 10

server cn.ntp.org.cn prefer iburst minpoll 4 maxpoll 10
server ntp.aliyun.com iburst minpoll 4 maxpoll 10
server ntp.tuna.tsinghua.edu.cn iburst minpoll 4 maxpoll 10
server time.ustc.edu.cn iburst minpoll 4 maxpoll 10
# 啟動NTP時間服務並設定服務開機自啟
[root@k8s-master01 ~]# systemctl start ntpd
[root@k8s-master01 ~]# systemctl enable ntpd
[root@k8s-master01 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-03-21 02:59:43 EDT; 4min 52s ago
  Process: 27106 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
[root@k8s-master01 ~]# ntpstat
synchronised to NTP server (120.25.108.11) at stratum 3
   time correct to within 70 ms
   polling server every 16 s

其他主機均從k8s-master01主機上同步時間：

# 安裝NTP時間服務和NTP客戶端
[root@k8s-master02 ~]# yum -y install epel-release.noarch
[root@k8s-master02 ~]# yum -y install ntp ntpdate
# 使用NTP客戶端從NTP時間伺服器同步本機時間
[root@k8s-master02 ~]# ntpdate 192.168.124.128
# 配置NTP時間服務
[root@k8s-master02 ~]# vim /etc/ntp.conf
# 設定從剛剛搭建的NTP時間伺服器主動同步時間
# 如果NTP時間伺服器連線失敗時則以本機時間為準
server 127.127.1.0
Fudge 127.127.1.0 stratum 10

server 192.168.124.128 prefer iburst minpoll 4 maxpoll 10
# 啟動NTP時間服務並設定服務開機自啟
[root@k8s-master02 ~]# systemctl start ntpd
[root@k8s-master02 ~]# systemctl enable ntpd
[root@k8s-master02 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-03-21 02:59:43 EDT; 4min 52s ago
  Process: 27106 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
[root@k8s-master02 ~]# ntpstat
synchronised to NTP server (192.168.124.128) at stratum 3
   time correct to within 70 ms
   polling server every 16 s

6、關閉SWAP

SWAP可能導致容器出現效能下降問題。
所有主機上：

[root@k8s-master01 ~]# swapoff -a  # 臨時關閉
[root@k8s-master01 ~]# free -mh
              total        used        free      shared  buff/cache   available
Mem:           1.8G        133M        1.4G        9.5M        216M        1.5G
Swap:            0B          0B          0B
[root@k8s-master01 ~]# vim /etc/fstab  # 永久關閉
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

7、關閉Firewalld

Kubernetes中的kube-proxy元件需要利用IPtables或者IPVS建立Service物件，CentOS7預設使用Firewalld防火牆服務，為了避免衝突，所以需要禁用和關閉它。
所有主機上：

[root@k8s-master01 ~]# systemctl stop firewalld
[root@k8s-master01 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

8、關閉SeLinux

所有主機上：

[root@k8s-master01 ~]# setenforce 0  # 臨時關閉
[root@k8s-master01 ~]# sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux  # 永久關閉

9、啟用bridge-nf功能

開啟IPtables的網橋透明工作模式，即二層的流量也會受到IPtables規則影響。
如果該功能模組開機沒有載入，則需要載入"br_netfilter"模組。
所有主機上：

[root@k8s-master01 ~]# modprobe br_netfilter
[root@k8s-master01 ~]# lsmod | grep br_netfilter
br_netfilter           22256  0 
bridge                151336  1 br_netfilter
[root@k8s-master01 ~]# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
[root@k8s-master01 ~]# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@k8s-master01 ~]# sysctl --system

10、安裝並啟用IPVS

kube-proxy元件支援三種工作模式轉發流量到Pod：userspace、iptables、ipvs。
如果想要使用ipvs模式則需要安裝IPVS。
所有主機上：

[root@k8s-master01 ~]# yum -y install kernel-devel
[root@k8s-master01 ~]# cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
[root@k8s-master01 ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules
[root@k8s-master01 ~]# bash /etc/sysconfig/modules/ipvs.modules
[root@k8s-master01 ~]# lsmod |grep ip_vs
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  0 
ip_vs                 145497  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          133095  2 ip_vs,nf_conntrack_ipv4
libcrc32c              12644  3 xfs,ip_vs,nf_conntrack
[root@k8s-master01 ~]# yum -y install ipset ipvsadm

2.3.安裝容器執行平臺（Docker）

容器執行平臺用於承載和管理執行容器應用。
配置Docker線上映象源為國內映象源，官方推薦使用的cgroup驅動為"systemd"。
所有主機上：

# 安裝指定版本的docker
[root@k8s-master01 ~]# yum -y install epel-release.noarch yum-utils
[root@k8s-master01 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@k8s-master01 ~]# yum -y install device-mapper-persistent-data  lvm2
[root@k8s-master01 ~]# yum list docker-ce --showduplicates | sort -r
[root@k8s-master01 ~]# yum -y install docker-ce-19.03.0
[root@k8s-master01 ~]# systemctl start docker
[root@k8s-master01 ~]# systemctl enable docker
# 配置docker
[root@k8s-master01 ~]# cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": [
        "https://7mimmp7p.mirror.aliyuncs.com",
        "https://registry.docker-cn.com",
        "http://hub-mirror.c.163.com",
        "https://docker.mirrors.ustc.edu.cn"
        ],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF
[root@k8s-master01 ~]# systemctl restart docker
[root@k8s-master01 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-03-21 06:26:38 EDT; 4s ago
[root@k8s-master01 ~]# docker info | grep Cgroup
 Cgroup Driver: systemd

2.4.安裝Kubeadm、Kubelet、Kubectl

kubeadm，引導構建叢集所使用的工具。
kubelet，在叢集中所有機器上要執行的元件，用於管理Pod和容器。
kubectl，在命令列操作和使用叢集的客戶端工具。
在所有主機上：

[root@k8s-master01 ~]# cat <<EOF >/etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@k8s-master01 ~]# yum install -y kubelet-1.23.0 kubectl-1.23.0 kubeadm-1.23.0 --disableexcludes=kubernetes --nogpgcheck
[root@k8s-master01 ~]# systemctl enable kubelet
[root@k8s-master01 ~]# cat > /etc/sysconfig/kubelet <<EOF
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
EOF

2.5.建立負載均衡器（HAProxy+Keepalived）

參考文件：https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#options-for-software-load-balancing
當存在多個控制平面時，kube-apiserver也存在多個，可以使用Nginx+Keepalived、HAProxy+Keepalived等工具實現多個kube-apiserver的負載均衡和高可用。
推薦使用HAProxy+Keepalived這個組合，因為HAProxy可以提高更高效能的四層負載均衡功能，這也是大多數人的選擇。

1、安裝HAProxy、Keepalived

HAProxy可以實現對後端APIServer的負載均衡與健康檢查，不會轉發請求到不可用的APIServer，以避免失敗的請求。

[root@k8s-master01 ~]# yum -y install haproxy keepalived

2、配置並啟動HAProxy

啟動的HAProxy服務由於後端的api-server還沒有部署執行，需要等待Kubernetes初始化完成才可以正常接受處理請求！
在所有master主機上：

[root@k8s-master01 ~]# vim /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2 emerg info

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserver
    bind *:9443
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    mode tcp
    balance     roundrobin
    server k8s-master01 192.168.124.128:6443 check
    server k8s-master02 192.168.124.130:6443 check
    server k8s-master03 192.168.124.131:6443 check
[root@k8s-master01 ~]# haproxy -f /etc/haproxy/haproxy.cfg -c
Configuration file is valid
[root@k8s-master01 ~]# systemctl start haproxy
[root@k8s-master01 ~]# systemctl enable haproxy
[root@k8s-master01 ~]# netstat -lnupt |grep 9443
tcp        0      0 0.0.0.0:9443            0.0.0.0:*               LISTEN      44965/haproxy  
[root@k8s-master01 ~]# curl localhost:9443
curl: (52) Empty reply from server

3、配置並重啟rsyslog

HAProxy採用rsyslog記錄日誌，日誌有助於後續我們觀察和分析問題。
所有master主機上：

[root@k8s-master01 ~]# vim /etc/rsyslog.conf
local2.*                       /var/log/haproxy.log
[root@k8s-master01 ~]# systemctl restart rsyslog
[root@k8s-master01 ~]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2022-04-12 16:39:50 CST; 2h 11min ago

4、配置Keeaplived

配置Keepalived以實現HAProxy的高可用性，當A主負載均衡器不可用時，還有B、C備用負載均衡器繼續提供服務。
配置基於指令碼(vrrp_script)的健康檢查，當檢查失敗時，權重-2，即優先順序-2，這時候就會發生主備切換。
k8s-master01（MASTER）：

[root@k8s-master01 ~]# vim /etc/keepalived/keepalived.conf
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id 192.168.124.128
}
vrrp_script check_haproxy {
  script "bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi'"
  interval 3
  weight -2
  fall 3
  rise 3
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 50
    priority 100
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        192.168.124.100
    }
    track_script {
        check_haproxy
    }
}

k8s-master02（BACKUP）：

[root@k8s-master02 ~]# vim /etc/keepalived/keepalived.conf
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id 192.168.124.130
}
vrrp_script check_haproxy {
  script "bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi'"
  interval 3
  weight -2
  fall 3
  rise 3
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 50
    priority 99
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        192.168.124.100
    }
    track_script {
        check_haproxy
    }
}

k8s-master03（BACKUP）：

[root@k8s-master03 ~]# vim /etc/keepalived/keepalived.conf
! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id 192.168.124.131
}
vrrp_script check_haproxy {
  script "bash -c 'if [ $(ss -alnupt |grep 9443|wc -l) -eq 0 ];then exit 1;fi'"
  interval 3
  weight -2
  fall 3
  rise 3
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 50
    priority 98
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        192.168.124.100
    }
    track_script {
        check_haproxy
    }
}

5、啟動並設定Keepalived開機自啟

所有master主機上：

[root@k8s-master01 ~]# systemctl start keepalived
[root@k8s-master01 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-03-21 11:48:06 EDT; 4min 0s ago
 Main PID: 48653 (keepalived)
[root@k8s-master01 ~]# systemctl enable keepalived

6、檢視VIP是否在MASTER主機上

[root@k8s-master01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:40:e3:9f brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.128/24 brd 192.168.124.255 scope global dynamic ens33
       valid_lft 1057sec preferred_lft 1057sec
    inet 192.168.124.100/32 scope global ens33
       valid_lft forever preferred_lft forever

7、測試：主故障時自動切換至備

主故障自動切換：
停止MASTER主機上的HAProxy服務，這個時候檢查指令碼觸發優先順序-2，則就會發生主備切換，VIP則會漂移到另外一臺優先順序較低的BACKUP主機上以代替成為新的MASTER。
以下可以看出VIP已經漂移到了k8s-master02上。

[root@k8s-master01 ~]# systemctl stop haproxy
[root@k8s-master01 ~]# ip addr
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:40:e3:9f brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.128/24 brd 192.168.124.255 scope global dynamic ens33
       valid_lft 1451sec preferred_lft 1451sec
[root@k8s-master02 ~]# ip addr
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:c4:65:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.130/24 brd 192.168.124.255 scope global dynamic ens33
       valid_lft 1320sec preferred_lft 1320sec
    inet 192.168.124.100/32 scope global ens33
       valid_lft forever preferred_lft forever

主故障後恢復：
當MASTER主機上的HAProxy服務恢復時，這個時候檢查指令碼觸發優先順序+2，也會發生切換，VIP會漂移到優先順序更高的已恢復正常的MASTER主機繼續作為MASTER提供服務。

[root@k8s-master01 ~]# systemctl start haproxy
[root@k8s-master01 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:40:e3:9f brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.128/24 brd 192.168.124.255 scope global dynamic ens33
       valid_lft 1175sec preferred_lft 1175sec
    inet 192.168.124.100/32 scope global ens33
       valid_lft forever preferred_lft forever

2.6.部署並構建Kubernetes叢集

1、準備映象

可以使用以下命令檢視kubeadm-v1.23.0部署kubernetes-v1.23.0所需要的映象列表以及預設所使用的的映象來源。
所有主機上：

[root@k8s-master01 ~]# kubeadm config print init-defaults |grep imageRepository
imageRepository: k8s.gcr.io
[root@k8s-master01 ~]# kubeadm config images list --kubernetes-version 1.23.0
k8s.gcr.io/kube-apiserver:v1.23.0
k8s.gcr.io/kube-controller-manager:v1.23.0
k8s.gcr.io/kube-scheduler:v1.23.0
k8s.gcr.io/kube-proxy:v1.23.0
k8s.gcr.io/pause:3.6
k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6

由於訪問k8s.gcr.io可能需要FQ，所以我們可以在國內的映象倉庫中下載它們（比如使用阿里雲映象倉庫。阿里雲代理映象倉庫地址：registry.aliyuncs.com/google_containers
如果你需要在更多臺主機上使用它們，則可以考慮使用Harbor或Docker Register搭建私有化映象倉庫。
所有主機上：

# 從映象倉庫中拉取映象
[root@k8s-master01 ~]# kubeadm config images pull --kubernetes-version=v1.23.0 --image-repository=registry.aliyuncs.com/google_containers
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.23.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.6
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.1-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6
# 檢視本地映象列表
[root@k8s-master01 ~]# docker images |grep 'registry.aliyuncs.com/google_containers'
registry.aliyuncs.com/google_containers/kube-apiserver            v1.23.0   e6bf5ddd4098   4 months ago   135MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.23.0   37c6aeb3663b   4 months ago   125MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.23.0   e03484a90585   4 months ago   112MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.23.0   56c5af1d00b5   4 months ago   53.5MB
registry.aliyuncs.com/google_containers/etcd                      3.5.1-0   25f8c7f3da61   5 months ago   293MB
registry.aliyuncs.com/google_containers/coredns                   v1.8.6    a4ca41631cc7   6 months ago   46.8MB
registry.aliyuncs.com/google_containers/pause                     3.6       6270bb605e12   7 months ago   683kB

2、準備kubeadm-init配置檔案

kubeadm相關配置可以參考文件：https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/
k8s-master01:

[root@k8s-master01 ~]# kubeadm config print init-defaults > kubeadm-init.yaml
[root@k8s-master01 ~]# vim kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: "0" # 設定引導令牌的永不過期
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.124.128 # 繫結APIServer要監聽的本機IP地址
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: k8s-master01 # 節點名稱
  taints: null
---
controlPlaneEndpoint: "192.168.124.100:9443" # 控制平面入口點地址："負載均衡器VIP或DNS:負載均衡器埠"
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers # 設定部署叢集時要使用的映象倉庫地址
kind: ClusterConfiguration
kubernetesVersion: 1.23.0 # 設定要部署的kubernetes版本
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs # 設定kube-proxy工作模式為ipvs

3、安裝kubeadm_src，防止kubernetes證書過期（可選步驟）

在Kubernetes中，客戶端與APIServer通訊需要使用X509證書，各元件之間也是使用證書進行身份驗證的，由於官方預設使用kubeadm建立的相關證書有效期只有一年，如果證書到期後可能導致叢集不可用，這非常嚴重。
所以我們這裡對kubernetes原始碼進行修改後編譯生成後的kubeadm(重新命名為kubeadm_src)初始化控制平面節點，在初始化的過程中會生成有效期為其100年的的kubernetes證書！
k8s-master01:

# 安裝GO
[root@k8s-master01 ~]# wget https://go.dev/dl/go1.17.8.linux-amd64.tar.gz
[root@k8s-master01 ~]# tar xzvf go1.17.8.linux-amd64.tar.gz -C /usr/local
[root@k8s-master01 ~]# vim /etc/profile
export PATH=$PATH:/usr/local/go/bin
export GO111MODULE=auto
export GOPROXY=https://goproxy.cn
[root@k8s-master01 ~]# source /etc/profile
[root@k8s-master01 ~]# go version
go version go1.17.8 linux/amd64
# 從GITHUB克隆官方程式碼
[root@k8s-master01 ~]# yum -y install git
[root@k8s-master01 ~]# git clone https://github.91chi.fun/https://github.com/kubernetes/kubernetes.git
[root@k8s-master01 ~]# cd kubernetes
[root@k8s-master01 kubernetes]# git tag -l
...
v1.23.0
...
[root@k8s-master01 kubernetes]# git checkout -b v1.23.0 v1.23.0
# 修改證書有效期相關程式碼
[root@k8s-master01 kubernetes]# vim cmd/kubeadm/app/constants/constants.go
const (
...
        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        // CertificateValidity = time.Hour * 24 * 365
        CertificateValidity = time.Hour * 24 * 365 * 100
...
}
[root@k8s-master01 kubernetes]# vim staging/src/k8s.io/client-go/util/cert/cert.go
...
// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
        now := time.Now()
        tmpl := x509.Certificate{
                SerialNumber: new(big.Int).SetInt64(0),
                Subject: pkix.Name{
                        CommonName:   cfg.CommonName,
                        Organization: cfg.Organization,
                },
                DNSNames:              []string{cfg.CommonName},
                NotBefore:             now.UTC(),
                //NotAfter:              now.Add(duration365d * 10).UTC(),
                NotAfter:              now.Add(duration365d * 100).UTC(),
                KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
                BasicConstraintsValid: true,
                IsCA:                  true,
        }
}
...
# 編譯生成新的kubeadm命令，這將會輸出到_output/bin/目錄下
[root@k8s-master01 kubernetes]# make WHAT=cmd/kubeadm GOFLAGS=-v
# 拷貝kubeadm到所有節點主機的/usr/bin目錄下，並重新命名為kubeadm_src
[root@k8s-master01 kubernetes]# cd _output/bin/ && cp -rf kubeadm /usr/bin/kubeadm_src
[root@k8s-master01 bin]# scp kubeadm root@k8s-master02:/usr/bin/kubeadm_src
[root@k8s-master01 bin]# scp kubeadm root@k8s-master03:/usr/bin/kubeadm_src
[root@k8s-master01 bin]# scp kubeadm root@k8s-node01:/usr/bin/kubeadm_src

4、基於kubeadm-init配置並使用原始碼編譯的kubeadm建立一個初始化控制平面節點

kubeadm在初始化控制平面時會生成部署Kubernetes叢集中各個元件所需的相關配置檔案在/etc/kubernetes目錄下，可以供我們參考。
注：由於原始碼生成kubeadm在初始化節點的時候無法正確的配置YUM安裝的kubelet服務，所以需要YUM安裝的kubeadm先配置kubelet服務！
初始化完成後的提示資訊和後續執行命令需要儲存一下！
k8s-master01:

# 使用YUM安裝的kubeadm初始配置kubelet
[root@k8s-master01 ~]# kubeadm init phase kubelet-start --config kubeadm-init.yaml

# 使用原始碼編譯的kubeadm初始化控制平面節點
[root@k8s-master01 ~]# kubeadm_src init --config kubeadm-init.yaml --upload-certs
Your Kubernetes control-plane has initialized successfully!
你的Kubernetes控制平面已初始化成功！

To start using your cluster, you need to run the following as a regular user:
你的叢集是啟動狀態，如果你是普通使用者的話請繼續執行以下命令：

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:
或者，如果你是root使用者的話請繼續執行以下命令：

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
你還需要在叢集上部署一個Pod網路外掛！

You can now join any number of the control-plane node running the following command on each as root:
你如果需要更多數量的控制平面節點加入到叢集的話，請使用root使用者在節點執行以下命令：

  kubeadm join 192.168.124.100:9443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:64c918139d7d344b64b0720244077b60ea10f5572717f92113c08fe9c56be3c9 \
	--control-plane --certificate-key 5d87ca735c040ba6b04de388f2857530bbd9de094cbd43810904afe9a6aec50d

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
證書的訪問金鑰有效期只有2小時，如果你還需要部署更多的節點，請先執行"kubeadm init phase upload-certs --upload-certs"重新上傳證書到kubeadm-certs！

Then you can join any number of worker nodes by running the following on each as root:
你如果需要更多數量的工作節點加入到叢集的話，請使用root使用者在節點上執行以下命令：

kubeadm join 192.168.124.100:9443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:64c918139d7d344b64b0720244077b60ea10f5572717f92113c08fe9c56be3c9

5、將正確的kubelet服務配置檔案拷貝到其他主機

k8s-master01：

[root@k8s-master01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
[root@k8s-master01 ~]# scp -r /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf root@k8s-master02:/usr/lib/systemd/system/kubelet.service.d
[root@k8s-master01 ~]# scp -r /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf root@k8s-master03:/usr/lib/systemd/system/kubelet.service.d
[root@k8s-master01 ~]# scp -r /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf root@k8s-node01:/usr/lib/systemd/system/kubelet.service.d

其他主機上過載服務配置：

[root@k8s-master02 ~]# systemctl daemon-reload

6、其他節點加入到叢集

請使用原始碼編譯後的kubeadm命令！
其他控制平面節點加入到叢集：

[root@k8s-master02 ~]# kubeadm_src join 192.168.124.100:9443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:64c918139d7d344b64b0720244077b60ea10f5572717f92113c08fe9c56be3c9 --control-plane --certificate-key 5d87ca735c040ba6b04de388f2857530bbd9de094cbd43810904afe9a6aec50d

工作節點加入到叢集：

[root@k8s-node01 ~]# kubeadm_src join 192.168.124.100:9443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:64c918139d7d344b64b0720244077b60ea10f5572717f92113c08fe9c56be3c9

7、觀察Etcd

可見Etcd是以叢集的方式執行的！

[root@k8s-master03 ~]# ps aux |grep etcd
root       1971  5.4  4.5 11283128 84128 ?      Ssl  16:33   1:00 etcd --advertise-client-urls=https://192.168.124.131:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://192.168.124.131:2380 --initial-cluster=k8s-master03=https://192.168.124.131:2380,k8s-master01=https://192.168.124.128:2380,k8s-master02=https://192.168.124.130:2380 --initial-cluster-state=existing --key-file=/etc/kubernetes/pki/etcd/server.key --listen-client-urls=https://127.0.0.1:2379,https://192.168.124.131:2379 --listen-metrics-urls=http://127.0.0.1:2381 --listen-peer-urls=https://192.168.124.131:2380 --name=k8s-master03 --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/etc/kubernetes/pki/etcd/peer.key --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

8、檢視Kubernetes證書有效期

[root@k8s-master01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 18, 2122 04:38 UTC   99y                                     no      
apiserver                  Mar 18, 2122 04:38 UTC   99y             ca                      no      
apiserver-etcd-client      Mar 18, 2122 04:38 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   Mar 18, 2122 04:38 UTC   99y             ca                      no      
controller-manager.conf    Mar 18, 2122 04:38 UTC   99y                                     no      
etcd-healthcheck-client    Mar 18, 2122 04:38 UTC   99y             etcd-ca                 no      
etcd-peer                  Mar 18, 2122 04:38 UTC   99y             etcd-ca                 no      
etcd-server                Mar 18, 2122 04:38 UTC   99y             etcd-ca                 no      
front-proxy-client         Mar 18, 2122 04:38 UTC   99y             front-proxy-ca          no      
scheduler.conf             Mar 18, 2122 04:38 UTC   99y                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 18, 2122 04:38 UTC   99y             no      
etcd-ca                 Mar 18, 2122 04:38 UTC   99y             no      
front-proxy-ca          Mar 18, 2122 04:38 UTC   99y             no

9、設定kubectl客戶端以連線到叢集

節點在部署完成時，會生成用於kubectl登入所使用的kubeconfig配置檔案在"/etc/kubernetes/admin.conf"!
所有master主機上：

[root@k8s-master01 ~]# mkdir -p $HOME/.kube
[root@k8s-master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

10、檢視叢集中節點列表

可見所有節點都是"NotReady"未就緒狀態，這需要在叢集中安裝Pod網路外掛！

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS     ROLES                  AGE    VERSION
k8s-master01   NotReady   control-plane,master   145m   v1.23.0
k8s-master02   NotReady   control-plane,master   144m   v1.23.0
k8s-master03   NotReady   control-plane,master   143m   v1.23.0
k8s-node01     NotReady   <none>                 76m    v1.23.0
[root@k8s-master01 ~]# kubectl describe nodes k8s-master01
Name:               k8s-master01
...
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                      
KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

3.安裝一些必要的外掛

3.1.安裝Pod網路外掛（CNI）-Calico

Calico是一個開源的虛擬化網路方案，支援基礎的Pod網路通訊和網路策略功能。
Kubernetes有一種資源型別"NetworkPolicy"，用於描述Pod的網路策略，要想使用該資源型別，則需要Pod網路外掛支援網路策略功能。
參考文件：https://github.com/projectcalico/calico
任意一臺master主機上：

[root@k8s-master01 ~]# wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate
[root@k8s-master01 ~]# kubectl apply -f calico.yaml 
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionudget.policy/calico-kube-controllers created

安裝完成後可以看到Calico會在叢集中建立並執行對應Pod，並且此時所有的Node已經是就緒狀態：

[root@k8s-master01 ~]# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS         AGE
calico-kube-controllers-56fcbf9d6b-phf49   1/1     Running   0                12m
calico-node-8frvw                          1/1     Running   0                12m
calico-node-mzpmv                          1/1     Running   0                12m
calico-node-rblcg                          1/1     Running   0                12m
calico-node-vh9c9                          1/1     Running   0                12m
coredns-6d8c4cb4d-85b5t                    1/1     Running   0                39m
coredns-6d8c4cb4d-h7ttw                    1/1     Running   0                38m
[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES                  AGE   VERSION
k8s-master01   Ready    control-plane,master   28h   v1.23.0
k8s-master02   Ready    control-plane,master   28h   v1.23.0
k8s-master03   Ready    control-plane,master   28h   v1.23.0
k8s-node01     Ready    <none>                 27h   v1.23.0

3.2.安裝Ingress控制器-Ingress-Nginx

Ingress是Kubernetes標準的資源型別之一，用於描述Service的七層實現，實現基於HTTP協議的反向代理功能，這在Web專案中是經常要用的。
"Ingress"功能的提供由Ingress控制器（外掛）實現，ingress-nginx是常用的Ingress控制器。
參考文件：
https://github.com/kubernetes/ingress-nginx
https://kubernetes.github.io/ingress-nginx/deploy/

1、檢視相容版本

Ingress-NGINX version	k8s supported version	        Alpine Version	Nginx Version
v1.1.3	                1.23, 1.22, 1.21, 1.20, 1.19	3.14.4	        1.19.10†
v1.1.2	                1.23, 1.22, 1.21, 1.20, 1.19	3.14.2	        1.19.9†
v1.1.1	                1.23, 1.22, 1.21, 1.20, 1.19	3.14.2	        1.19.9†

2、搜尋國內映象源

注：這邊需要修改一下映象源為國內克隆映象源，否則可能無法下載映象。
可以去DockerHUB中搜尋一下對應版本的相關映象！

3、安裝Ingress-Nginx-Controller

[root@k8s-master01 ~]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.2/deploy/static/provider/cloud/deploy.yaml -O ingress-nginx.yaml
[root@k8s-master01 ~]# vim ingress-nginx.yaml
#image: k8s.gcr.io/ingress-nginx/controllerv1.1.2@...
image: willdockerhub/ingress-nginx-controller:v1.1.2
#image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1@...
image: liangjw/kube-webhook-certgen:v1.1.1
[root@k8s-master01 ~]# kubectl apply -f ingress-nginx.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created

4、檢視執行狀態

[root@k8s-master01 ~]# kubectl get pods --namespace=ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-6xk5t        0/1     Completed   0          11m
ingress-nginx-admission-patch-sp6w2         0/1     Completed   0          11m
ingress-nginx-controller-7bc7476f95-gdxkz   1/1     Running     0          11m

5、使用外部負載均衡器關聯Ingress控制器

外部主機想要訪問到Pod-Ingress控制器需要通過Service，預設情況下使用.yaml安裝Ingress-nginx-controller時會建立LoadBalancer型別的Service，以用於外部負載均衡器關聯並將訪問請求轉發至Ingress控制器處理。
LoadBalancer型別的Service是NodePort型別的上層實現，同理它會在每臺節點主機上都開放一個對映埠，可用於外部負載均衡器進行關聯。

[root@k8s-master01 ~]# kubectl get service --namespace=ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.103.77.111   <pending>     80:30408/TCP,443:32686/TCP   20m
ingress-nginx-controller-admission   ClusterIP      10.98.133.60    <none>        443/TCP                      20m
[root@k8s-master01 ~]# netstat -lnupt  |grep -E '30408|32686'
tcp        1      0 0.0.0.0:30408           0.0.0.0:*               LISTEN      41631/kube-proxy    
tcp        0      0 0.0.0.0:32686           0.0.0.0:*               LISTEN      41631/kube-proxy

3.3.安裝Metrics-Server

Metrices-Server，指標伺服器，Metrices-Server是Kubernetes中的一個常用外掛，它類似於Top命令，可以檢視Kubernetes中Node和Pod的CPU和記憶體資源使用情況。
Metrices-Server每15秒收集一次指標，它在叢集中的每個節點中執行，可擴充套件支援多達5000個節點的叢集。
Metrices-Server從0.5版本開始預設情況下要求節點上需要的資源請求為100m的CPU和200MiB的記憶體，以保證100+節點數量的效能是良好的。
參考文件：https://github.com/kubernetes-sigs/metrics-server

1、檢視與Kuberneres的相容性

Metrics Server	Metrics API group/version	Supported Kubernetes version
0.6.x	       metrics.k8s.io/v1beta1	        1.19+
0.5.x	       metrics.k8s.io/v1beta1	        *1.8+
0.4.x	       metrics.k8s.io/v1beta1	        *1.8+
0.3.x	       metrics.k8s.io/v1beta1	        1.8-1.21

2、搜尋國內克隆映象

官方的安裝清單components.yaml預設情況下使用的映象倉庫為k8s.gcr.io，在沒有FQ的情況下Pod執行可能無法正常獲取到Metrics-Server的安裝映象。

3、安裝Metrics-Server

Metrics-Server預設情況下在啟動的時候需要驗證kubelet提供的CA證書，這可能會導致其啟動失敗，所以需要新增引數"--kubelet-insecure-tls"禁用此校驗證書功能。

[root@k8s-master01 ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -O metrics-server.yaml
[root@k8s-master01 ~]# vim metrics-server.yaml
    spec:
      containers:
      - args:
        - --kubelet-insecure-tls
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        image: bitnami/metrics-server:0.6.1
[root@k8s-master01 ~]# kubectl apply -f metrics-server.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
[root@k8s-master01 ~]# kubectl get pods --namespace=kube-system |grep -E 'NAME|metrics-server'
NAME                                       READY   STATUS    RESTARTS       AGE
metrics-server-599b4c96ff-njg8b            1/1     Running   0              76s

4、檢視叢集中節點的資源使用情況

[root@k8s-master01 ~]# kubectl top node
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master01   331m         8%     1177Mi          68%       
k8s-master02   419m         10%    1216Mi          70%       
k8s-master03   344m         8%     1155Mi          67%       
k8s-node01     246m         6%     997Mi           57%

5、檢視叢集中指定名稱空間下Pod的資源使用情況

[root@k8s-master01 ~]# kubectl top pod --namespace=kube-system
NAME                                       CPU(cores)   MEMORY(bytes)   
calico-kube-controllers-56fcbf9d6b-phf49   5m           29Mi            
calico-node-8frvw                          98m          120Mi           
calico-node-mzpmv                          71m          121Mi           
...

3.4.安裝Dashboard

Kubernetes Dashboard是Kubernetes叢集的通用、基於Web的UI。它允許使用者管理叢集中執行的應用程式並對其進行故障排除，以及管理叢集本身。
Dashboard是Kubernetes的一個外掛，由APIServer提供的一個URL提供訪問入口：/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy
當前你也可以通過Service直接訪問到DashBoard！
參考文件：
https://github.com/kubernetes/dashboard
https://github.com/kubernetes/dashboard/blob/master/docs/user/accessing-dashboard/README.md#login-not-available

1、安裝Dashboard

根據配置清單安裝Dashboard，會建立Cluster型別的Service，僅只能從叢集內部主機訪問到Dashboard，所以這邊需要簡單修改一下，將Service修改為NodePort型別，這樣外部主機也可以訪問它。

[root@k8s-master01 ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml -O dashboard.yaml
[root@k8s-master01 ~]# vim dashboard.yaml
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
[root@k8s-master01 ~]# kubectl apply -f dashboard.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
[root@k8s-master01 ~]# kubectl get pod --namespace=kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-799d786dbf-xx9j7   1/1     Running   0          3m16s
kubernetes-dashboard-fb8648fd9-rgc2z         1/1     Running   0          3m17s

2、訪問到Dashboard

[root@k8s-master01 ~]# kubectl get service --namespace=kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.97.23.158    <none>        8000/TCP        4m6s
kubernetes-dashboard        NodePort    10.103.40.153   <none>        443:32358/TCP   4m7s
[root@k8s-master01 ~]# netstat -lnupt |grep 32358
tcp        0      0 0.0.0.0:32358           0.0.0.0:*               LISTEN      41631/kube-proxy

瀏覽器輸入：https://<任一節點主機IP>:/#/login

3、選擇登入到Dashboard要使用的身份認證方式

登入進入Dashboard需要進行身份認證。
Dashboard服務在Pod中執行，Pod想要訪問並獲取到叢集相關資訊的話則需要建立一個ServiceAccount以驗證身份。
Dashboard想要管理Kubernetes叢集需要進行身份認證，目前支援Token和Kubeconfig兩種方式。
Token：
建立一個擁有叢集角色"cluster-admin"的服務賬戶"dashboard-admin"，然後使用dashboard-admin的Token即可！當然你也可以根據特殊需要建立擁有指定許可權的叢集角色將其繫結到對應的服務賬戶上，以管理叢集中指定資源。

# 建立一個專用於Dashboard的服務賬戶"dashboard-admin"
[root@k8s-master01 ~]# kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard
serviceaccount/dashboard-admin created
# 為服務賬戶"dashboard-admin"繫結到擁有超級管理員許可權的叢集角色"cluster-admin"
# 則dashboard-admin就擁有了超級管理員許可權
[root@k8s-master01 ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created
# 建立的服務賬戶，會自動生成一個Token，它是Secret型別的資源物件
# 我們可以使用以下操作獲取到服務賬戶"dashboard-admin"的Token以用於Dashboard身份驗證
[root@k8s-master01 ~]# kubectl get secrets -n kubernetes-dashboard |grep dashboard-admin-token
dashboard-admin-token-2bxfl        kubernetes.io/service-account-token   3      66s
[root@k8s-master01 ~]# kubectl describe secrets/dashboard-admin-token-2bxfl -n kubernetes-dashboard
Name:         dashboard-admin-token-2bxfl
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 492a031e-db41-4a65-a8d4-af0e240e7f9d

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1103 bytes
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImFXTzZFUElaS2RoTUpScHFwNzJSNUN5eU1lcFNSZEZqNWNNbi1VbFV2Zk0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tMmJ4ZmwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDkyYTAzMWUtZGI0MS00YTY1LWE4ZDQtYWYwZTI0MGU3ZjlkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.l5VEIPd9nIsJuXMh86rjFHhkIoZmg5nlDw7Bixn0b3-KT1r6o7WRegq8DJyVk_iiIfRnrrz5jjuOOkCKwXwvI1NCfVdsuBKXFwFZ1Crc-BwHjIxWbGuZfEGxSbN8du4T4xcUuNU-7HuZQcGDY23uy68aPqWSm8UoIcOFwUgVcYkKlOuW76tIXxG_upxWpWZz74aMDUIkjar7sdWXzMr1m5G43TLE9Z_lKCgoV-hc4Fo9_Er-TIAPqDG6-sfZZZ9Raldvn3j380QDYahUKaGKabnOFDXbODKOQ1VKRizgiRTOqt-z9YRPTcyxQzfheKC8DTb2X8D-E4x6azulenNgqw

Kubeconfig：
Token是很長的複雜的金鑰字串，使用它進行身份認證並不方便，所以Dashboard支援使用Kubeconfig檔案的方式登陸到Dashboard。
基於上面Token的建立的服務賬戶，建立一個Kubeconfig配置檔案。

# 檢視叢集資訊
[root@k8s-master01 ~]# kubectl cluster-info
Kubernetes control plane is running at https://192.168.124.100:9443
# 建立kubeconfig檔案並設定叢集相關
[root@k8s-master01 ~]# kubectl config set-cluster kubernetes --embed-certs=true --server="https://192.168.124.100:9443" --certificate-authority=/etc/kubernetes/pki/ca.crt --kubeconfig=dashboard-admin.kubeconfig
# 設定認證相關到kubeconfig檔案
# 預設情況下服務賬戶的Token是base64編碼格式，如果需要將其寫到kubeconfig中的則需要使用"base64 -d"進行解
# 碼
[root@k8s-master01 ~]# Token=$(kubectl get secrets/dashboard-admin-token-2bxfl -n kubernetes-dashboard -o jsonpath={.data.token} |base64 -d)
[root@k8s-master01 ~]# kubectl config set-credentials dashboard-admin --token=${Token} --kubeconfig=./dashboard-admin.kubeconfig 
# 設定上下文相關到kubeconfig檔案
[root@k8s-master01 ~]# kubectl config set-context dashboard-admin --cluster=kubernetes  --user=dashboard-admin --kubeconfig=./dashboard-admin.kubeconfig 
# 設定當前要使用的上下文到kubeconfig檔案
[root@k8s-master01 ~]# kubectl config use-context dashboard-admin --cluster=kubernetes  --user=dashboard-admin --kubeconfig=./dashboard-admin.kubeconfig
# 最後得到以下檔案
[root@k8s-master01 ~]# cat dashboard-admin.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBRENDQWVpZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQ0FYRFRJeU1EUXhNVEEwTXpnME1Gb1lEekl4TWpJd016RTRNRFF6T0RRd1dqQVZNUk13RVFZRApWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBCjR0RDRmU2ZmcHU1WS9KUGJrQWgvdG0xS1lSeWQ5YU9MVk9xTDQyc1M5YmxiZGh0WU9QSHYvWEpVb1k1ZSs5MXgKUE9NbnZnWmhiR29uditHQWVFazRKeUl4MTNiTm1XUk1DZ1QyYnJIWlhvcm5QeGE0ZlVsNHg5K2swVEc5ejdIMAo0cjF5MFkzWXNXaGJIeHBwL0hvQzNRR2JVWVJyMm03NVgxTWUvdFFCL25FcUNybUZxNkRveEU3REIxMkRnemE4CjBrM3FwZllGZHBOcnZBakdIcUlSZ0ZxT24ybDVkb0c3bGVhbkIrY2wxQWltUnZCMDdQdlVKdVhrK1N5NUhmdnMKNzYyYXJRYklNMUlISkJ0ZXBaQzVjYi9pNGZhcWNrTXJaeTZvanlnN2JPcjBuMlpQcHV5SnR5QjhLMnJDZCtYZApTeXlrZG44S0MxRlRSR0p6dkdpaVRRSURBUUFCbzFrd1Z6QU9CZ05WSFE4QkFmOEVCQU1DQXFRd0R3WURWUjBUCkFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVucEhxdGJzZ01CcSt4Q1MzTVErWnk4akFpeFV3RlFZRFZSMFIKQkE0d0RJSUthM1ZpWlhKdVpYUmxjekFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBRHhpR3c2bk5NV1hRMnVlRgppK2R2Nittc1FUT0JCWWJENkhIblVETUtnK0loaEcwclA5MkFRRjFWcUZaN1ZDSTgyWmJ5VnVWSmVvMjlkdjZpClBDeFJzWERxdHl0TG1CMkFPRUxXOFdqSCtheTZ5a3JYbGIwK1NIZ1Q3Q1NJRHhRdG9TeE8rK094WjhBb1JGMmUKTy94U1YxM0E0eG45RytmUEJETkVnWUJHbWd6L1RjSjZhYnljZnNNaGNwZ1kwKzJKZlJDemZBeFNiMld6TzBqaApucFRONUg2dG1ST3RlQ2h3anRWVDYrUXBUSzdkN0hjNmZlZ0w0S1pQZDEwZ0hyRFV1eWtpY01UNkpWNXNJSjArCmw5eWt2V1R2M2hEN0NJSmpJWnUySjdod0FGeW1hSmxzekZuZEpNZUFEL21pcDBMQk40OUdER2M2UFROdUw0WHEKeUxrYUhRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.124.100:9443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: dashboard-admin
  name: dashboard-admin
current-context: dashboard-admin
kind: Config
preferences: {}
users:
- name: dashboard-admin
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImFXTzZFUElaS2RoTUpScHFwNzJSNUN5eU1lcFNSZEZqNWNNbi1VbFV2Zk0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tMmJ4ZmwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDkyYTAzMWUtZGI0MS00YTY1LWE4ZDQtYWYwZTI0MGU3ZjlkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.l5VEIPd9nIsJuXMh86rjFHhkIoZmg5nlDw7Bixn0b3-KT1r6o7WRegq8DJyVk_iiIfRnrrz5jjuOOkCKwXwvI1NCfVdsuBKXFwFZ1Crc-BwHjIxWbGuZfEGxSbN8du4T4xcUuNU-7HuZQcGDY23uy68aPqWSm8UoIcOFwUgVcYkKlOuW76tIXxG_upxWpWZz74aMDUIkjar7sdWXzMr1m5G43TLE9Z_lKCgoV-hc4Fo9_Er-TIAPqDG6-sfZZZ9Raldvn3j380QDYahUKaGKabnOFDXbODKOQ1VKRizgiRTOqt-z9YRPTcyxQzfheKC8DTb2X8D-E4x6azulenNgqw

4、選擇Kubeconfig檔案登陸Dashboard即可


附錄-檢視Kubernetes與Docker相容性

訪問網址：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.23.md

附錄-重置節點

當在使用"kubeadm init"或"kubeadm join"部署節點出現失敗狀況時，可以使用以下操作對節點進行重置！
注：重置會將節點恢復到未部署前狀態，若叢集已正常工作則無需重置，否則將引起不可恢復的叢集故障！

[root@k8s-master01 ~]# kubeadm reset -f
[root@k8s-master01 ~]# ipvsadm --clear
[root@k8s-master01 ~]# iptables -F && iptables -X && iptables -Z

附錄-常用檢視命令

更多的操作請完整學習Kubernetes的資源和叢集管理！
檢視令牌（Token）列表：

[root@k8s-master01 ~]# kubeadm token list
TOKEN                     TTL         EXPIRES   USAGES                   DESCRIPTION                                                EXTRA GROUPS
abcdef.0123456789abcdef   <forever>   <never>   authentication,signing   <none>                                                     system:bootstrappers:kubeadm:default-node-token
ek6xtl.s3dk4vjxzp83bcx3   1h          2022-04-06T13:30:39Z   <none>                   Proxy for managing TTL for the kubeadm-certs secret        <none>

檢視kubernetes叢集中證書到期時間：

[root@k8s-master01 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 18, 2122 04:02 UTC   99y                                     no      
apiserver                  Mar 18, 2122 04:02 UTC   99y             ca                      no      
apiserver-etcd-client      Mar 18, 2122 04:02 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   Mar 18, 2122 04:02 UTC   99y             ca                      no      
controller-manager.conf    Mar 18, 2122 04:02 UTC   99y                                     no      
etcd-healthcheck-client    Mar 18, 2122 04:02 UTC   99y             etcd-ca                 no      
etcd-peer                  Mar 18, 2122 04:02 UTC   99y             etcd-ca                 no      
etcd-server                Mar 18, 2122 04:02 UTC   99y             etcd-ca                 no      
front-proxy-client         Mar 18, 2122 04:02 UTC   99y             front-proxy-ca          no      
scheduler.conf             Mar 18, 2122 04:02 UTC   99y                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 18, 2122 04:02 UTC   99y             no      
etcd-ca                 Mar 18, 2122 04:02 UTC   99y             no      
front-proxy-ca          Mar 18, 2122 04:02 UTC   99y             no

檢視節點執行狀態：

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES                  AGE   VERSION
k8s-master01   Ready    control-plane,master   40h   v1.23.0
k8s-master02   Ready    control-plane,master   40h   v1.23.0
k8s-master03   Ready    control-plane,master   40h   v1.23.0
k8s-node01     Ready    <none>                 39h   v1.23.0

檢視Kubeadm初始化控制平面預設使用的配置資訊:

[root@k8s-master ~]# kubeadm config print init-defaults

檢視Kubeadm部署安裝Kubernetes叢集所要使用的容器映象列表：

[root@k8s-master ~]# kubeadm config images list

檢視叢集中的名稱空間（NameSpace）：

[root@k8s-master01 ~]# kubectl get namespace
NAME                   STATUS   AGE
default                Active   44h
ingress-nginx          Active   13h
kube-node-lease        Active   44h
kube-public            Active   44h
kube-system            Active   44h
kubernetes-dashboard   Active   7h7m

檢視指定名稱空間下Pod執行狀態：

[root@k8s-master01 ~]# kubectl get pod --namespace=kube-system
NAME                                       READY   STATUS    RESTARTS        AGE
calico-kube-controllers-56fcbf9d6b-phf49   1/1     Running   0               16h
calico-node-8frvw                          1/1     Running   0               16h
calico-node-mzpmv                          1/1     Running   0               16h
calico-node-rblcg                          1/1     Running   0               16h
calico-node-vh9c9                          1/1     Running   0               16h
coredns-6d8c4cb4d-85b5t                    1/1     Running   0               16h
coredns-6d8c4cb4d-h7ttw                    1/1     Running   0               16h
etcd-k8s-master01                          1/1     Running   73 (19h ago)    44h
etcd-k8s-master02                          1/1     Running   51 (19h ago)    43h
etcd-k8s-master03                          1/1     Running   1 (24h ago)     43h
...

附錄-新的Pod可以排程到Master節點上執行嗎？

可以，預設情況下Master節點在建立的時候，就已經被填充了汙點"taints"，如果想要在Master節點上執行Pod，只需要將"taints"刪除即可！（不建議的操作）

[root@k8s-master01 ~]# kubectl describe nodes/k8s-master01
Name:               k8s-master01
...
Taints:             node-role.kubernetes.io/master:NoSchedule
...
[root@k8s-master ~]# kubectl taint nodes --all node-role.kubernetes.io/master-

附錄-叢集最大支援多少個節點那？

參考文件：https://kubernetes.io/docs/setup/best-practices/cluster-large/
Kubernetes叢集是由一組執行有Kubernetes代理的節點（物理機/虛擬機器）組成，由控制平面節點管理著工作節點。
Kubernetes-v1.23.x理論上支援5000個節點的叢集，其中：

  • 每個節點不超過110個Pod;
  • 叢集中總的Pod數量不超過150000個;
  • 叢集中總的容器數量不超過300000個。

以上資料僅是官方實踐後得出的結論！
工作節點由一個或多個控制平面節點管理，控制平面節點可以管理工作節點的數量取決於控制平面節點所在物理主機的CPU、記憶體、磁碟IO和空間使用情況！那麼這時候對主機以及相關元件做好監控是非常重要的！
其他人員的經驗：
一臺1核2GB的控制平面節點可以管理約5個工作節點！
一臺32核120GB的控制平面節點可以管理約500個工作節點！
以上資料僅供參考！

相關文章