基於containerd 部署 kubernetes 1.28叢集

1、準備說明

• 8臺Linux主機，安裝Ubuntu 20.04系統，其中2臺haproxy，3臺master節點，3臺work節點
• 每臺主機不低於2GB 記憶體大小，CPU大於2核心
• 叢集中的所有主機網路互通
• 節點中不能存在重複的主機名、mac地址或者product_uuid
• 交換分割槽配置。kubelet預設是在節點上檢測到交換分割槽時，無法啟動kubelet。kubelet從1.22版本開始支援交換分割槽。1.28版本，僅針對cgroup v2支援交換分割槽。如果kubelet沒有配置好對swap分割槽的支援，則必須進行關閉交換分割槽swapoff -a，進行暫時關於交換分割槽。

1.1、確保節點的mac地址和product_uuid的唯一性

#每一個節點進行檢視
root@K8SMS0001:~# ip link	#檢視mac地址
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:b6:97:cd brd ff:ff:ff:ff:ff:ff

root@K8SMS0001:~# cat /sys/class/dmi/id/product_uuid	#檢視uuid的唯一
42364795-73ca-edde-a606-9ab6b26b6516

2、主機系統引數配置

2.1、主機名配置說明

主機名	角色	ip地址
K8SHA0001	HA主節點	10.64.143.31
K8SHA0002	HA從節點	10.64.143.32
K8SMS0001	K8S-master01	10.64.143.33
K8SMS0002	K8S-master02	10.64.143.34
K8SMS0003	K8S-master03	10.64.143.35
K8SWK0001	K8S-worker01	10.64.143.36
K8SWK0002	K8S-worker02	10.64.143.37
K8SWK0003	K8S-worker03	10.64.143.38

2.2、hosts解析配置

# 每個節點進行配置
$ cat >> /etc/hosts << EOF
K8SHA0001 10.64.143.31
K8SHA0002 10.64.143.32
K8SMS0001 10.64.143.33
K8SMS0002 10.64.143.34
K8SMS0003 10.64.143.35
K8SWK0001 10.64.143.36
K8SWK0002 10.64.143.37
K8SWK0003 10.64.143.38

2.3、selinux、防火牆關閉

$ systemctl stop firewalld
$ firewall-cmd --state
$ setenforce 0
$ getenforce
$ sed -i 's/SELINUX=enforceing/SELINUX=disabled/g' /etc/selinux/config

2.4、時間同步

$ crontab -e
*/5 * * * * ntpdate time1.aliyun.com
$ ntpdate time1.aliyun.com

2.5、配置核心轉發和網橋過濾

# 新增網橋過濾及核心轉發配置檔案
# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

載入br_netfilter模組
# modprobe br_netfilter
# sysctl -p /etc/sysctl.d/k8s.conf

#檢視是否載入	
# lsmod |grep br_netfilter	
br_netfilter           28672  0
bridge                176128  1 br_netfilter
# lsmod | grep overlay
overlay               118784  0

# sysctl -a |grep 'bridge-nf-call-ip\|ip_forward = 1'
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

2.6、安裝ipset和ipvsadm

# yum install -y ipset ipvsadm

配置ipvsadm模組載入
# cat >> /etc/sysconfig/modules/ipvs.modules << EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

# 授權並載入模組
# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod |grep -e ip_vs -e nf_conntrack

2.7、關閉SWAP分割槽

# swappoff -a

3、HAproxy和keepalived安裝配置

可以參考1.22.8的配置

# yum install haproxy keepalived -y

4、容器執行時安裝

4.1、下載cri-containerd

github containerd地址
cri-containerd-1.7.14版本下載

# 下載cri-containerd 1.7.14版本
# wget https://github.com/containerd/containerd/releases/download/v1.7.14/cri-containerd-1.7.14-linux-amd64.tar.gz

4.2、二進位制安裝cri-containerd

# mkdir containerd && tar -zxf cri-containerd-1.7.14-linux-amd64.tar.gz -C containerd/ && cp ./usr/local/bin/* /usr/local/bin/ && cp ./etc/systemd/system/containerd.service /usr/lib/systemd/system/ && systemctl daemon-reload && containerd --version

4.3、生成並配置containerd預設配置檔案

# 生成containerd配置檔案，然後/etc/containerd/config.toml中修改如下配置項：
# mkdir /etc/containerd && containerd config default > /etc/containerd/config.toml
# vim /etc/containerd/config.toml
......          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
......
            SystemdCgroup = true	#修改SystemdCgroup
[plugins."io.containerd.grpc.v1.cri"]
......
    #修改pause映象路徑
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"	
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
......
    runtime_type = 'io.containerd.runc.v2'	#修改預設的runtime_type，否則後面crictl image存在報錯資訊

# 重啟containerd服務
# systemctl restart containerd
# systemctl enable containerd

4.4、下載並安裝runc

github runc下載地址
runc 1.1.12版本下載

# wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64
# mv runc.amd64 /usr/sbin/runc && chmod +x /usr/sbin/runc && runc -v

5、安裝kubeadm、kubelet、kubectl

需要在每臺機器上安裝以下的軟體包

• kubeadm：用來初始化叢集的指令。
• kubelet：在叢集中的每個節點上用來啟動 Pod 和容器等。
• kubectl：用來與叢集通訊的命令列工具。
  kubeadm 無法進行安裝或者管理 kubelet 或 kubectl， 所以需要確保它們與透過 kubeadm 安裝的控制平面的版本相匹配。 如果不這樣做，則存在發生版本偏差的風險，可能會導致一些預料之外的錯誤和問題。

5.1、增加阿里雲的kubernetes源

# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.28/rpm/repodata/repomd.xml.key
EOF

# yum clean all
# yum makecache

5.2、安裝指定1.28.8版本kubeadm、kubectl、kubelet

# yum --showduplicate list kubeadm kubectl kublet

# yum install -y kubelet-1.28.8 kubeadm-1.28.8 kubectl-1.28.8

5.3、修改kubelet預設cgroup驅動

# vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"

# systemctl enable kubelet --now

6、初始化叢集

6.1、檢視初始化需要的映象並下載

映象所有的節點都先下好

# kubeadm config image list

# crictl pull registry.aliyuncs.com/google_containers/pause:3.9
# crictl pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/kube-proxy:v1.28.8
# crictl pull registry.aliyuncs.com/google_containers/etcd:3.5.12-0

6.2、叢集初始化

root@K8SMS0001:~# kubeadm init --control-plane-endpoint 10.64.143.30:6443 --kubernetes-version=v1.28.8 --pod-network-cidr 172.16.0.0/16 --service-cidr 10.96.0.0/16 --service-dns-domain cluster.local --image-repository registry.aliyuncs.com/google_containers --upload-certs
[init] Using Kubernetes version: v1.28.8
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local sphqjdk8sms0001] and IPs [10.96.0.1 10.64.143.33 10.64.143.30]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost sphqjdk8sms0001] and IPs [10.64.143.33 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost sphqjdk8sms0001] and IPs [10.64.143.33 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.037389 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node sphqjdk8sms0001 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node sphqjdk8sms0001 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: xy67nn.3yagma81us8dzatp
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96 \
        --control-plane 

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96 

6.3、複製配置

# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
# kubectl get node

6.4、master叢集加入和work節點叢集加入

master節點加入指令

  kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96 \
        --control-plane 

work節點加入指令

kubeadm join 10.64.143.30:6443 --token xy67nn.3yagma81us8dzatp \
        --discovery-token-ca-cert-hash sha256:e4c4f8a7495c9f3a07e208200f355f1f6c97cfb7c5cd562f6758dbf172acef96 

7、calico網路外掛部署

官方指導文件

安裝calico自定義資源

# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/tigera-operator.yaml

下載calico基礎配置

# wget https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/custom-resources.yaml

# vim custom-resources.yaml
修改pod地址段，和初始化保持一致
cidr: 172.16.0.0/16

# watch kubectl get pods -n calico-system

calico相關pod都正常running後，叢集就正常了。