[Kubernetes]node節點pod無法啟動/節點刪除網路重置

node1之前反覆新增過,新增之前需要清除下網路

root@master1:/var/lib/kubelet# kubectl get pod -o wide
NAME                   READY     STATUS              RESTARTS   AGE       IP           NODE
nginx-8586cf59-6zw9k   1/1       Running             0          9m        10.244.3.3   node2
nginx-8586cf59-jk5pc   0/1       ContainerCreating   0          9m        <none>       node1
nginx-8586cf59-vm9h4   0/1       ContainerCreating   0          9m        <none>       node1
nginx-8586cf59-zjb84   1/1       Running             0          9m        10.244.3.2   node2

root@node1:~# journalctl -u kubelet
 failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "nginx-8586cf59-rm4sh_default" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.2.1/24
12252 cni.go:227] Error while adding to cni network: failed to set bridge addr: "cni0" already

排查節點日誌：journalctl  -u kubelet

Nov 15 09:03:15 mvxl0851 kubelet[783]: E1115 09:03:15.577421     783 cni.go:259] Error adding network: failed to set bridge addr: "cni0" already has an IP address different from 10.16.2.1/24
Nov 15 09:03:15 mvxl0851 kubelet[783]: E1115 09:03:15.577465     783 cni.go:227] Error while adding to cni network: failed to set bridge addr: "cni0" already has an IP address different from 10.16.2.1/24
Nov 15 09:03:15 mvxl0851 kubelet[783]: E1115 09:03:15.652288     783 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "htt
Nov 15 09:03:15 mvxl0851 kubelet[783]: E1115 09:03:15.652386     783 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-0050568a39f2)" failed: rpc er
Nov 15 09:03:15 mvxl0851 kubelet[783]: E1115 09:03:15.652411     783 kuberuntime_manager.go:647] createPodSandbox for pod "httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-0050568a39f2)" failed: rpc e
Nov 15 09:03:15 mvxl0851 kubelet[783]: E1115 09:03:15.652516     783 pod_workers.go:186] Error syncing pod 8a6c7915-e871-11e8-af2a-0050568a39f2 ("httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-00505
Nov 15 09:03:16 mvxl0851 kubelet[783]: W1115 09:03:16.266688     783 pod_container_deletor.go:77] Container "2af4863fef146c5be51a704837c7943c3beab015fbd5cc2c705caee7216f5a6f" not found in pod's containers
Nov 15 09:03:18 mvxl0851 kubelet[783]: E1115 09:03:18.147904     783 cni.go:259] Error adding network: failed to set bridge addr: "cni0" already has an IP address different from 10.16.2.1/24
Nov 15 09:03:18 mvxl0851 kubelet[783]: E1115 09:03:18.147970     783 cni.go:227] Error while adding to cni network: failed to set bridge addr: "cni0" already has an IP address different from 10.16.2.1/24
Nov 15 09:03:18 mvxl0851 kubelet[783]: E1115 09:03:18.202769     783 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "htt
Nov 15 09:03:18 mvxl0851 kubelet[783]: E1115 09:03:18.202868     783 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-0050568a39f2)" failed: rpc er
Nov 15 09:03:18 mvxl0851 kubelet[783]: E1115 09:03:18.202896     783 kuberuntime_manager.go:647] createPodSandbox for pod "httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-0050568a39f2)" failed: rpc e
Nov 15 09:03:18 mvxl0851 kubelet[783]: E1115 09:03:18.203047     783 pod_workers.go:186] Error syncing pod 8a6c7915-e871-11e8-af2a-0050568a39f2 ("httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-00505
Nov 15 09:03:18 mvxl0851 kubelet[783]: W1115 09:03:18.332581     783 pod_container_deletor.go:77] Container "780e7ab6407c660cb6e4b4b0696fb24a9ba3b605b0c5e005ea4f51679f7eaf28" not found in pod's containers
Nov 15 09:03:19 mvxl0851 kubelet[783]: I1115 09:03:19.301582     783 reconciler.go:191] operationExecutor.UnmountVolume started for volume "default-token-qwrr2" (UniqueName: "kubernetes.io/secret/8a6c7915-e871-
Nov 15 09:03:19 mvxl0851 kubelet[783]: I1115 09:03:19.305596     783 operation_generator.go:633] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6c7915-e871-11e8-af2a-0050568a39f2-default-to
Nov 15 09:03:19 mvxl0851 kubelet[783]: I1115 09:03:19.402052     783 reconciler.go:297] Volume detached for volume "default-token-qwrr2" (UniqueName: "kubernetes.io/secret/8a6c7915-e871-11e8-af2a-0050568a39f2-d
Nov 15 09:03:20 mvxl0851 kubelet[783]: E1115 09:03:20.188278     783 cni.go:259] Error adding network: failed to set bridge addr: "cni0" already has an IP address different from 10.16.2.1/24
Nov 15 09:03:20 mvxl0851 kubelet[783]: E1115 09:03:20.188325     783 cni.go:227] Error while adding to cni network: failed to set bridge addr: "cni0" already has an IP address different from 10.16.2.1/24
Nov 15 09:03:20 mvxl0851 kubelet[783]: E1115 09:03:20.254502     783 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "htt
Nov 15 09:03:20 mvxl0851 kubelet[783]: E1115 09:03:20.254585     783 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-0050568a39f2)" failed: rpc er
Nov 15 09:03:20 mvxl0851 kubelet[783]: E1115 09:03:20.254612     783 kuberuntime_manager.go:647] createPodSandbox for pod "httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-0050568a39f2)" failed: rpc e
Nov 15 09:03:20 mvxl0851 kubelet[783]: E1115 09:03:20.254712     783 pod_workers.go:186] Error syncing pod 8a6c7915-e871-11e8-af2a-0050568a39f2 ("httpd-app-5fbccd7c6c-khnj2_default(8a6c7915-e871-11e8-af2a-00505
Nov 15 09:03:20 mvxl0851 kubelet[783]: W1115 09:03:20.410453     783 pod_container_deletor.go:77] Container "73e22f0f076e0472b31cfc83ed6f855d50048993e8bbd8508d2886ad3ea45b48" not found in pod's containers
Nov 15 09:03:27 mvxl0851 kubelet[783]: E1115 09:03:27.860718     783 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring

重點關注：

cni.go:259] Error adding network: failed to set bridge addr: "cni0" already has an IP address different from 10.16.2.1/24

在Node上執行如下操作：

重置kubernetes服務，重置網路。刪除網路配置，link

kubeadm reset
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
systemctl start docker

獲取master的join token

kubeadm token create --print-join-command

加入節點

kubeadm join --token 55c2c6.2a4bde1bc73a6562 192.168.1.144:6443 --discovery-token-ca-cert-hash sha256:0fdf8cfc6fecc18fded38649a4d9a81d043bf0e4bf57341239250dcc62d2c832