k8s裡node 當機後如何提高pod遷移速度

fengjian1585發表於2024-06-07
K8S

大概的配置引數：

node故障後，pod會遷移到 正常的node上， 遷移時間大概8分鐘左右， 如果是微服務，註冊到nacos，服務不受影響，但是對於其他的服務，
請求中會有大量失敗。

需要幾個流程：

kubelet 自身會定期更新狀態到 apiserver，透過kubelet的引數 node-status-update-frequency 配置上報頻率，預設 10s 上報一次。

kube-controller-manager定期去探測kubelet的執行狀態，預設5s問kubelet，使用--node-monitor-grace-period引數

kube-controller-manager詢問kubelet，如果5分鐘沒有回覆，認為kubelet有問題，node-monitor-grace-period，是kubelet引數node-status-update-frequency的整數倍

如果有問題，故障node節點被設定成汙點

  • node.kubernetes.io/unreachable:NoExecute
  • node.kubernetes.io/unreachable:NoSchedule

節點被設定了汙點，pod節點上還是running，apiserver的引數--default-unreachable-toleration-seconds是驅逐掉時間。

vim /var/lib/kubelet/config.yaml

memorySwap: {}
#修改kubelet向apiserver 彙報時間，由10秒變成4秒
node-status-update-frequency: 4s
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

vim /etc/kubernetes/manifests/kube-apiserver.yaml

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.148.131
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    #當node節點為notready或者 unreachable時，5秒鐘驅逐掉，預設300秒
    - --default-unreachable-toleration-seconds=5
    - --default-not-ready-toleration-seconds=5

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --use-service-account-credentials=true
    - --v=5
    # kube-controller-manager 修改成2秒主動探測，預設值5秒
    - --node-monitor-period=2s
    # 將一個node標記為不健康之前允許其無響應的上線
    - --node-monitor-grace-period=12s

驗證指令碼，中間關閉node， 檢視pod漂移時間

while true; do echo `date +%F-%T`; kubectl get node ;echo "####################################################################"; kubectl describe nodes node2 | grep -A5 Taint;echo "";echo "##############################################################"; kubectl get pod -o wide; sleep 5; done

注意：以前的pod需要重建。

相關文章