Kubernetes叢集證書過期解決方案：使用kubeadm為證書續期

人生的哲理發表於2024-11-01

一.系統環境
二.前言
三.Kubernetes證書過期及續期簡介
四.使用kubeadm為Kubernetes叢集證書續期
- 4.1 檢視k8s叢集證書過期時間
- 4.2 為master節點續期證書
- 4.3 為worker節點替換最新的證書
五.總結

一.系統環境

本文主要基於Kubernetes1.22.2和Linux作業系統Ubuntu 18.04。

伺服器版本	docker軟體版本	Kubernetes(k8s)叢集版本	CPU架構
Ubuntu 18.04.5 LTS	Docker version 20.10.14	v1.22.2	x86_64

Kubernetes叢集架構：k8scludes1作為master節點，k8scludes2，k8scludes3作為worker節點。

伺服器	作業系統版本	CPU架構	程序	功能描述
k8scludes1/192.168.110.128	Ubuntu 18.04.5 LTS	x86_64	docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico	k8s master節點
k8scludes2/192.168.110.129	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節點
k8scludes3/192.168.110.130	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節點

二.前言

Kubernetes叢集的證書對於叢集的安全性和穩定性至關重要。然而，隨著時間的推移，這些證書會過期，導致叢集服務不可用。本文將詳細介紹如何使用kubeadm工具為Kubernetes叢集續期證書。

給Kubernetes叢集證書續期的前提是已經有一套可以正常執行的Kubernetes叢集，關於Kubernetes(k8s)叢集的安裝部署，可以檢視部落格《Ubuntu 安裝部署Kubernetes(k8s)叢集》https://www.cnblogs.com/renshengdezheli/p/17632858.html，如果你的作業系統是RHEL的，可以檢視部落格《Centos7 安裝部署Kubernetes(k8s)叢集》https://www.cnblogs.com/renshengdezheli/p/16686769.html。

三.Kubernetes證書過期及續期簡介

Kubernetes叢集在初始化時，會自動生成一系列證書，包括API伺服器證書、CA證書、Kubelet證書等。這些證書通常有1年的有效期。當證書過期後，Kubernetes叢集的某些服務可能會受到影響，例如API伺服器無法訪問。為了解決證書過期的問題，我們可以使用kubeadm工具進行證書續期。

四.使用kubeadm為Kubernetes叢集證書續期

4.1 檢視k8s叢集證書過期時間

現在k8s叢集已經不能正常執行了，查詢pod報錯，可以看到報錯資訊為：“連線API伺服器拒絕”。

root@k8scludes1:~# kubectl get pod -o wie
The connection to the server 192.168.110.128:6443 was refused - did you specify the right host or port?

master節點的/etc/kubernetes/pki/目錄下存的是各個元件的證書。

root@k8scludes1:~# ls /etc/kubernetes/pki/
apiserver.crt              apiserver-etcd-client.key  apiserver-kubelet-client.crt  ca.crt  ca.srl  front-proxy-ca.crt  front-proxy-ca.srl      front-proxy-client.key  sa.key
apiserver-etcd-client.crt  apiserver.key              apiserver-kubelet-client.key  ca.key  etcd    front-proxy-ca.key  front-proxy-client.crt  mytok.csv               sa.pub

檢視master節點的apiserver證書有效期，可以看到證書在2023年4月16號就過期了，證書已經過期一年多了。

root@k8scludes1:~# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep Not
            Not Before: Apr 16 14:57:44 2022 GMT
            Not After : Apr 16 14:57:44 2023 GMT

在master節點檢視各個元件的證書過期時間。

root@k8scludes1:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 21, 2023 14:25 UTC   9y                                      no      
apiserver                  Oct 21, 2023 14:25 UTC   9y              ca                      no      
apiserver-etcd-client      Oct 21, 2023 14:25 UTC   9y              etcd-ca                 no      
apiserver-kubelet-client   Oct 21, 2023 14:25 UTC   9y              ca                      no      
controller-manager.conf    Oct 21, 2023 14:25 UTC   9y                                      no      
etcd-healthcheck-client    Oct 21, 2023 14:25 UTC   9y              etcd-ca                 no      
etcd-peer                  Oct 21, 2023 14:25 UTC   9y              etcd-ca                 no      
etcd-server                Oct 21, 2023 14:25 UTC   9y              etcd-ca                 no      
front-proxy-client         Oct 21, 2023 14:25 UTC   9y              front-proxy-ca          no      
scheduler.conf             Oct 21, 2023 14:25 UTC   9y                                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 13, 2023 14:57 UTC   7y              no      
etcd-ca                 Apr 13, 2023 14:57 UTC   7y              no      
front-proxy-ca          Apr 13, 2023 14:57 UTC   7y              no

檢視master節點的kubelet證書過期時間。

root@k8scludes1:~# ls /var/lib/kubelet/pki/
kubelet-client-2022-04-16-22-57-47.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key

root@k8scludes1:~# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text  |grep Not
            Not Before: Apr 16 14:57:44 2022 GMT
            Not After : Apr 16 14:57:46 2023 GMT

4.2 為master節點續期證書

在master節點給各個元件續簽證書。

root@k8scludes1:~# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

在master節點再次檢視各個元件的證書過期時間，可以看到證書續簽了一年。

root@k8scludes1:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 24, 2025 02:53 UTC   364d                                    no      
apiserver                  Oct 24, 2025 02:53 UTC   364d            ca                      no      
apiserver-etcd-client      Oct 24, 2025 02:53 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Oct 24, 2025 02:53 UTC   364d            ca                      no      
controller-manager.conf    Oct 24, 2025 02:53 UTC   364d                                    no      
etcd-healthcheck-client    Oct 24, 2025 02:53 UTC   364d            etcd-ca                 no      
etcd-peer                  Oct 24, 2025 02:53 UTC   364d            etcd-ca                 no      
etcd-server                Oct 24, 2025 02:53 UTC   364d            etcd-ca                 no      
front-proxy-client         Oct 24, 2025 02:53 UTC   364d            front-proxy-ca          no      
scheduler.conf             Oct 24, 2025 02:53 UTC   364d                                    no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 13, 2025 14:57 UTC   7y              no      
etcd-ca                 Apr 13, 2025 14:57 UTC   7y              no      
front-proxy-ca          Apr 13, 2025 14:57 UTC   7y              no

當前kubernetes各個元件所使用的kubecong檔案都在/etc/kubernetes/裡。

root@k8scludes1:~# ls /etc/kubernetes/
admin.conf  admission-control-config-file  audit  controller-manager.conf  kubelet.conf  manifests  pki  scheduler.conf

檔案字尾為conf的都是各個元件所需的kubeconfig檔案，但是這些檔案裡使用的證書都是之前過期的證書，需要把conf檔案刪除並重新生成。

root@k8scludes1:~# ls /etc/kubernetes/*.conf
/etc/kubernetes/admin.conf  /etc/kubernetes/controller-manager.conf  /etc/kubernetes/kubelet.conf  /etc/kubernetes/scheduler.conf

root@k8scludes1:~# mkdir k8sconf_bak

root@k8scludes1:~# cp /etc/kubernetes/*.conf k8sconf_bak/

root@k8scludes1:~# ls k8sconf_bak/
admin.conf  controller-manager.conf  kubelet.conf  scheduler.conf

root@k8scludes1:~# rm -rf /etc/kubernetes/*.conf

root@k8scludes1:~# ls /etc/kubernetes/
admission-control-config-file  audit  manifests  pki

為k8s的各個元件重新生成kubeconfig檔案。

root@k8scludes1:~# kubeadm init --kubernetes-version=v1.22.2  phase kubeconfig all
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file

root@k8scludes1:~# ls /etc/kubernetes/
admin.conf  admission-control-config-file  audit  controller-manager.conf  kubelet.conf  manifests  pki  scheduler.conf

替換管理員所用的kubeconfig檔案。

root@k8scludes1:~# ls ~/.kube/config
/root/.kube/config

root@k8scludes1:~# rm -rf ~/.kube/config

root@k8scludes1:~# ls ~/.kube/
cache  config.old-20241023  kubens

root@k8scludes1:~# cp /etc/kubernetes/admin.conf ~/.kube/config

root@k8scludes1:~# ls ~/.kube/config
/root/.kube/config

重啟kube-scheduler。

root@k8scludes1:~# docker ps | grep kube-scheduler
47ac8592cf5c   b51ddc1014b0                                        "kube-scheduler --au…"   6 minutes ago   Up 6 minutes             k8s_kube-scheduler_kube-scheduler-k8scludes1_kube-system_f637e8449089a70204a39d176f936bc7_289
6e65a5b16329   registry.aliyuncs.com/google_containers/pause:3.5   "/pause"                 6 minutes ago   Up 6 minutes             k8s_POD_kube-scheduler-k8scludes1_kube-system_f637e8449089a70204a39d176f936bc7_75

root@k8scludes1:~# docker ps | awk '/kube-scheduler /{print $1}'
47ac8592cf5c

root@k8scludes1:~# docker rm -f $(docker ps | awk '/kube-scheduler /{print $1}')
47ac8592cf5c

root@k8scludes1:~# kubectl get pods -n kube-system | grep scheduler
kube-scheduler-k8scludes1                  1/1     Running   289 (2y120d ago)   2y191d

檢視master節點的kubelet當前使用的證書，kubelet-client-current.pem軟連結到了kubelet-client-2024-10-24-11-08-14.pem，說明現在kubelet使用的是最新的證書。

root@k8scludes1:~# ls /var/lib/kubelet/pki/
kubelet-client-2022-04-16-22-57-47.pem  kubelet-client-2024-10-24-11-05-29.pem  kubelet-client-2024-10-24-11-08-14.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key

root@k8scludes1:~# ls /var/lib/kubelet/pki/kubelet-client-current.pem -l
lrwxrwxrwx 1 root root 59 Oct 24 11:08 /var/lib/kubelet/pki/kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2024-10-24-11-08-14.pem

如果kubelet-client-current.pem軟連結到kubelet-client-2022-04-16-22-57-47.pem，說明kubelet使用的是舊的證書，重啟kebelet即可。

root@k8scludes1:~# systemctl restart kubelet

root@k8scludes1:~# ls /var/lib/kubelet/pki/
kubelet-client-2022-04-16-22-57-47.pem  kubelet-client-2024-10-24-11-05-29.pem  kubelet-client-2024-10-24-11-08-14.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key

root@k8scludes1:~# ls -l /var/lib/kubelet/pki/kubelet-client-current.pem 
lrwxrwxrwx 1 root root 59 Oct 24 11:08 /var/lib/kubelet/pki/kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2024-10-24-11-08-14.pem

在master節點上檢視證書籤名請求（簡稱為CSR），如果CONDITION顯示的是Approved,Issued，說明證書籤名請求CSR已經被批准，則不需要執行kubectl certificate approve csr-htp29，如果CONDITION顯示的是Pending，則需要手動批准證書籤名請求CSR，語法為：kubectl certificate approve CSR名。

root@k8scludes1:~# kubectl get csr
NAME        AGE   SIGNERNAME                                    REQUESTOR                REQUESTEDDURATION   CONDITION
csr-htp29   12m   kubernetes.io/kube-apiserver-client-kubelet   system:node:k8scludes1   <none>              Approved,Issued

#批准證書籤名請求CSR
root@k8scludes1:~# kubectl certificate approve csr-htp29
certificatesigningrequest.certificates.k8s.io/csr-rn8xc approved

在master節點檢視kubelet的證書過期時間，到2025年才過期了。

root@k8scludes1:~# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text  |grep Not
            Not Before: Oct 24 03:03:14 2024 GMT
            Not After : Oct 24 03:03:14 2025 GMT

檢視k8s叢集狀態，可以發現k8scludes1節點已經正常了，但是兩個worker節點還是不正常，原因是兩個worker節點的證書還是舊的，需要替換為最新的證書。

root@k8scludes1:~# kubectl get node
NAME         STATUS     ROLES                  AGE      VERSION
k8scludes1   Ready      control-plane,master   2y191d   v1.22.2
k8scludes2   NotReady   <none>                 2y191d   v1.22.2
k8scludes3   NotReady   <none>                 2y191d   v1.22.2

4.3 為worker節點替換最新的證書

k8scludes2節點的kubelet使用的還是舊的證書。

root@k8scludes2:~# ls /var/lib/kubelet/pki/
kubelet-client-2022-04-17-01-59-26.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key

root@k8scludes2:~# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text  |grep Not
            Not Before: Apr 16 17:54:26 2022 GMT
            Not After : Apr 16 17:54:26 2023 GMT

在master節點生成k8scludes2節點所需的kubelet.conf檔案，把kubelet.conf檔案放在/tmp/目錄。

root@k8scludes1:~# kubeadm init --kubernetes-version=v1.22.2 phase kubeconfig kubelet --node-name k8scludes2 --kubeconfig-dir /tmp/
[kubeconfig] Writing "kubelet.conf" kubeconfig file
 
root@k8scludes1:~# ls /tmp/
kubelet.conf  systemd-private-3e6f81ffe01748ec8909700ec12195cb-systemd-resolved.service-kzxcft  systemd-private-3e6f81ffe01748ec8909700ec12195cb-systemd-timesyncd.service-Uop8xG  vmware-root_751-4290559920

複製檔案到k8scludes2節點的/etc/kubernetes/目錄下。

root@k8scludes1:~# scp /tmp/kubelet.conf 192.168.110.129:/etc/kubernetes/
root@192.168.110.129's password: 
kubelet.conf                                                                                                                                                                   100% 5671     1.4MB/s   00:00

k8scludes2節點重啟kubelet。

root@k8scludes2:~# systemctl restart kubelet

再次檢視k8scludes2節點的kubelet證書，現在已經是最新的證書了。

root@k8scludes2:~# ls /var/lib/kubelet/pki/
kubelet-client-2022-04-17-01-59-26.pem  kubelet-client-2024-10-24-11-29-31.pem  kubelet-client-2024-10-24-11-29-40.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key

root@k8scludes2:~# ls -l /var/lib/kubelet/pki/kubelet-client-current.pem 
lrwxrwxrwx 1 root root 59 Oct 24 11:29 /var/lib/kubelet/pki/kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2024-10-24-11-29-40.pem

root@k8scludes2:~# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text  |grep Not
            Not Before: Oct 24 03:24:40 2024 GMT
            Not After : Oct 24 03:24:40 2025 GMT

k8scludes3節點也是類似的操作。

在master節點生成k8scludes3節點所需的kubelet.conf檔案，把kubelet.conf檔案放在/tmp/目錄。

root@k8scludes1:~# rm -rf /tmp/* ;  ls /tmp/

root@k8scludes1:~# kubeadm init --kubernetes-version=v1.22.2 phase kubeconfig kubelet --node-name k8scludes3 --kubeconfig-dir /tmp/
[kubeconfig] Writing "kubelet.conf" kubeconfig file

複製kubelet.conf到k8scludes3節點的/etc/kubernetes/目錄下

root@k8scludes1:~# scp /tmp/kubelet.conf 192.168.110.130:/etc/kubernetes/
root@192.168.110.130's password: 
kubelet.conf                                                                                                                                                                   100% 5671     2.6MB/s   00:00

重啟k8scludes3節點的kubelet，現在kubelet使用的是最新的證書了。

root@k8scludes3:~# systemctl restart kubelet

root@k8scludes3:~# ls /var/lib/kubelet/pki/
kubelet-client-2022-04-17-01-59-29.pem  kubelet-client-2024-10-24-11-34-49.pem  kubelet-client-2024-10-24-11-34-57.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key

root@k8scludes3:~# ls -l /var/lib/kubelet/pki/kubelet-client-current.pem 
lrwxrwxrwx 1 root root 59 Oct 24 11:34 /var/lib/kubelet/pki/kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2024-10-24-11-34-57.pem

root@k8scludes3:~# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text  |grep Not
            Not Before: Oct 24 03:29:57 2024 GMT
            Not After : Oct 24 03:29:57 2025 GMT

檢視k8s叢集狀態，現在叢集恢復正常了。

root@k8scludes1:~# kubectl get node
NAME         STATUS   ROLES                  AGE      VERSION
k8scludes1   Ready    control-plane,master   2y191d   v1.22.2
k8scludes2   Ready    <none>                 2y191d   v1.22.2
k8scludes3   Ready    <none>                 2y191d   v1.22.2

五.總結

本文介紹瞭如何使用kubeadm工具為Kubernetes叢集續期證書。透過定期檢查和續期證書，可以確保Kubernetes叢集的安全性和穩定性。

在進行證書續期時，請確保有足夠的磁碟空間來儲存新的證書檔案；
如果你的叢集配置了自動化工具或指令碼來管理Kubernetes叢集，確保這些工具和指令碼也更新為處理新的證書；
在生產環境中，建議設定證書到期提醒，以便提前進行維護工作。

監控Kubernetes叢集證書過期時間的三種方案
2022-12-08
7.Kubernetes 證書續期
2024-08-01
kubernetes證書過期處理
2022-07-26
SSL證書為什麼會過期？證書過期會有哪些影響？
2022-12-01
rancher證書過期
2020-05-27
SSL證書會不會過期？域名SSL證書過期了怎麼辦？
2021-08-04
如果網站ssl證書過期怎麼解決
2020-09-22
網站
Kubeadm證書過期問題修復方法之一：通過修改kubeadm原始碼
2019-02-25
原始碼
Mac安裝homebrew證書過期
2022-04-04
Mac
kubernetes叢集證書期限修改（三）
2024-07-02
SSL證書為什麼會有有效期？如何有效避免SSL證書過期？
2021-11-10
Kubernetes-高可用叢集證書更新
2024-09-20
GlobalSign證書過期不續費還可繼續訪問嗎
2022-06-13
npm ERR! code CERT_HAS_EXPIRED：解決證書過期問題
2024-09-07
NPM
東半球最好用的SSL證書續期工具
2024-09-27
harbor映象倉庫證書過期問題
2020-09-23
SSL證書過期怎麼辦？只需幾步就能輕鬆解決！
2021-08-31
win11部分功能將因為證書過期無法使用，好在微軟已著手解決
2021-12-15
微軟
如何快速給你的微軟考試證書續期？
2022-03-13
微軟
acme+cloudflare生成免費證書（自動續期）
2024-09-23
ACMCloud
網站安裝SSL證書就安全了嗎？不，SSL證書會過期！
2021-07-12
網站
VMware許可證過期解決
2018-02-02
win10系統下安全證書過期如何解決
2018-10-30
Win10
關於IOS開發者證書過期的問題
2017-12-19
iOS
win10證書即將過期如何解決_win10許可證即將過期的處理方法
2020-07-17
Win10
附025.kubeadm部署Kubernetes更新證書
2020-07-24
【VMware vCenter】vCenter Server 各版本證書有效期及過期解決辦法。
2024-09-14
Server
SSL 證書過期事件頻發，切忌不要因小失大！
2020-08-13
事件
Kubelet證書自動續簽（為kubelet配置證書輪換）
2024-09-05
如何下載電子版證書以及如何申請紙質證書（申請紙質證書的入口已過期）
2013-11-25
K8s叢集證書更新
2024-05-31
K8S
中級通訊工程師證書會過期嗎
2020-07-28
工程師
10 刪除個人儲存中過期的證書薦
2013-06-25
證書過期?私鑰洩露？原來，企業證書管理不當竟有這麼多安全風險！
2020-03-23
SSL證書過期後，網站還能正常訪問嗎？
2022-03-23
網站
根證書過期問題大嗎？有什麼影響
2023-03-01
WAS證書過期替換之獨立WAS Server之文字操作版
2010-12-31
Server
Tailscale 的 TLS 證書過期，網站掛了 90 分鐘！
2024-04-09
AITLS網站