本文分享自華為雲社群《K8s 映象快取管理 kube-fledged 認知》，作者： 山河已無恙。
我們知道 k8s 上的容器排程需要在排程的節點行拉取當前容器的映象，在一些特殊場景中，
-
需要
快速啟動和/或擴充套件的應用程式。例如，由於資料量激增，執行實時資料處理的應用程式需要快速擴充套件。 -
映象比較龐大，涉及多個版本，節點儲存有限，需要動態清理不需要的映象
-
無伺服器函式通常需要在幾分之一秒內立即對傳入事件和啟動容器做出反應。 -
在邊緣裝置上執行的
IoT 應用程式，需要容忍邊緣裝置和映象映象倉庫之間的間歇性網路連線。 -
如果需要從
專用倉庫中拉取映象，並且無法授予每個人從此映象倉庫拉取映象的訪問許可權，則可以在群集的節點上提供映象。 -
如果叢集管理員或操作員需要對應用程式進行升級，並希望事先驗證是否可以成功拉取新映象。
kube-fledged 是一個 kubernetes operator，用於直接在 Kubernetes 叢集的 worker 節點上建立和管理容器映象快取。它允許使用者定義映象列表以及這些映象應快取到哪些工作節點上（即拉取）。因此，應用程式 Pod 幾乎可以立即啟動，因為不需要從映象倉庫中提取映象。
kube-fledged 提供了 CRUD API 來管理映象快取的生命週期，並支援多個可配置的引數，可以根據自己的需要自定義功能。
Kubernetes 具有內建的映象垃圾回收機制。節點中的 kubelet 會定期檢查磁碟使用率是否達到特定閾值（可透過標誌進行配置）。一旦達到這個閾值，kubelet 會自動刪除節點中所有未使用的映象。
需要在建議的解決方案中實現自動和定期重新整理機制。如果映象快取中的映象被 kubelet 的 gc 刪除，下一個重新整理週期會將已刪除的映象拉入映象快取中。這可確保映象快取是最新的。
設計流程
https://github.com/senthilrch/kube-fledged/blob/master/docs/kubefledged-architecture.png
部署 kube-fledged
Helm 方式部署
──[root@vms100.liruilongs.github.io]-[~/ansible] └─$mkdir kube-fledged ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$cd kube-fledged ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$export KUBEFLEDGED_NAMESPACE=kube-fledged ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$kubectl create namespace ${KUBEFLEDGED_NAMESPACE} namespace/kube-fledged created ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/ "kubefledged-charts" has been added to your repositories ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "kubefledged-charts" chart repository ...Successfully got an update from the "kubescape" chart repository ...Successfully got an update from the "rancher-stable" chart repository ...Successfully got an update from the "skm" chart repository ...Successfully got an update from the "openkruise" chart repository ...Successfully got an update from the "awx-operator" chart repository ...Successfully got an update from the "botkube" chart repository Update Complete. ⎈Happy Helming!⎈ ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$helm install --verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait
實際部署中發現，由於網路問題，chart 無法下載，所以透過 make deploy-using-yaml 使用 yaml 方式部署
Yaml 檔案部署
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$git clone https://github.com/senthilrch/kube-fledged.git 正克隆到 'kube-fledged'... remote: Enumerating objects: 10613, done. remote: Counting objects: 100% (1501/1501), done. remote: Compressing objects: 100% (629/629), done. remote: Total 10613 (delta 845), reused 1357 (delta 766), pack-reused 9112 接收物件中: 100% (10613/10613), 34.58 MiB | 7.33 MiB/s, done. 處理 delta 中: 100% (4431/4431), done. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$ls kube-fledged ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$cd kube-fledged/ ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged] └─$make deploy-using-yaml kubectl apply -f deploy/kubefledged-namespace.yaml
第一次部署，發現映象拉不下來
┌──[root@vms100.liruilongs.github.io]-[~] └─$kubectl get all -n kube-fledged NAME READY STATUS RESTARTS AGE pod/kube-fledged-controller-df69f6565-drrqg 0/1 CrashLoopBackOff 35 (5h59m ago) 21h pod/kube-fledged-webhook-server-7bcd589bc4-b7kg2 0/1 Init:CrashLoopBackOff 35 (5h58m ago) 21h pod/kubefledged-controller-55f848cc67-7f4rl 1/1 Running 0 21h pod/kubefledged-webhook-server-597dbf4ff5-l8fbh 0/1 Init:CrashLoopBackOff 34 (6h ago) 21h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-fledged-webhook-server ClusterIP 10.100.194.199 <none> 3443/TCP 21h service/kubefledged-webhook-server ClusterIP 10.101.191.206 <none> 3443/TCP 21h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kube-fledged-controller 0/1 1 0 21h deployment.apps/kube-fledged-webhook-server 0/1 1 0 21h deployment.apps/kubefledged-controller 0/1 1 0 21h deployment.apps/kubefledged-webhook-server 0/1 1 0 21h NAME DESIRED CURRENT READY AGE replicaset.apps/kube-fledged-controller-df69f6565 1 1 0 21h replicaset.apps/kube-fledged-webhook-server-7bcd589bc4 1 1 0 21h replicaset.apps/kubefledged-controller-55f848cc67 1 1 0 21h replicaset.apps/kubefledged-webhook-server-597dbf4ff5 1 1 0 21h ┌──[root@vms100.liruilongs.github.io]-[~] └─$
這裡我們找一下要拉取的映象
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat *.yaml | grep image: - image: senthilrch/kubefledged-controller:v0.10.0 - image: senthilrch/kubefledged-webhook-server:v0.10.0 - image: senthilrch/kubefledged-webhook-server:v0.10.0
單獨拉取一些，當前使用 ansible 在所有工作節點批次操作
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible k8s_node -m shell -a "docker pull docker.io/senthilrch/kubefledged-cri-client:v0.10.0" -i host.yaml
其他相關的映象都拉取一下
操作完成之後容器狀態全部正常
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl -n kube-fledged get all NAME READY STATUS RESTARTS AGE pod/kube-fledged-controller-df69f6565-wdb4g 1/1 Running 0 13h pod/kube-fledged-webhook-server-7bcd589bc4-j8xxp 1/1 Running 0 13h pod/kubefledged-controller-55f848cc67-klxlm 1/1 Running 0 13h pod/kubefledged-webhook-server-597dbf4ff5-ktbsh 1/1 Running 0 13h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kube-fledged-webhook-server ClusterIP 10.100.194.199 <none> 3443/TCP 36h service/kubefledged-webhook-server ClusterIP 10.101.191.206 <none> 3443/TCP 36h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kube-fledged-controller 1/1 1 1 36h deployment.apps/kube-fledged-webhook-server 1/1 1 1 36h deployment.apps/kubefledged-controller 1/1 1 1 36h deployment.apps/kubefledged-webhook-server 1/1 1 1 36h NAME DESIRED CURRENT READY AGE replicaset.apps/kube-fledged-controller-df69f6565 1 1 1 36h replicaset.apps/kube-fledged-webhook-server-7bcd589bc4 1 1 1 36h replicaset.apps/kubefledged-controller-55f848cc67 1 1 1 36h replicaset.apps/kubefledged-webhook-server-597dbf4ff5 1 1 1 36h
驗證是否安裝成功
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged] └─$kubectl get pods -n kube-fledged -l app=kubefledged NAME READY STATUS RESTARTS AGE kubefledged-controller-55f848cc67-klxlm 1/1 Running 0 16h kubefledged-webhook-server-597dbf4ff5-ktbsh 1/1 Running 0 16h ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged] └─$kubectl get imagecaches -n kube-fledged No resources found in kube-fledged namespace.
使用 kubefledged
建立映象快取物件
根據 Demo 檔案，建立映象快取物件
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged] └─$cd deploy/ ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat kubefledged-imagecache.yaml --- apiVersion: kubefledged.io/v1alpha2 kind: ImageCache metadata: # Name of the image cache. A cluster can have multiple image cache objects name: imagecache1 namespace: kube-fledged # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference labels: app: kubefledged kubefledged: imagecache spec: # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled). cacheSpec: # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster - images: - ghcr.io/jitesoft/nginx:1.23.1 # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector - images: - us.gcr.io/k8s-artifacts-prod/cassandra:v7 - us.gcr.io/k8s-artifacts-prod/etcd:3.5.4-0 nodeSelector: tier: backend # Specifies a list of image pull secrets to pull images from private repositories into the cache imagePullSecrets: - name: myregistrykey
官方的 Demo 中對應的 映象拉取不下來，所以換一下
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$docker pull us.gcr.io/k8s-artifacts-prod/cassandra:v7 Error response from daemon: Get "https://us.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
為了測試選擇器標籤的使用,我們找一個節點的標籤單獨做映象快取
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl get nodes --show-labels
同時我們直接從公有倉庫拉取映象，所以不需要 imagePullSecrets 物件
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$vim kubefledged-imagecache.yaml
修改後的 yaml 檔案
-
新增了一個所有節點的 liruilong/my-busybox:latest 映象快取
-
新增了一個
kubernetes.io/hostname: vms105.liruilongs.github.io對應標籤選擇器的liruilong/hikvision-sdk-config-ftp:latest映象快取
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat kubefledged-imagecache.yaml --- apiVersion: kubefledged.io/v1alpha2 kind: ImageCache metadata: # Name of the image cache. A cluster can have multiple image cache objects name: imagecache1 namespace: kube-fledged # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference labels: app: kubefledged kubefledged: imagecache spec: # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled). cacheSpec: # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster - images: - liruilong/my-busybox:latest # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector - images: - liruilong/hikvision-sdk-config-ftp:latest nodeSelector: kubernetes.io/hostname: vms105.liruilongs.github.io # Specifies a list of image pull secrets to pull images from private repositories into the cache #imagePullSecrets: #- name: myregistrykey ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
直接建立報錯了
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl create -f kubefledged-imagecache.yaml Error from server (InternalError): error when creating "kubefledged-imagecache.yaml": Internal error occurred: failed calling webhook "validate-image-cache.kubefledged.io": failed to call webhook: Post "https://kubefledged-webhook-server.kube-fledged.svc:3443/validate-image-cache?timeout=1s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefledged.io") ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl get imagecaches -n kube-fledged No resources found in kube-fledged namespace. ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
解決辦法，刪除對應的物件，重新建立
我在當前專案的一個 issues 下面找到了解決辦法 https://github.com/senthilrch/kube-fledged/issues/76
看起來這是因為 Webhook CA 是硬編碼的，但是當 webhook 伺服器啟動時，會生成一個新的 CA 捆綁包並更新 webhook 配置。當發生另一個部署時，將重新應用原始 CA 捆綁包，並且 Webhook 請求開始失敗，直到再次重新啟動 Webhook 元件以修補捆綁包init-server
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged] └─$make remove-kubefledged-and-operator # Remove kubefledged kubectl delete -f deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml error: resource mapping not found for name: "kube-fledged" namespace: "kube-fledged" from "deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml": no matches for kind "KubeFledged" in version "charts.helm.kubefledged.io/v1alpha2" ensure CRDs are installed first ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged] └─$make deploy-using-yaml kubectl apply -f deploy/kubefledged-namespace.yaml namespace/kube-fledged created kubectl apply -f deploy/kubefledged-crd.yaml customresourcedefinition.apiextensions.k8s.io/imagecaches.kubefledged.io unchanged .................... kubectl rollout status deployment kubefledged-webhook-server -n kube-fledged --watch Waiting for deployment "kubefledged-webhook-server" rollout to finish: 0 of 1 updated replicas are available... deployment "kubefledged-webhook-server" successfully rolled out kubectl get pods -n kube-fledged NAME READY STATUS RESTARTS AGE kubefledged-controller-55f848cc67-76c4v 1/1 Running 0 112s kubefledged-webhook-server-597dbf4ff5-56h6z 1/1 Running 0 66s
重新建立快取物件，建立成功
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl create -f kubefledged-imagecache.yaml imagecache.kubefledged.io/imagecache1 created ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl get imagecaches -n kube-fledged NAME AGE imagecache1 10s ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
檢視當前被納管的映象快取
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$kubectl get imagecaches imagecache1 -n kube-fledged -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 83, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20169836", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "liruilong/my-busybox:latest" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "completionTime": "2024-03-02T01:06:47Z", "message": "All requested images pulled succesfully to respective nodes", "reason": "ImageCacheRefresh", "startTime": "2024-03-02T01:05:33Z", "status": "Succeeded" } } ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged] └─$
透過 ansible 來驗證
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml 192.168.26.102 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.101 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.103 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.105 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.100 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.106 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/hikvision-sdk-config-ftp" -i host.yaml 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.105 | CHANGED | rc=0 >> liruilong/hikvision-sdk-config-ftp latest a02cd03b4342 4 months ago 830MB 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$
開啟自動重新整理
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache= imagecache.kubefledged.io/imagecache1 annotated ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$
新增映象快取
新增一個新的映象快取
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl get imagecaches.kubefledged.io -n kube-fledged imagecache1 -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 92, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20175233", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "liruilong/my-busybox:latest", "liruilong/jdk1.8_191:latest" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "completionTime": "2024-03-02T01:43:32Z", "message": "All requested images pulled succesfully to respective nodes", "reason": "ImageCacheUpdate", "startTime": "2024-03-02T01:40:34Z", "status": "Succeeded" } } ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$
透過 ansible 確認
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml 192.168.26.101 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.102 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.100 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.103 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.105 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB 192.168.26.106 | CHANGED | rc=0 >> liruilong/jdk1.8_191 latest 17dbd4002a8c 5 years ago 170MB ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$
刪除映象快取
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl edit imagecaches imagecache1 -n kube-fledged imagecache.kubefledged.io/imagecache1 edited ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl get imagecaches.kubefledged.io -n kube-fledged imagecache1 -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 94, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20175766", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "liruilong/jdk1.8_191:latest" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "message": "Image cache is being updated. Please view the status after some time", "reason": "ImageCacheUpdate", "startTime": "2024-03-02T01:48:03Z", "status": "Processing" } }
透過 Ansible 確認，可以看到無論是 mastere 上的節點還是 work 的節點，對應的映象快取都被清理
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml 192.168.26.102 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.101 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | CHANGED | rc=0 >> liruilong/my-busybox latest 497b83a63aad 11 months ago 1.24MB 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$
這裡需要注意如果清除所有的映象快取，那麼需要把 images 下的陣列 寫成 "".
┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl edit imagecaches imagecache1 -n kube-fledged imagecache.kubefledged.io/imagecache1 edited ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml 192.168.26.102 | FAILED | rc=1 >> non-zero return code 192.168.26.101 | FAILED | rc=1 >> non-zero return code 192.168.26.100 | FAILED | rc=1 >> non-zero return code 192.168.26.105 | FAILED | rc=1 >> non-zero return code 192.168.26.103 | FAILED | rc=1 >> non-zero return code 192.168.26.106 | FAILED | rc=1 >> non-zero return code ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubectl get imagecaches.kubefledged.io -n kube-fledged imagecache1 -o json { "apiVersion": "kubefledged.io/v1alpha2", "kind": "ImageCache", "metadata": { "creationTimestamp": "2024-03-01T15:08:42Z", "generation": 98, "labels": { "app": "kubefledged", "kubefledged": "imagecache" }, "name": "imagecache1", "namespace": "kube-fledged", "resourceVersion": "20176849", "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72" }, "spec": { "cacheSpec": [ { "images": [ "" ] }, { "images": [ "liruilong/hikvision-sdk-config-ftp:latest" ], "nodeSelector": { "kubernetes.io/hostname": "vms105.liruilongs.github.io" } } ] }, "status": { "completionTime": "2024-03-02T01:52:16Z", "message": "All cached images succesfully deleted from respective nodes", "reason": "ImageCacheUpdate", "startTime": "2024-03-02T01:51:47Z", "status": "Succeeded" } } ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$
如果透過下面的方式刪除，直接註釋調對應的標籤
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$cat kubefledged-imagecache.yaml --- apiVersion: kubefledged.io/v1alpha2 kind: ImageCache metadata: # Name of the image cache. A cluster can have multiple image cache objects name: imagecache1 namespace: kube-fledged # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference labels: app: kubefledged kubefledged: imagecache spec: # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled). cacheSpec: # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster #- images: #- liruilong/my-busybox:latest # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector - images: - liruilong/hikvision-sdk-config-ftp:latest nodeSelector: kubernetes.io/hostname: vms105.liruilongs.github.io # Specifies a list of image pull secrets to pull images from private repositories into the cache #imagePullSecrets: #- name: myregistrykey ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$
那麼會報下面的錯
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy] └─$kubectl edit imagecaches imagecache1 -n kube-fledged error: imagecaches.kubefledged.io "imagecache1" could not be patched: admission webhook "validate-image-cache.kubefledged.io" denied the request: Mismatch in no. of image lists You can run `kubectl replace -f /tmp/kubectl-edit-4113815075.yaml` to try this update again.
博文部分內容參考
© 文中涉及參考連結內容版權歸原作者所有，如有侵權請告知,如果你認可它不要吝嗇星星哦 :)
https://github.com/senthilrch/kube-fledged
點選關注，第一時間瞭解華為雲新鮮技術~