詳解K8s 映象快取管理kube-fledged

华为云开发者联盟發表於2024-04-15

本文分享自華為雲社群《K8s 映象快取管理 kube-fledged 認知》，作者：山河已無恙。

我們知道 k8s 上的容器排程需要在排程的節點行拉取當前容器的映象，在一些特殊場景中，

需要快速啟動和/或擴充套件的應用程式。例如，由於資料量激增，執行實時資料處理的應用程式需要快速擴充套件。
映象比較龐大，涉及多個版本，節點儲存有限，需要動態清理不需要的映象
無伺服器函式通常需要在幾分之一秒內立即對傳入事件和啟動容器做出反應。
在邊緣裝置上執行的 IoT 應用程式，需要容忍邊緣裝置和映象映象倉庫之間的間歇性網路連線。
如果需要從專用倉庫中拉取映象，並且無法授予每個人從此映象倉庫拉取映象的訪問許可權，則可以在群集的節點上提供映象。
如果叢集管理員或操作員需要對應用程式進行升級，並希望事先驗證是否可以成功拉取新映象。

kube-fledged 是一個 kubernetes operator，用於直接在 Kubernetes 叢集的 worker 節點上建立和管理容器映象快取。它允許使用者定義映象列表以及這些映象應快取到哪些工作節點上（即拉取）。因此，應用程式 Pod 幾乎可以立即啟動，因為不需要從映象倉庫中提取映象。

kube-fledged 提供了 CRUD API 來管理映象快取的生命週期，並支援多個可配置的引數，可以根據自己的需要自定義功能。

Kubernetes 具有內建的映象垃圾回收機制。節點中的 kubelet 會定期檢查磁碟使用率是否達到特定閾值（可透過標誌進行配置）。一旦達到這個閾值，kubelet 會自動刪除節點中所有未使用的映象。

需要在建議的解決方案中實現自動和定期重新整理機制。如果映象快取中的映象被 kubelet 的 gc 刪除，下一個重新整理週期會將已刪除的映象拉入映象快取中。這可確保映象快取是最新的。

設計流程

https://github.com/senthilrch/kube-fledged/blob/master/docs/kubefledged-architecture.png

部署 kube-fledged

Helm 方式部署

──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$mkdir  kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$cd kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$export KUBEFLEDGED_NAMESPACE=kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$kubectl create namespace ${KUBEFLEDGED_NAMESPACE}
namespace/kube-fledged created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$helm repo add kubefledged-charts https://senthilrch.github.io/kubefledged-charts/
"kubefledged-charts" has been added to your repositories
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kubefledged-charts" chart repository
...Successfully got an update from the "kubescape" chart repository
...Successfully got an update from the "rancher-stable" chart repository
...Successfully got an update from the "skm" chart repository
...Successfully got an update from the "openkruise" chart repository
...Successfully got an update from the "awx-operator" chart repository
...Successfully got an update from the "botkube" chart repository
Update Complete. ⎈Happy Helming!⎈
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$helm install --verify kube-fledged kubefledged-charts/kube-fledged -n ${KUBEFLEDGED_NAMESPACE} --wait

實際部署中發現，由於網路問題，chart 無法下載，所以透過 make deploy-using-yaml 使用 yaml 方式部署

Yaml 檔案部署

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$git clone https://github.com/senthilrch/kube-fledged.git
正克隆到 'kube-fledged'...
remote: Enumerating objects: 10613, done.
remote: Counting objects: 100% (1501/1501), done.
remote: Compressing objects: 100% (629/629), done.
remote: Total 10613 (delta 845), reused 1357 (delta 766), pack-reused 9112
接收物件中: 100% (10613/10613), 34.58 MiB | 7.33 MiB/s, done.
處理 delta 中: 100% (4431/4431), done.
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$ls
kube-fledged
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$cd kube-fledged/
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$make deploy-using-yaml
kubectl apply -f deploy/kubefledged-namespace.yaml

第一次部署，發現映象拉不下來

┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get all -n kube-fledged
NAME                                               READY   STATUS                  RESTARTS         AGE
pod/kube-fledged-controller-df69f6565-drrqg        0/1     CrashLoopBackOff        35 (5h59m ago)   21h
pod/kube-fledged-webhook-server-7bcd589bc4-b7kg2   0/1     Init:CrashLoopBackOff   35 (5h58m ago)   21h
pod/kubefledged-controller-55f848cc67-7f4rl        1/1     Running                 0                21h
pod/kubefledged-webhook-server-597dbf4ff5-l8fbh    0/1     Init:CrashLoopBackOff   34 (6h ago)      21h

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   21h
service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   21h

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-fledged-controller       0/1     1            0           21h
deployment.apps/kube-fledged-webhook-server   0/1     1            0           21h
deployment.apps/kubefledged-controller        0/1     1            0           21h
deployment.apps/kubefledged-webhook-server    0/1     1            0           21h

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/kube-fledged-controller-df69f6565        1         1         0       21h
replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         0       21h
replicaset.apps/kubefledged-controller-55f848cc67        1         1         0       21h
replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         0       21h
┌──[root@vms100.liruilongs.github.io]-[~]
└─$

這裡我們找一下要拉取的映象

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat *.yaml | grep image:
      - image: senthilrch/kubefledged-controller:v0.10.0
      - image: senthilrch/kubefledged-webhook-server:v0.10.0
      - image: senthilrch/kubefledged-webhook-server:v0.10.0

單獨拉取一些，當前使用 ansible 在所有工作節點批次操作

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible k8s_node -m shell -a "docker pull docker.io/senthilrch/kubefledged-cri-client:v0.10.0" -i host.yaml

其他相關的映象都拉取一下

操作完成之後容器狀態全部正常

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl -n kube-fledged get all
NAME                                               READY   STATUS    RESTARTS   AGE
pod/kube-fledged-controller-df69f6565-wdb4g        1/1     Running   0          13h
pod/kube-fledged-webhook-server-7bcd589bc4-j8xxp   1/1     Running   0          13h
pod/kubefledged-controller-55f848cc67-klxlm        1/1     Running   0          13h
pod/kubefledged-webhook-server-597dbf4ff5-ktbsh    1/1     Running   0          13h

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/kube-fledged-webhook-server   ClusterIP   10.100.194.199   <none>        3443/TCP   36h
service/kubefledged-webhook-server    ClusterIP   10.101.191.206   <none>        3443/TCP   36h

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-fledged-controller       1/1     1            1           36h
deployment.apps/kube-fledged-webhook-server   1/1     1            1           36h
deployment.apps/kubefledged-controller        1/1     1            1           36h
deployment.apps/kubefledged-webhook-server    1/1     1            1           36h

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/kube-fledged-controller-df69f6565        1         1         1       36h
replicaset.apps/kube-fledged-webhook-server-7bcd589bc4   1         1         1       36h
replicaset.apps/kubefledged-controller-55f848cc67        1         1         1       36h
replicaset.apps/kubefledged-webhook-server-597dbf4ff5    1         1         1       36h

驗證是否安裝成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$kubectl get pods -n kube-fledged -l app=kubefledged
NAME                                          READY   STATUS    RESTARTS   AGE
kubefledged-controller-55f848cc67-klxlm       1/1     Running   0          16h
kubefledged-webhook-server-597dbf4ff5-ktbsh   1/1     Running   0          16h
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$kubectl get imagecaches -n kube-fledged
No resources found in kube-fledged namespace.

使用 kubefledged

建立映象快取物件

根據 Demo 檔案，建立映象快取物件

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$cd deploy/
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat kubefledged-imagecache.yaml
---
apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
metadata:
  # Name of the image cache. A cluster can have multiple image cache objects
  name: imagecache1
  namespace: kube-fledged
  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  labels:
    app: kubefledged
    kubefledged: imagecache
spec:
  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  cacheSpec:
  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  - images:
    - ghcr.io/jitesoft/nginx:1.23.1
  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  - images:
    - us.gcr.io/k8s-artifacts-prod/cassandra:v7
    - us.gcr.io/k8s-artifacts-prod/etcd:3.5.4-0
    nodeSelector:
      tier: backend
  # Specifies a list of image pull secrets to pull images from private repositories into the cache
  imagePullSecrets:
  - name: myregistrykey

官方的 Demo 中對應的映象拉取不下來，所以換一下

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$docker pull us.gcr.io/k8s-artifacts-prod/cassandra:v7
Error response from daemon: Get "https://us.gcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

為了測試選擇器標籤的使用,我們找一個節點的標籤單獨做映象快取

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl get nodes  --show-labels

同時我們直接從公有倉庫拉取映象，所以不需要 imagePullSecrets 物件

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$vim kubefledged-imagecache.yaml

修改後的 yaml 檔案

新增了一個所有節點的 liruilong/my-busybox:latest 映象快取
新增了一個 kubernetes.io/hostname: vms105.liruilongs.github.io 對應標籤選擇器的 liruilong/hikvision-sdk-config-ftp:latest 映象快取

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat kubefledged-imagecache.yaml
---
apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
metadata:
  # Name of the image cache. A cluster can have multiple image cache objects
  name: imagecache1
  namespace: kube-fledged
  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  labels:
    app: kubefledged
    kubefledged: imagecache
spec:
  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  cacheSpec:
  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  - images:
    - liruilong/my-busybox:latest
  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  - images:
    - liruilong/hikvision-sdk-config-ftp:latest
    nodeSelector:
      kubernetes.io/hostname: vms105.liruilongs.github.io
  # Specifies a list of image pull secrets to pull images from private repositories into the cache
  #imagePullSecrets:
  #- name: myregistrykey
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

直接建立報錯了

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl create -f kubefledged-imagecache.yaml
Error from server (InternalError): error when creating "kubefledged-imagecache.yaml": Internal error occurred: failed calling webhook "validate-image-cache.kubefledged.io": failed to call webhook: Post "https://kubefledged-webhook-server.kube-fledged.svc:3443/validate-image-cache?timeout=1s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubefledged.io")
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl get imagecaches -n kube-fledged
No resources found in kube-fledged namespace.
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

解決辦法，刪除對應的物件，重新建立

我在當前專案的一個 issues 下面找到了解決辦法 https://github.com/senthilrch/kube-fledged/issues/76

看起來這是因為 Webhook CA 是硬編碼的，但是當 webhook 伺服器啟動時，會生成一個新的 CA 捆綁包並更新 webhook 配置。當發生另一個部署時，將重新應用原始 CA 捆綁包，並且 Webhook 請求開始失敗，直到再次重新啟動 Webhook 元件以修補捆綁包init-server

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$make remove-kubefledged-and-operator
# Remove kubefledged
kubectl delete -f deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml
error: resource mapping not found for name: "kube-fledged" namespace: "kube-fledged" from "deploy/kubefledged-operator/deploy/crds/charts.helm.kubefledged.io_v1alpha2_kubefledged_cr.yaml": no matches for kind "KubeFledged" in version "charts.helm.kubefledged.io/v1alpha2"
ensure CRDs are installed first
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged]
└─$make deploy-using-yaml
kubectl apply -f deploy/kubefledged-namespace.yaml
namespace/kube-fledged created
kubectl apply -f deploy/kubefledged-crd.yaml
customresourcedefinition.apiextensions.k8s.io/imagecaches.kubefledged.io unchanged
....................
kubectl rollout status deployment kubefledged-webhook-server -n kube-fledged --watch
Waiting for deployment "kubefledged-webhook-server" rollout to finish: 0 of 1 updated replicas are available...
deployment "kubefledged-webhook-server" successfully rolled out
kubectl get pods -n kube-fledged
NAME                                          READY   STATUS    RESTARTS   AGE
kubefledged-controller-55f848cc67-76c4v       1/1     Running   0          112s
kubefledged-webhook-server-597dbf4ff5-56h6z   1/1     Running   0          66s

重新建立快取物件，建立成功

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl create -f kubefledged-imagecache.yaml
imagecache.kubefledged.io/imagecache1 created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl get imagecaches -n kube-fledged
NAME          AGE
imagecache1   10s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

檢視當前被納管的映象快取

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$kubectl get imagecaches imagecache1 -n kube-fledged -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 83,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20169836",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    "liruilong/my-busybox:latest"
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "completionTime": "2024-03-02T01:06:47Z",
        "message": "All requested images pulled succesfully to respective nodes",
        "reason": "ImageCacheRefresh",
        "startTime": "2024-03-02T01:05:33Z",
        "status": "Succeeded"
    }
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged]
└─$

透過 ansible 來驗證

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
192.168.26.102 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.101 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.103 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.105 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.100 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.106 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/hikvision-sdk-config-ftp" -i host.yaml
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.105 | CHANGED | rc=0 >>
liruilong/hikvision-sdk-config-ftp                                          latest            a02cd03b4342   4 months ago    830MB
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

開啟自動重新整理

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl annotate imagecaches imagecache1 -n kube-fledged kubefledged.io/refresh-imagecache=
imagecache.kubefledged.io/imagecache1 annotated
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

新增映象快取

新增一個新的映象快取

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 92,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20175233",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    "liruilong/my-busybox:latest",
                    "liruilong/jdk1.8_191:latest"
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "completionTime": "2024-03-02T01:43:32Z",
        "message": "All requested images pulled succesfully to respective nodes",
        "reason": "ImageCacheUpdate",
        "startTime": "2024-03-02T01:40:34Z",
        "status": "Succeeded"
    }
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

透過 ansible 確認

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
192.168.26.101 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
192.168.26.102 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
192.168.26.100 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest    17dbd4002a8c   5 years ago     170MB
192.168.26.103 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest                                      17dbd4002a8c   5 years ago     170MB
192.168.26.105 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB
192.168.26.106 | CHANGED | rc=0 >>
liruilong/jdk1.8_191                                                        latest            17dbd4002a8c   5 years ago     170MB
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

刪除映象快取

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl edit imagecaches imagecache1 -n kube-fledged
imagecache.kubefledged.io/imagecache1 edited
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 94,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20175766",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    "liruilong/jdk1.8_191:latest"
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "message": "Image cache is being updated. Please view the status after some time",
        "reason": "ImageCacheUpdate",
        "startTime": "2024-03-02T01:48:03Z",
        "status": "Processing"
    }
}

透過 Ansible 確認，可以看到無論是 mastere 上的節點還是 work 的節點，對應的映象快取都被清理

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
192.168.26.102 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.101 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | CHANGED | rc=0 >>
liruilong/my-busybox                                                        latest    497b83a63aad   11 months ago   1.24MB
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/my-busybox" -i host.yaml
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

這裡需要注意如果清除所有的映象快取，那麼需要把 images 下的陣列寫成 "".

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl edit imagecaches imagecache1 -n kube-fledged
imagecache.kubefledged.io/imagecache1 edited
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$ansible all -m shell -a "docker images | grep liruilong/jdk1.8_191" -i host.yaml
192.168.26.102 | FAILED | rc=1 >>
non-zero return code
192.168.26.101 | FAILED | rc=1 >>
non-zero return code
192.168.26.100 | FAILED | rc=1 >>
non-zero return code
192.168.26.105 | FAILED | rc=1 >>
non-zero return code
192.168.26.103 | FAILED | rc=1 >>
non-zero return code
192.168.26.106 | FAILED | rc=1 >>
non-zero return code
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get imagecaches.kubefledged.io  -n kube-fledged  imagecache1 -o json
{
    "apiVersion": "kubefledged.io/v1alpha2",
    "kind": "ImageCache",
    "metadata": {
        "creationTimestamp": "2024-03-01T15:08:42Z",
        "generation": 98,
        "labels": {
            "app": "kubefledged",
            "kubefledged": "imagecache"
        },
        "name": "imagecache1",
        "namespace": "kube-fledged",
        "resourceVersion": "20176849",
        "uid": "3a680a57-d8ab-444f-b9c9-4382459c5c72"
    },
    "spec": {
        "cacheSpec": [
            {
                "images": [
                    ""
                ]
            },
            {
                "images": [
                    "liruilong/hikvision-sdk-config-ftp:latest"
                ],
                "nodeSelector": {
                    "kubernetes.io/hostname": "vms105.liruilongs.github.io"
                }
            }
        ]
    },
    "status": {
        "completionTime": "2024-03-02T01:52:16Z",
        "message": "All cached images succesfully deleted from respective nodes",
        "reason": "ImageCacheUpdate",
        "startTime": "2024-03-02T01:51:47Z",
        "status": "Succeeded"
    }
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$

如果透過下面的方式刪除，直接註釋調對應的標籤

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$cat kubefledged-imagecache.yaml
---
apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
metadata:
  # Name of the image cache. A cluster can have multiple image cache objects
  name: imagecache1
  namespace: kube-fledged
  # The kubernetes namespace to be used for this image cache. You can choose a different namepace as per your preference
  labels:
    app: kubefledged
    kubefledged: imagecache
spec:
  # The "cacheSpec" field allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pre-pulled).
  cacheSpec:
  # Specifies a list of images (nginx:1.23.1) with no node selector, hence these images will be cached in all the nodes in the cluster
  #- images:
    #- liruilong/my-busybox:latest
  # Specifies a list of images (cassandra:v7 and etcd:3.5.4-0) with a node selector, hence these images will be cached only on the nodes selected by the node selector
  - images:
    - liruilong/hikvision-sdk-config-ftp:latest
    nodeSelector:
      kubernetes.io/hostname: vms105.liruilongs.github.io
  # Specifies a list of image pull secrets to pull images from private repositories into the cache
  #imagePullSecrets:
  #- name: myregistrykey
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$

那麼會報下面的錯

┌──[root@vms100.liruilongs.github.io]-[~/ansible/kube-fledged/kube-fledged/deploy]
└─$kubectl edit imagecaches imagecache1 -n kube-fledged
error: imagecaches.kubefledged.io "imagecache1" could not be patched: admission webhook "validate-image-cache.kubefledged.io" denied the request: Mismatch in no. of image lists
You can run `kubectl replace -f /tmp/kubectl-edit-4113815075.yaml` to try this update again.

博文部分內容參考

https://github.com/senthilrch/kube-fledged

點選關注，第一時間瞭解華為雲新鮮技術~

Spring 整合 Ehcache 管理快取詳解
2016-10-23
Spring快取
Mybatis快取詳解
2019-09-10
MyBatis快取
Redis詳解（十二）------ 快取穿透、快取擊穿、快取雪崩
2020-06-02
Redis快取穿透
深入Nginx + PHP 快取詳解
2019-03-04
NginxPHP快取
JuiceFS 快取預熱詳解
2022-04-19
UI快取
瀏覽器快取詳解
2017-06-15
瀏覽器快取
快取穿透詳解及解決方案
2022-02-17
快取穿透
nginx快取使用詳解，nginx快取使用及配置步驟
2022-05-10
Nginx快取
http強制快取、協商快取、指紋ETag詳解
2021-06-30
HTTP快取
詳解cookie、session和HTTP快取
2018-02-08
CookieSessionHTTP快取
解決k8s中node拉取映象失敗問題
2021-12-08
K8S
iOS OpenGL ES FBO 幀快取區渲染快取區詳解
2017-10-11
iOS快取
Hibernate中一級快取和二級快取使用詳解
2014-10-14
快取
深度詳解GaussDB bufferpool快取策略
2020-09-15
快取
瀏覽器快取機制詳解
2018-06-03
瀏覽器快取
MySQL查詢快取引數詳解
2016-09-08
MySql快取
Android Bitmap快取池使用詳解
2015-01-02
Android快取
Hibernate 所有快取機制詳解
2015-10-25
快取
SpringBoot快取管理（一）預設快取管理
2021-07-05
Spring Boot快取
docker 映象詳解
2017-10-16
Docker
App快取管理
2013-11-28
APP快取
Nginx 快取機制詳解！非常詳細實用
2021-09-30
Nginx快取
【docker專欄5】詳解docker映象管理命令
2022-07-12
Docker
Mybatis 一級快取和二級快取原理區別 (圖文詳解)
2022-09-21
MyBatis快取
k8s使用secret從私有倉庫拉取映象
2020-12-02
K8S
快取問題(四) 快取穿透、快取雪崩、快取併發解決案例
2020-11-10
快取穿透
詳解NSURLCache快取引發的安全漏洞
2019-02-18
快取
詳解快取更新策略及如何選擇
2023-05-12
快取
用Java寫一個分散式快取——快取管理
2023-01-27
Java分散式快取
快取穿透、快取擊穿、快取雪崩概念及解決方案
2018-10-12
快取穿透
REDIS快取穿透，快取擊穿，快取雪崩原因+解決方案
2020-10-27
Redis快取穿透
【Redis】快取穿透，快取擊穿，快取雪崩及解決方案
2020-11-08
Redis快取穿透
淺解強快取和協商快取
2018-12-12
快取
邊緣叢集場景下的映象快取
2021-06-09
快取
Dockerfile生成映象的時候是如何快取的？
2017-11-24
Docker快取
瀏覽器 HTTP 協議快取機制詳解
2016-10-19
瀏覽器HTTP協議快取
PHP快取技術：memcache函式詳解之一
2017-11-15
PHP快取函式
Android中圖片的三層快取詳解
2016-07-10
Android快取