准入控制器(Admission Controller):ResourceQuota，ImagePolicyWebhook

人生的哲理發表於2024-06-21

一.系統環境
二.前言
三.准入控制器簡介
四.為什麼需要准入控制器
五.啟用/禁用ResourceQuota資源配額
- 5.1 檢視預設啟用/禁用的准入控制器外掛
- 5.2 ResourceQuota資源配額示例
- 5.3 禁用ResourceQuota
六.配置ImagePolicyWebhook准入控制器禁止使用字尾為latest的映象
- 6.1 搭建Webhook伺服器
- 6.2 配置kubernetes連線後端webhook伺服器
- 6.3 驗證
七.總結

一.系統環境

本文主要基於Kubernetes1.22.2和Linux作業系統Ubuntu 18.04。

伺服器版本	docker軟體版本	Kubernetes(k8s)叢集版本	CPU架構
Ubuntu 18.04.5 LTS	Docker version 20.10.14	v1.22.2	x86_64

Kubernetes叢集架構：k8scludes1作為master節點，k8scludes2，k8scludes3作為worker節點。

伺服器	作業系統版本	CPU架構	程序	功能描述
k8scludes1/192.168.110.128	Ubuntu 18.04.5 LTS	x86_64	docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico	k8s master節點
k8scludes2/192.168.110.129	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節點
k8scludes3/192.168.110.130	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節點

二.前言

在本文中，我們將探討Kubernetes中的准入控制器（Admission Controller）。准入控制器是Kubernetes API server的一部分，負責處理來自外部的API請求。它們確保只有滿足特定條件的請求才能訪問叢集資源。透過使用准入控制器，我們可以實現對叢集資源的細粒度控制，從而提高叢集的安全性和可靠性。

使用准入控制器（Admission Controller）的前提是已經有一套可以正常執行的Kubernetes叢集，關於Kubernetes(k8s)叢集的安裝部署，可以檢視部落格《Ubuntu 安裝部署Kubernetes(k8s)叢集》https://www.cnblogs.com/renshengdezheli/p/17632858.html。

三.准入控制器簡介

當使用者賬戶或者service account要執行一些操作的時候，首先要進行賬號認證，認證透過之後，如果賬號已經被授權相關資源，就可以進行相關操作了。有時授權之後也會有Admission control准入控制器(可以理解為一系列規則，可以啟用或者關閉某個功能，准入控制器也支援一系列的webhook，在webhook裡可以定義一系列規則)，滿足Admission control准入控制器的規則就可以建立相關資源了。OPA Gatekeeper也是利用的准入控制器功能，詳情請檢視部落格《OPA Gatekeeper：Kubernetes的策略和管理》。

准入控制器是一段程式碼，它會在請求透過認證和鑑權之後、物件被持久化之前攔截到達 API 伺服器的請求。

准入控制器可以執行驗證（Validating）和/或變更（Mutating）操作。變更（mutating）控制器可以根據被其接受的請求更改相關物件；驗證（validating）控制器則不行。

准入控制器限制建立、刪除、修改物件的請求。准入控制器也可以阻止自定義動作，例如透過 API 伺服器代理連線到 Pod 的請求。准入控制器不會（也不能）阻止讀取（get、watch 或 list）物件的請求。

Kubernetes 1.30 中的准入控制器編譯進 kube-apiserver 可執行檔案，並且只能由叢集管理員配置。在該列表中，有兩個特殊的控制器：MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook。它們根據 API 中的配置，分別執行變更和驗證准入控制 Webhook。

准入控制過程分為兩個階段。第一階段，執行變更准入控制器。第二階段，執行驗證准入控制器。再次提醒，某些控制器既是變更准入控制器又是驗證准入控制器。

如果兩個階段之一的任何一個控制器拒絕了某請求，則整個請求將立即被拒絕，並向終端使用者返回錯誤。

最後，除了對物件進行變更外，准入控制器還可能有其它副作用：將相關資源作為請求處理的一部分進行變更。增加配額用量就是一個典型的示例，說明了這樣做的必要性。此類用法都需要相應的回收或回撥過程，因為任一準入控制器都無法確定某個請求能否透過所有其它准入控制器。

四.為什麼需要准入控制器

Kubernetes 的若干重要功能都要求啟用一個准入控制器，以便正確地支援該特性。因此，沒有正確配置准入控制器的 Kubernetes API 伺服器是不完整的，它無法支援你所期望的所有特性。

五.啟用/禁用ResourceQuota資源配額

5.1 檢視預設啟用/禁用的准入控制器外掛

本文中的kube-apiserver是以pod的方式執行的，名字為：kube-apiserver-k8scludes1。

root@k8scludes1:~# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS        AGE
calico-kube-controllers-65898446b5-qd9q6   1/1     Running   21 (2d8h ago)   28d
calico-node-d6564                          1/1     Running   58 (2d8h ago)   59d
calico-node-jgvjb                          1/1     Running   67 (10h ago)    59d
calico-node-snkxp                          1/1     Running   58 (2d8h ago)   59d
coredns-7f6cbbb7b8-5gpjt                   1/1     Running   20 (2d8h ago)   28d
coredns-7f6cbbb7b8-xm6pl                   1/1     Running   20 (2d8h ago)   28d
etcd-k8scludes1                            1/1     Running   54 (2d8h ago)   58d
kube-apiserver-k8scludes1                  1/1     Running   15 (10h ago)    19d
kube-controller-manager-k8scludes1         1/1     Running   249 (10h ago)   59d
kube-proxy-464tx                           1/1     Running   52 (2d8h ago)   59d
kube-proxy-mkwpx                           1/1     Running   52 (2d8h ago)   59d
kube-proxy-vsc9q                           1/1     Running   53 (2d8h ago)   59d
kube-scheduler-k8scludes1                  1/1     Running   247 (10h ago)   59d

檢視kube-apiserver的幫助資訊。

root@k8scludes1:~# kubectl exec -it kube-apiserver-k8scludes1 -n kube-system -- kube-apiserver -h
The Kubernetes API server validates and configures data
for the api objects which include pods, services, replicationcontrollers, and
others. The API Server services REST operations and provides the frontend to the
     ......
      --goaway-chance float                                                                                                                                                                              
                To prevent HTTP/2 clients from getting stuck on a single apiserver, randomly close a connection (GOAWAY). The client's other in-flight requests won't be affected, and the client will
                reconnect, likely landing on a different apiserver after going through the load balancer again. This argument sets the fraction of requests that will be sent a GOAWAY. Clusters with single
                apiservers, or which don't use a load balancer, should NOT enable this. Min is 0 (off), Max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.
      --livez-grace-period duration                                                                                                                                                                      
                This option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has
                elapsed, /livez will assume that unfinished post-start hooks will complete successfully and therefore return true.
     
      --vmodule moduleSpec                                                                                                                                                                               
                comma-separated list of pattern=N settings for file-filtered logging

檢視預設啟用/禁止的准入控制器外掛有哪些？可以看到預設啟用的准入控制器外掛有：CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook。

root@k8scludes1:~# kubectl exec -it kube-apiserver-k8scludes1 -n kube-system -- kube-apiserver -h | grep admission-plugins
      --admission-control strings              Admission is divided into two phases. In the first phase, only mutating admission plugins run. In the second phase, only validating admission plugins run. The names in the below list may represent a validating plugin, a mutating plugin, or both. The order of plugins in which they are passed to this flag does not matter. Comma-delimited list of: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. (DEPRECATED: Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.)
      --disable-admission-plugins strings      admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
      --enable-admission-plugins strings       admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.

5.2 ResourceQuota資源配額示例

建立目錄存放yaml檔案。

root@k8scludes1:~# mkdir admissioncontr

root@k8scludes1:~# cd admissioncontr/

建立名稱空間。

root@k8scludes1:~/admissioncontr# kubectl create ns admissioncontr
namespace/admissioncontr created

切換名稱空間到admissioncontr。

root@k8scludes1:~/admissioncontr# kubens admissioncontr
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "admissioncontr".

編輯pod配置檔案，使用nginx映象生成pod。

root@k8scludes1:~/admissioncontr# vim pod.yaml 

root@k8scludes1:~/admissioncontr# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #當需要關閉容器時，立即殺死容器而不等待預設的30秒優雅停機時長。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已經存在該映象，則不重新下載；否則從遠端 Docker Hub 下載該映象
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

建立pod。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
pod/podtest created

使用相同的yaml檔案，生成另外兩個pod。

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
pod/podtest2 created

現在沒有任何限制，可以正常建立多個pod。

root@k8scludes1:~/admissioncontr# kubectl get pod -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest    1/1     Running   0          61s   10.244.218.179   k8scludes2   <none>           <none>
podtest1   1/1     Running   0          15s   10.244.218.133   k8scludes2   <none>           <none>
podtest2   1/1     Running   0          9s    10.244.1.84      k8scludes3   <none>           <none>

刪除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1 podtest2
pod "podtest" deleted
pod "podtest1" deleted
pod "podtest2" deleted

現在沒有設定resourcequota資源配額。

root@k8scludes1:~/admissioncontr# kubectl get quota
No resources found in admissioncontr namespace.

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
No resources found in admissioncontr namespace.

編輯ResourceQuota配置檔案，表示建立一個resourcequota資源配額，當前名稱空間只能建立2個pod。

resourcequota 資源配額用來設定一個名稱空間最多能建立多少個資源物件，比如能建立多少svc，能建立多少pod或者deploy，詳細資訊請檢視部落格《Kubernetes(k8s) 資源限制：resources，LimitRange，ResourceQuota》。

root@k8scludes1:~/admissioncontr# vim resourcequota.yaml

root@k8scludes1:~/admissioncontr# cat resourcequota.yaml 
apiVersion: v1 
kind: ResourceQuota 
metadata: 
  name: myresourcequota 
spec: 
  #hard：指定資源的硬性限制，即最大允許使用的資源數量。
  hard: 
    pods: "2"

建立ResourceQuota。

root@k8scludes1:~/admissioncontr# kubectl apply -f resourcequota.yaml 
resourcequota/myresourcequota created

檢視resourcequota資源配額，可以看到pod資源限額是2，現在有0個pod。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE   REQUEST     LIMIT
myresourcequota   11s   pods: 0/2

再次建立pod。

root@k8scludes1:~/admissioncontr#  kubectl apply -f pod.yaml
pod/podtest created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

建立第三個pod的時候報錯了，pod資源配額是2，超過2個pod就建立失敗了。

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
Error from server (Forbidden): error when creating "STDIN": pods "podtest2" is forbidden: exceeded quota: myresourcequota, requested: pods=1, used: pods=2, limited: pods=2

檢視resourcequota，可以看到pods: 2/2，資源配額滿了。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE     REQUEST     LIMIT
myresourcequota   5m48s   pods: 2/2

刪除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1
pod "podtest" deleted
pod "podtest1" deleted

5.3 禁用ResourceQuota

resourcequota 資源配額是一種准入控制器外掛，預設是啟用的。我們可以把resourcequota禁用了。

修改kube-apiserver.yaml檔案，禁用resourcequota 資源配額。

disable-admission-plugins=ResourceQuota表示禁用准入控制器外掛ResourceQuota。

root@k8scludes1:~/admissioncontr# vim /etc/kubernetes/manifests/kube-apiserver.yaml

root@k8scludes1:~/admissioncontr# grep disable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml
    - --disable-admission-plugins=ResourceQuota

重啟kubelet使配置生效。

root@k8scludes1:~/admissioncontr# systemctl restart kubelet

現在resourcequota 資源配額還在，還是要求當前名稱空間只能建立2個pod。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE   REQUEST     LIMIT
myresourcequota   16m   pods: 0/2

一口氣建立了5個pod還是沒有報錯。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
pod/podtest created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
pod/podtest2 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest3/' pod.yaml | kubectl apply -f -
pod/podtest3 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest4/' pod.yaml | kubectl apply -f -
pod/podtest4 created

root@k8scludes1:~/admissioncontr# kubectl get pod 
NAME       READY   STATUS    RESTARTS   AGE
podtest    1/1     Running   0          30s
podtest1   1/1     Running   0          21s
podtest2   1/1     Running   0          16s
podtest3   1/1     Running   0          11s
podtest4   1/1     Running   0          6s

可以發現，禁用resourcequota功能之後，就算有resourcequota物件存在，資源配額也不生效。

root@k8scludes1:~/admissioncontr# kubectl get resourcequota
NAME              AGE     REQUEST     LIMIT
myresourcequota   3m59s   pods: 5/2

刪除pod。

root@k8scludes1:~# kubectl delete pod podtest podtest1 podtest2 podtest3 podtest4
pod "podtest" deleted
pod "podtest1" deleted
pod "podtest2" deleted
pod "podtest3" deleted
pod "podtest4" deleted

root@k8scludes1:~# kubectl get pod 
No resources found in admissioncontr namespace.

編輯kube-apiserver.yaml，去掉disable-admission-plugins=ResourceQuota，讓ResourceQuota預設啟用。

root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:~# grep disable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml 
    #- --disable-admission-plugins=ResourceQuota

重啟kubelet使配置生效。

root@k8scludes1:~# systemctl restart kubelet

現在resourcequota 資源配額還是要求當前名稱空間只能建立2個pod。

root@k8scludes1:~# kubectl get resourcequota
NAME              AGE     REQUEST     LIMIT
myresourcequota   5h31m   pods: 0/2

現在ResourceQuota資源配額又生效了，只能建立2個pod。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml
pod/podtest created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest1/' pod.yaml | kubectl apply -f -
pod/podtest1 created

root@k8scludes1:~/admissioncontr# sed 's/podtest/podtest2/' pod.yaml | kubectl apply -f -
Error from server (Forbidden): error when creating "STDIN": pods "podtest2" is forbidden: exceeded quota: myresourcequota, requested: pods=1, used: pods=2, limited: pods=2

刪除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest podtest1 
pod "podtest" deleted
pod "podtest1" deleted

刪除ResourceQuota資源配額。

root@k8scludes1:~/admissioncontr# kubectl delete resourcequota myresourcequota 
resourcequota "myresourcequota" deleted

root@k8scludes1:~/admissioncontr# kubectl get resourcequotas 
No resources found in admissioncontr namespace.

六.配置ImagePolicyWebhook准入控制器禁止使用字尾為latest的映象

6.1 搭建Webhook伺服器

ImagePolicyWebhook的類別為驗證。ImagePolicyWebhook 准入控制器允許使用後端 Webhook 做出准入決策。此准入控制器預設被禁用。

准入 Webhook 是一種用於接收准入請求並對其進行處理的 HTTP 回撥機制。可以定義兩種型別的准入 webhook，即驗證性質的准入 Webhook 和修改性質的准入 Webhook。修改性質的准入 Webhook 會先被呼叫。它們可以更改傳送到 API 伺服器的物件，以執行自定義的設定預設值操作。在完成了所有物件修改並且 API 伺服器也驗證了所傳入的物件之後，驗證性質的 Webhook 會被呼叫，並透過拒絕請求的方式來強制實施自定義的策略。

說明：如果准入 Webhook 需要保證它們所看到的是物件的最終狀態以實施某種策略。則應使用驗證性質的准入 Webhook，因為物件被修改性質 Webhook 看到之後仍然可能被修改。

首先需要找一臺機器搭建Webhook伺服器，我們使用etcd2機器當Webhook伺服器。

首先需要安裝docker。

[root@etcd2 ~]# yum -y install docker-ce

檢視docker版本。

[root@etcd2 ~]# docker -v
Docker version 20.10.12, build e91ed57

配置docker映象加速器。

[root@etcd2 ~]# cat /etc/docker/daemon.json
{
    "registry-mirrors": [
        "https://frz7i079.mirror.aliyuncs.com"
    ]
}

拉取flavio/kube-image-bouncer映象，flavio/kube-image-bouncer這個映象的功能為：不允許使用字尾為latest的映象。

[root@etcd2 ~]# docker pull flavio/kube-image-bouncer
Using default tag: latest
latest: Pulling from flavio/kube-image-bouncer
ff3a5c916c92: Pull complete 
4947cf5a98cf: Pull complete 
37ff781c0e4d: Pull complete 
1545c1d26daf: Pull complete 
dd581c9cf10b: Pull complete 
Digest: sha256:01e1e0873d20cee1ffefb45421c798cb26250b7a9384978d34ffb1f57403d501
Status: Downloaded newer image for flavio/kube-image-bouncer:latest
docker.io/flavio/kube-image-bouncer:latest

檢視flavio/kube-image-bouncer映象歷史資訊，可以發現開放埠為1323(EXPOSE 1323)，使用的是web賬號登入(USER web)。關於映象Dockerfile的詳細資訊，請檢視部落格《構建自定義映象並最佳化dockerfile檔案》。

[root@etcd2 ~]# docker history flavio/kube-image-bouncer
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
3974e07cc0c8   3 years ago   /bin/sh -c #(nop)  EXPOSE 1323                  0B        
<missing>      3 years ago   /bin/sh -c #(nop)  ENTRYPOINT ["./kube-image…   0B        
<missing>      3 years ago   /bin/sh -c #(nop)  USER web                     0B        
<missing>      3 years ago   /bin/sh -c chown -R web:web *                   19.9MB    
<missing>      3 years ago   /bin/sh -c #(nop) COPY file:de99d16a3731b75b…   19.9MB    
<missing>      3 years ago   /bin/sh -c adduser -h /app -D web               4.79kB    
<missing>      3 years ago   /bin/sh -c #(nop) WORKDIR /app                  0B        
<missing>      4 years ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B        
<missing>      4 years ago   /bin/sh -c #(nop) ADD file:093f0723fa46f6cdb…   4.15MB

使用flavio/kube-image-bouncer映象建立容器。

-vpwd/webhook-key.pem:/certs/webhook-key.pem:ro -v pwd/webhook.pem:/certs/webhook.pem:ro 表示做目錄對映（-v 本地目錄：容器目錄），webhook-key.pem公鑰和webhook.pem私鑰不存在也沒問題，ro表示容器只有只讀許可權。

-p 1323:1323表示做埠對映（物理機埠1323：容器埠1323）。關於容器更多詳細資訊，請檢視部落格《一文搞懂docker容器基礎：docker映象管理，docker容器管理》。

[root@etcd2 ~]# docker run -dit --name=c1 --restart=always -v `pwd`/webhook-key.pem:/certs/webhook-key.pem:ro -v `pwd`/webhook.pem:/certs/webhook.pem:ro -p 1323:1323 flavio/kube-image-bouncer
7c677fb67522073932617f07ad72cd1a17e60eddc6a68dd25ffe0b8c4f80aaae

容器建立成功了，這時候訪問192.168.110.131:1323 就訪問到該webhook伺服器了。

[root@etcd2 ~]# docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED          STATUS         PORTS                                       NAMES
7c677fb67522   flavio/kube-image-bouncer     "./kube-image-bouncer"   10 seconds ago   Up 8 seconds   0.0.0.0:1323->1323/tcp, :::1323->1323/tcp   c1

6.2 配置kubernetes連線後端webhook伺服器

回到kubernetes叢集。

root@k8scludes1:~/admissioncontr# cd /etc/kubernetes/

root@k8scludes1:/etc/kubernetes# ls
admin.conf  admission-control-config-file  controller-manager.conf  kubelet.conf  manifests  pki  scheduler.conf

root@k8scludes1:/etc/kubernetes# cd admission-control-config-file/

root@k8scludes1:/etc/kubernetes/admission-control-config-file# ls
admission_configuration.json  apiserver-client-key.pem  apiserver-client.pem  kubeconfig.yaml  webhook-key.pem  webhook.pem

root@k8scludes1:/etc/kubernetes/admission-control-config-file# pwd
/etc/kubernetes/admission-control-config-file

ImagePolicyWebhook 使用配置檔案來為後端行為設定選項。該檔案可以是 JSON 或 YAML。

admission_configuration.json是ImagePolicyWebhook 的配置檔案，引數解釋如下：

allowTTL以秒計的時長，控制批准請求的快取時間；
denyTTL以秒計的時長，控制拒絕請求的快取時間；
retryBackoff以毫秒計的時長，控制重試間隔；
defaultAllow確定 Webhook 後端失效時的行為；
kubeConfigFile指定kubeconfig檔案的路徑。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim admission_configuration.json 

root@k8scludes1:/etc/kubernetes/admission-control-config-file# cat admission_configuration.json 
{
  "imagePolicy": {
     "kubeConfigFile": "/etc/kubernetes/admission-control-config-file/kubeconfig.yaml",
     "allowTTL": 50,
     "denyTTL": 50,
     "retryBackoff": 500,
     "defaultAllow": true
  }
}

root@k8scludes1:/etc/kubernetes/admission-control-config-file# ls /etc/kubernetes/admission-control-config-file/admission_configuration.json 
/etc/kubernetes/admission-control-config-file/admission_configuration.json

kubeconfig.yaml用來設定與後端Webhook伺服器的連線，要求後端使用 TLS 進行通訊。

kubeconfig.yaml檔案的 clusters 欄位需要指向遠端Webhook服務，server指定webhook伺服器進行連線。

users 欄位需要包含已返回的授權者。關於kubeconfig檔案的詳細資訊，請檢視部落格《Kubernetes(k8s)訪問控制：身份認證》。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim kubeconfig.yaml 

#指定各種證書
root@k8scludes1:/etc/kubernetes/admission-control-config-file# cat kubeconfig.yaml 
apiVersion: v1
kind: Config
clusters:
- cluster:
    # CA 用於驗證遠端服務
    certificate-authority: /etc/kubernetes/admission-control-config-file/webhook.pem
    # 要查詢的遠端服務的 URL
    server: http://192.168.110.131:1323/image_policy
  name: bouncer_webhook
contexts:
- context:
    cluster: bouncer_webhook
    user: api-server
  name: bouncer_validator
current-context: bouncer_validator
preferences: {}
users:
- name: api-server
  user:
    # Webhook 准入控制器使用的證書
    client-certificate: /etc/kubernetes/admission-control-config-file/apiserver-client.pem
    # 證書匹配的金鑰
    client-key:  /etc/kubernetes/admission-control-config-file/apiserver-client-key.pem

ImagePolicyWebhook預設是沒有開啟的，現在啟用ImagePolicyWebhook。

enable-admission-plugins=NodeRestriction,ImagePolicyWebhook表示啟用ImagePolicyWebhook准入控制器。

透過 --admission-control-config-file 引數指定準入控制器ImagePolicyWebhook配置檔案的位置。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:/etc/kubernetes/admission-control-config-file# grep admission /etc/kubernetes/manifests/kube-apiserver.yaml
    - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
    - --admission-control-config-file=/etc/kubernetes/admission-control-config-file/admission_configuration.json

重啟kubelet使配置生效。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# systemctl restart kubelet

可以發現kube-apiserver.yaml新增引數之後，連線不上叢集了。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# kubectl get node
The connection to the server 192.168.110.128:6443 was refused - did you specify the right host or port?

下面開始排查問題，kubectl使用不了，那就只能使用docker排查問題了，使用docker檢視容器，可以發現k8s_kube-apiserver容器處於退出狀態。

root@k8scludes1:/etc/kubernetes/admission-control-config-file# cd

root@k8scludes1:~# docker ps -a | grep api
2aacfa2bdbc4   e64579b7d886                                        "kube-apiserver --ad…"   19 seconds ago      Exited (1) 18 seconds ago             k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_054ae2c0df7fd5edd4ebb3b7057a0462_8
cb0e2e181dda   registry.aliyuncs.com/google_containers/pause:3.5   "/pause"                 4 minutes ago       Up 4 minutes                          k8s_POD_kube-apiserver-k8scludes1_kube-system_054ae2c0df7fd5edd4ebb3b7057a0462_0
ac234a8ee92b   e64579b7d886                                        "kube-apiserver --ad…"   4 minutes ago       Exited (1) 4 minutes ago              k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_f38197908cec7cb32a42e0862e367c1c_0
49e9b9c1b389   registry.aliyuncs.com/google_containers/pause:3.5   "/pause"                 4 minutes ago       Up 4 minutes                          k8s_POD_kube-apiserver-k8scludes1_kube-system_f38197908cec7cb32a42e0862e367c1c_0
5d4985a8833c   e64579b7d886                                        "kube-apiserver --ad…"   5 minutes ago       Exited (1) 5 minutes ago              k8s_kube-apiserver_kube-apiserver-k8scludes1_kube-system_3dc64835bcbf5bdf937660d07f41b90b_87
c674a6a6c794   registry.aliyuncs.com/google_containers/pause:3.5   "/pause"                 About an hour ago   Up About an hour                      k8s_POD_kube-apiserver-k8scludes1_kube-system_3dc64835bcbf5bdf937660d07f41b90b_2

檢視k8s_kube-apiserver容器日誌，報錯為：“open /etc/kubernetes/admission-control-config-file/admission_configuration.json: no such file or directory“，發現admission_configuration.json檔案找不到。

root@k8scludes1:~# docker logs 2aacfa2bdbc4
I0616 01:39:59.927352       1 server.go:553] external host was not specified, using 192.168.110.128
I0616 01:39:59.927965       1 server.go:161] Version: v1.22.2
Error: failed to initialize admission: failed to read plugin config: unable to read admission control configuration from "/etc/kubernetes/admission-control-config-file/admission_configuration.json" [open /etc/kubernetes/admission-control-config-file/admission_configuration.json: no such file or directory]

/etc/kubernetes/admission-control-config-file/admission_configuration.json檔案是存在的，但是報錯。

原因為：/etc/kubernetes/admission-control-config-file/admission_configuration.json這個檔案是在宿主機裡的，並不在容器裡。

解決方法：做一個資料卷，把宿主機裡的檔案對映到容器裡。

root@k8scludes1:~# ls /etc/kubernetes/admission-control-config-file/admission_configuration.json
/etc/kubernetes/admission-control-config-file/admission_configuration.json

注意：生產環境裡可以先備份下kube-apiserver.yaml 檔案再修改，以免改不回來。

定義一個資料卷把宿主機的/etc/kubernetes/admission-control-config-file目錄掛載到容器/etc/kubernetes/admission-control-config-file目錄。

root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:~# grep -A3 admission-control-config-file /etc/kubernetes/manifests/kube-apiserver.yaml
    - --admission-control-config-file=/etc/kubernetes/admission-control-config-file/admission_configuration.json
    #- --disable-admission-plugins=ResourceQuota
    #- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
    - --token-auth-file=/etc/kubernetes/pki/mytok.csv
--
    - mountPath: /etc/kubernetes/admission-control-config-file
      name: admissionconfile
      readOnly: true
  hostNetwork: true
--
      path: /etc/kubernetes/admission-control-config-file
      type: DirectoryOrCreate
    name: admissionconfile
  - hostPath:

直接上截圖可能更直觀：

重啟kubelet使配置生效。

root@k8scludes1:~# systemctl restart kubelet

現在k8s正常了。

root@k8scludes1:~# kubectl get pod 
No resources found in admissioncontr namespace.

6.3 驗證

編輯pod配置檔案，表示使用hub.c.163.com/library/nginx:latest映象建立pod。

root@k8scludes1:~# cd admissioncontr/

root@k8scludes1:~/admissioncontr# vim pod.yaml 

root@k8scludes1:~/admissioncontr# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #當需要關閉容器時，立即殺死容器而不等待預設的30秒優雅停機時長。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已經存在該映象，則不重新下載；否則從遠端 Docker Hub 下載該映象
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

建立pod，可以發現：使用tag為latest的映象不能建立pod。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
Error from server (Forbidden): error when creating "pod.yaml": pods "podtest" is forbidden: image policy webhook backend denied one or more images: Images using latest tag are not allowed

修改pod配置檔案，表示使用hub.c.163.com/library/nginx:test映象建立pod。

root@k8scludes1:~/admissioncontr# vim pod.yaml 

root@k8scludes1:~/admissioncontr# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #當需要關閉容器時，立即殺死容器而不等待預設的30秒優雅停機時長。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:test
  #- image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已經存在該映象，則不重新下載；否則從遠端 Docker Hub 下載該映象
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

映象tag為test，pod建立成功。

root@k8scludes1:~/admissioncontr# kubectl apply -f pod.yaml 
pod/podtest created

root@k8scludes1:~/admissioncontr# kubectl get pod -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP               NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          6s    10.244.218.145   k8scludes2   <none>           <none>

刪除pod。

root@k8scludes1:~/admissioncontr# kubectl delete pod podtest 
pod "podtest" deleted

七.總結

准入控制器是Kubernetes中一個非常重要的元件，它負責攔截和處理來自使用者或其他應用程式的API請求。透過使用准入控制器，我們可以實現對叢集資源的細粒度控制，從而提高叢集的安全性和可靠性。

搞懂 Kubernetes 准入控制（Admission Controller）
2021-11-30
Controller
Kubernetes 准入控制 Admission Controller 介紹
2018-09-20
Controller
kubernetes實踐之四十七：ResourceQuota Controller
2018-05-28
Controller
imagepolicywebhook
2024-03-17
WebHook
Kubernetes Admission Controller 簡介 - 注入 sidacar 示例
2023-04-17
Controller
laravel 5 Controller 控制器
2017-04-07
LaravelController
@Controller基本註解的控制器
2017-08-18
Controller
Jmeter之邏輯控制器(Logic Controller)
2016-02-20
JMeterController
手把手教你在容器服務 TKE 中使用動態准入控制器
2021-02-04
.NET/ASP.NET MVC Controller 控制器（深入解析控制器執行原理）
2013-10-27
ASP.NETMVCController
Jmeter邏輯控制器之If Controller的使用解析
2020-08-19
JMeterController
【劉文彬】 Controller：EOS區塊鏈核心控制器
2018-12-07
Controller區塊鏈
.NET/ASP.NET MVC Controller 控制器（IController控制器的建立過程）
2013-11-14
ASP.NETMVCController
Jmeter系列（55）- 詳解 Throughput Controller 吞吐量控制器
2020-08-20
JMeterController
Jmeter-邏輯控制器If Controller的例項運用
2021-06-13
JMeterController
Kafka實戰寶典：Kafka的控制器controller詳解
2020-09-21
KafkaController
第四章 Controller介面控制器詳解（1）
2016-03-21
Controller
第四章 Controller介面控制器詳解（2）
2016-03-21
Controller
既准入和加密
2018-03-17
加密
Yii2 在模組內對映 Controller 到其它控制器
2019-06-20
Controller
深入解析Kubernetes admission webhooks
2022-07-11
WebHook
ADAS域控制器方案介紹（ADAS Domain Controller Unit，ADCU）
2019-06-26
AIController
Laravel 控制器 Controller 傳值到檢視 View 的幾種方法總結
2018-10-16
LaravelControllerView
如何使用 jMeter Parallel Controller - 並行控制器以及一些常犯的錯誤
2021-10-17
JMeterParallelController並行
SAP UI5 檢視控制器 View Controller 的生命週期方法 - Lifecycle methods
2021-10-11
UIViewController
Kubernetes:kube-apiserver 之准入
2023-11-14
APIServer
MySQL准入規範及容量評估
2017-02-25
MySql
Jmeter——ForEach Controller&Loop Controller
2020-11-05
JMeterControllerOOP
kubernetes高階之動態准入控制
2019-06-25
K8s准入控制與RBAC比較
2021-06-12
K8S
Kubernetes筆記（四）：詳解Namespace與資源限制ResourceQuota，LimitRange
2020-06-03
筆記namespaceMIT
Kubernetes可擴充套件Admission進入Beta階段
2018-12-14
套件
Kubernetes1.3：Quota配額管理----在kubernetes的controller-manager模組中有一個ResourceQuotaController控制器，通過這個控制器管理所
2016-11-07
Controller
使用 Admission Webhook 機制實現多叢集資源配額控制
2021-01-12
WebHook
詳解 View Controller
2017-03-13
ViewController
JTAG TAP Controller
2013-07-12
Controller
有關service controller
2003-03-05
Controller
HSI Controller IP Core
2016-01-22
Controller

准入控制器(Admission Controller):ResourceQuota，ImagePolicyWebhook

一.系統環境

二.前言

三.准入控制器簡介

四.為什麼需要准入控制器

五.啟用/禁用ResourceQuota資源配額

5.1 檢視預設啟用/禁用的准入控制器外掛

5.2 ResourceQuota資源配額示例

5.3 禁用ResourceQuota

六.配置ImagePolicyWebhook准入控制器禁止使用字尾為latest的映象

6.1 搭建Webhook伺服器

6.2 配置kubernetes連線後端webhook伺服器

6.3 驗證

七.總結

相關文章