Kubernetes 審計（Auditing）

目錄

• 一.系統環境
• 二.前言
• 三.Kubernetes 審計簡介
• 四.審計策略簡介
• 五.啟用審計
  • 5.1 引入審計
  • 5.2 啟用審計
• 六.審計策略
  • 6.1 記錄審計階段為：ResponseStarted，審計級別為Metadata，apiVersion為group: "" 的日誌
  • 6.2 只記錄audit名稱空間裡的日誌
  • 6.3 只記錄audit名稱空間的pods操作日誌
  • 6.4 只記錄audit名稱空間的pods,services,deployments操作日誌
  • 6.5 只記錄audit名稱空間的pods操作,審計級別為RequestResponse
  • 6.6 只記錄audit名稱空間下的tom使用者的pods操作，其他使用者操作不記錄
  • 6.7 rules規則是從上往下匹配的，第一條規則已經匹配了，第二條就不匹配了
  • 6.8 在 Metadata 級別為所有請求生成日誌
• 七.總結

一.系統環境

本文主要基於Kubernetes1.22.2和Linux作業系統Ubuntu 18.04。

伺服器版本	docker軟體版本	Kubernetes(k8s)叢集版本	CPU架構
Ubuntu 18.04.5 LTS	Docker version 20.10.14	v1.22.2	x86_64

Kubernetes叢集架構：k8scludes1作為master節點，k8scludes2，k8scludes3作為worker節點。

伺服器	作業系統版本	CPU架構	程序	功能描述
k8scludes1/192.168.110.128	Ubuntu 18.04.5 LTS	x86_64	docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico	k8s master節點
k8scludes2/192.168.110.129	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節點
k8scludes3/192.168.110.130	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker節點

二.前言

在當今的雲端計算時代，容器編排技術已成為企業部署和管理應用程式的關鍵組成部分。Kubernetes 作為最流行的容器編排平臺，提供了強大的功能來部署、管理和擴充套件容器化應用程式。然而，隨著應用程式的日益複雜和部署環境的擴大，確保 Kubernetes 叢集的安全性變得越來越重要。在這篇部落格中，我們將深入瞭解 Kubernetes 審計的概念，並探討如何在 Kubernetes 環境中實施審計策略。

使用Kubernetes 審計（Auditing）的前提是已經有一套可以正常執行的Kubernetes叢集，關於Kubernetes(k8s)叢集的安裝部署，可以檢視部落格《Ubuntu 安裝部署Kubernetes(k8s)叢集》https://www.cnblogs.com/renshengdezheli/p/17632858.html。

三.Kubernetes 審計簡介

Kubernetes 審計（Auditing） 功能提供了與安全相關的、按時間順序排列的記錄集， 記錄每個使用者、使用 Kubernetes API 的應用以及控制面自身引發的活動。

審計功能使得叢集管理員能夠回答以下問題：

• 發生了什麼？
• 什麼時候發生的？
• 誰觸發的？
• 活動發生在哪個（些）物件上？
• 在哪觀察到的？
• 它從哪觸發的？
• 活動的後續處理行為是什麼？

審計記錄最初產生於 kube-apiserver 內部。每個請求在不同執行階段都會生成審計事件；這些審計事件會根據特定策略被預處理並寫入後端。 策略確定要記錄的內容和用來儲存記錄的後端，當前的後端支援日誌檔案和 webhook。

每個請求都可被記錄其相關的階段（stage），階段（stage）可以理解為什麼時候記錄。。已定義的階段有：

• RequestReceived - 此階段對應審計處理器接收到請求後， 並且在委託給其餘處理器之前生成的事件。
• ResponseStarted - 在響應訊息的頭部傳送後，響應訊息體傳送前生成的事件。 只有長時間執行的請求（例如 watch）才會生成這個階段。
• ResponseComplete - 當響應訊息體完成並且沒有更多資料需要傳輸的時候。
• Panic - 當 panic 發生時生成。

審計日誌記錄功能會增加 API server 的記憶體消耗，因為需要為每個請求儲存審計所需的某些上下文。 記憶體消耗取決於審計日誌記錄的配置。

Kubernetes 審計是一種監控和記錄 Kubernetes 叢集中資源操作的方法，用於確保叢集的安全性和符合性。透過審計，管理員可以跟蹤對叢集資源的訪問和修改，以便在發生安全事件時進行調查和響應。Kubernetes 提供了審計日誌記錄的框架，允許管理員自定義審計策略，以確定哪些資源操作應該被記錄。

四.審計策略簡介

審計策略定義了關於應記錄哪些事件以及應包含哪些資料的規則。 審計策略物件結構定義在 audit.k8s.io API 組。 處理事件時，將按順序與規則列表進行比較。第一個匹配規則設定事件的審計級別（Audit Level），審計級別（Audit Level）可以理解為記錄什麼？ 已定義的審計級別有：

• None - 符合這條規則的日誌將不會記錄。
• Metadata - 記錄請求的後設資料（請求的使用者、時間戳、資源、動詞等等）， 但是不記錄請求或者響應的訊息體。
• Request - 記錄事件的後設資料和請求的訊息體，但是不記錄響應的訊息體。 這不適用於非資源型別的請求。
• RequestResponse - 記錄事件的後設資料，請求和響應的訊息體。這不適用於非資源型別的請求。

你可以使用 --audit-policy-file 標誌將包含策略的檔案傳遞給 kube-apiserver。 如果不設定該標誌，則不記錄事件。 注意 rules 欄位必須在審計策略檔案中提供。沒有（0）規則的策略將被視為非法配置。

審計策略定義了哪些資源操作應該被審計以及審計記錄的格式。在 Kubernetes 中，審計策略透過 Admission Controllers 實現，可以透過 Webhook 的方式進行整合。審計策略可以根據資源的型別、操作的型別和使用者身份等資訊進行過濾，以滿足不同場景下的審計需求。

五.啟用審計

5.1 引入審計

本次使用etcd2機器作為kubernetes叢集的客戶端，kctom是kubeconfig檔案，etcd2機器使用kctom連線kubernetes叢集，現在使用者tom沒有許可權，需要先授權。kubeconfig檔案在部落格《Kubernetes(k8s)訪問控制：身份認證》裡已經詳細介紹過了，這裡不再贅述。

[root@etcd2 ~]# ls kc* -lh
-rw------- 1 root root 5.5K 5月   5 17:46 kctom

[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
Error from server (Forbidden): nodes is forbidden: User "tom" cannot list resource "nodes" in API group "" at the cluster scope

去kubernetes叢集給tom使用者授予cluster-admin許可權，也可以授予部分許可權。關於授權的詳細資訊，請檢視部落格《Kubernetes(k8s)訪問控制：許可權管理之RBAC授權/鑑權》。

root@k8scludes1:~# kubectl create clusterrolebinding crbindtom --clusterrole=cluster-admin --user=tom
clusterrolebinding.rbac.authorization.k8s.io/crbindtom created

授予cluster-admin許可權之後，客戶端就具有許可權了。

[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 65d   v1.22.2
k8scludes3   Ready    <none>                 65d   v1.22.2

編輯pod配置檔案，表示使用nginx映象建立pod。

root@k8scludes1:~# vim pod.yaml 

root@k8scludes1:~# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #當需要關閉容器時，立即殺死容器而不等待預設的30秒優雅停機時長。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已經存在該映象，則不重新下載；否則從遠端 Docker Hub 下載該映象
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

在admissioncontr名稱空間建立一個pod。

root@k8scludes1:~# kubectl apply -f pod.yaml 
pod/podtest created

在客戶端etcd2機器可以看到這個pod。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n admissioncontr
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          34s

在客戶端刪除pod。

[root@etcd2 ~]# kubectl delete pod podtest --kubeconfig=kctom -n admissioncontr
pod "podtest" deleted

kubectl get ev （檢視事件），可以看到pod被刪除掉了，但是看不到pod被誰刪除掉了！

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
LAST SEEN   TYPE     REASON    OBJECT        SUBOBJECT                  SOURCE                MESSAGE                                                                            FIRST SEEN   COUNT   NAME
3m37s       Normal   Pulled    pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Container image "hub.c.163.com/library/nginx:latest" already present on machine   3m37s        1       podtest.16fab43e14342a3f
3m37s       Normal   Created   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Created container podtest                                                          3m37s        1       podtest.16fab43e1db484f4
3m36s       Normal   Started   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Started container podtest                                                          3m36s        1       podtest.16fab43e3315ada7
2m47s       Normal   Killing   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Stopping container podtest                                                         2m47s        1       podtest.16fab449c468f9af

我們的k8s叢集會被各種使用者連線使用，我們想知道連線k8s叢集的各個使用者做了什麼操作？應該怎麼辦？透過審計解決！

5.2 啟用審計

建立audit目錄存放yaml檔案。

root@k8scludes1:~# mkdir audit

root@k8scludes1:~# cd audit/

root@k8scludes1:~/audit# pwd
/root/audit

建立名稱空間audit。

root@k8scludes1:~/audit# kubectl create ns audit
namespace/audit created

切換到名稱空間audit。

root@k8scludes1:~/audit# kubens audit
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "audit".

root@k8scludes1:~/audit# kubectl get pod 
No resources found in audit namespace.

kubernetes預設並沒有啟用審計，需要啟用審計功能。指定審計策略檔案的路徑：--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml 

root@k8scludes1:~# grep audit-policy-file /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml

建立審計策略檔案，檔案內容可以先為空。

root@k8scludes1:~# mkdir /etc/kubernetes/audit/

root@k8scludes1:~# cd /etc/kubernetes/audit/

root@k8scludes1:/etc/kubernetes/audit# touch audit-policy.yaml

root@k8scludes1:/etc/kubernetes/audit# ls /etc/kubernetes/audit/audit-policy.yaml
/etc/kubernetes/audit/audit-policy.yaml

指定審計日誌存放的路徑：--audit-log-path=/var/log/kubernetes/audit/audit.log 。審計日誌格式為JSONlines 格式 。

root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml

root@k8scludes1:/etc/kubernetes/audit# grep audit-log-path /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log

建立審計日誌檔案。

root@k8scludes1:/etc/kubernetes/audit# mkdir -p /var/log/kubernetes/audit/

root@k8scludes1:/etc/kubernetes/audit# touch /var/log/kubernetes/audit/audit.log

root@k8scludes1:/etc/kubernetes/audit# ls /var/log/kubernetes/audit/audit.log
/var/log/kubernetes/audit/audit.log

審計日誌引數如下：

• --audit-log-path 指定用來寫入審計事件的日誌檔案路徑。不指定此標誌會禁用日誌後端。- 意味著標準化
• --audit-log-maxage 定義保留舊審計日誌檔案的最大天數
• --audit-log-maxbackup 定義要保留的審計日誌檔案的最大數量
• --audit-log-maxsize 定義審計日誌檔案輪轉之前的最大大小（兆位元組）

配置審計日誌的引數。

root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml

root@k8scludes1:/etc/kubernetes/audit# grep audit /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=365
    - --audit-log-maxbackup=730
    - --audit-log-maxsize=250

如果你的叢集控制面以 Pod 的形式執行 kube-apiserver，需要透過 hostPath 資料捲來訪問策略檔案和日誌檔案所在的目錄，這樣審計記錄才會持久儲存下來。

現在/etc/kubernetes/audit/audit-policy.yaml和/var/log/kubernetes/audit/audit.log在宿主機裡是存在的，但是kube-apiserver是以pod的方式執行的，pod裡並不存在這兩個檔案，需要透過hostPath 資料捲進行對映。

/etc/kubernetes/manifests/kube-apiserver.yaml所有的修改如下截圖：

注意：/var/log/kubernetes/audit/的readOnly為false，否則不能寫入日誌！

自此，成功啟用審計了，下面開始配置審計策略。

六.審計策略

6.1 記錄審計階段為：ResponseStarted，審計級別為Metadata，apiVersion為group: "" 的日誌

先不用重啟kubelet，因為審計策略檔案還沒寫好。

現在編寫審計策略檔案，該審計策略表示什麼都不記錄。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: None
    resources:
    - group: ""

重啟kubelet使配置生效。

root@k8scludes1:~# systemctl restart kubelet

root@k8scludes1:~# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 66d   v1.22.2
k8scludes3   Ready    <none>                 66d   v1.22.2

在客戶端etcd2機器執行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide。

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

檢視審計日誌，沒有任何日誌記錄。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log 

審計級別修改為Metadata，表示記錄請求的後設資料（請求的使用者、時間戳、資源、動詞等等）， 但是不記錄請求或者響應的訊息體。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""

重啟kubelet使配置生效。

root@k8scludes1:~# systemctl daemon-reload ; systemctl restart kubelet

在客戶端etcd2機器執行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide 。

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

檢視審計日誌，發現還是沒有日誌產生，看來重啟kubelet還不行，現在重啟機器。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log

現在直接重啟k8scludes1機器，k8s worker節點不用重啟。

root@k8scludes1:~# reboot

重啟k8scludes1機器之後，現在/var/log/kubernetes/audit/audit.log裡有很多資料，先刪除資料，然後客戶端執行命令，檢視審計日誌。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
3317

清空日誌。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

客戶端執行kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide命令。

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

現在不停的產生日誌。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
32

這是最新的日誌。

root@k8scludes1:~# tail -1 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"72736d33-508f-4279-89a5-d95aea7b98c5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader","verb":"update","user":{"username":"system:serviceaccount:ingress-nginx:ingress-nginx","uid":"70a36f2c-225f-450d-849e-2432db224f40","groups":["system:serviceaccounts","system:serviceaccounts:ingress-nginx","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["ingress-nginx-controller-684bbc4b45-jmpxk"],"authentication.kubernetes.io/pod-uid":["308fdf30-854b-49f7-b041-4e5d770c0667"]}},"sourceIPs":["192.168.110.130"],"userAgent":"nginx-ingress-controller/v1.0.0 (linux/amd64) ingress-nginx/041eb167c7bfccb1d1653f194924b0c5fd885e10","objectRef":{"resource":"configmaps","namespace":"ingress-nginx","name":"ingress-controller-leader","uid":"db625974-8e33-4bd6-a2e2-1816301af942","apiVersion":"v1","resourceVersion":"2739119"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:03:36.751232Z","stageTimestamp":"2022-06-22T13:03:36.759442Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ingress-nginx/ingress-nginx\" of Role \"ingress-nginx\" to ServiceAccount \"ingress-nginx/ingress-nginx\"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"}}

客戶端不執行命令就產生了太多審計日誌了，需要修改審計策略。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
1174

6.2 只記錄audit名稱空間裡的日誌

修改審計策略，現在配置只記錄某個名稱空間裡的審計日誌，namespaces: ["audit"]表示只記錄audit名稱空間裡的日誌。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
    namespaces: ["audit"]

重啟kubelet使其生效，但是審計策略沒有生效，重啟機器。

root@k8scludes1:~# systemctl restart kubelet

重啟k8s master節點。

root@k8scludes1:~# reboot

root@k8scludes1:~# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 66d   v1.22.2
k8scludes3   Ready    <none>                 66d   v1.22.2

重啟k8s master節點之後，清空日誌。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

客戶端執行kubectl get pod --kubeconfig=kctom -n default -o wide。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n default -o wide
No resources found in default namespace.

客戶端執行 kubectl get pod --kubeconfig=kctom -n audit -o wide。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

檢視審計日誌，只記錄了audit名稱空間的操作，default名稱空間的操作沒有記錄。

root@k8scludes1:~# tail -2 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.732751Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.741665Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}

6.3 只記錄audit名稱空間的pods操作日誌

修改審計策略，該審計策略表示只記錄audit名稱空間的pods操作。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重啟機器使審計策略生效。

root@k8scludes1:~# reboot

重啟之後，清空日誌。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

在客戶端執行kubectl get pod --kubeconfig=kctom -n audit -o wide。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

在客戶端執行 kubectl get svc --kubeconfig=kctom -n audit -o wide 。

[root@etcd2 ~]# kubectl get svc --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

檢視日誌，發現只記錄了audit名稱空間的pod操作，svc操作沒有記錄。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.351036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.434147Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}

6.4 只記錄audit名稱空間的pods,services,deployments操作日誌

編輯審計策略檔案，表示只記錄audit名稱空間的pods,services,deployments操作，因為deployments的apiVersion的父級為apps，所以需要group: "apps" 。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods","services"]
    - group: "apps"
      resources: ["deployments"]
    namespaces: ["audit"]

6.5 只記錄audit名稱空間的pods操作,審計級別為RequestResponse

編輯審計策略檔案，表示只記錄audit名稱空間的pods操作,審計級別為RequestResponse，記錄事件的後設資料，請求和響應的訊息體。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重啟機器使審計策略生效。

root@k8scludes1:~# reboot

建立一個pod。

root@k8scludes1:~# kubectl apply -f pod.yaml 
pod/podtest created

root@k8scludes1:~# kubectl get pod
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          5s

清空審計日誌。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

客戶端執行kubectl get pod --kubeconfig=kctom -n audit -o wide 。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          48s   10.244.1.97   k8scludes3   <none>           <none>

檢視審計日誌，記錄的內容很豐富，後設資料，請求和響應的訊息體都顯示出來了。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"d6836f9b-4918-49f2-80dc-ad02e246949b","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.782241Z"}
......
23T00:42:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PodScheduled\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.97\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]}}}]},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.785883Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}

6.6 只記錄audit名稱空間下的tom使用者的pods操作，其他使用者操作不記錄

編輯審計策略檔案，設定審計策略：只記錄audit名稱空間下的tom使用者的pods操作，其他使用者的操作不記錄。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    users: ["tom"]
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重啟機器使審計策略生效。

root@k8scludes1:~# reboot 

清空日誌。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

使用管理員使用者檢視pod。

root@k8scludes1:~# kubectl get pod -n audit
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          29m

使用tom使用者在客戶端檢視pod 。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          29m   10.244.1.97   k8scludes3   <none>           <none>

檢視日誌，可以發現只記錄了tom使用者的pod查詢操作。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.103036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.107872Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}

6.7 rules規則是從上往下匹配的，第一條規則已經匹配了，第二條就不匹配了

編輯審計策略檔案，有兩條策略，一條是：apiVersion為group: ""的操作不記錄日誌，另外一條是：只記錄audit名稱空間下tom使用者的pod操作。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  #apiVersion為group: ""的操作不記錄日誌
  - level: None
    resources:
    - group: ""
  #只記錄audit名稱空間下tom使用者的pod操作
  - level: Metadata
    users: ["tom"]
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重啟機器使配置生效。

root@k8scludes1:~# reboot 

清空日誌。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log 

使用管理員使用者查詢audit名稱空間的pod。

root@k8scludes1:~# kubectl get pod -n audit
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          37m

使用tom使用者在客戶端機器etcd2上查詢audit名稱空間的pod。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          37m   10.244.1.97   k8scludes3   <none>           <none>

檢視審計日誌，可以發現在客戶端使用tom使用者查詢pod和使用管理員使用者檢視pod都沒有生成審計日誌，rules規則是從上往下匹配的，第一條規則已經匹配了，第二條就不匹配了。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log

6.8 在 Metadata 級別為所有請求生成日誌

編輯審計策略檔案，在 Metadata 級別為所有請求生成日誌。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 

root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata

七.總結

在 Kubernetes 1.22.2 環境中實施審計策略可以幫助管理員監控和記錄叢集中的資源操作，確保叢集的安全性和符合性。透過啟用審計 Admission Controller 和配置相應的審計策略，我們可以靈活地控制審計記錄的格式和範圍。