深入解析Kubernetes admission webhooks

Cylon發表於2022-07-11
WebHook

BACKGROUND

admission controllers的特點：

  • 可定製性：准入功能可針對不同的場景進行調整。
  • 可預防性：審計則是為了檢測問題，而准入控制器可以預防問題發生
  • 可擴充套件性：在kubernetes自有的驗證機制外，增加了另外的防線，彌補了RBAC僅能對資源提供安全保證。

下圖，顯示了使用者操作資源的流程，可以看出 admission controllers 作用是在通過身份驗證資源持久化之前起到攔截作用。在准入控制器的加入會使kubernetes增加了更高階的安全功能。

image

圖：Kubernetes API 請求的請求處理步驟圖
Source：https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

這裡找到一個大佬部落格畫的圖，通過兩張圖可以很清晰的瞭解到admission webhook流程，與官方給出的不一樣的地方在於，這裡清楚地定位了kubernetes admission webhook 處於准入控制中，RBAC之後，push 之前。

image

圖：Kubernetes API 請求的請求處理步驟圖（詳細）
Source：https://www.armosec.io/blog/kubernetes-admission-controller/

兩種控制器有什麼區別？

根據官方提供的說法是

Mutating controllers may modify related objects to the requests they admit; validating controllers may not

從結構圖中也可以看出，validating 是在持久化之前，而 Mutating 是在結構驗證前，根據這些特性我們可以使用 Mutating 修改這個資源物件內容（如增加驗證的資訊），在 validating 中驗證是否合法。

composition of admission controllers

kubernetes中的 admission controllers 由兩部分組成：

  • 內建在APIServer中的准入控制器 build-in list
  • 特殊的控制器；也是內建在APIServer中，但提供一些自定義的功能
    • MutatingAdmission
    • ValidatingAdmission

Mutating 控制器可以修改他們處理的資源物件，Validating 控制器不會。當在任何一個階段中的任何控制器拒絕這個了請求，則會立即拒絕整個請求，並將錯誤返回。

admission webhook

由於准入控制器是內建在 kube-apiserver 中的，這種情況下就限制了admission controller的可擴充套件性。在這種背景下，kubernetes提供了一種可擴充套件的准入控制器 extensible admission controllers，這種行為叫做動態准入控制 Dynamic Admission Control，而提供這個功能的就是 admission webhook 。

admission webhook 通俗來講就是 HTTP 回撥，通過定義一個http server，接收准入請求並處理。使用者可以通過kubernetes提供的兩種型別的 admission webhook，validating admission webhook 和 mutating admission webhook。來完成自定義的准入策略的處理。

webhook 就是

注：從上面的流程圖也可以看出，admission webhook 也是有順序的。首先呼叫mutating webhook，然後會呼叫validating webhook。

如何使用准入控制器

使用條件：kubernetes v1.16 使用 admissionregistration.k8s.io/v1 ；kubernetes v1.9 使用 admissionregistration.k8s.io/v1beta1。

如何在叢集中開啟准入控制器? ：檢視kube-apiserver 的啟動引數 --enable-admission-plugins ；通過該引數來配置要啟動的准入控制器，如 --enable-admission-plugins=NodeRestriction 多個准入控制器以 , 分割，順序無關緊要。 反之使用 --disable-admission-plugins 引數可以關閉相應的准入控制器（Refer to apiserver opts）。

通過 kubectl 命令可以看到，當前kubernetes叢集所支援准入控制器的版本

$ kubectl api-versions | grep admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1

webhook工作原理

通過上面的學習，已經瞭解到了兩種webhook的工作原理如下所示：

mutating webhook，會在持久化前攔截在 MutatingWebhookConfiguration 中定義的規則匹配的請求。MutatingAdmissionWebhook 通過向 mutating webhook 伺服器傳送准入請求來執行驗證。

validaing webhook，會在持久化前攔截在 ValidatingWebhookConfiguration 中定義的規則匹配的請求。ValidatingAdmissionWebhook 通過將准入請求傳送到 validating webhook server來執行驗證。

那麼接下來將從原始碼中看這個在這個工作流程中，究竟做了些什麼？

資源型別

對於 1.9 版本之後，也就是 v1 版本 ，admission 被定義在 k8s.io\api\admissionregistration\v1\types.go ，大同小異，因為本地只有1.18叢集，所以以這個講解。

對於 Validating Webhook 來講實現主要都在webhook中

type ValidatingWebhookConfiguration struct {
    // 每個api必須包含下列的metadata，這個是kubernetes規範，可以在註釋中的url看到相關文件
	metav1.TypeMeta `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
	// Webhooks在這裡被表示為[]ValidatingWebhook，表示我們可以註冊多個
	// +optional
	// +patchMergeKey=name
	// +patchStrategy=merge
	Webhooks []ValidatingWebhook `json:"webhooks,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,2,rep,name=Webhooks"`
}

webhook，則是對這種型別的webhook提供的操作、資源等。對於這部分不做過多的註釋了，因為這裡本身為kubernetes API資源，官網有很詳細的例子與說明。這裡更多欄位的意思的可以參考官方 doc

type ValidatingWebhook struct {
	//  admission webhook的名詞，Required
	Name string `json:"name" protobuf:"bytes,1,opt,name=name"`

	// ClientConfig 定義了與webhook通訊的方式 Required
	ClientConfig WebhookClientConfig `json:"clientConfig" protobuf:"bytes,2,opt,name=clientConfig"`

	// rule表示了webhook對於哪些資源及子資源的操作進行關注
	Rules []RuleWithOperations `json:"rules,omitempty" protobuf:"bytes,3,rep,name=rules"`

	// FailurePolicy 對於無法識別的value將如何處理，allowed/Ignore optional
	FailurePolicy *FailurePolicyType `json:"failurePolicy,omitempty" protobuf:"bytes,4,opt,name=failurePolicy,casttype=FailurePolicyType"`

	// matchPolicy 定義瞭如何使用“rules”列表來匹配傳入的請求。
	MatchPolicy *MatchPolicyType `json:"matchPolicy,omitempty" protobuf:"bytes,9,opt,name=matchPolicy,casttype=MatchPolicyType"`
	NamespaceSelector *metav1.LabelSelector `json:"namespaceSelector,omitempty" protobuf:"bytes,5,opt,name=namespaceSelector"`
	SideEffects *SideEffectClass `json:"sideEffects" protobuf:"bytes,6,opt,name=sideEffects,casttype=SideEffectClass"`
	AdmissionReviewVersions []string `json:"admissionReviewVersions" protobuf:"bytes,8,rep,name=admissionReviewVersions"`
}

到這裡瞭解了一個webhook資源的定義，那麼這個如何使用呢？通過 Find Usages 找到一個 k8s.io/apiserver/pkg/admission/plugin/webhook/accessors.go 在使用它。這裡沒有註釋，但在結構上可以看出，包含客戶端與一系列選擇器組成

type mutatingWebhookAccessor struct {
	*v1.MutatingWebhook
	uid               string
	configurationName string

	initObjectSelector sync.Once
	objectSelector     labels.Selector
	objectSelectorErr  error

	initNamespaceSelector sync.Once
	namespaceSelector     labels.Selector
	namespaceSelectorErr  error

	initClient sync.Once
	client     *rest.RESTClient
	clientErr  error
}

accessor 因為包含了整個webhookconfig定義的一些動作（這裡個人這麼覺得）。

accessor.go 下面 有一個 GetRESTClient 方法 ，通過這裡可以看出，這裡做的就是使用根據 accessor 構造一個客戶端。

func (m *mutatingWebhookAccessor) GetRESTClient(clientManager *webhookutil.ClientManager) (*rest.RESTClient, error) {
	m.initClient.Do(func() {
		m.client, m.clientErr = clientManager.HookClient(hookClientConfigForWebhook(m))
	})
	return m.client, m.clientErr
}

到這步驟已經沒必要往下看了，因已經知道這裡是請求webhook前的步驟了，下面就是何時請求了。

k8s.io\apiserver\pkg\admission\plugin\webhook\validating\dispatcher.go 下面有兩個方法，Dispatch去請求我們自己定義的webhook

func (d *validatingDispatcher) Dispatch(ctx context.Context, attr admission.Attributes, o admission.ObjectInterfaces, hooks []webhook.WebhookAccessor) error {
	var relevantHooks []*generic.WebhookInvocation
	// Construct all the versions we need to call our webhooks
	versionedAttrs := map[schema.GroupVersionKind]*generic.VersionedAttributes{}
	for _, hook := range hooks {
		invocation, statusError := d.plugin.ShouldCallHook(hook, attr, o)
		if statusError != nil {
			return statusError
		}
		if invocation == nil {
			continue
		}
		relevantHooks = append(relevantHooks, invocation)
		// If we already have this version, continue
		if _, ok := versionedAttrs[invocation.Kind]; ok {
			continue
		}
		versionedAttr, err := generic.NewVersionedAttributes(attr, invocation.Kind, o)
		if err != nil {
			return apierrors.NewInternalError(err)
		}
		versionedAttrs[invocation.Kind] = versionedAttr
	}

	if len(relevantHooks) == 0 {
		// no matching hooks
		return nil
	}

	// Check if the request has already timed out before spawning remote calls
	select {
	case <-ctx.Done():
		// parent context is canceled or timed out, no point in continuing
		return apierrors.NewTimeoutError("request did not complete within requested timeout", 0)
	default:
	}

	wg := sync.WaitGroup{}
	errCh := make(chan error, len(relevantHooks))
	wg.Add(len(relevantHooks))
    // 迴圈所有相關的註冊的hook
	for i := range relevantHooks {
		go func(invocation *generic.WebhookInvocation) {
			defer wg.Done()
            // invacation 中有一個 Accessor,Accessor註冊了一個相關的webhookconfig
            // 也就是我們 kubectl -f 註冊進來的那個webhook的相關配置
			hook, ok := invocation.Webhook.GetValidatingWebhook()
			if !ok {
				utilruntime.HandleError(fmt.Errorf("validating webhook dispatch requires v1.ValidatingWebhook, but got %T", hook))
				return
			}
			versionedAttr := versionedAttrs[invocation.Kind]
			t := time.Now()
            // 呼叫了callHook去請求我們自定義的webhook
			err := d.callHook(ctx, hook, invocation, versionedAttr)
			ignoreClientCallFailures := hook.FailurePolicy != nil && *hook.FailurePolicy == v1.Ignore
			rejected := false
			if err != nil {
				switch err := err.(type) {
				case *webhookutil.ErrCallingWebhook:
					if !ignoreClientCallFailures {
						rejected = true
						admissionmetrics.Metrics.ObserveWebhookRejection(hook.Name, "validating", string(versionedAttr.Attributes.GetOperation()), admissionmetrics.WebhookRejectionCallingWebhookError, 0)
					}
				case *webhookutil.ErrWebhookRejection:
					rejected = true
					admissionmetrics.Metrics.ObserveWebhookRejection(hook.Name, "validating", string(versionedAttr.Attributes.GetOperation()), admissionmetrics.WebhookRejectionNoError, int(err.Status.ErrStatus.Code))
				default:
					rejected = true
					admissionmetrics.Metrics.ObserveWebhookRejection(hook.Name, "validating", string(versionedAttr.Attributes.GetOperation()), admissionmetrics.WebhookRejectionAPIServerInternalError, 0)
				}
			}
			admissionmetrics.Metrics.ObserveWebhook(time.Since(t), rejected, versionedAttr.Attributes, "validating", hook.Name)
			if err == nil {
				return
			}

			if callErr, ok := err.(*webhookutil.ErrCallingWebhook); ok {
				if ignoreClientCallFailures {
					klog.Warningf("Failed calling webhook, failing open %v: %v", hook.Name, callErr)
					utilruntime.HandleError(callErr)
					return
				}

				klog.Warningf("Failed calling webhook, failing closed %v: %v", hook.Name, err)
				errCh <- apierrors.NewInternalError(err)
				return
			}

			if rejectionErr, ok := err.(*webhookutil.ErrWebhookRejection); ok {
				err = rejectionErr.Status
			}
			klog.Warningf("rejected by webhook %q: %#v", hook.Name, err)
			errCh <- err
		}(relevantHooks[i])
	}
	wg.Wait()
	close(errCh)

	var errs []error
	for e := range errCh {
		errs = append(errs, e)
	}
	if len(errs) == 0 {
		return nil
	}
	if len(errs) > 1 {
		for i := 1; i < len(errs); i++ {
			// TODO: merge status errors; until then, just return the first one.
			utilruntime.HandleError(errs[i])
		}
	}
	return errs[0]
}

callHook 可以理解為真正去請求我們自定義的webhook服務的動作

func (d *validatingDispatcher) callHook(ctx context.Context, h *v1.ValidatingWebhook, invocation *generic.WebhookInvocation, attr *generic.VersionedAttributes) error {
   if attr.Attributes.IsDryRun() {
      if h.SideEffects == nil {
         return &webhookutil.ErrCallingWebhook{WebhookName: h.Name, Reason: fmt.Errorf("Webhook SideEffects is nil")}
      }
      if !(*h.SideEffects == v1.SideEffectClassNone || *h.SideEffects == v1.SideEffectClassNoneOnDryRun) {
         return webhookerrors.NewDryRunUnsupportedErr(h.Name)
      }
   }

   uid, request, response, err := webhookrequest.CreateAdmissionObjects(attr, invocation)
   if err != nil {
      return &webhookutil.ErrCallingWebhook{WebhookName: h.Name, Reason: err}
   }
   // 發生請求，可以看到，這裡從上面的講到的地方獲取了一個客戶端
   client, err := invocation.Webhook.GetRESTClient(d.cm)
   if err != nil {
      return &webhookutil.ErrCallingWebhook{WebhookName: h.Name, Reason: err}
   }
   trace := utiltrace.New("Call validating webhook",
      utiltrace.Field{"configuration", invocation.Webhook.GetConfigurationName()},
      utiltrace.Field{"webhook", h.Name},
      utiltrace.Field{"resource", attr.GetResource()},
      utiltrace.Field{"subresource", attr.GetSubresource()},
      utiltrace.Field{"operation", attr.GetOperation()},
      utiltrace.Field{"UID", uid})
   defer trace.LogIfLong(500 * time.Millisecond)

   // 這裡設定超時，超時時長就是在yaml資源清單中設定的那個值
   if h.TimeoutSeconds != nil {
      var cancel context.CancelFunc
      ctx, cancel = context.WithTimeout(ctx, time.Duration(*h.TimeoutSeconds)*time.Second)
      defer cancel()
   }
   // 直接用post請求我們自己定義的webhook介面
   r := client.Post().Body(request)

   // if the context has a deadline, set it as a parameter to inform the backend
   if deadline, hasDeadline := ctx.Deadline(); hasDeadline {
      // compute the timeout
      if timeout := time.Until(deadline); timeout > 0 {
         // if it's not an even number of seconds, round up to the nearest second
         if truncated := timeout.Truncate(time.Second); truncated != timeout {
            timeout = truncated + time.Second
         }
         // set the timeout
         r.Timeout(timeout)
      }
   }

   if err := r.Do(ctx).Into(response); err != nil {
      return &webhookutil.ErrCallingWebhook{WebhookName: h.Name, Reason: err}
   }
   trace.Step("Request completed")

   result, err := webhookrequest.VerifyAdmissionResponse(uid, false, response)
   if err != nil {
      return &webhookutil.ErrCallingWebhook{WebhookName: h.Name, Reason: err}
   }

   for k, v := range result.AuditAnnotations {
      key := h.Name + "/" + k
      if err := attr.Attributes.AddAnnotation(key, v); err != nil {
         klog.Warningf("Failed to set admission audit annotation %s to %s for validating webhook %s: %v", key, v, h.Name, err)
      }
   }
   if result.Allowed {
      return nil
   }
   return &webhookutil.ErrWebhookRejection{Status: webhookerrors.ToStatusErr(h.Name, result.Result)}
}

走到這裡基本上對 admission webhook 有了大致的瞭解，可以知道這個操作是由 apiserver 完成的。下面就實際操作下自定義一個webhook。

這裡還有兩個概念，就是請求引數 AdmissionRequest 和相應引數 AdmissionResponse，這些可以在 callHook 中看到，這兩個引數被定義在 k8s.io\api\admission\v1\types.go ；這兩個引數也就是我們在自定義 webhook 時需要處理接收到的body的結構，以及我們響應內容資料結構。

如何編寫一個自定義的admission webhook

通過上面的學習瞭解到了，自定義的webhook就是做為kubernetes提供給使用者兩種admission controller來驗證自定義業務的一箇中介軟體 admission webhook。本質上他是一個HTTP Server，使用者可以使用任何語言來完成這部分功能。當然，如果涉及到需要對kubernetes叢集資源操作的話，還是建議使用kubernetes官方提供了SDK的程式語言來完成自定義的webhook。

那麼完成一個自定義admission webhook需要兩個步驟：

  • 將相關的webhook config註冊給kubernetes，也就是讓kubernetes知道你的webhook
  • 準備一個http server來處理 apiserver發過來驗證的資訊

注：這裡使用go net/http包，本身不區分方法處理HTTP的何種請求，如果用其他框架實現的，如django，需要指定對應方法需要為POST

向kubernetes註冊webhook物件

kubernetes提供的兩種型別可自定義的准入控制器，和其他資源一樣，可以利用資源清單，動態配置那些資源要被adminssion webhook處理。 kubernetes將這種形式抽象為兩種資源：

  • ValidatingWebhookConfiguration

  • MutatingWebhookConfiguration

ValidatingAdmission

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "pod-policy.example.com"
webhooks:
- name: "pod-policy.example.com"
  rules:
  - apiGroups:   [""] # 攔截資源的Group "" 表示 core。"*" 表示所有。
    apiVersions: ["v1"] # 攔截資源的版本
    operations:  ["CREATE"] # 什麼請求下攔截
    resources:   ["pods"]  # 攔截什麼資源
    scope:       "Namespaced" # 生效的範圍，cluster還是namespace "*"表示沒有範圍限制。
  clientConfig: # 我們部署的webhook服務，
    service: # service是在cluster-in模式下
      namespace: "example-namespace"
      name: "example-service"
      port: 443 # 服務的埠
      path: "/validate" # path是對應用於驗證的介面
    # caBundle是提供給 admission webhook CA證書  
    caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle containing the CA that signed the webhook's serving certificate>...tLS0K"
  admissionReviewVersions: ["v1", "v1beta1"]
  sideEffects: None
  timeoutSeconds: 5 # 1-30s直接，表示請求api的超時時間

MutatingAdmission

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "valipod-policy.example.com"
webhooks:
- name: "valipod-policy.example.com"
  rules:
    - apiGroups:   ["apps"] # 攔截資源的Group "" 表示 core。"*" 表示所有。
      apiVersions: ["v1"] # 攔截資源的版本
      operations:  ["CREATE"] # 什麼請求下攔截
      resources:   ["deployments"]  # 攔截什麼資源
      scope:       "Namespaced" # 生效的範圍，cluster還是namespace "*"表示沒有範圍限制。
  clientConfig: # 我們部署的webhook服務，
    url: "https://10.0.0.1:81/validate" # 這裡是外部模式
    #      service: # service是在cluster-in模式下
    #        namespace: "default"
    #        name: "admission-webhook"
    #        port: 81 # 服務的埠
    #        path: "/mutate" # path是對應用於驗證的介面
    # caBundle是提供給 admission webhook CA證書
    caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle containing the CA that signed the webhook's serving certificate>...tLS0K"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5 # 1-30s直接，表示請求api的超時時間

注：對於webhook，也可以引入外部的服務，並非必須部署到叢集內部

對於外部服務來講，需要 clientConfig 中的 service , 更換為 url ; 通過 url 引數可以將一個外部的服務引入

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
...
webhooks:
- name: my-webhook.example.com
  clientConfig:
    url: "https://my-webhook.example.com:9443/my-webhook-path"
  ...

注：這裡的url規則必須準守下列形式：

  • scheme://host:port/path
  • 使用了url 時，這裡不應填寫叢集內的服務
  • scheme 必須是 https，不能為http，這就意味著，引入外部時也需要
  • 配置時使用了，?xx=xx 的引數也是不被允許的（官方說法是這樣的，通過原始碼學習瞭解到因為會傳送特定的請求體，所以無需管引數）

更多的配置可以參考kubernetes官方提供的 doc

準備一個webhook

讓我們編寫我們的 webhook server。將建立兩個鉤子，/mutate 與 /validate；

  • /mutate 將在建立deployment資源時，基於版本，給資源加上註釋 webhook.example.com/allow: true
  • /validate 將對 /mutate 增加的 allow:true 那麼則繼續，否則拒絕。

這裡為了方便，全部寫在一起了，實際上不符合軟體的設計。在kubernetes程式碼庫中也提供了一個webhook server，可以參考他這個webhook server來學習具體要做什麼

package main

import (
	"context"
	"crypto/tls"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"os"
	"os/signal"
	"strings"
	"syscall"

	v1admission "k8s.io/api/admission/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/serializer"

	appv1 "k8s.io/api/apps/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/klog"
)

type patch struct {
	Op    string            `json:"op"`
	Path  string            `json:"path"`
	Value map[string]string `json:"value"`
}

func serve(w http.ResponseWriter, r *http.Request) {

	var body []byte
	if data, err := ioutil.ReadAll(r.Body); err == nil {
		body = data
	}
	klog.Infof(fmt.Sprintf("receive request: %v....", string(body)[:130]))
	if len(body) == 0 {
		klog.Error(fmt.Sprintf("admission request body is empty"))
		http.Error(w, fmt.Errorf("admission request body is empty").Error(), http.StatusBadRequest)
		return
	}
	var admission v1admission.AdmissionReview
	codefc := serializer.NewCodecFactory(runtime.NewScheme())
	decoder := codefc.UniversalDeserializer()
	_, _, err := decoder.Decode(body, nil, &admission)

	if err != nil {
		msg := fmt.Sprintf("Request could not be decoded: %v", err)
		klog.Error(msg)
		http.Error(w, msg, http.StatusBadRequest)
		return
	}

	if admission.Request == nil {
		klog.Error(fmt.Sprintf("admission review can't be used: Request field is nil"))
		http.Error(w, fmt.Errorf("admission review can't be used: Request field is nil").Error(), http.StatusBadRequest)
		return
	}

	switch strings.Split(r.RequestURI, "?")[0] {
	case "/mutate":
		req := admission.Request
		var admissionResp v1admission.AdmissionReview
		admissionResp.APIVersion = admission.APIVersion
		admissionResp.Kind = admission.Kind
		klog.Infof("AdmissionReview for Kind=%v, Namespace=%v Name=%v UID=%v Operation=%v",
			req.Kind.Kind, req.Namespace, req.Name, req.UID, req.Operation)
		switch req.Kind.Kind {
		case "Deployment":
			var (
				respstr []byte
				err     error
				deploy  appv1.Deployment
			)
			if err = json.Unmarshal(req.Object.Raw, &deploy); err != nil {
				respStructure := v1admission.AdmissionResponse{Result: &metav1.Status{
					Message: fmt.Sprintf("could not unmarshal resouces review request: %v", err),
					Code:    http.StatusInternalServerError,
				}}
				klog.Error(fmt.Sprintf("could not unmarshal resouces review request: %v", err))
				if respstr, err = json.Marshal(respStructure); err != nil {
					klog.Error(fmt.Errorf("could not unmarshal resouces review response: %v", err))
					http.Error(w, fmt.Errorf("could not unmarshal resouces review response: %v", err).Error(), http.StatusInternalServerError)
					return
				}
				http.Error(w, string(respstr), http.StatusBadRequest)
				return
			}

			current_annotations := deploy.GetAnnotations()
			pl := []patch{}
			for k, v := range current_annotations {
				pl = append(pl, patch{
					Op:   "add",
					Path: "/metadata/annotations",
					Value: map[string]string{
						k: v,
					},
				})
			}
			pl = append(pl, patch{
				Op:   "add",
				Path: "/metadata/annotations",
				Value: map[string]string{
					deploy.Name + "/Allow": "true",
				},
			})

			annotationbyte, err := json.Marshal(pl)

			if err != nil {
				http.Error(w, err.Error(), http.StatusInternalServerError)
				return
			}
			respStructure := &v1admission.AdmissionResponse{
				UID:     req.UID,
				Allowed: true,
				Patch:   annotationbyte,
				PatchType: func() *v1admission.PatchType {
					t := v1admission.PatchTypeJSONPatch
					return &t
				}(),
				Result: &metav1.Status{
					Message: fmt.Sprintf("could not unmarshal resouces review request: %v", err),
					Code:    http.StatusOK,
				},
			}
			admissionResp.Response = respStructure

			klog.Infof("sending response: %s....", admissionResp.Response.String()[:130])
			respByte, err := json.Marshal(admissionResp)
			if err != nil {
				klog.Errorf("Can't encode response messages: %v", err)
				http.Error(w, err.Error(), http.StatusInternalServerError)
			}
			klog.Infof("prepare to write response...")
			w.Header().Set("Content-Type", "application/json")
			if _, err := w.Write(respByte); err != nil {
				klog.Errorf("Can't write response: %v", err)
				http.Error(w, fmt.Sprintf("could not write response: %v", err), http.StatusInternalServerError)
			}

		default:
			klog.Error(fmt.Sprintf("unsupport resouces review request type"))
			http.Error(w, "unsupport resouces review request type", http.StatusBadRequest)
		}

	case "/validate":
		req := admission.Request
		var admissionResp v1admission.AdmissionReview
		admissionResp.APIVersion = admission.APIVersion
		admissionResp.Kind = admission.Kind
		klog.Infof("AdmissionReview for Kind=%v, Namespace=%v Name=%v UID=%v Operation=%v",
			req.Kind.Kind, req.Namespace, req.Name, req.UID, req.Operation)
		var (
			deploy  appv1.Deployment
			respstr []byte
		)
		switch req.Kind.Kind {
		case "Deployment":
			if err = json.Unmarshal(req.Object.Raw, &deploy); err != nil {
				respStructure := v1admission.AdmissionResponse{Result: &metav1.Status{
					Message: fmt.Sprintf("could not unmarshal resouces review request: %v", err),
					Code:    http.StatusInternalServerError,
				}}
				klog.Error(fmt.Sprintf("could not unmarshal resouces review request: %v", err))
				if respstr, err = json.Marshal(respStructure); err != nil {
					klog.Error(fmt.Errorf("could not unmarshal resouces review response: %v", err))
					http.Error(w, fmt.Errorf("could not unmarshal resouces review response: %v", err).Error(), http.StatusInternalServerError)
					return
				}
				http.Error(w, string(respstr), http.StatusBadRequest)
				return
			}
		}
		al := deploy.GetAnnotations()
		respStructure := v1admission.AdmissionResponse{
			UID: req.UID,
		}
		if al[fmt.Sprintf("%s/Allow", deploy.Name)] == "true" {
			respStructure.Allowed = true
			respStructure.Result = &metav1.Status{
				Code: http.StatusOK,
			}
		} else {
			respStructure.Allowed = false
			respStructure.Result = &metav1.Status{
				Code: http.StatusForbidden,
				Reason: func() metav1.StatusReason {
					return metav1.StatusReasonForbidden
				}(),
				Message: fmt.Sprintf("the resource %s couldn't to allow entry.", deploy.Kind),
			}
		}

		admissionResp.Response = &respStructure

		klog.Infof("sending response: %s....", admissionResp.Response.String()[:130])
		respByte, err := json.Marshal(admissionResp)
		if err != nil {
			klog.Errorf("Can't encode response messages: %v", err)
			http.Error(w, err.Error(), http.StatusInternalServerError)
		}
		klog.Infof("prepare to write response...")
		w.Header().Set("Content-Type", "application/json")
		if _, err := w.Write(respByte); err != nil {
			klog.Errorf("Can't write response: %v", err)
			http.Error(w, fmt.Sprintf("could not write response: %v", err), http.StatusInternalServerError)
		}
	}
}

func main() {
	var (
		cert, key string
	)

	if cert = os.Getenv("TLS_CERT"); len(cert) == 0 {
		cert = "./tls/tls.crt"
	}

	if key = os.Getenv("TLS_KEY"); len(key) == 0 {
		key = "./tls/tls.key"
	}

	ca, err := tls.LoadX509KeyPair(cert, key)
	if err != nil {
		klog.Error(err.Error())
		return
	}

	server := &http.Server{
		Addr: ":81",
		TLSConfig: &tls.Config{
			Certificates: []tls.Certificate{
				ca,
			},
		},
	}

	httpserver := http.NewServeMux()

	httpserver.HandleFunc("/validate", serve)
	httpserver.HandleFunc("/mutate", serve)
	httpserver.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
		klog.Info(fmt.Sprintf("%s %s", r.RequestURI, "pong"))
		fmt.Fprint(w, "pong")
	})
	server.Handler = httpserver

	go func() {
		if err := server.ListenAndServeTLS("", ""); err != nil {
			klog.Errorf("Failed to listen and serve webhook server: %v", err)
		}
	}()

	klog.Info("starting serve.")
	signalChan := make(chan os.Signal, 1)
	signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
	<-signalChan

	klog.Infof("Got shut signal, shutting...")
	if err := server.Shutdown(context.Background()); err != nil {
		klog.Errorf("HTTP server Shutdown: %v", err)
	}
}

對應的Dockerfile

FROM golang:alpine AS builder
MAINTAINER cylon
WORKDIR /admission
COPY ./ /admission
ENV GOPROXY https://goproxy.cn,direct
RUN \
    sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories && \
    apk add upx  && \
    GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags "-s -w" -o webhook main.go && \
    upx -1 webhook && \
    chmod +x webhook

FROM alpine AS runner
WORKDIR /go/admission
COPY --from=builder /admission/webhook .
VOLUME ["/admission"]

叢集內部部署所需的資源清單

apiVersion: v1
kind: Service
metadata:
  name: admission-webhook
  labels:
    app: admission-webhook
spec:
  ports:
    - port: 81
      targetPort: 81
  selector:
    app: simple-webhook
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: simple-webhook
  name: simple-webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: simple-webhook
  template:
    metadata:
      labels:
        app: simple-webhook
    spec:
      containers:
        - image: cylonchau/simple-webhook:v0.0.2
          imagePullPolicy: IfNotPresent
          name: webhook
          command: ["./webhook"]
          env:
            - name: "TLS_CERT"
              value: "./tls/tls.crt"
            - name: "TLS_KEY"
              value: "./tls/tls.key"
            - name: NS_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 81
          volumeMounts:
            - name: tlsdir
              mountPath: /go/admission/tls
              readOnly: true
      volumes:
        - name: tlsdir
          secret:
            secretName: webhook
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: "pod-policy.example.com"
webhooks:
  - name: "pod-policy.example.com"
    rules:
      - apiGroups:   ["apps"] # 攔截資源的Group "" 表示 core。"*" 表示所有。
        apiVersions: ["v1"] # 攔截資源的版本
        operations:  ["CREATE"] # 什麼請求下攔截
        resources:   ["deployments"]  # 攔截什麼資源
        scope:       "Namespaced" # 生效的範圍，cluster還是namespace "*"表示沒有範圍限制。
    clientConfig: # 我們部署的webhook服務，
      url: "https://10.0.0.1:81/mutate"
#      service: # service是在cluster-in模式下
#        namespace: "default"
#        name: "admission-webhook"
#        port: 81 # 服務的埠
#        path: "/mutate" # path是對應用於驗證的介面
      # caBundle是提供給 admission webhook CA證書
      caBundle: Put you CA (base64 encode) in here
    admissionReviewVersions: ["v1"]
    sideEffects: None
    timeoutSeconds: 5 # 1-30s直接，表示請求api的超時時間
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "valipod-policy.example.com"
webhooks:
- name: "valipod-policy.example.com"
  rules:
    - apiGroups:   ["apps"] # 攔截資源的Group "" 表示 core。"*" 表示所有。
      apiVersions: ["v1"] # 攔截資源的版本
      operations:  ["CREATE"] # 什麼請求下攔截
      resources:   ["deployments"]  # 攔截什麼資源
      scope:       "Namespaced" # 生效的範圍，cluster還是namespace "*"表示沒有範圍限制。
  clientConfig: # 我們部署的webhook服務，
    #      service: # service是在cluster-in模式下
    #        namespace: "default"
    #        name: "admission-webhook"
    #        port: 81 # 服務的埠
    #        path: "/mutate" # path是對應用於驗證的介面
    # caBundle是提供給 admission webhook CA證書
    caBundle: Put you CA (base64 encode) in here
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5 # 1-30s直接，表示請求api的超時時間

這裡需要主義的問題

證書問題

如果需要 cluster-in ，那麼則需要對對應webhookconfig資源配置 service ；如果使用的是外部部署，則需要配置對應訪問地址，如："https://xxxx:port/method"

這兩種方式的證書均需要對應的 subjectAltName ，cluster-in 模式 需要對應service名稱，如，至少包含serviceName.NS.svc 這一個域名。

下面就是證書類問題的錯誤

Failed calling webhook, failing closed pod-policy.example.com: failed calling webhook "pod-policy.example.com": Post https://admission-webhook.default.svc:81/mutate?timeout=5s: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "admission-webhook-ca")

相應資訊問題

上面我們瞭解到的APIServer是去發出 v1admission.AdmissionReview 也就是 Request 和 Response型別的，所以，為了更清晰的表示出問題所在，需要對響應格式中的 Reason 與 Message 配置，這也就是我們在客戶端看到的報錯資訊。

&metav1.Status{
    Code: http.StatusForbidden,
    Reason: func() metav1.StatusReason {
        return metav1.StatusReasonForbidden
    }(),
    Message: fmt.Sprintf("the resource %s couldn't to allow entry.", deploy.Kind),
}

通過上面的設定使用者可以看到下列錯誤

$ kubectl apply -f nginx.yaml 
Error from server (Forbidden): error when creating "nginx.yaml": admission webhook "valipod-policy.example.com" denied the request: the resource Deployment couldn't to allow entry.

注：必須的引數還包含，UID，allowed，這兩個是必須的，上面闡述的只是對使用者友好的提示資訊

下面的報錯就是對相應格式設定錯誤

Error from server (InternalError): error when creating "nginx.yaml": Internal error occurred: failed calling webhook "pod-policy.example.com": the server rejected our request for an unknown reason

相應資訊版本問題

相應資訊也需要指定一個版本，這個與請求來的結構中拿即可

admissionResp.APIVersion = admission.APIVersion
admissionResp.Kind = admission.Kind

下面是沒有為對應相應資訊配置對應KV的值出現的報錯

Error from server (InternalError): error when creating "nginx.yaml": Internal error occurred: failed calling webhook "pod-policy.example.com": expected webhook response of admission.k8s.io/v1, Kind=AdmissionReview, got /, Kind=

關於patch

kubernetes中patch使用的是特定的規範，如 jsonpatch

kubernetes當前唯一支援的 patchType 是 JSONPatch。 有關更多詳細資訊，請參見 JSON patch

對於 jsonpatch 是一個固定的型別，在go中必須定義其結構體

{
	"op": "add", // 做什麼操作
	"path": "/spec/replicas", // 操作的路徑
	"value": 3 // 對應新增的key value
}

下面就是字串型別設定為布林型產生的報錯

Error from server (InternalError): error when creating "nginx.yaml": Internal error occurred: v1.Deployment.ObjectMeta: v1.ObjectMeta.Annotations: ReadString: expects " or n, but found t, error found in #10 byte of ...|t/Allow":true},"crea|..., bigger context ...|tadata":{"annotations":{"nginx-deployment/Allow":true},"creationTimestamp":null,"managedFields":[{"m|..

準備證書

Ubuntu

touch ./demoCAindex.txt
touch ./demoCA/serial 
touch ./demoCA/crlnumber
echo 01 > ./demoCA/serial
mkdir ./demoCA/newcerts

openssl genrsa -out cakey.pem 2048

openssl req -new \
	-x509 \
	-key cakey.pem \
	-out cacert.pem \
	-days 3650 \
	-subj "/CN=admission webhook ca"

openssl genrsa -out tls.key 2048

openssl req -new \
	-key tls.key \
	-subj "/CN=admission webhook client" \
	-reqexts webhook \
	-config <(cat /etc/ssl/openssl.cnf \
	<(printf "[webhook]\nsubjectAltName=DNS: admission-webhook, DNS: admission-webhook.default.svc, DNS: admission-webhook.default.svc.cluster.local, IP:10.0.0.1,  IP:10.0.0.4")) \
	-out tls.csr

sed -i 's/= match/= optional/g' /etc/ssl/openssl.cnf

openssl ca \
	-in tls.csr \
	-cert cacert.pem \
	-keyfile cakey.pem \
	-out tls.crt \
	-days 300 \
	-extensions webhook \
	-extfile <(cat /etc/ssl/openssl.cnf \
    <(printf "[webhook]\nsubjectAltName=DNS: admission-webhook, DNS: admission-webhook.default.svc, DNS: admission-webhook.default.svc.cluster.local, IP:10.0.0.1,  IP:10.0.0.4"))

CentOS

touch /etc/pki/CA/index.txt
touch /etc/pki/CA/serial # 下一個要頒發的編號 16進位制
touch /etc/pki/CA/crlnumber
echo 01 > /etc/pki/CA/serial

openssl req -new \
	-x509 \
	-key cakey.pem \
	-out cacert.pem \
	-days 3650 \
	-subj "/CN=admission webhook ca"

openssl genrsa -out tls.key 2048

openssl req -new \
	-key tls.key \
	-subj "/CN=admission webhook client" \
	-reqexts webhook \
	-config <(cat /etc/pki/tls/openssl.cnf \
	<(printf "[webhook]\nsubjectAltName=DNS: admission-webhook, DNS: admission-webhook.default.svc, DNS: admission-webhook.default.svc.cluster.local, IP:10.0.0.1,  IP:10.0.0.4")) \
	-out tls.csr

sed -i 's/= match/= optional/g' /etc/ssl/openssl.cnf

openssl ca \
	-in tls.csr \
	-cert cacert.pem \
	-keyfile cakey.pem \
	-out tls.crt \
	-days 300 \
	-extensions webhook \
	-extfile <(cat /etc/pki/tls/openssl.cnf \
    <(printf "[webhook]\nsubjectAltName=DNS: admission-webhook, DNS: admission-webhook.default.svc, DNS: admission-webhook.default.svc.cluster.local, IP:10.0.0.1,  IP:10.0.0.4"))

通過部署測試結果

可以看到我們自己注入的 annotation nginx-deployment/Allow: true，在該示例中，僅為演示過程，而不是真的策略，實際環境中可以根據情況進行定製自己的策略。

結果可以看出，當在 mutating 中不通過，即缺少對應的 annotation 標籤 , 則 validating 會不允許准入

$ kubectl describe deploy nginx-deployment
Name:                   nginx-deployment
Namespace:              default
CreationTimestamp:      Mon, 11 Jul 2022 20:25:16 +0800
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
                        nginx-deployment/Allow: true
Selector:               app=nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        nginx:1.14.2

Reference

extensible admission controllers

K8S client-go Patch example

admission controllers response

a guide to kubernetes admission controllers

相關文章