開發一個MutatingWebhook

leason001發表於2024-08-22
WebHook

介紹

Webhook就是一種HTTP回撥，用於在某種情況下執行某些動作，Webhook不是K8S獨有的，很多場景下都可以進行Webhook，比如在提交完程式碼後呼叫一個Webhook自動構建docker映象

准入 Webhook 是一種用於接收准入請求並對其進行處理的 HTTP 回撥機制。 可以定義兩種型別的准入 Webhook， 即驗證性質的准入 Webhook 和變更性質的准入 Webhook。 變更性質的准入 Webhook 會先被呼叫。它們可以修改傳送到 API 伺服器的物件以執行自定義的設定預設值操作。

在完成了所有物件修改並且 API 伺服器也驗證了所傳入的物件之後， 驗證性質的 Webhook 會被呼叫，並透過拒絕請求的方式來強制實施自定義的策略。

Admission Webhook使用較多的場景如下

  1. 在資源持久化到ETCD之前進行修改（Mutating Webhook），比如增加init Container或者sidecar Container
  2. 在資源持久化到ETCD之前進行校驗（Validating Webhook），不滿足條件的資源直接拒絕並給出相應資訊

組成

  1. webhook 服務
  2. webhook 配置
  3. webhook 證書

建立核心元件Pod的Webhook

使用kubebuilder新建webhook專案

kubebuilder init --domain test.com --repo gitlab.qima-inc.com/test-operator

(base) (⎈ |kubernetes-admin@qa-u03:qa)➜  test-operator kubebuilder init --domain test.com --repo gitlab.qima-inc.com/test-operator
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.17.2
INFO Update dependencies:
$ go mod tidy
go: go.mod file indicates go 1.21, but maximum version supported by tidy is 1.19
Error: failed to initialize project: unable to run post-scaffold tasks of "base.go.kubebuilder.io/v4": exit status 1

因為我預設是是go1.19所以版本達不到要求，這裡兩種處理方式

  1. 指定 --plugins go/v3 --project-version 3
  2. 切換高版本golang 這裡我切換了go1.22
(base) (⎈ |kubernetes-admin@qa-u03:qa)➜  test-operator kubebuilder init --domain test.com --repo gitlab.qima-inc.com/test-operator                                    
INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.17.2 
INFO Update dependencies:
$ go mod tidy           
Next: define a resource with:
$ kubebuilder create api

生成核心元件Pod的API

(base) (⎈ |kubernetes-admin@qa-u03:qa)➜  test-operator kubebuilder create api --group core --version v1 --kind Pod                                                    
INFO Create Resource [y/n]                        
n
INFO Create Controller [y/n]                      
n
INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO Update dependencies:
$ go mod tidy

這裡有兩個選項，建立資源和建立控制器
因為是內建資源Pod所以不需要建立資源，也不需要控制器
假如是自定義資源，需要建立資源，建立控制器

建立webhook

(base) (⎈ |kubernetes-admin@qa-u03:qa)➜  test-operator kubebuilder create webhook --group core --version v1 --kind Pod --defaulting --programmatic-validation
INFO Writing kustomize manifests for you to edit... 
ERRO Unable to find the target(s) #- path: patches/webhook/* to uncomment in the file config/crd/kustomization.yaml. 
ERRO Unable to find the target(s) #configurations:
#- kustomizeconfig.yaml to uncomment in the file config/crd/kustomization.yaml. 
INFO Writing scaffold for you to edit...          
INFO api/v1/pod_webhook.go                        
INFO api/v1/pod_webhook_test.go                   
INFO api/v1/webhook_suite_test.go                 
INFO Update dependencies:
$ go mod tidy           
INFO Running make:
$ make generate                
mkdir -p /Users/xxxx/test-operator/bin
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.14.0
/Users/xxxx/test-operator/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new Webhook and generate the manifests with:
$ make manifests

程式碼結構

.
├── Dockerfile
├── Makefile
├── PROJECT
├── README.md
├── api
│   └── v1
│       ├── pod_webhook.go
│       ├── pod_webhook_test.go
│       └── webhook_suite_test.go
├── bin
│   └── controller-gen-v0.14.0
├── cmd
│   └── main.go
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── crd
│   │   └── patches
│   │       ├── cainjection_in_pods.yaml
│   │       └── webhook_in_pods.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_config_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── role.yaml
│   │   ├── role_binding.yaml
│   │   └── service_account.yaml
│   └── webhook
│       ├── kustomization.yaml
│       ├── kustomizeconfig.yaml
│       ├── manifests.yaml
│       └── service.yaml
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
└── test
├── e2e
│   ├── e2e_suite_test.go
│   └── e2e_test.go
└── utils
└── utils.go

實現Webhook相關程式碼

因為只有Webhook，沒有Controller 所以只需要實現Webhook相關程式碼即可，同時需要註釋掉一些程式碼如：
Dockerfile中的

# COPY internal/controller/ internal/controller/

修改api/v1/xxx_suite_test.go
因為核心元件Pod的Webhook和一般的CRD的webhook不一樣，此處生成的pod_webhook.go只有Default()這個function，因此，我們需要直接重寫整個程式碼，最重要的是Handle()方法。

/*
Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
	"fmt"
	"net/http"
	"sigs.k8s.io/controller-runtime/pkg/client"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
)

// log is for logging in this package.
var podlog = logf.Log.WithName("pod-resource")

// 定義核心元件pod的webhook的主struct，類似於java的Class
type PodWebhookMutate struct {
	Client  client.Client
	decoder *admission.Decoder
}

// +kubebuilder:webhook:path=/mutate-core-v1-pod,mutating=true,failurePolicy=fail,sideEffects=None,groups=core,resources=pods,verbs=create;update,versions=v1,name=mpod.kb.io,admissionReviewVersions=v1
func (a *PodWebhookMutate) Handle(ctx context.Context, req admission.Request) admission.Response {
	pod := &corev1.Pod{}
	err := a.decoder.Decode(req, pod)
	if err != nil {
		return admission.Errored(http.StatusBadRequest, err)
	}

	// TODO: 變數marshaledPod是一個Map，可以直接修改pod的一些屬性
	marshaledPod, err := json.Marshal(pod)
	if err != nil {
		return admission.Errored(http.StatusInternalServerError, err)
	}
	// 列印
	fmt.Println("======================================================")
	fmt.Println(string(marshaledPod))
	return admission.PatchResponseFromRaw(req.Object.Raw, marshaledPod)
}

func (a *PodWebhookMutate) InjectDecoder(d *admission.Decoder) error {
	a.decoder = d
	return nil
}

修改main.go檔案：

if os.Getenv("ENABLE_WEBHOOKS") != "false" {
    //if err = (&corev1.Pod{}).SetupWebhookWithManager(mgr); err != nil {
    //	setupLog.Error(err, "unable to create webhook", "webhook", "Pod")
    //	os.Exit(1)
    //}
    mgr.GetWebhookServer().Register("/mutate-core-v1-pod", &webhook.Admission{Handler: &v1.PodWebhookMutate{Client: mgr.GetClient()}})
}

生成mainfests

make manifests generate

證書

手動簽發證書

https://cuisongliu.github.io/2020/07/kubernetes/admission-webhook/

自動簽發證書

webhook 服務啟動時自動生成證書，授權證書

  1. 建立CA根證書以及服務的證書
  2. 將服務端、CA證書寫入 k8s Secret，並且支援find or create
  3. 本地寫入證書
  4. 獲取MutatingWebhookConfiguration和ValidatingWebhookConfiguration將caCert寫入webhook config中的ClientConfig.CABundle（這裡有個問題是webhook需要提前建立，CABundle可以寫個臨時值，等webhook server 啟動覆蓋）

自動簽發證書參考專案： https://github.com/koordinator-sh/koordinator/blob/main/pkg/webhook/util/controller/webhook_controller.go#L187

相關文章