Operator介紹

0x00000發表於2022-06-26

一、Operator簡介

      在Kubernetes中我們經常使用Deployment、DaemonSet、Service、ConfigMap等資源，這些資源都是Kubernetes的內建資源，他們的建立、更新、刪除等均有Controller Manager負責管理。

 

二、Operator組成

      Operator(Controller+CRD），Operator是由Kubernetes自定義資源（CRD）和控制器（Controller）構成的元原生擴充套件服務，其中CRD定義了每個Operator需要建立和管理的自定義資源物件，底層實際就是通過APIServer介面在ETCD中註冊一種新的資源型別，註冊完成後就可以建立該資源型別的物件了。但是僅註冊資源和建立資源物件是沒有任何實際意義的，CRD最重要的是需要配合對應的Controller來實現自定義資源的功能達到自定義資源期望的狀態，比如內建的Deployment Controller用來控制Department資源物件的功能，根據配置生成特定數量的Pod監控其狀態，並根據事件做出相應的動作。

 

三、Operator使用

     使用者想為自己的自定義資源構建一個Kubernetes Operator，有很多工具可供選擇比如Operator SDK、Kubebuilder，甚至可以使用Operator SDK（HELM、Ansible、Go）。這些工具建立Kubernetes Operator用來監控自定義資源，並且根據資源的變化調整資源狀態。

     Operator作為自定義擴充套件資源以Deployment的方式部署到Kubernetes中，通過List-Watch方式監聽對應資源的變化，當使用者修改自定義資源中的任何內容，Operator會監控資源的更改，並根據更改內容執行特定的操作，這些操作通常會對Kubernetes API中某些資源進行呼叫。

 

四、Operator應用案例

 Kubebuilder介紹

    Kubebuilder是一個用Go語言構建Kubenetes API控制器和CRD的腳手架工具，通過使用Kubebuilder,使用者可以遵循一套簡單的程式設計框架，編寫Operator使用例項。

  • kubenetes: v1.23.8
  • go
tar xf go1.18.3.linux-amd64.tar.gz
mv go /usr/local/
vim /etc/profile.d/go183.sh
export GO111MODULE=on
export GOROOT=/usr/local/go 
export GOPATH=/root/gopath
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

source /etc/profile.d/go183.sh
  • kubebuilder: 3.5.0
yum -y install gcc
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

kubebuilder version

案例

　　Welcome案例主要實現使用Operator和CRD部署一套完整的應用環境，可以實現根據自定義型別建立資源，通過建立一個Welcome型別的資源，後臺自動建立Deployment和Service，通過Web頁面訪問Service呈現應用部署，通過自定義控制器方式進行控制管理

　　我們西藥建立Welcome自定義資源及對應的Controllers，最終我們可以通過類似如下程式碼的YAML檔案部署簡單的Web應用

apiVersion: webapp.demo.welcome.domain/v1
kind: Welcome
metadata:
  name: welcome-sample
spec:
  name: myfriends

Web應用介紹

  本案例中，我們使用Go語言HTTP模組建立一個Web服務，使用者訪問頁面後會自動載入NAME及PORT環境變數並渲染index.html靜態檔案中，程式碼如下：

package main

import (
	"fmt"
	"net/http"
	"os"
)

func main() {
	name := os.Getenv("NAME")
	hello := fmt.Sprintf("Hello %s", name)
	http.Handle("/hello/", http.StripPrefix("/hello/", http.FileServer(http.Dir("static"))))
	f, err := os.OpenFile("./static/index.html", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
	if err != nil {
		panic(err)
	}
	defer f.Close()
	if _, err = f.WriteString(hello); err != nil {
		panic(err)
	}
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
}

    其中，NAME環境變數通過我們在Welcome中定義的name欄位獲取，我們在下面的控制器編寫中會詳細介紹獲取欄位的詳細方法。我們將index.html放在Static資料夾下，並將工程檔案打包為Docker映象，Dockerfile如下：

FROM golang:1.17 as builder

WORKDIR /
COPY . .
COPY static

RUN CGO_ENABLED=0 GOOS=linux go build -v -o main
FROM alpine
RUN apk add --no-cache ca-certificates
COPY --from=builder /main /usr/local/main
COPY --from=builder static /static
CMD ["/usr/local/main"]

專案初始化

使用Kubebuilder命令映象專案初始化

mkdir demo
cd demo
go mod init welcome_demo.domain
kubebuilder init --domain demo.welcome.domain

初始化專案後，kubebuilder生成專案介面如下

[root@k8s-01 demo]# tree .
.
├── config
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   └── rbac
│       ├── auth_proxy_client_clusterrole.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_service.yaml
│       ├── kustomization.yaml
│       ├── leader_election_role_binding.yaml
│       ├── leader_election_role.yaml
│       ├── role_binding.yaml
│       └── service_account.yaml
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

6 directories, 25 files

 

建立`Welcome` Kind和其對應的控制器

[root@k8s-01 demo]# kubebuilder create api --group webapp --kind Welcome --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y

   輸入兩次y, Kubebuilder 分別建立了資源和控制器的模板，此處的group、version、kind這3個屬性組合起來標識一個k8s的CRD，建立完成後，Kubebuilder新增檔案如下：

[root@k8s-01 demo]# tree .
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── welcome_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_welcomes.yaml
│   │       └── webhook_in_welcomes.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── service_account.yaml
│   │   ├── welcome_editor_role.yaml
│   │   └── welcome_viewer_role.yaml
│   └── samples
│       └── webapp_v1_welcome.yaml
├── controllers
│   ├── suite_test.go
│   └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

13 directories, 38 files

後續需要執行兩步操作：

  1. 修改Resource Type
  2. 修改Controller 邏輯

 

修改Resource Type

 此處Resource Type為需要定義的資源欄位，用於在Yaml檔案中進行宣告，本案例中需要新增name欄位用於“Welcome”Kind中的Web應用，程式碼如下：

vim api/v1/welcome_types.go

// WelcomeSpec defines the desired state of Welcome
type WelcomeSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        // Foo is an example field of Welcome. Edit welcome_types.go to remove/update
        // Foo string `json:"foo,omitempty"`
        Name string `json:"name,omitempty"`
}

修改Controller邏輯

    在Controller中需要通過Reconcile方法完成Deployment和Service部署，並最終達到期望的狀態。

vim controllers/welcome_controller.go

//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.1/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        // _ = log.FromContext(ctx)

        // TODO(user): your logic here
        log := r.Log.WithValues("welcome", req.NamespacedName)
        log.Info("reconcilling welcome")
        return ctrl.Result{}, nil
}
  • 最終程式碼如下
/*
Copyright 2022.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
	"context"
	"fmt"
	appsv1 "k8s.io/api/apps/v1"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"
	"sigs.k8s.io/controller-runtime/pkg/log"

	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	webappv1 "welcome_demo.domain/api/v1"
)

// WelcomeReconciler reconciles a Welcome object
type WelcomeReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=webapp.demo.welcome.domain,resources=welcomes/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Welcome object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.1/pkg/reconcile
func (r *WelcomeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here
	welcome := &webappv1.Welcome{}
	if err := r.Client.Get(ctx, req.NamespacedName, welcome); err != nil {
		return ctrl.Result{}, err
	}
	deployment, err := r.createWelcomeDeployment(welcome)
	if err != nil {
		return ctrl.Result{}, err
	}
	fmt.Println("create deployment success!")
	svc, err := r.createService(welcome)
	if err != nil {
		return ctrl.Result{}, err
	}
	fmt.Println("create service success!")
	applyOpts := []client.PatchOption{client.ForceOwnership, client.FieldOwner("welcome_controller")}
	err = r.Patch(ctx, &deployment, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}
	err = r.Patch(ctx, &svc, client.Apply, applyOpts...)
	if err != nil {
		return ctrl.Result{}, err
	}
	return ctrl.Result{}, nil
}

func (r *WelcomeReconciler) createWelcomeDeployment(welcome *webappv1.Welcome) (appsv1.Deployment, error) {
	defOne := int32(1)
	name := welcome.Spec.Name
	if name == "" {
		name = "world"
	}
	depl := appsv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			APIVersion: appsv1.SchemeGroupVersion.String(),
			Kind:       "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      welcome.Name,
			Namespace: welcome.Namespace,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &defOne,
			Selector: &metav1.LabelSelector{
				MatchLabels: map[string]string{"welcome": welcome.Name},
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: map[string]string{"welcome": welcome.Name},
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Name: "welcome",
							Env: []corev1.EnvVar{
								{Name: "NAME", Value: name},
							},
							Ports: []corev1.ContainerPort{
								{ContainerPort: 8080,
									Name:     "http",
									Protocol: "TCP",
								},
							},
							Image: "sdfcdwefe/operatordemo:v1",
							Resources: corev1.ResourceRequirements{
								Requests: corev1.ResourceList{
									corev1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI),
									corev1.ResourceMemory: *resource.NewMilliQuantity(100000, resource.BinarySI),
								},
							},
						},
					},
				},
			},
		},
	}

	return depl, nil
}

func (r *WelcomeReconciler) createService(welcome *webappv1.Welcome) (corev1.Service, error) {
	svc := corev1.Service{
		TypeMeta: metav1.TypeMeta{
			APIVersion: corev1.SchemeGroupVersion.String(),
			Kind:       "Service"},
		ObjectMeta: metav1.ObjectMeta{
			Name:      welcome.Name,
			Namespace: welcome.Namespace,
		},
		Spec: corev1.ServiceSpec{
			Ports: []corev1.ServicePort{
				{Name: "http",
					Port:       8080,
					Protocol:   "TCP",
					TargetPort: intstr.FromString("http")},
			},
			Selector: map[string]string{"welcome": welcome.Name},
			Type:     corev1.ServiceTypeLoadBalancer,
		},
	}
	return svc, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *WelcomeReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&webappv1.Welcome{}).
		Complete(r)
}

 

Welcome應部署

  •  生成CRD資源
[root@k8s-01 demo]# make manifests
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

[root@k8s-01 demo]# tree .
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── welcome_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   └── controller-gen
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── webapp.demo.welcome.domain_welcomes.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_welcomes.yaml
│   │       └── webhook_in_welcomes.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   ├── service_account.yaml
│   │   ├── welcome_editor_role.yaml
│   │   └── welcome_viewer_role.yaml
│   └── samples
│       └── webapp_v1_welcome.yaml
├── controllers
│   ├── suite_test.go
│   └── welcome_controller.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
├── PROJECT
└── README.md

14 directories, 40 files


[root@k8s-01 demo]# cat config/crd/bases/webapp.demo.welcome.domain_welcomes.yaml 
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.9.0
  creationTimestamp: null
  name: welcomes.webapp.demo.welcome.domain
spec:
  group: webapp.demo.welcome.domain
  names:
    kind: Welcome
    listKind: WelcomeList
    plural: welcomes
    singular: welcome
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Welcome is the Schema for the welcomes API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: WelcomeSpec defines the desired state of Welcome
            properties:
              name:
                description: Foo is an example field of Welcome. Edit welcome_types.go
                  to remove/update Foo string `json:"foo,omitempty"`
                type: string
            type: object
          status:
            description: WelcomeStatus defines the observed state of Welcome
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
  •  建立Welcome型別資源
[root@k8s-01 demo]# kubectl create -f config/crd/bases/
customresourcedefinition.apiextensions.k8s.io/welcomes.webapp.demo.welcome.domain created
[root@k8s-01 demo]# kubectl create -f config/samples/webapp_v1_welcome.yaml 
welcome.webapp.demo.welcome.domain/welcome-sample created
  •  使用`kubectl get crd`命令檢視自定義物件
[root@k8s-01 demo]# kubectl get crd | grep welcome
welcomes.webapp.demo.welcome.domain          2022-06-25T09:10:37Z
  • 通過kubectl get welcome命令可以看到建立的welcome物件
[root@k8s-01 demo]# kubectl get welcome
NAME             AGE
welcome-sample   2m20s

 此時CRD並不會完成任何工作，只是在ETCD中建立了一條記錄，我們需要執行Controller才能幫助我們完成工作並最終達到welcome定義的狀態。

[root@k8s-01 demo]# make run
/root/demo/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/demo/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
api/v1/welcome_types.go
go vet ./...
go run ./main.go
1.6561485987622015e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6561485987624757e+09	INFO	setup	starting manager
1.6561485987638762e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.656148598763948e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.656148598764167e+09	INFO	Starting EventSource	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "source": "kind source: *v1.Welcome"}
1.6561485987641926e+09	INFO	Starting Controller	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome"}
1.6561485988653958e+09	INFO	Starting workers	{"controller": "welcome", "controllerGroup": "webapp.demo.welcome.domain", "controllerKind": "Welcome", "worker count": 1}
create deployment success!
create service success!

 以上方式在本地啟動控制器，方便除錯和驗證

  •  檢視建立的deployment、service
[root@k8s-01 demo]# kubectl get deploy
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
welcome-sample   1/1     1            1           34s
[root@k8s-01 demo]# kubectl get svc
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes       ClusterIP      10.96.0.1       <none>        443/TCP          4d2h
welcome-sample   LoadBalancer   10.106.36.129   <pending>     8080:30037/TCP   39s

 

相關文章