概述startDeploymentController 入口邏輯DeploymentController 物件DeploymentController 型別定義DeploymentController 初始化ResourceEventHandlerDeployment 變更事件ReplicaSet 變更事件DeploymentController 啟動Run()syncDeployment小結
概述
原始碼版本：kubernetes-v1.22.3 / commit-id: c92036
Deployment 是最常用的 Kubernetes 原生 Workload 資源之一，我們一開始嘗試使用 Kubernetes 的時候大概率就是從執行一個 Deployment 型別的工作負載開始的。
在上一篇《Kubernetes Deployment 原始碼分析（一）》中我們過了下 Deployment 的全部特性，主要介紹“滾動更新”和“回滾”等主要功能，從而心中有個概念，知道 Deployment 的能力邊界在那裡，所以今天以此為基礎，我們繼續從原始碼角度看下 Deployment 的實現。
注意：閱讀 Deployment 原始碼需要有一定的自定義控制器工作原理基礎，裡面涉及到了 Informer 工作機制、workqueue（延時/限速工作佇列）、ResourceEventHandler 等等邏輯，沒有相關知識儲備直接看本文會有一定挑戰，建議先閱讀《深入理解 K8S 原理與實現》系列目錄裡列的相關文章。
《Kubernetes Deployment Controller 原始碼分析》分為兩講：
- 《Kubernetes Deployment 原始碼分析（一）》 - 功能特性
- 《Kubernetes Deployment 原始碼分析（二）》 - 原始碼流程
startDeploymentController 入口邏輯
DeploymentController 的初始化和啟動入口是 startDeploymentController() 函式
- cmd/kube-controller-manager/app/apps.go:72
1func startDeploymentController(ctx ControllerContext) (http.Handler, bool, error) {
2 dc, err := deployment.NewDeploymentController(
3 ctx.InformerFactory.Apps().V1().Deployments(),
4 ctx.InformerFactory.Apps().V1().ReplicaSets(),
5 ctx.InformerFactory.Core().V1().Pods(),
6 ctx.ClientBuilder.ClientOrDie("deployment-controller"),
7 )
8 if err != nil {
9 return nil, true, fmt.Errorf("error creating Deployment controller: %v", err)
10 }
11 go dc.Run(int(ctx.ComponentConfig.DeploymentController.ConcurrentDeploymentSyncs), ctx.Stop)
12 return nil, true, nil
13}
在 startDeploymentController() 函式中先通過 NewDeploymentController() 方法初始化一個 DeploymentController 例項，這裡的引數 DeploymentInformer、ReplicaSetInformer、PodInformer 和 Clientset，因而 DeploymentController 也就具備了獲取Deployment、 ReplicaSet、Pod 三類資源變更事件以及 CURD apiserver 操作各種資源的能力。接著這個函式中又呼叫了 DeploymentController 的 Run() 方法來啟動 DeploymentController，這裡的引數 ConcurrentDeploymentSyncs 預設值是 5，也就是預設情況下併發調諧的 Deployment 數量是 5 個。
DeploymentController 物件
繼續看下 DeploymentController 物件的定義和初始化。
DeploymentController 型別定義
接著來看 DeploymentController 型別的定義
- pkg/controller/deployment/deployment_controller.go:68
1type DeploymentController struct {
2 // ReplicaSet 操控器
3 rsControl controller.RSControlInterface
4 client clientset.Interface
5 eventRecorder record.EventRecorder
6
7 syncHandler func(dKey string) error
8 // 測試用
9 enqueueDeployment func(deployment *apps.Deployment)
10
11 // 用來從 cache 裡 get/list Deployment
12 dLister appslisters.DeploymentLister
13 // 用來從 cache 裡 get/list ReplicaSet
14 rsLister appslisters.ReplicaSetLister
15 // 用來從 cache 裡 get/list Pod
16 podLister corelisters.PodLister
17
18 dListerSynced cache.InformerSynced
19 rsListerSynced cache.InformerSynced
20 podListerSynced cache.InformerSynced
21
22 // 工作佇列，限速佇列實現
23 queue workqueue.RateLimitingInterface
24}
25
DeploymentController 初始化
- pkg/controller/deployment/deployment_controller.go:101
1func NewDeploymentController(dInformer appsinformers.DeploymentInformer, rsInformer appsinformers.ReplicaSetInformer, podInformer coreinformers.PodInformer, client clientset.Interface) (*DeploymentController, error) {
2 // Event 相關邏輯
3 eventBroadcaster := record.NewBroadcaster()
4 eventBroadcaster.StartStructuredLogging(0)
5 eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: client.CoreV1().Events("")})
6
7 // ……
8
9 // new dc
10 dc := &DeploymentController{
11 client: client,
12 eventRecorder: eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "deployment-controller"}),
13 queue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "deployment"),
14 }
15 // 主要是 clientset
16 dc.rsControl = controller.RealRSControl{
17 KubeClient: client,
18 Recorder: dc.eventRecorder,
19 }
20 // ResourceEventHandler 配置，後面會分析
21 dInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
22 AddFunc: dc.addDeployment,
23 UpdateFunc: dc.updateDeployment,
24 DeleteFunc: dc.deleteDeployment,
25 })
26 rsInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
27 AddFunc: dc.addReplicaSet,
28 UpdateFunc: dc.updateReplicaSet,
29 DeleteFunc: dc.deleteReplicaSet,
30 })
31 podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
32 DeleteFunc: dc.deletePod,
33 })
34
35 // 這裡有主要邏輯，後面會講
36 dc.syncHandler = dc.syncDeployment
37 dc.enqueueDeployment = dc.enqueue
38 // 各種 lister
39 dc.dLister = dInformer.Lister()
40 dc.rsLister = rsInformer.Lister()
41 dc.podLister = podInformer.Lister()
42 dc.dListerSynced = dInformer.Informer().HasSynced
43 dc.rsListerSynced = rsInformer.Informer().HasSynced
44 dc.podListerSynced = podInformer.Informer().HasSynced
45 return dc, nil
46}
ResourceEventHandler
上面提到了幾個 ResourceEventHandler 回撥函式：
- addDeployment
- updateDeployment
- deleteDeployment
- addReplicaSet
- updateReplicaSet
- deleteReplicaSet
- deletePod
逐個分析下。
Deployment 變更事件
這裡邏輯比較簡單，三個方法一起看：
- pkg/controller/deployment/deployment_controller.go:167
1func (dc *DeploymentController) addDeployment(obj interface{}) {
2 d := obj.(*apps.Deployment)
3 klog.V(4).InfoS("Adding deployment", "deployment", klog.KObj(d))
4 // 新增 Deployment 時直接 enqueue
5 dc.enqueueDeployment(d)
6}
7
8func (dc *DeploymentController) updateDeployment(old, cur interface{}) {
9 oldD := old.(*apps.Deployment)
10 curD := cur.(*apps.Deployment)
11 klog.V(4).InfoS("Updating deployment", "deployment", klog.KObj(oldD))
12 // old Deployment 只用來列印一個日誌，cur Deployment enqueue
13 dc.enqueueDeployment(curD)
14}
15
16func (dc *DeploymentController) deleteDeployment(obj interface{}) {
17 d, ok := obj.(*apps.Deployment)
18 if !ok {
19 // 處理 DeletedFinalStateUnknown 場景
20 tombstone, ok := obj.(cache.DeletedFinalStateUnknown)
21 if !ok {
22 utilruntime.HandleError(fmt.Errorf("couldn't get object from tombstone %#v", obj))
23 return
24 }
25 d, ok = tombstone.Obj.(*apps.Deployment)
26 if !ok {
27 utilruntime.HandleError(fmt.Errorf("tombstone contained object that is not a Deployment %#v", obj))
28 return
29 }
30 }
31 klog.V(4).InfoS("Deleting deployment", "deployment", klog.KObj(d))
32 // 入隊
33 dc.enqueueDeployment(d)
34}
ReplicaSet 變更事件
然後來看 ReplicaSet 相關回撥函式的實現。
1、Added
- pkg/controller/deployment/deployment_controller.go:199
1func (dc *DeploymentController) addReplicaSet(obj interface{}) {
2 rs := obj.(*apps.ReplicaSet)
3 // 如果是準備刪除了，重啟的過程會收到 Added 事件，這時候直接呼叫刪除操作
4 if rs.DeletionTimestamp != nil {
5 dc.deleteReplicaSet(rs)
6 return
7 }
8 // 查詢對應的 Deployment
9 if controllerRef := metav1.GetControllerOf(rs); controllerRef != nil {
10 d := dc.resolveControllerRef(rs.Namespace, controllerRef)
11 if d == nil {
12 return
13 }
14 klog.V(4).InfoS("ReplicaSet added", "replicaSet", klog.KObj(rs))
15 // 將這個 Deployment 加入工作佇列
16 dc.enqueueDeployment(d)
17 return
18 }
19
20 // 如果是一個孤兒 ReplicaSet，則看是不是能找到一個 Deployment 來領養
21 ds := dc.getDeploymentsForReplicaSet(rs)
22 if len(ds) == 0 {
23 return
24 }
25 klog.V(4).InfoS("Orphan ReplicaSet added", "replicaSet", klog.KObj(rs))
26 // 一般只有一個 Deployment，但是也不能排出多個的情況，所以這裡用的是 ds 列表，迴圈 enqueue
27 for _, d := range ds {
28 dc.enqueueDeployment(d)
29 }
30}
2、Updated
- pkg/controller/deployment/deployment_controller.go:256
1func (dc *DeploymentController) updateReplicaSet(old, cur interface{}) {
2 curRS := cur.(*apps.ReplicaSet)
3 oldRS := old.(*apps.ReplicaSet)
4 if curRS.ResourceVersion == oldRS.ResourceVersion {
5 // Resync 的時候 RV 相同，不做處理
6 return
7 }
8
9 curControllerRef := metav1.GetControllerOf(curRS)
10 oldControllerRef := metav1.GetControllerOf(oldRS)
11 controllerRefChanged := !reflect.DeepEqual(curControllerRef, oldControllerRef)
12 if controllerRefChanged && oldControllerRef != nil {
13 // 如果 rs 的 ref 變更了，就需要通知老的 ref 對應的 Deployment
14 if d := dc.resolveControllerRef(oldRS.Namespace, oldControllerRef); d != nil {
15 dc.enqueueDeployment(d)
16 }
17 }
18
19 if curControllerRef != nil {
20 d := dc.resolveControllerRef(curRS.Namespace, curControllerRef)
21 if d == nil {
22 return
23 }
24 klog.V(4).InfoS("ReplicaSet updated", "replicaSet", klog.KObj(curRS))
25 // 當前 rs 對應 dp 入隊
26 dc.enqueueDeployment(d)
27 return
28 }
29
30 // 孤兒 rs 的場景，和 Added 時處理邏輯一樣
31 labelChanged := !reflect.DeepEqual(curRS.Labels, oldRS.Labels)
32 if labelChanged || controllerRefChanged {
33 ds := dc.getDeploymentsForReplicaSet(curRS)
34 if len(ds) == 0 {
35 return
36 }
37 klog.V(4).InfoS("Orphan ReplicaSet updated", "replicaSet", klog.KObj(curRS))
38 for _, d := range ds {
39 dc.enqueueDeployment(d)
40 }
41 }
42}
3、Deleted
- pkg/controller/deployment/deployment_controller.go:304
1func (dc *DeploymentController) deleteReplicaSet(obj interface{}) {
2 rs, ok := obj.(*apps.ReplicaSet)
3
4 // 刪除場景需要處理的 DeletedFinalStateUnknown 場景
5 if !ok {
6 tombstone, ok := obj.(cache.DeletedFinalStateUnknown)
7 if !ok {
8 utilruntime.HandleError(fmt.Errorf("couldn't get object from tombstone %#v", obj))
9 return
10 }
11 rs, ok = tombstone.Obj.(*apps.ReplicaSet)
12 if !ok {
13 utilruntime.HandleError(fmt.Errorf("tombstone contained object that is not a ReplicaSet %#v", obj))
14 return
15 }
16 }
17
18 // 孤兒 rs 被刪除時沒有 Deployment 需要關心
19 controllerRef := metav1.GetControllerOf(rs)
20 if controllerRef == nil {
21 return
22 }
23 d := dc.resolveControllerRef(rs.Namespace, controllerRef)
24 if d == nil {
25 return
26 }
27 klog.V(4).InfoS("ReplicaSet deleted", "replicaSet", klog.KObj(rs))
28 // 入隊
29 dc.enqueueDeployment(d)
30}
DeploymentController 啟動
前面看了哪些 Event 會向 workqueue 中新增 item，接著看下這些 item 是怎麼被消費的。
Run()
Run() 方法本身很簡潔，根據給定的併發數，也就是預設 5 併發，啟動 dc.worker
- pkg/controller/deployment/deployment_controller.go:149
1func (dc *DeploymentController) Run(workers int, stopCh <-chan struct{}) {
2 defer utilruntime.HandleCrash()
3 defer dc.queue.ShutDown()
4
5 klog.InfoS("Starting controller", "controller", "deployment")
6 defer klog.InfoS("Shutting down controller", "controller", "deployment")
7
8 if !cache.WaitForNamedCacheSync("deployment", stopCh, dc.dListerSynced, dc.rsListerSynced, dc.podListerSynced) {
9 return
10 }
11
12 for i := 0; i < workers; i++ {
13 go wait.Until(dc.worker, time.Second, stopCh)
14 }
15
16 <-stopCh
17}
繼續看 worker 的內容
- pkg/controller/deployment/deployment_controller.go:460
1func (dc *DeploymentController) worker() {
2 for dc.processNextWorkItem() {
3 }
4}
5
6func (dc *DeploymentController) processNextWorkItem() bool {
7 key, quit := dc.queue.Get() // 從 workqueue 中獲取一個 item
8 if quit {
9 return false
10 }
11 defer dc.queue.Done(key)
12 // 主要邏輯
13 err := dc.syncHandler(key.(string))
14 dc.handleErr(err, key)
15
16 return true
17}
這裡從 workqueue 裡拿到一個 key 之後，通過呼叫 syncHandler() 方法來處理，前面強調過這行程式碼：
- dc.syncHandler = dc.syncDeployment
所以接著我們繼續跟 dc.syncDeployment 的實現。
syncDeployment
syncDeployment() 方法做的事情是拿著 workqueue 裡出隊的 key，根據這個 key 來 sync 對應的 Deployment，繼續看下具體的邏輯。
- pkg/controller/deployment/deployment_controller.go
1func (dc *DeploymentController) syncDeployment(key string) error {
2 // 從 key 中分割出 namespace 和 name
3 namespace, name, err := cache.SplitMetaNamespaceKey(key)
4 if err != nil {
5 klog.ErrorS(err, "Failed to split meta namespace cache key", "cacheKey", key)
6 return err
7 }
8
9 startTime := time.Now()
10 klog.V(4).InfoS("Started syncing deployment", "deployment", klog.KRef(namespace, name), "startTime", startTime)
11 defer func() {
12 klog.V(4).InfoS("Finished syncing deployment", "deployment", klog.KRef(namespace, name), "duration", time.Since(startTime))
13 }()
14 // 根據 namespace 和 name 從 cache 中檢索對應 Deployment 物件
15 deployment, err := dc.dLister.Deployments(namespace).Get(name)
16 if errors.IsNotFound(err) {
17 klog.V(2).InfoS("Deployment has been deleted", "deployment", klog.KRef(namespace, name))
18 return nil
19 }
20 if err != nil {
21 return err
22 }
23
24 // 為了不改動這個 cache，這是一個 ThreadSafeStore
25 d := deployment.DeepCopy()
26
27 // 空 LabelSelector 會匹配到所有 pods，發一個 Warning Event，更新 .Status.ObservedGeneration 然後返回
28 everything := metav1.LabelSelector{}
29 if reflect.DeepEqual(d.Spec.Selector, &everything) {
30 dc.eventRecorder.Eventf(d, v1.EventTypeWarning, "SelectingAll", "This deployment is selecting all pods. A non-empty selector is required.")
31 if d.Status.ObservedGeneration < d.Generation {
32 d.Status.ObservedGeneration = d.Generation
33 dc.client.AppsV1().Deployments(d.Namespace).UpdateStatus(context.TODO(), d, metav1.UpdateOptions{})
34 }
35 return nil
36 }
37
38 // 獲取當前 Deployment 擁有的所有 ReplicaSet，同時會更新這些 ReplicaSet 的 ControllerRef
39 rsList, err := dc.getReplicaSetsForDeployment(d)
40 if err != nil {
41 return err
42 }
43 // 這個 map 是 map[types.UID][]*v1.Pod 型別，key 是 rs 的 UID，value 是對應 rs 管理的所有 pod 列表
44 podMap, err := dc.getPodMapForDeployment(d, rsList)
45 if err != nil {
46 return err
47 }
48
49 // 已經標記要刪除了，這時候只更新狀態
50 if d.DeletionTimestamp != nil {
51 return dc.syncStatusOnly(d, rsList)
52 }
53
54 // 根據 .Spec.Pause配置看是否更新 Deployment 的 conditions
55 if err = dc.checkPausedConditions(d); err != nil {
56 return err
57 }
58
59 if d.Spec.Paused {
60 // Pause 或 scale 時的調諧邏輯
61 return dc.sync(d, rsList)
62 }
63
64 // 應該過期了，老版本的 "deprecated.deployment.rollback.to" 註解回滾邏輯
65 if getRollbackTo(d) != nil {
66 // 回滾到舊版本的邏輯
67 return dc.rollback(d, rsList)
68 }
69 // 如果是 scale
70 scalingEvent, err := dc.isScalingEvent(d, rsList)
71 if err != nil {
72 return err
73 }
74 if scalingEvent {
75 // Pause 或 scale 時的調諧邏輯
76 return dc.sync(d, rsList)
77 }
78
79 switch d.Spec.Strategy.Type {
80 // 重建策略
81 case apps.RecreateDeploymentStrategyType:
82 return dc.rolloutRecreate(d, rsList, podMap)
83 // 滾動更新策略
84 case apps.RollingUpdateDeploymentStrategyType:
85 return dc.rolloutRolling(d, rsList)
86 }
87 return fmt.Errorf("unexpected deployment strategy type: %s", d.Spec.Strategy.Type)
88}
小結
syncDeployment() 方法看完之後，Deployment 控制器的邏輯就算過完一遍了。當然這個方法內部涉及到的一些小方法的呼叫這裡只是簡單介紹其功能，並沒有深究所有實現細節，不過這些小方法的邏輯都不難，就暫不贅述了。
對 Deployment Controller 的程式碼分析在 client-go 和 Job Controller 之後，所以這裡看起來感覺會很簡單，對應有些描述也沒有那麼詳細，如果大家看過前面我發的相關文章，看本文的邏輯應該也不會感覺吃力，反正我希望你先回過頭看下我之前發的相關文章，最新版可以在我的部落格網站 Daniel Hu's Blog 查閱。
轉載請保留本文原始連結 https://www.danielhu.cn）
