K8S 高階排程方式

klvchen發表於2018-11-29

可以使用高階排程分為：

節點選擇器： nodeSelector、nodeName
節點親和性排程： nodeAffinity
Pod親和性排程：PodAffinity
Pod反親和性排程：podAntiAffinity

nodeSelector, nodeName

cd; mkdir schedule; cd schedule/

vi pod-demo.yaml
# 內容為
apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
  labels:
    app: myapp
    tier: frontend
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
  nodeSelector:
    disktype: harddisk


kubectl apply -f pod-demo.yaml 
kubectl get pods

kubectl describe pod  pod-demo
# 執行結果:
Warning  FailedScheduling  2m3s (x25 over 3m15s)  default-scheduler  0/3 nodes are available: 3 node(s) didn`t match node selector.

# 打上標籤
kubectl label node node2 disktype=harddisk

# 正常啟動
kubectl get pods

nodeAffinity

requiredDuringSchedulingIgnoredDuringExecution 硬親和性必須滿足親和性。
preferredDuringSchedulingIgnoredDuringExecution 軟親和性能滿足最好，不滿足也沒關係。

硬親和性：
matchExpressions : 匹配表示式,這個標籤可以指定一段，例如pod中定義的key為zone，operator為In(包含那些)，values為 foo和bar。就是在node節點中包含foo和bar的標籤中排程
matchFields : 匹配欄位和上面的意思不過他可以不定義標籤值，可以定義

# 選擇在 node 有 zone 標籤值為 foo 或 bar 值的節點上執行 pod
vi pod-nodeaffinity-demo.yaml 
# 內容為
apiVersion: v1
kind: Pod
metadata:
  name: pod-node-affinity-demo
  labels:
    app: myapp
    tier: frontend
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: zone
            operator: In
            values:
            - foo
            - bar

kubectl apply -f pod-nodeaffinity-demo.yaml

kubectl describe pod pod-node-affinity-demo
# 執行結果:
Warning  FailedScheduling  2s (x8 over 20s)  default-scheduler  0/3 nodes are available: 3 node(s) didn`t match node selector.

# 給其中一個node打上foo的標籤
kubectl label node node1 zone=foo

# 正常啟動
kubectl get pods

軟親和性 :

cp pod-nodeaffinity-demo.yaml pod-nodeaffinity-demo-2.yaml 

vi pod-nodeaffinity-demo-2.yaml 
# 內容為
apiVersion: v1
kind: Pod
metadata:
  name: pod-node-affinity-demo-2
  labels:
    app: myapp
    tier: frontend
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: zone
            operator: In
            values:
            - foo
            - bar
        weight: 60

kubectl apply -f pod-nodeaffinity-demo-2.yaml

podAffinity

Pod親和性場景，我們的k8s叢集的節點分佈在不同的區域或者不同的機房，當服務A和服務B要求部署在同一個區域或者同一機房的時候，我們就需要親和性排程了。

labelSelector : 選擇跟那組Pod親和
namespaces : 選擇哪個名稱空間
topologyKey : 指定節點上的哪個鍵

kubectl get pods
kubectl delete pod pod-node-affinity-demo pod-node-affinity-demo-2 pod-demo

cd ~/schedule/

vi pod-required-affinity-demo.yaml 
# 內容為：
apiVersion: v1
kind: Pod
metadata:
  name: pod-first
  labels:
    app: myapp
    tier: frontend
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-second
  labels:
    app: db
    tier: db
spec:
  containers:
  - name: busybox
    image: busybox
    imagePullPolicy: IfNotPresent
    command: ["sh","-c","sleep 3600"]
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - {key: app, operator: In, values: ["myapp"]}
        topologyKey: kubernetes.io/hostname


kubectl apply -f pod-required-affinity-demo.yaml 

kubectl get pods -o wide
# 執行結果，兩個 pod 在同一 node 節點上
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE
pod-first    1/1     Running   0          11s   10.244.1.6   node1
pod-second   1/1     Running   0          11s   10.244.1.5   node1

podAntiAffinity

Pod反親和性場景，當應用服務A和資料庫服務B要求儘量不要在同一臺節點上的時候。

kubectl delete -f pod-required-affinity-demo.yaml 

cp pod-required-affinity-demo.yaml pod-required-anti-affinity-demo.yaml 

vi pod-required-anti-affinity-demo.yaml 
# 內容為
apiVersion: v1
kind: Pod
metadata:
  name: pod-first
  labels:
    app: myapp
    tier: frontend
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-second
  labels:
    app: backend
    tier: db
spec:
  containers:
  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: ["sh","-c","sleep 3600"]
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - {key: app, operator: In, values: ["myapp"]}
        topologyKey: kubernetes.io/hostname

kubectl apply -f pod-required-anti-affinity-demo.yaml 

kubectl get pods -o wide
# 執行結果，兩個 pod 不在同一個 node
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE
pod-first    1/1     Running   0          5s    10.244.2.4   node2
pod-second   1/1     Running   0          5s    10.244.1.7   node1

kubectl delete -f pod-required-anti-affinity-demo.yaml 


# 如果硬反親和性定義的標籤兩個節點都有，則第二個 Pod 沒法進行排程，如下面的的 zone=foo
# 給兩個 node 打上同一個標籤 zone=foo
kubectl label nodes node2 zone=foo
kubectl label nodes node1 zone=foo

vi pod-required-anti-affinity-demo.yaml 
# 內容為：
apiVersion: v1
kind: Pod
metadata:
  name: pod-first
  labels:
    app: myapp
    tier: frontend
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-second
  labels:
    app: backend
    tier: db
spec:
  containers:
  - name: busybox
    image: busybox:latest
    imagePullPolicy: IfNotPresent
    command: ["sh","-c","sleep 3600"]
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - {key: app, operator: In, values: ["myapp"]}
        topologyKey: zone


kubectl apply -f pod-required-anti-affinity-demo.yaml 

kubectl get pods -o wide
# 結果如下，pod-second 沒法啟動
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE
pod-first    1/1     Running   0          12s   10.244.1.8   node1
pod-second   0/1     Pending   0          12s   <none>       <none>

kubectl delete -f pod-required-anti-affinity-demo.yaml

汙點容忍排程（Taint和Toleration）

taints and tolerations 允許將某個節點做標記，以使得所有的pod都不會被排程到該節點上。但是如果某個pod明確制定了 tolerates 則可以正常排程到被標記的節點上。

# 可以使用命令列為 Node 節點新增 Taints：
kubectl taint nodes node1 key=value:NoSchedule

operator可以定義為：
Equal：表示key是否等於value，預設
Exists：表示key是否存在，此時無需定義value

tain 的 effect 定義對 Pod 排斥效果:
NoSchedule：僅影響排程過程，對現存的Pod物件不產生影響；
NoExecute：既影響排程過程，也影響顯著的Pod物件；不容忍的Pod物件將被驅逐
PreferNoSchedule: 表示儘量不排程

# 檢視節點的 taint
kubectl describe node master
kubectl get pods -n kube-system
kubectl describe pods kube-apiserver-master -n kube-system

# 為 node1 打上汙點
kubectl taint node node1 node-type=production:NoSchedule

vi deploy-demo.yaml 
# 內容為：
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v1
        ports:
        - name: http
          containerPort: 80



kubectl apply -f deploy-demo.yaml 

kubectl get pods -o wide
# 執行結果：
NAME                            READY   STATUS    RESTARTS   AGE   IP           NODE
myapp-deploy-69b47bc96d-cwt79   1/1     Running   0          5s    10.244.2.6   node2
myapp-deploy-69b47bc96d-qqrwq   1/1     Running   0          5s    10.244.2.5   node2


# 為 node2 打上汙點
kubectl taint node node2 node-type=dev:NoExecute

# NoExecute 將會驅逐沒有容忍該汙點的 pod，因兩個node節點都有汙點，pod沒有定義容忍，導致沒有節點可以啟動pod
kubectl get pods -o wide
# 執行結果：
NAME                            READY   STATUS    RESTARTS   AGE   IP       NODE
myapp-deploy-69b47bc96d-psl8f   0/1     Pending   0          14s   <none>   <none>
myapp-deploy-69b47bc96d-q296k   0/1     Pending   0          14s   <none>   <none>


# 定義Toleration(容忍)
vi deploy-demo.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
      tolerations:
      - key: "node-type"
        operator: "Equal"
        value: "production"
        effect: "NoSchedule"


kubectl apply -f deploy-demo.yaml

# pod 容忍 node1 的 tain ，可以在 node1 上執行
ubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE
myapp-deploy-65cc47f858-tmpnz   1/1     Running   0          10s   10.244.1.10   node1
myapp-deploy-65cc47f858-xnklh   1/1     Running   0          13s   10.244.1.9    node1


# 定義Toleration，是否存在 node-type 這個key 且 effect 值為 NoSchedule 
vi deploy-demo.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
      tolerations:
      - key: "node-type"
        operator: "Exists"
        value: ""
        effect: "NoSchedule"

kubectl apply -f deploy-demo.yaml

kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE
myapp-deploy-559f559bcc-6jfqq   1/1     Running   0          10s   10.244.1.11   node1
myapp-deploy-559f559bcc-rlwp2   1/1     Running   0          9s    10.244.1.12   node1


##定義Toleration，是否存在 node-type 這個key 且 effect 值為空，則包含所有的值 
vi deploy-demo.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
      tolerations:
      - key: "node-type"
        operator: "Exists"
        value: ""
        effect: ""

kubectl apply -f deploy-demo.yaml

# 兩個 pod 均衡排程到兩個節點
kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE
myapp-deploy-5d9c6985f5-hn4k2   1/1     Running   0          2m    10.244.1.13   node1
myapp-deploy-5d9c6985f5-lkf9q   1/1     Running   0          2m    10.244.2.7    node2

k8s排程器介紹（排程框架版本）
2021-10-15
K8S框架
k8s排程器
2019-11-17
K8S
k8s 排程 GPU
2021-02-04
K8SGPU
k8s之pod排程
2021-07-14
K8S
APS高階計劃排程系統和生產排產系統
2021-12-19
Kubernetes高階排程- Taint和Toleration、Node Affinity分析
2019-04-10
AI
程式執行緒排程方式
2015-01-08
執行緒
Quartz排程系統入門和排程高可用實現方案
2018-01-30
quartz
解決k8s排程不均衡問題
2022-06-20
K8S
K8s容器debug高階技巧
2023-12-11
K8S
GitHub的高階搜尋方式
2020-11-03
Github
【APS】90張專業PPT學習高階計劃與排程（APS）
2018-04-27
Flink排程之排程器、排程策略、排程模式
2023-03-08
模式
K8S原來如此簡單（六）Pod排程
2022-03-25
K8S
Abaqus熱分析高階課程
2020-04-08
Spring排程定時任務的方式
2019-01-13
Spring
高階k8s HA 叢集搭建（一）
2021-09-09
K8S
從零開始入門 K8s | 排程器的排程流程和演算法介紹
2020-01-08
K8S演算法
使用 K8s 進行作業排程實戰分享
2020-08-27
K8S
Net作業排程(三) — Quartz.Net進階
2014-11-17
quartz
K8s中的排程程式是如何工作的？ - daniele
2020-09-25
K8S
k8s之資料儲存-高階儲存
2021-08-18
K8S
[譯] Kotlin 協程高階使用技巧
2019-03-03
Kotlin
Oracle資料倉儲高階課程
2007-04-17
Oracle
上海攜程java高階面試題（一）
2024-04-14
Java面試題
Linux核心排程分析（程式排程）
2018-03-01
Linux
[杭州] 阿里系統軟體事業部排程團隊 Java/Golang高階開發工程師
2018-03-27
阿里JavaGolang工程師
Spark中資源排程和任務排程
2021-11-12
Spark
K8s Scheduler 在排程 pod 過程中遺漏部分節點的問題排查
2021-05-13
K8S
Web前端------JS高階繼承的實現方式
2018-07-07
Web前端JS繼承
Go語言排程器之主動排程(20)
2019-05-28
Go
Go排程器系列（3）圖解排程原理
2019-04-06
Go圖解
OpenMP For Construct dynamic 排程方式實現原理和原始碼分析
2023-02-03
Struct原始碼
百度自動內網流量排程進階實戰
2018-01-16
內網
排程器簡介，以及Linux的排程策略
2020-03-26
Linux
Go排程器系列（2）巨集觀看排程器
2019-03-27
Go
Go語言排程器之排程main goroutine（14）
2019-05-09
GoAI
【Spark篇】---Spark資源排程和任務排程
2018-02-05
Spark