要點01：軟體配置項的來源主要有2塊：命令列引數 和 配置檔案
- k8s中的元件一般都是遵循上面的模式
- 我們以kube-scheduler為例
- 命令引數舉例 ：可以看到--xxx=xxx的傳參，當然所有引數都是有預設值的，如果你不傳就走預設
/usr/local/bin/kube-scheduler --log-dir=/var/log/k8s --logtostderr=false --alsologtostderr=true --config=/etc/k8s/kube-scheduler.yaml --kube-api-qps=500 --kube-api-burst=500 --authentication-kubeconfig=/etc/k8s/scheduler.conf --authorization-kubeconfig=/etc/k8s/scheduler.conf --kubeconfig=/etc/k8s/scheduler.conf --leader-elect=true --v=2- 上面的命令列引數中可以看到--config=xxx.yaml指定了 配置檔案的路徑
- 比如我們可以檢視 這個配置檔案的內容 ：發現基本就是官方的預設配置
cat /etc/k8s/kube-scheduler.yaml
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: "/etc/k8s/scheduler.conf"
leaderElection:
leaderElect: true
metricsBindAddress: 0.0.0.0:10251
要點02：k8s元件提供了一個應用執行時檢視生效配置的介面
思考為何要提供？主要原因有下面幾點
- 來自命令列和配置檔案的配置2塊可能有些覆蓋的地方
- 配置專案太多了
在沒有配置熱更新的情況下：檢視變更是否生效：
- 配置檔案已經更改，但忘記是應用重啟前還是重啟後改的了
- 所以講究的專案裡面都會留有一個http介面
- 直接將當前應用記憶體中生效的配置專案列印出來，方便排查問題
- 所以我們後面寫golang的專案也可以仿照這個介面
要點03 : 如何請求scheduler的配置介面
k8s元件對應檢視配置的介面就是configz
訪問元件介面需要鑑權，我們可以透過sa來實現
如何在1.24叢集中建立rbac
apiVersion: rbac.authorization.k8s.io/v1 # api的version
kind: ClusterRole # 型別
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources: # 資源
- nodes
- nodes/metrics
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups:
- extensions
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus # 自定義名字
namespace: kube-system # 名稱空間
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef: # 選擇需要繫結的Role
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects: # 物件
- kind: ServiceAccount
name: prometheus
namespace: kube-system
---
apiVersion: v1
kind: Secret
metadata:
namespace: kube-system
name: prometheus
annotations:
kubernetes.io/service-account.name: prometheus
type: kubernetes.io/service-account-token
獲取token命令
root@k8s-master01:~# TOKEN=$(kubectl -n kube-system get secret prometheus -o jsonpath='{.data.token}'| base64 --decode )
root@k8s-master01:~# echo $TOKEN
eyJhbGciOiJSUzI1NiIsImtpZCI6ImFVMS1mYlhobWIxcF92djBwbUIxZDhTVlFWd0VNa3VpNDlmOUhqcG9qSlkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2OTg2NWUwYy0yOGE4LTQ3YTEtYWEzYy03NThmNDlkYjA1YWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06cHJvbWV0aGV1cyJ9.SlORtZtVvLptYWE3zvblaOMxHNMBrHVTTHra7fO1RrwxdK3Bzc42ETLvkzVfQmAQlWrq5yiB4HiKFLe4qY2KVwK3qLDS_sWADLI16Sv8-O1Dt0oOQ0UZD0VOSGY0XEq2EGUxgxnx_JWllgEuMd0rjxtAtyjFh9wjCo_07lFCj44BffGGFp6Kovd8Dl_CJKpORakaJW-haIvTmTlbFPbRKojRTyKvtNCVn0zIXsz8Esp7z9XZmtUvZmHqNlY7bAFtGM9qLWUY_PkM1C0lQ2ZDKASdhZpx6LJr1Wo4WSNILCVfECT0sd6TnFHbgd1NwBc0kcTct5VbST76AJwUpG5esA
在http請求中把上面的TOKEN 作為beare-token傳過去就可以了
請求的curl命令如下(在kube-scheduler部署的節點上執行，通常是master)
curl -k -s https://localhost:10259/configz --header "Authorization: Bearer $TOKEN" |python -m json.tool要點04 根據返回的json配置段來分析外掛開啟的情況
- json資料如下
{
"componentconfig": {
"AlgorithmSource": {
"Policy": null,
"Provider": "DefaultProvider"
},
"ClientConnection": {
"AcceptContentTypes": "",
"Burst": 100,
"ContentType": "application/vnd.kubernetes.protobuf",
"Kubeconfig": "/etc/k8s/scheduler.conf",
"QPS": 50
},
"EnableContentionProfiling": true,
"EnableProfiling": true,
"Extenders": null,
"HealthzBindAddress": "0.0.0.0:10251",
"LeaderElection": {
"LeaderElect": true,
"LeaseDuration": "15s",
"RenewDeadline": "10s",
"ResourceLock": "leases",
"ResourceName": "kube-scheduler",
"ResourceNamespace": "kube-system",
"RetryPeriod": "2s"
},
"MetricsBindAddress": "0.0.0.0:10251",
"Parallelism": 16,
"PercentageOfNodesToScore": 0,
"PodInitialBackoffSeconds": 1,
"PodMaxBackoffSeconds": 10,
"Profiles": [
{
"PluginConfig": [
{
"Args": {
"MinCandidateNodesAbsolute": 100,
"MinCandidateNodesPercentage": 10
},
"Name": "DefaultPreemption"
},
{
"Args": {
"HardPodAffinityWeight": 1
},
"Name": "InterPodAffinity"
},
{
"Args": {
"AddedAffinity": null
},
"Name": "NodeAffinity"
},
{
"Args": {
"IgnoredResourceGroups": null,
"IgnoredResources": null
},
"Name": "NodeResourcesFit"
},
{
"Args": {
"Resources": [
{
"Name": "cpu",
"Weight": 1
},
{
"Name": "memory",
"Weight": 1
}
]
},
"Name": "NodeResourcesLeastAllocated"
},
{
"Args": {
"DefaultConstraints": null,
"DefaultingType": "System"
},
"Name": "PodTopologySpread"
},
{
"Args": {
"BindTimeoutSeconds": 600
},
"Name": "VolumeBinding"
}
],
"Plugins": {
"Bind": {
"Disabled": null,
"Enabled": [
{
"Name": "DefaultBinder",
"Weight": 0
}
]
},
"Filter": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeUnschedulable",
"Weight": 0
},
{
"Name": "NodeName",
"Weight": 0
},
{
"Name": "TaintToleration",
"Weight": 0
},
{
"Name": "NodeAffinity",
"Weight": 0
},
{
"Name": "NodePorts",
"Weight": 0
},
{
"Name": "NodeResourcesFit",
"Weight": 0
},
{
"Name": "VolumeRestrictions",
"Weight": 0
},
{
"Name": "EBSLimits",
"Weight": 0
},
{
"Name": "GCEPDLimits",
"Weight": 0
},
{
"Name": "NodeVolumeLimits",
"Weight": 0
},
{
"Name": "AzureDiskLimits",
"Weight": 0
},
{
"Name": "VolumeBinding",
"Weight": 0
},
{
"Name": "VolumeZone",
"Weight": 0
},
{
"Name": "PodTopologySpread",
"Weight": 0
},
{
"Name": "InterPodAffinity",
"Weight": 0
}
]
},
"Permit": {
"Disabled": null,
"Enabled": null
},
"PostBind": {
"Disabled": null,
"Enabled": null
},
"PostFilter": {
"Disabled": null,
"Enabled": [
{
"Name": "DefaultPreemption",
"Weight": 0
}
]
},
"PreBind": {
"Disabled": null,
"Enabled": [
{
"Name": "VolumeBinding",
"Weight": 0
}
]
},
"PreFilter": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeResourcesFit",
"Weight": 0
},
{
"Name": "NodePorts",
"Weight": 0
},
{
"Name": "PodTopologySpread",
"Weight": 0
},
{
"Name": "InterPodAffinity",
"Weight": 0
},
{
"Name": "VolumeBinding",
"Weight": 0
}
]
},
"PreScore": {
"Disabled": null,
"Enabled": [
{
"Name": "InterPodAffinity",
"Weight": 0
},
{
"Name": "PodTopologySpread",
"Weight": 0
},
{
"Name": "TaintToleration",
"Weight": 0
}
]
},
"QueueSort": {
"Disabled": null,
"Enabled": [
{
"Name": "PrioritySort",
"Weight": 0
}
]
},
"Reserve": {
"Disabled": null,
"Enabled": [
{
"Name": "VolumeBinding",
"Weight": 0
}
]
},
"Score": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeResourcesBalancedAllocation",
"Weight": 1
},
{
"Name": "ImageLocality",
"Weight": 1
},
{
"Name": "InterPodAffinity",
"Weight": 1
},
{
"Name": "NodeResourcesLeastAllocated",
"Weight": 1
},
{
"Name": "NodeAffinity",
"Weight": 1
},
{
"Name": "NodePreferAvoidPods",
"Weight": 10000
},
{
"Name": "PodTopologySpread",
"Weight": 2
},
{
"Name": "TaintToleration",
"Weight": 1
}
]
}
},
"SchedulerName": "default-scheduler"
}
]
}
}
前置知識 k8s的排程框架 scheduler framework
- 文件位置
- 這張架構圖一定要爛熟於心
我們發現在 componentconfig.Profiles其實主要分兩塊：
Plugins段代表排程框架每個擴充套件點的外掛開啟和禁用的情況：
- 可以理解為在各個階段：需要依次執行開啟列表中的外掛，不執行禁用列表中的外掛
- 比如在Filter階段(我節選了一部分) 都是我們們經常能看到的外掛：去掉NodeUnschedulable 、過濾NodeName、過濾TaintToleration 還有根據request資源情況的NodeResourcesFit
"Filter": {
"Disabled": null,
"Enabled": [
{
"Name": "NodeUnschedulable",
"Weight": 0
},
{
"Name": "NodeName",
"Weight": 0
},
{
"Name": "TaintToleration",
"Weight": 0
},
{
"Name": "NodeAffinity",
"Weight": 0
},
{
"Name": "NodePorts",
"Weight": 0
},
{
"Name": "NodeResourcesFit",
"Weight": 0
},
]
},- PluginConfig段代表各個外掛的配置欄位：這裡是個map 的k-v形式：很好理解就是每個外掛的配置欄位不一樣，直接用map最方便
- 比如 NodeResourcesLeastAllocated外掛
{
"Args": {
"Resources": [
{
"Name": "cpu",
"Weight": 1
},
{
"Name": "memory",
"Weight": 1
}
]
},
"Name": "NodeResourcesLeastAllocated"
},要點6：關於外掛權重的討論
- 在上面的配置中可以看到很多 階段如Filter和Score 的外掛中都有權重的配置
{
"Name": "NodePreferAvoidPods",
"Weight": 10000
},- 但是隻有在Score中的外掛的Weight值才大於0
- 其餘階段的外掛的Weight值都是0
結合程式碼看看 ：找到 Plugin.Weight (1.22分支)
- 位置 D:\go_path\src\github.com\kubernetes\kubernetes\pkg\scheduler\apis\config\types.go
// Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.
type Plugin struct {
// Name defines the name of plugin
Name string
// Weight defines the weight of plugin, only used for Score plugins.
Weight int32
}
- 我們發現Plugin結構體就是一個名字和權重
- 並且從註釋中明確的看到了權重只有在 Score階段的外掛中才有作用