1款開源工具，實現自動化升級K3S叢集！

k3s中文社群發表於2020-12-18

即便你的叢集能夠平穩執行，Kubernetes升級依舊是一項艱難的任務。由於每3個月Kubernetes會釋出一個新版本，所以升級是十分必要的。如果一年內你不升級你的Kubernetes叢集，你就會落後許多。Rancher致力於解決開發運維人員的痛點，於是建立了新的開源專案System Upgrade Controller可以幫助開發人員平滑升級。

System Upgrade Controller引入了一個新的Kubernetes自定義資源定義（CRD），稱為Plan。現在Plan是處理升級程式的主要元件。以下是從git repo獲取的架構圖：

在這裡插入圖片描述

使用System Upgrade Controller自動升級K3s

升級K3s Kubernetes叢集有兩個主要要求：

CRD安裝
建立Plan

首先，讓我們檢查當前正在執行的K3s叢集版本。

執行以下命令，即可快速安裝：

#For master install:
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.16.3-k3s.2 sh

#For joining nodes:
K3S_TOKEN is created at /var/lib/rancher/k3s/server/node-token on the server.
For adding nodes, K3S_URL and K3S_TOKEN needs to be passed:

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=XXX sh -


KUBECONFIG file is create at /etc/rancher/k3s/k3s.yaml location

kubectl get nodes

NAME               STATUS   ROLES    AGE   VERSION
kube-node-c155     Ready    <none>   25h   v1.16.3-k3s.2
kube-node-2404     Ready    <none>   25h   v1.16.3-k3s.2
kube-master-303d   Ready    master   25h   v1.16.3-k3s.2

現在，我們部署CRD：

kind: Namespace
metadata:
  name: system-upgrade
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: system-upgrade
  namespace: system-upgrade
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name:  system-upgrade
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: system-upgrade
  namespace: system-upgrade
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: default-controller-env
  namespace: system-upgrade
data:
  SYSTEM_UPGRADE_CONTROLLER_DEBUG: "false"
  SYSTEM_UPGRADE_CONTROLLER_THREADS: "2"
  SYSTEM_UPGRADE_JOB_ACTIVE_DEADLINE_SECONDS: "900"
  SYSTEM_UPGRADE_JOB_BACKOFF_LIMIT: "99"
  SYSTEM_UPGRADE_JOB_IMAGE_PULL_POLICY: "Always"
  SYSTEM_UPGRADE_JOB_KUBECTL_IMAGE: "rancher/kubectl:v1.18.3"
  SYSTEM_UPGRADE_JOB_PRIVILEGED: "true"
  SYSTEM_UPGRADE_JOB_TTL_SECONDS_AFTER_FINISH: "900"
  SYSTEM_UPGRADE_PLAN_POLLING_INTERVAL: "15m"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: system-upgrade-controller
  namespace: system-upgrade
spec:
  selector:
    matchLabels:
      upgrade.cattle.io/controller: system-upgrade-controller
  template:
    metadata:
      labels:
        upgrade.cattle.io/controller: system-upgrade-controller # necessary to avoid drain
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - {key: "node-role.kubernetes.io/master", operator: In, values: ["true"]}
      serviceAccountName: system-upgrade
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
        - key: "node-role.kubernetes.io/master"
          operator: "Exists"
          effect: "NoSchedule"
      containers:
        - name: system-upgrade-controller
          image: rancher/system-upgrade-controller:v0.5.0
          imagePullPolicy: IfNotPresent
          envFrom:
            - configMapRef:
                name: default-controller-env
          env:
            - name: SYSTEM_UPGRADE_CONTROLLER_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['upgrade.cattle.io/controller']
            - name: SYSTEM_UPGRADE_CONTROLLER_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
            - name: etc-ssl
              mountPath: /etc/ssl
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: etc-ssl
          hostPath:
            path: /etc/ssl
            type: Directory
        - name: tmp
          emptyDir: {}

將上面的yaml分解，它將建立以下元件：

system-upgrade名稱空間
system-upgrade服務賬戶
system-upgrade ClusterRoleBinding
用於設定容器中環境變數的config map
實際部署

現在，我們來部署yaml：

#Get the Lateest release tag
curl -s "https://api.github.com/repos/rancher/system-upgrade-controller/releases/latest" | awk -F '"' '/tag_name/{print $4}'
v0.6.2

# Apply the controller manifest
kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/v0.6.2/manifests/system-upgrade-controller.yaml

namespace/system-upgrade created
serviceaccount/system-upgrade created
clusterrolebinding.rbac.authorization.k8s.io/system-upgrade created
configmap/default-controller-env created
deployment.apps/system-upgrade-controller created

# Verify everything is running
kubectl get all -n system-upgrade

NAME                                             READY   STATUS    RESTARTS   AGE
pod/system-upgrade-controller-7fff98589f-blcxs   1/1     Running   0          5m26s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/system-upgrade-controller   1/1     1            1           5m28s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/system-upgrade-controller-7fff98589f   1         1         1       5m28s

建立一個K3s升級Plan

現在，是時候建立一個升級Plan。我們將使用在Git repo示例資料夾中提到的示例Plan。

---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-server
  namespace: system-upgrade
  labels:
    k3s-upgrade: server
spec:
  concurrency: 1
  version: v1.17.4+k3s1
  nodeSelector:
    matchExpressions:
      - {key: k3s-upgrade, operator: Exists}
      - {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
      - {key: k3s.io/hostname, operator: Exists}
      - {key: k3os.io/mode, operator: DoesNotExist}
      - {key: node-role.kubernetes.io/master, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  cordon: true
#  drain:
#    force: true
  upgrade:
    image: rancher/k3s-upgrade
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-agent
  namespace: system-upgrade
  labels:
    k3s-upgrade: agent
spec:
  concurrency: 2
  version: v1.17.4+k3s1
  nodeSelector:
    matchExpressions:
      - {key: k3s-upgrade, operator: Exists}
      - {key: k3s-upgrade, operator: NotIn, values: ["disabled", "false"]}
      - {key: k3s.io/hostname, operator: Exists}
      - {key: k3os.io/mode, operator: DoesNotExist}
      - {key: node-role.kubernetes.io/master, operator: NotIn, values: ["true"]}
  serviceAccountName: system-upgrade
  prepare:
    # Since v0.5.0-m1 SUC will use the resolved version of the plan for the tag on the prepare container.
    # image: rancher/k3s-upgrade:v1.17.4-k3s1
    image: rancher/k3s-upgrade
    args: ["prepare", "k3s-server"]
  drain:
    force: true
  upgrade:
    image: rancher/k3s-upgrade

拆解以上yaml，它將建立：

與表示式匹配的Plan，以瞭解需要升級的內容。所以在上述例子中，我們有2個plan：k3s-server和k3s-agent。node-role.kubernetes.io/master為true和k3s-upgrade的節點將被server Plan佔用。帶false的將由client Plan佔用。所以標籤必須要設定正確。接下來，我們來apply Plan。

#Set the Node Labels

kubectl label node kube-master-303d node-role.kubernetes.io/master=true


# Apply the plan manifest
kubectl apply -f https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/examples/k3s-upgrade.yaml
plan.upgrade.cattle.io/k3s-server created
plan.upgrade.cattle.io/k3s-agent created

# We see that the jobs have started
kubectl get jobs -n system-upgrade
NAME                                                              COMPLETIONS   DURATION   AGE
apply-k3s-server-on-kube-master-303d-with-9efdeac5f6ede78-125aa   0/1           40s        40s
apply-k3s-agent-on-kube-node-2404-with-9efdeac5f6ede78917-07df3   0/1           39s        39s
apply-k3s-agent-on-kube-node-c155-with-9efdeac5f6ede78917-9a585   0/1           39s        39s



# Upgrade in-progress, completed on the `node-role.kubernetes.io/master=true` node
kubectl get nodes
NAME               STATUS                     ROLES    AGE   VERSION
kube-node-2404     Ready,SchedulingDisabled   <none>   26h   v1.16.3-k3s.2
kube-node-c155     Ready,SchedulingDisabled   <none>   26h   v1.16.3-k3s.2
kube-master-303d   Ready                      master   26h   v1.17.4+k3s1

# In a few minutes all nodes get upgraded to latest version as per the plan
kubectl get nodes
NAME               STATUS   ROLES    AGE   VERSION
kube-node-2404     Ready    <none>   26h   v1.17.4+k3s1
kube-node-c155     Ready    <none>   26h   v1.17.4+k3s1
kube-master-303d   Ready    master   26h   v1.17.4+k3s1

我們的K3s Kubernetes升級完成！極為輕鬆而且十分順利。Project可以更新底層作業系統並重啟節點。歡迎嘗試喲！

Github地址：

https://github.com/rancher/system-upgrade-controller

叢集自動化工具--Ansible
2024-12-08
Kubernetes 叢集無損升級實踐
2021-12-20
Elasticsearch叢集升級指引
2021-04-08
Elasticsearch
Captial One如何實現Artifactory HA叢集的自動化維護
2019-12-31
APT
Slack使用開源元件構建自動驅動Kafka叢集
2022-03-13
元件Kafka
在Rainbond中實現資料庫結構自動化升級
2022-02-10
AI資料庫
[自動化]基於kolla的自動化部署ceph叢集
2022-02-28
flyway實現java 自動升級SQL指令碼
2021-07-07
JavaSQL指令碼
使用開源元件構建自動運維Kafka叢集 - Slack
2022-06-06
元件運維Kafka
CapitalOne - Artifactory高可用叢集的自動化部署實踐
2020-02-17
API
k8s 叢集升級
2024-06-06
K8S
Openshift叢集3.9升級到3.10
2021-09-09
Kubernetes 叢集升級指南：從理論到實踐
2020-10-09
用Ansible 自動化搭建本地Kubernetes叢集
2020-04-05
vivo大規模Kubernetes叢集自動化運維實踐
2023-04-25
運維
vivo大規模 Kubernetes 叢集自動化運維實踐
2022-06-13
運維
多雲搭建 K3S 叢集
2021-11-19
java 自動升級sql指令碼 flyway 工具
2021-07-08
JavaSQL指令碼
升級kubeadm 叢集（只有master單節點）
2024-03-13
AST
k8s叢集自動化維護POD
2020-09-15
K8S
跨雲廠商部署 k3s 叢集
2020-06-19
關閉chrome自動升級的教程 chrome如何取消自動升級
2020-11-21
Chrome
Oracle叢集技術 | 叢集的自啟動系列（一）
2019-01-07
Oracle
使用開源API Logic Server實現業務邏輯模型自動化
2022-07-03
APIServer模型
升級 kubeadm 部署的 k8s 叢集
2024-08-17
K8S
Centos7升級K8S叢集
2024-06-17
CentOSK8S
Selenium自動化實現web自動化-1
2021-09-12
Web
通過memberlist庫實現gossip管理叢集以及叢集資料互動
2022-07-12
Go
IDEA升級開源框架
2021-11-02
Idea框架
「實戰篇」開源專案docker化運維部署-搭建mysql叢集（四）
2019-03-28
Docker運維MySql
vivo 萬臺規模 HDFS 叢集升級 HDFS 3.x 實踐
2022-05-16
Memcache/Redis叢集管理探索與實現：美圖開源PaaS平臺資源閘道器
2018-10-10
Redis
安全啟動和安全升級的實現
2024-04-08
iOS 元件化開發（四）：fastlane實現pod自動化
2018-04-06
iOS元件化AST
Kubernetes 叢集無損升級實踐轉至後設資料結尾
2021-12-20
螞蟻大規模 Kubernetes 叢集無損升級實踐指南【探索篇】
2022-02-11
如何安裝一個高可用K3s叢集？
2020-10-30
6 個開源的家庭自動化工具
2018-03-05

1款開源工具，實現自動化升級K3S叢集！

使用System Upgrade Controller自動升級K3s

建立一個K3s升級Plan

相關文章