開篇
- 《K3s 系列文章》
- 《Rancher 系列文章》
方案
在騰訊雲上安裝 K3S
後續會在這套 K3S 叢集上安裝 Rancher
方案目標
- 高可用
- 3 臺master 的 k3s 叢集
- 資料備份
- k3s 資料備份到 騰訊雲物件儲存 cos
- 儘量複用公有云的能力
Tencent Cloud Controller Manager(❌ 因為騰訊雲已經放棄維護相關原始碼, 所以無法複用)SVC LoadBalancer 呼叫 CLB(❌ 因為騰訊雲已經放棄維護相關原始碼, 所以無法複用)- 備份 - 使用騰訊雲 COS
前提條件
-
有騰訊雲賬戶,賬戶至少擁有如下許可權:auto k3s 安裝 - 設定 CAM 以及這些許可權:
QcloudTAGFullAccess
-
該騰訊雲賬號有對應的 API 金鑰,地址:訪問金鑰 - 控制檯 (tencent.com) ,或者擁有相關許可權:
cam:QueryCollApiKey
和cam:CreateCollApiKey
-
一臺 linux 操作機,用於部署 autok3s
-
一個物件儲存通 cos,用於備份
-
已有的映象倉庫的一些賬號密碼或認證資訊,包括:quay,docker,騰訊雲 (用於加速 pull push映象)
ℹ️ Info:
騰訊雲 tcr 廣州 提供免費個人版例項,可以使用並新增:
K3S 安裝注意事項
- 透過autok3s部署
- 透過 autok3s 安裝後, 預設 k8s api 透過 公網 IP 進行通訊, 需要調整 systemd 配置使其透過內網進行通訊.
- ⚠️付費模式,安裝後可根據具體情況在將付費模式控制檯改為:包年包月
K3S 安裝引數
本次 K3s 安裝引數如下:
- Master
3
臺 - Worker:
0
- Region:shanghai (
ap-shanghai
) - zone:二區(
ap-shanghai-2
) - Instance Type:
S5.MEDIUM8
- Image:
img-22trbn9x
(ubuntu 20.04) - instanceChargeType:預設後付費,且無法調整。⚠️安裝後控制檯改為:PREPAID
- Disk:
CLOUD_SSD
(ℹ️CLOUD_PREMIUM
(高效能雲盤),CLOUD_SSD
(SSD雲硬碟)) - Disk Size:
50
G - VPC ID: 空(autok3s 會自動建立)
- Subnet ID: 空(autok3s 會自動建立)
- Internet Max Bandwidth Out:
5
(可以按需調小) - Security Group Ids:空, 讓 autok3s 自動建立, 叢集建立好之後再調整安全組, 縮小入口範圍
- EIP:是否使用彈性公網IP
false
- Tags (見下文)
- K3s Version:
v1.21.7+k3s1
- Cluster:
true
- Master Extra Args: 見下文
- Cluster模式:
true
- Registry(見下文)
- UI:
true
⚠️ Warning:
執行
autok3s
建立前, 如果選擇已有的安全組, 那麼 CVM 例項至少需要應用以下安全組規則:Rule Protocol Port Source Description InBound TCP 22 ALL SSH Connect Port InBound TCP 6443 K3s agent nodes Kubernetes API InBound TCP 10250 K3s server & agent Kubelet InBound UDP 8472 K3s server & agent (Optional) Required only for Flannel VXLAN InBound TCP 2379,2380 K3s server nodes (Optional) Required only for embedded ETCD OutBound ALL ALL ALL Allow All
特別是: 22 埠必須要對操作機的公網IP 開放
原因: autok3s 自動部署公有云時, 透過公網IP 上傳 KeyPair, 如果沒有以上安全組,
autok3s
會執行失敗. 報錯如下: (101.34.46.218
就是公網 IP)level=error msg="[ssh-dialer] init dialer [101.34.46.218:22] error: [tencent] calling getInstanceStatus error. region: ap-shanghai, zone: ap-shanghai-2, instanceName: [ins-ggxozpyl ins-cfi2vio1 ins-78rkem0b], message: not `RUNNING` status"
安裝步驟
AutoK3s
在操作機上安裝,命令如下:
curl -sS http://rancher-mirror.cnrancher.com/autok3s/install.sh | INSTALL_AUTOK3S_MIRROR=cn sh
過程如下:
Downloading package http://rancher-mirror.rancher.cn/autok3s/v0.4.6/autok3s_linux_amd64 as /tmp/autok3s_linux_amd64
Download complete.
Running with sufficient permissions to attempt to move autok3s to /usr/local/bin
New version of autok3s installed to /usr/local/bin
Version: {"gitVersion":"v0.4.6","gitCommit":"4537e6ee2aea8b204a72f7b6c377edb154f7c058","gitTreeState":"","buildDate":"2021-12-28T04:15:30Z","goVersion":"go1.16.2","compiler":"gc","platform":"linux/amd64"}
Downloading package http://rancher-mirror.rancher.cn/kube-explorer/v0.2.7/kube-explorer-linux-amd64 as /tmp/kube-explorer-linux-amd64
Download complete.
Running with sufficient permissions to attempt to move kube-explorer to /usr/local/bin
New version of kube-explorer installed to /usr/local/bin
Skipping /usr/local/bin/kubectl symlink to autok3s, already exists
您可以透過以下 CLI 命令啟動本地 UI。
autok3s serve --bind-address 0.0.0.0 --bind-port 8087
⚠️ Warning:
頁面無登入認證,確保最小許可權開放以及用完後及時關閉。
輸出如下:
INFO[0000] run as daemon, listening on 127.0.0.1:8087
訪問 UI:http://<操作機IP>:8087
AutoK3s UI 模板
如果今後要多次安裝,可以在 UI 上建立可複用的模板,模板包括如下固定引數:
-
Credential Options:
- 騰訊雲 Secret Id
- 騰訊雲 Secret Key
-
Instance Options
- Basic
- Region:
ap-shanghai
- Zone:
ap-shanghai-2
- Instance Type:
S5.MEDIUM8
- Image:
img-22trbn9x
(ubuntu 20.04) - Disk Category:
CLOUD_SSD
- Disk Size:
50
G
- Region:
- Network
- Internet Max Bandwidth Out:
5
- EIP:
Disable
- ⚠️ 注意:另外 3 個引數:VPC ID、SubnetID、Security Group Ids 每次建立時需要按需填寫或留空
- Internet Max Bandwidth Out:
- SSH Public
- SSH User:
ubuntu
- SSH Port:
22
- Keypair Id : 留空(⚠️ 注意:如果 Keypair Id 留空,會自動生成 Keypair)
- SSH User:
- SSH Private
- SSH Agent Auth:
Disable
- SSH Key Path: 留空(⚠️ 注意:如果上面選擇了 Keypair Id, 那麼對應的SSH Key Path 也要填寫)
- SSH Agent Auth:
- Advance
- 打了 3 個 tags,方便後續管理:
app=rancher
env=prod
provider=k3s
- 打了 3 個 tags,方便後續管理:
- Basic
-
K3s Options
-
Basic
- K3s Channel:
stable
- K3s Version:
v1.21.7+k3s1
(ℹ️ Info: 202201 根據 suse 官網選型的最新穩定版,k3s v1.21.7+k3s1,後面會按需調整版本) - Cluster:
Enable
(啟用叢集模式,使用 etcd 組成高可用叢集) - K3s Install Script:
http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh
- K3s Channel:
-
Master
-
Master:
3
-
Master Extra Args:
--write-kubeconfig-mode "644" --pause-image registry.cn-hangzhou.aliyuncs.com/rancher/pause:3.6 --etcd-s3 --etcd-snapshot-schedule-cron 0 0 * * * --etcd-s3-endpoint cos.ap-shanghai.myqcloud.com --etcd-s3-access-key <your-cos-access-key> --etcd-s3-secret-key <your-cos-secret-key> --etcd-s3-bucket <your-cos-bucket> --etcd-s3-folder /rancher/k3s
-
-
Worker
- Worker:
0
- Worker:
-
Advance: 留空
-
TLS Sans: 留空 (⚠️ 如果前面會使用 CLB 作為負載均衡,那麼建議填上 CLB VIP)
-
Registry, 見下面
registries.yaml
-
-
Additional Options
- UI:
explorer
- UI:
registries.yaml
:
mirrors:
docker.io:
endpoint:
- "https://mirror.ccs.tencentyun.com"
- "https://registry.cn-hangzhou.aliyuncs.com"
- "https://docker.mirrors.ustc.edu.cn"
quay.io:
endpoint:
- "https://mirror.ccs.tencentyun.com"
configs:
'ccr.ccs.tencentyun.com':
auth:
username: <your-account-id>
password: <your-registry-password>
AutoK3s 透過 UI 建立 K3S 叢集
訪問 UI 介面, 點選 Quick Start, Provider 選擇 tencent;
然後在下方填入自定義的資訊, 主要是填寫 Network 的資訊, 如下圖:
點選 Create, 等待返回結果即可.
AutoK3s CLI 命令
您也可以透過以下 CLI 在 騰訊雲上快速建立一個 3 master, 0 worker 節點的 K3s 高可用叢集。
autok3s create --provider tencent --cluster --enable [ "explorer" ] --k3s-channel stable --k3s-install-mirror INSTALL_K3S_MIRROR=cn --k3s-install-script http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh --k3s-version v1.21.7+k3s1 --master 3 --master-extra-args '--write-kubeconfig-mode "644" --pause-image registry.cn-hangzhou.aliyuncs.com/rancher/pause:3.6 --etcd-s3 --etcd-snapshot-schedule-cron 0 0 * * * --etcd-s3-endpoint cos.ap-shanghai.myqcloud.com --etcd-s3-access-key <your-cos-access-key> --etcd-s3-secret-key <your-cos-secret-key> --etcd-s3-bucket <your-cos-bucket> --etcd-s3-folder /rancher/k3s' --name rancher-1 --ssh-port 22 --ssh-user ubuntu --tls-sans <your-clb-ip> --worker 0 --disk-category CLOUD_SSD --disk-size 50 --image img-22trbn9x --instance-type S5.MEDIUM8 --internet-max-bandwidth-out 5 --keypair-id <your-keypair-id> --region ap-shanghai --secret-id <your-tencent-secret-id> --secret-key <your-tencent-secret-key> --tags 'app=rancher' --tags 'env=prod' --tags 'provider=k3s' --zone ap-shanghai-2 --vpc <your-vpc-id> --subnet <your-subnet-id> --registry /etc/autok3s/registries.yaml
安裝成功日誌顯示如下:
time="2022-02-12T14:52:16+08:00" level=info msg="[tencent] executing create logic..."
INFO[0000] [tencent] use existing key pair
time="2022-02-12T14:52:16+08:00" level=info msg="[tencent] 3 masters and 0 workers will be added"
time="2022-02-12T14:52:16+08:00" level=info msg="[tencent] check default security group autok3s in region ap-shanghai"
time="2022-02-12T14:52:16+08:00" level=info msg="[tencent] create default security group autok3s in region ap-shanghai"
time="2022-02-12T14:52:16+08:00" level=info msg="[tencent] check rules of security group autok3s"
time="2022-02-12T14:52:18+08:00" level=info msg="[tencent] 3 number of master instances will be created"
time="2022-02-12T14:52:23+08:00" level=info msg="[tencent] 3 number of master instances successfully created"
time="2022-02-12T14:52:23+08:00" level=info msg="[tencent] waiting for the instances [ins-xxxxx] to be in `RUNNING` status..."
time="2022-02-12T14:52:54+08:00" level=info msg="[tencent] instances [ins-xxxxx] are in `RUNNING` status"
time="2022-02-12T14:52:54+08:00" level=info msg="[tencent] executing init k3s cluster logic..."
time="2022-02-12T14:52:54+08:00" level=info msg="[tencent] creating k3s master-1..."
mirrors:
docker.io:
endpoint:
- https://mirror.ccs.tencentyun.com
- https://registry.cn-hangzhou.aliyuncs.com
- https://docker.mirrors.ustc.edu.cn
quay.io:
endpoint:
- https://mirror.ccs.tencentyun.com
configs:
ccr.ccs.tencentyun.com:
auth:
username:
password:
auth: ""
identity_token: ""
tls: null
auths: {}
time="2022-02-12T14:53:26+08:00" level=info msg="[cluster] k3s master command: curl -sLS http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_TOKEN='xxxxxxx' INSTALL_K3S_EXEC='server --tls-san xxxxx --tls-san xxxxxxxx --tls-san xxxxxxx --node-external-ip xxxxxx --write-kubeconfig-mode \"644\" --pause-image registry.cn-hangzhou.aliyuncs.com/rancher/pause:3.6 --disable-cloud-controller --cluster-cidr 10.42.0.0/16 --cluster-init' INSTALL_K3S_VERSION='v1.21.7+k3s1' sh -"
[INFO] Using v1.21.7+k3s1 as release
[INFO] Downloading hash http://rancher-mirror.cnrancher.com/k3s/v1.21.7-k3s1/sha256sum-amd64.txt
[INFO] Downloading binary http://rancher-mirror.cnrancher.com/k3s/v1.21.7-k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
time="2022-02-12T14:53:59+08:00" level=info msg="[tencent] successfully created k3s master-1"
time="2022-02-12T14:53:59+08:00" level=info msg="[tencent] creating k3s master-2..."
...
time="2022-02-12T14:54:35+08:00" level=info msg="[tencent] successfully created k3s master-2"
time="2022-02-12T14:54:35+08:00" level=info msg="[tencent] creating k3s master-3..."
...
time="2022-02-12T14:55:06+08:00" level=info msg="[tencent] successfully created k3s master-3"
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ......
server: https://127.0.0.1:6443
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: ......
client-key-data: ......
time="2022-02-12T14:55:06+08:00" level=info msg="[tencent] deploying additional manifests"
time="2022-02-12T14:55:06+08:00" level=info msg="[tencent] successfully deployed additional manifests"
time="2022-02-12T14:55:06+08:00" level=info msg="[tencent] successfully executed init k3s cluster logic"
---
time="2022-02-12T14:55:07+08:00" level=info msg="[tencent] successfully deployed manifests"
time="2022-02-12T14:55:07+08:00" level=info msg="=========================== Prompt Info ==========================="
time="2022-02-12T14:55:07+08:00" level=info msg="Use 'autok3s kubectl config use-context prod-ha.ap-shanghai.tencent'"
time="2022-02-12T14:55:07+08:00" level=info msg="Use 'autok3s kubectl get pods -A' get POD status`"
? 到這裡, K3S 叢集建立完畢.
K3s 配置調整
? Important:
安裝後, 預設 k8s api 透過 公網 IP 進行通訊, 出於安全考慮,建議調整 systemd 配置使其透過內網進行通訊.
步驟如下:
修改後兩臺 master 的 /etc/systemd/system/k3s.service.env
的 K3S_URL
為第一臺 master 的內網地址
K3S_URL=https://<master1-internal-ip>:6443
第一臺 master, 修改 /etc/systemd/system/k3s.service
, 增加:
'--node-ip' \
'<master1-internal-ip>' \
'--advertise-address' \
'<master1-internal-ip>' \
另外2臺, 增加及修改如下:
'--server' \
'https://<master1-internal-ip>:6443' \
...
'--node-ip' \
'<other-master-internal-ip>' \
'--advertise-address' \
'<other-master-internal-ip>' \
重啟生效:
systemctl daemon-reload
systemctl restart k3s.service
驗證:
檢視 kubernetes
的 endpoint, 從公網地址變為內網地址, 如下:
apiVersion: v1
kind: Endpoints
metadata:
name: kubernetes
namespace: default
...
labels:
endpointslice.kubernetes.io/skip-mirror: 'true'
managedFields:
- manager: k3s
operation: Update
...
selfLink: /api/v1/namespaces/default/endpoints/kubernetes
subsets:
- addresses:
- ip: <master2-internal-ip>
- ip: <master1-internal-ip>
- ip: <master3-internal-ip>
ports:
- name: https
port: 6443
protocol: TCP
收尾工作
調整安全組
入站規則:
- TCP:22(SSH) 埠許可權收緊
- TCP:6443(K8S API) 埠許可權收緊
- UDP:8472(K3s vxlan) 只開放給內網
- TCP:10250(kube api-server) 只開放給內網
最終效果如下: (應該可以進一步收緊)
總結
??? 至此, 完成騰訊雲上 K3S 高可用叢集的搭建, 並配置備份.
為後面的 Rancher 搭建做好了基礎。
以下是安裝後的相關輸出資訊:
K3s
-
3 個 Master 和 Server hostname、內外網IP
-
K3S API Server 地址:
https://<以上6個IP地址任一個或CLB的IP>:6443
-
K3S kubeconfig 配置:位於 k3s 的
/etc/rancher/k3s/k3s.yaml
以及操作機的/root/.autok3s/.kube/config
三人行, 必有我師; 知識共享, 天下為公. 本文由東風微鳴技術部落格 EWhisper.cn 編寫.