sealos踩坑記錄

前言

記錄下我安裝sealos的踩坑歷程,全網基本沒有什麼類似的可靠資料,也許是因為太小眾了吧,希望能幫助到搜尋到此文的人.

sealos是什麼

Sealos 是以 kubernetes 為核心的雲作業系統發行版, 單機作業系統如同 linux 發行版本可以在上面安裝和使用各種單機應用，如 PPT，Word，Excel 等。 雲作業系統只需要把這些單機應用替換成各種雲應用，如資料庫，物件儲存，訊息佇列等，就很容易理解了，這些應用都是分散式高可用的。 Sealos 就是能支撐執行各種分散式應用的雲作業系統。有了 Sealos 就擁有了一朵雲。
主要資料參考這裡介紹 | sealos 這裡不做贅述

資料

• 介紹 | sealos
• labring/sealos: Sealos is a Kubernetes distribution, a general-purpose Cloud Operating System designed for managing cloud-native applications. Demo: https://cloud.sealos.io (github.com)
• sealerio/sealer: Build, Share and Run Both Your Kubernetes Cluster and Distributed Applications (Project under CNCF) --- sealerio/sealer：構建、共享和執行您的 Kubernetes 叢集和分散式應用程式（CNCF 下的專案） (github.com)
• 用到的基本映象可以從這裡找到: labring's Profile | Docker Hub
• 安裝4.17版本 https://github.com/labring/sealos/releases/download/v4.1.7/sealos_4.1.7_linux_amd64.tar.gz
• 安裝crictl命令 https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz
• 命令參考 Kubernetes 生命週期管理 | sealos

架構

• 本身資料中沒有畫,要麼從程式碼中提煉
• 閱讀程式碼,瞭解設計模式和程式碼架構,瞭解基礎操作和實現

安裝

官方操作

4.0版本的sealos

# 安裝前必讀

1.目前只支援root使用者，不支援非root和sudo
2.目前只支援在叢集內的節點執行安裝命令
3.提前解除安裝掉已安裝的docker
4.3.0版本的k8s離線包無法使用4.0版本的sealos安裝
5.run命令時如果密碼有特殊字元，請加英文單引號
6.離線安裝示例：
4.0離線安裝示例:

---
# 映象打包, 在有外網的機器上執行

sealos pull labring/kubernetes:v1.24.0
sealos pull labring/calico:v3.22.1
sealos save -o kubernetes.tar labring/kubernetes:v1.24.0
sealos save -o calico.tar labring/calico:v3.22.1

---
# 載入映象, 內網機器執行

sealos load -i kubernetes.tar
sealos load -i calico.tar

主機

主機	用途
10.55.10.107	計劃作為sealos的安裝機,以及master節點
10.55.10.106	node節點1
10.55.10.97	node節點2

可以選擇打通免密,方便定位問題

ssh-keygen -t rsa
cat id_rsa.pub >> authorized_keys
vim authorized_keys # 新增秘鑰
vim /etc/ssh/sshd_config # 修改允許root登入 PermitRootLogin yes

systemctl restart sshd

前置檢查和檔案準備

# 主機只有掛載的/data01磁碟支援overlay,所以註定了沒法向上面官方檔案給出的那麼簡單的就能安裝完成
[root@test-d-010055010107 data01]# xfs_info  /data01
meta-data=/dev/vdb               isize=512    agcount=4, agsize=5242880 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=20971520, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=10240, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


lsmod | grep -e ip_vs -e nf_conntrack_ipv4

# 檔案準備,從遠處pull下來,然後save成映象包
ctr image import kubernetes.tar
ctr image import calico.tar
ctr images export calico.tar docker.io/labring/calico:v3.22.1

wget https://github.com/labring/sealos/releases/download/v4.1.4/sealos_4.1.4_linux_amd64.tar.gz \
   && tar zxvf sealos_4.1.4_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin

# sealos_4.1.4 和 sealos_4.1.7 在Global Flags地方有區別,並且4.1.4有bug無法完成當前主機叢集的正常部署,需要使用4.1.7版本

單機安裝

# 遇到檔案格式問題,需要指定主目錄
[root@test-d-010055010107 data01]# ./sealos run
Error: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported
kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported

# 載入映象包有問題,需要指定映象解包格式
[root@test-d-010055010107 data01]# sealos --root /data01/ --runroot /data01/ load -i kubernetes.tar
Error: loading index: open /var/tmp/oci1097864579/index.json: no such file or directory
loading index: open /var/tmp/oci1097864579/index.json: no such file or directory

# 常用命令
mkdir /data01/sealos
sealos --debug --root /data01/sealos --runroot /data01/sealos/docker load -i calico.tar -t docker-archive
sealos --debug --root /data01/sealos --runroot /data01/sealos/docker load -i new-kubernetes.tar -t oci-archive
sealos load --help
sealos --debug --root /data01/sealos --runroot /data01/sealos/docker run localhost/labring/kuberentes:v1.24 --single # 透過映象名有問題,這裡直接用映象id
sealos --debug --root /data01/sealos --runroot /data01/sealos/docker run 133c6a0a0d5f --single

# 重置安裝
sealos --debug --root /data01/sealos --runroot /data01/sealos/docker reset

# 簡化命令
alias s="sealos --debug --root /data01/sealos --runroot /data01/sealos/docker "

s run 133c6a0a0d5f --single

[root@test-d-010055010107 sealos]# s images
REPOSITORY                     TAG       IMAGE ID       CREATED        SIZE
docker.io/labring/kubernetes   v1.24     133c6a0a0d5f   10 days ago    635 MB
docker.io/labring/helm         v3.8.2    1123e8b4b455   7 months ago   45.1 MB
docker.io/labring/calico       v3.22.1   29516dc98b4b   9 months ago   546 MB

# sealos version must >= v4.1.0
s reset
s run 133c6a0a0d5f 1123e8b4b455 29516dc98b4b --single

# 手動執行image-cri-shim啟動,還是有問題,檢視有報錯
/usr/bin/image-cri-shim -f /etc/image-cri-shim.yaml

fatal failed to setup image_shim, cri/shim: failed to register image service: falling using CRI v1 image API, please using other cri support v1 CRI API
fatal failed to setup image_shim, cri/shim: failed to register image service: falling using CRI v1alpha2 image API, please using other cri support v1alpha2 CRI API

# 排查containerd,看到有報錯資訊
[root@test-d-010055010107 sealos]# systemctl status containerd -l
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-03-27 17:49:48 CST; 16h ago
     Docs: https://containerd.io
 Main PID: 7077 (containerd)
   Memory: 13.9M
   CGroup: /system.slice/containerd.service
           └─7077 /usr/bin/containerd

Mar 27 17:49:48 test-d-010055010107 systemd[1]: Starting containerd container runtime...
Mar 27 17:49:48 test-d-010055010107 containerd[7077]: time="2023-03-27T17:49:48.229104592+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
Mar 27 17:49:48 test-d-010055010107 containerd[7077]: time="2023-03-27T17:49:48.229191393+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Mar 27 17:49:48 test-d-010055010107 containerd[7077]: time="2023-03-27T17:49:48.229403283+08:00" level=warning msg="could not use snapshotter overlayfs in metadata plugin" error="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs does not support d_type. If the backing filesystem is xfs, please reformat with ftype=1 to enable d_type support"
Mar 27 17:49:48 test-d-010055010107 containerd[7077]: time="2023-03-27T17:49:48.229420619+08:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Mar 27 17:49:48 test-d-010055010107 containerd[7077]: time="2023-03-27T17:49:48.238313538+08:00" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to find snapshotter \"overlayfs\""
Mar 27 17:49:48 test-d-010055010107 systemd[1]: Started containerd container runtime.


# 懷疑是 containerd 沒有安裝成功,嘗試安裝crictl命令來看看
tar zxvf crictl-v1.25.0-linux-amd6.tar.gz  -C /usr/local/bin

# 檢視資訊,確定是這個問題,嘗試修復
[root@test-d-010055010107 sealos]# crictl info
E0328 10:07:11.802780   10291 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
FATA[0000] getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

# 檢視containerd關於overlayfs的配置,以及修改目錄
cp -r /var/lib/container* /data01/
vim /etc/containerd/config.toml 修改 root = "/data01/containerd"

# 順利啟動containerd和image-cri-shim
systemctl restart containerd
systemctl restart image-cri-shim

# 遇到了 /root/.sealos/default/etc/admin.conf 找不到的問題,看著issue需要升級到4.1.7版本,問題解決但又然後發現重複安裝有問題,無法繼續上次安裝
s reset # 重新開始

# 但是安裝出來的containerd還是在/var/lib/containerd,需要找到改變此路徑的方法,翻閱檔案猜測指定criData環境變數可能有用

# 改變命令
s run 133c6a0a0d5f --single --env criData=/data01/containerd

# 的確有用,會把containerd安裝到/data01/containerd,但是/root/.sealos/default/Clusterfile中顯示的criData還是/var/lib/containerd

# 成功安裝

# 但是節點一直未就緒
[root@test-d-010055010107 sealos]# kubectl get node
NAME                             STATUS     ROLES           AGE     VERSION
test-d-010055010107   NotReady   control-plane   8m56s   v1.24.0

KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

[root@test-d-010055010107 sealos]# crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD
5d3572591a876       77b49675beae1       12 minutes ago      Running             kube-proxy                0                   dc61529f47415       kube-proxy-vjjqv
9559b3a7d80ec       aebe758cef4cd       12 minutes ago      Running             etcd                      0                   1a1846fb97f25       etcd-test-d-010055010107
00a5f23d7d227       529072250ccc6       12 minutes ago      Running             kube-apiserver            0                   b65e60cdc8996       kube-apiserver-test-d-010055010107
91b737d89b72e       e3ed7dee73e93       12 minutes ago      Running             kube-scheduler            0                   e682c3fb7cc11       kube-scheduler-test-d-010055010107
dd3a2ea10b7c7       88784fb4ac2f6       12 minutes ago      Running             kube-controller-manager   0                   d3177bd65479c       kube-controller-manager-test-d-010055010107

[root@test-d-010055010107 sealos]# kubectl get pod -A
NAMESPACE     NAME                                                     READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d4b75cb6d-qfnf5                                 0/1     Pending   0          3h24m
kube-system   coredns-6d4b75cb6d-xzjz5                                 0/1     Pending   0          3h24m
kube-system   etcd-test-d-010055010107                      1/1     Running   0          3h24m
kube-system   kube-apiserver-test-d-010055010107            1/1     Running   0          3h24m
kube-system   kube-controller-manager-test-d-010055010107   1/1     Running   0          3h24m
kube-system   kube-proxy-vjjqv                                         1/1     Running   0          3h24m
kube-system   kube-scheduler-test-d-010055010107            1/1     Running   0          3h24m

[root@test-d-010055010107 sealos]# journalctl -xeu kubelet
Mar 28 11:43:40 test-d-010055010107 kubelet[20385]: E0328 11:43:40.678552   20385 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=f
Mar 28 11:43:45 test-d-010055010107 kubelet[20385]: E0328 11:43:45.679314   20385 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=f

# 看issue上是說沒有安裝calico導致的,重新安裝
s reset # 並不會刪除/root/.sealos
s run 133c6a0a0d5f 1123e8b4b455 29516dc98b4b --single --env criData=/data01/containerd

# 看著一切正常
[root@test-d-010055010107 sealos]# kubectl get pod -A
NAMESPACE         NAME                                                     READY   STATUS    RESTARTS   AGE
calico-system     calico-kube-controllers-6b44b54755-qsmkl                 0/1     Pending   0          115s
calico-system     calico-node-7grz7                                        1/1     Running   0          115s
calico-system     calico-typha-6f9598cfd9-2sr27                            1/1     Running   0          115s
kube-system       coredns-6d4b75cb6d-6fncr                                 1/1     Running   0          2m2s
kube-system       coredns-6d4b75cb6d-b8czk                                 1/1     Running   0          2m2s
kube-system       etcd-test-d-010055010107                      1/1     Running   1          2m16s
kube-system       kube-apiserver-test-d-010055010107            1/1     Running   1          2m18s
kube-system       kube-controller-manager-test-d-010055010107   1/1     Running   1          2m16s
kube-system       kube-proxy-wnp2g                                         1/1     Running   0          2m3s
kube-system       kube-scheduler-test-d-010055010107            1/1     Running   1          2m16s
tigera-operator   tigera-operator-d7957f5cc-5wfc4                          1/1     Running   0          2m2s
[root@test-d-010055010107 sealos]#
[root@test-d-010055010107 sealos]#
[root@test-d-010055010107 sealos]# kubectl get node
NAME                             STATUS   ROLES           AGE     VERSION
test-d-010055010107   Ready    control-plane   2m25s   v1.24.0

叢集安裝

有了單機安裝的經驗,該踩的坑都踩了,直接開始安裝叢集

# 嘗試叢集安裝
alias s="sealos --debug --root /data01/sealos --runroot /data01/sealos/docker "
s run 133c6a0a0d5f 1123e8b4b455 29516dc98b4b -e defaultVIP=10.55.10.108 -e criData=/data01/containerd  --masters 10.55.10.107 --nodes 10.55.10.97,10.55.10.106 --passwd 112233
passwd 112233

[root@test-d-010055010107 ~]# kubectl get node -o wide
NAME                             STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
test-d-010055010097   Ready    <none>          65s   v1.24.0   10.55.10.97    <none>        CentOS Linux 7 (Core)   3.10.0-693.11.6.el7.x86_64   containerd://1.7.0
test-d-010055010106   Ready    <none>          76s   v1.24.0   10.55.10.106   <none>        CentOS Linux 7 (Core)   3.10.0-693.11.6.el7.x86_64   containerd://1.7.0
test-d-010055010107   Ready    control-plane   95s   v1.24.0   10.55.10.107   <none>        CentOS Linux 7 (Core)   3.10.0-693.11.6.el7.x86_64   containerd://1.7.0

# 看著沒啥問題

解決問題用到的參考連線

• unsupported graph driver: vfs · Issue #1576 · sealerio/sealer (github.com)
• 概覽 | sealer 有些問題可能也要參考這個檔案
• Question: Can sealos load -i use docker save -o image.tar? · Issue #2526 · labring/sealos --- 問：sealos可以載入-i use docker儲存-o image.tar嗎？·問題#2526 · labring/sealos (github.com)
• crictl安裝 - 小吉貓 - 部落格園 (cnblogs.com)
• (22條訊息) Containerd 安裝過程以及踩的坑_/var/lib/containerd_Aisaka81的部落格-CSDN部落格
• error Applied to cluster error: read admin.conf error in guest: open /root/.sealos/default/etc/admin.conf: no such file or directory · Issue #2548 · labring/sealos (github.com)
• sealos4.0首次安裝失敗，再次安裝沒有任何提示且安裝未成功 · Issue #1207 · labring/sealos (github.com)
• 單機安裝API Server未起來，kubelet也無法啟動 · Issue #2313 · labring/sealos (github.com)
• BUG: 單節點部署節點Notready · Issue #1663 · labring/sealos (github.com)
• sealos NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized · Issue #704 · labring/sealos (github.com)
• linux journalctl 命令 - sparkdev - 部落格園 (cnblogs.com) Linux系統檢視日誌命令

感想

• 版本變化多,命令引數有改動,bug隱藏的深
• 需要耐心抽絲剝繭的排查遇到的問題,可以提前安裝些k8s定位問題依賴的命令如ctr/crictl
• 也加入了官方的釘釘群,但基本不答覆問題和諮詢
• 關注issue,也是唯一有價值的參考資料了
• 禁止轉載