一文搞定 KubeKey 3.1.1 離線部署 KubeSphere 3.4.1 和 Kubernetes v1.28

kubesphere發表於2024-05-23

本文將詳細介紹，如何基於作業系統 openEuler 22.03 LTS SP3，利用 KubeKey 製作 KubeSphere 和 Kubernetes 離線安裝包，並實戰部署 KubeSphere 3.4.1 和 Kubernetes 1.28.8 叢集。

實戰伺服器配置 (架構 1:1 復刻小規模生產環境，配置略有不同)

主機名	IP	CPU	記憶體	系統盤	資料盤	用途
ksp-master-1	192.168.9.91	8	16	40	100	離線環境 KubeSphere/k8s-master
ksp-master-2	192.168.9.92	8	16	40	100	離線環境 KubeSphere/k8s-master
ksp-master-3	192.168.9.93	8	16	40	100	離線環境 KubeSphere/k8s-master
ksp-registry	192.168.9.90	4	8	40	100	離線環境部署節點和映象倉庫節點（Harbor）
ksp-deploy	192.168.9.89	4	8	40	100	聯網主機用於製作離線包
合計	4	32	64	200	500

實戰環境涉及軟體版本資訊

作業系統：openEuler 22.03 LTS SP3
KubeSphere：v3.4.1
Kubernetes：v1.28.8
KubeKey: v3.1.1
Harbor: v2.10.1

1. 離線部署資源製作

本文增加了一臺能聯網的 ksp-deploy 節點，用來製作離線部署資源包。

在該節點下載 KubeKey 最新版（v3.1.1）。具體 KubeKey 版本號可以在 KubeKey 發行頁面檢視。

1.1 下載 KubeKey

下載最新版的 KubeKey

cd ~
mkdir kubekey
cd kubekey/

# 選擇中文區下載(訪問 GitHub 受限時使用)
export KKZONE=cn

# 執行下載命令，獲取最新版的 kk（受限於網路，有時需要執行多次）
curl -sfL https://get-kk.kubesphere.io | sh -

1.2 建立 manifests 檔案

KubeKey v3.1.0 之前 manifests 檔案需要根據模板手動編寫，現在可以透過 KubeKey 的 create manifest 命令自動生成一份 manifests 模板。

create manifest 支援的引數如下

$ ./kk create manifest --help
Create an offline installation package configuration file

Usage:
  kk create manifest [flags]

Flags:
      --arch stringArray         Specify a supported arch (default [amd64])
      --debug                    Print detailed information
  -f, --filename string          Specify a manifest file path
  -h, --help                     help for manifest
      --ignore-err               Ignore the error message, remove the host which reported error and force to continue
      --kubeconfig string        Specify a kubeconfig file
      --name string              Specify a name of manifest object (default "sample")
      --namespace string         KubeKey namespace to use (default "kubekey-system")
      --with-kubernetes string   Specify a supported version of kubernetes
      --with-registry            Specify a supported registry components
  -y, --yes                      Skip confirm check

官方示例（支援多叢集、多架構）

# 示例：建立包含 kubernetes v1.24.17，v1.25.16，且 cpu 架構為 amd64、arm64 的 manifests 檔案。
./kk create manifest --with-kubernetes v1.24.17,v1.25.16 --arch amd64 --arch arm64

建立一個 amd64 架構 kubernetes v1.28.8 的 manifests 檔案

./kk create manifest --with-kubernetes v1.28.8 --arch amd64

生成的配置檔案如下

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  operatingSystems:
  - arch: amd64
    type: linux
    id: ubuntu
    version: "20.04"
    osImage: Ubuntu 20.04.6 LTS
    repository:
      iso:
        localPath:
        url:
  kubernetesDistributions:
  - type: kubernetes
    version: v1.28.8
  components:
    helm:
      version: v3.14.3
    cni:
      version: v1.2.0
    etcd:
      version: v3.5.13
    containerRuntimes:
    - type: docker
      version: 24.0.9
    - type: containerd
      version: 1.7.13
    calicoctl:
      version: v3.27.3
    crictl:
      version: v1.29.0

  images:
  - docker.io/kubesphere/pause:3.9
  - docker.io/kubesphere/kube-apiserver:v1.28.8
  - docker.io/kubesphere/kube-controller-manager:v1.28.8
  - docker.io/kubesphere/kube-scheduler:v1.28.8
  - docker.io/kubesphere/kube-proxy:v1.28.8
  - docker.io/coredns/coredns:1.9.3
  - docker.io/kubesphere/k8s-dns-node-cache:1.22.20
  - docker.io/calico/kube-controllers:v3.27.3
  - docker.io/calico/cni:v3.27.3
  - docker.io/calico/node:v3.27.3
  - docker.io/calico/pod2daemon-flexvol:v3.27.3
  - docker.io/calico/typha:v3.27.3
  - docker.io/flannel/flannel:v0.21.3
  - docker.io/flannel/flannel-cni-plugin:v1.1.2
  - docker.io/cilium/cilium:v1.15.3
  - docker.io/cilium/operator-generic:v1.15.3
  - docker.io/hybridnetdev/hybridnet:v0.8.6
  - docker.io/kubeovn/kube-ovn:v1.10.10
  - docker.io/kubesphere/multus-cni:v3.8
  - docker.io/openebs/provisioner-localpv:3.3.0
  - docker.io/openebs/linux-utils:3.3.0
  - docker.io/library/haproxy:2.9.6-alpine
  - docker.io/plndr/kube-vip:v0.7.2
  - docker.io/kubesphere/kata-deploy:stable
  - docker.io/kubesphere/node-feature-discovery:v0.10.0
  registry:
    auths: {}

修改配置檔案

KubeKey v3.1.1 生成的 manifests 配置檔案適用於 ubuntu 部署純 Kubernetes 叢集，並沒有部署 KubeSphere 的映象。因此，我們需要結合部署 KubeSphere 需要的映象列表，生成一份完整的 manifests 檔案。

下載 KubeSphere 映象列表

wget https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/images-list.txt

完整的映象一共 120 個，受限於篇幅，這裡不做展示，請檢視下文完整的 manifest 檔案。

1.3 獲取作業系統依賴包

本實驗環境使用的作業系統是 x64 的 openEuler 22.03 LTS SP3，需要自己製作安裝 Kubernetes 需要的作業系統依賴包映象 openEuler-22.03-rpms-amd64.iso，其他作業系統請讀者在 KubeKey releases 頁面下載。

個人建議在離線環境用 openEuler 的安裝 ISO，製做一個完整的離線軟體源。在利用 KubeKey 安裝離線叢集時，就不需要考慮作業系統依賴包的問題。

1.4 生成 manifest 檔案

根據上面的檔案及相關資訊，生成最終的 manifest 檔案 ksp-v3.4.1-v1.28.8-manifest.yaml。

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches:
  - amd64
  operatingSystems:
  - arch: amd64
    type: linux
    id: openEuler
    version: "22.03"
    osImage: openEuler 22.03 (LTS-SP3)
    repository:
      iso:
        localPath: "/data/kubekey/openEuler-22.03-rpms-amd64.iso"
        url:
  kubernetesDistributions:
  - type: kubernetes
    version: v1.28.8
  components:
    helm:
      version: v3.14.3
    cni:
      version: v1.2.0
    etcd:
      version: v3.5.13
    containerRuntimes:
    - type: docker
      version: 24.0.9
    - type: containerd
      version: 1.7.13
    calicoctl:
      version: v3.27.3
    crictl:
      version: v1.29.0
    docker-registry:
      version: "2"
    harbor:
      version: v2.5.3
    docker-compose:
      version: v2.2.2
  images:
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.9
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver:v1.28.8
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager:v1.28.8
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler:v1.28.8
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.28.8
  - registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.9.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pod2daemon-flexvol:v3.27.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/typha:v3.27.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel:v0.21.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/flannel-cni-plugin:v1.1.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hybridnet:v0.8.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/multus-cni:v3.8
  - registry.cn-beijing.aliyuncs.com/kubesphereio/provisioner-localpv:3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/linux-utils:3.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.9.6-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node-feature-discovery:v0.10.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-apiserver:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-console:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-controller-manager:v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.20.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kubefed:v0.8.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tower:v0.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/minio:RELEASE.2019-08-07T01-59-21Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/mc:RELEASE.2019-08-07T23-14-43Z
  - registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nginx-ingress-controller:v1.3.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/defaultbackend-amd64:1.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/metrics-server:v0.4.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:5.0.14-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/haproxy:2.0.25-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openldap:1.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/netshoot:v1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/cloudcore:v1.13.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/iptables-manager:v1.13.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/edgeservice:v0.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/gatekeeper:v3.5.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/openpitrix-jobs:v3.3.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-apiserver:ks-v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-controller:ks-v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/devops-tools:ks-v3.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/ks-jenkins:v3.4.0-2.319.3-1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/inbound-agent:4.10-2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-base:v3.2.2-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-nodejs:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-maven:v3.2.1-jdk11-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-python:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.0-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.16-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.17-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/builder-go:v3.2.2-1.18-podman
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2ioperator:v3.2.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2irun:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/s2i-binary:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/tomcat85-java8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-8-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/java-11-runtime:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-8-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-6-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nodejs-4-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-36-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-35-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-34-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/python-27-centos7:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd:v2.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/argocd-applicationset:v0.4.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/dex:v2.30.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/redis:6.2.6-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/configmap-reload:v0.7.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus:v2.39.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-config-reloader:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/prometheus-operator:v0.55.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-state-metrics:v2.6.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/node-exporter:v1.3.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/alertmanager:v0.23.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/thanos:v0.31.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/grafana:8.3.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-rbac-proxy:v0.11.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager-operator:v2.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-manager:v2.3.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/notification-tenant-sidecar:v3.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-curator:v5.7.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/opensearch-curator:v0.0.5
  - registry.cn-beijing.aliyuncs.com/kubesphereio/elasticsearch-oss:6.8.22
  - registry.cn-beijing.aliyuncs.com/kubesphereio/opensearch:2.6.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/opensearch-dashboards:2.6.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluentbit-operator:v0.14.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/docker:19.03
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluent-bit:v1.9.4
  - registry.cn-beijing.aliyuncs.com/kubesphereio/log-sidecar-injector:v1.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/filebeat:6.7.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-operator:v0.6.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-exporter:v0.6.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-events-ruler:v0.6.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-operator:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kube-auditing-webhook:v0.2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/pilot:1.14.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/proxyv2:1.14.6
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-operator:1.29
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-agent:1.29
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-collector:1.29
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-query:1.29
  - registry.cn-beijing.aliyuncs.com/kubesphereio/jaeger-es-index-cleaner:1.29
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali-operator:v1.50.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/kiali:v1.50
  - registry.cn-beijing.aliyuncs.com/kubesphereio/busybox:1.31.1
  - registry.cn-beijing.aliyuncs.com/kubesphereio/nginx:1.14-alpine
  - registry.cn-beijing.aliyuncs.com/kubesphereio/wget:1.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hello:plain-text
  - registry.cn-beijing.aliyuncs.com/kubesphereio/wordpress:4.8-apache
  - registry.cn-beijing.aliyuncs.com/kubesphereio/hpa-example:latest
  - registry.cn-beijing.aliyuncs.com/kubesphereio/fluentd:v1.4.2-2.0
  - registry.cn-beijing.aliyuncs.com/kubesphereio/perl:latest
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-productpage-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-reviews-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-reviews-v2:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-details-v1:1.16.2
  - registry.cn-beijing.aliyuncs.com/kubesphereio/examples-bookinfo-ratings-v1:1.16.3
  - registry.cn-beijing.aliyuncs.com/kubesphereio/scope:1.13.0
  registry:
    auths: {}

manifest 修改說明

開啟 harbor 和 docker-compose 配置項，為後面透過 KubeKey 自建 harbor 倉庫推送映象使用。

預設建立的 manifest 模板中的映象是從 docker.io 獲取，替換字首為 registry.cn-beijing.aliyuncs.com/kubesphereio

operatingSystems 配置刪除預設的 ubuntu，新增 openEuler 配置

1.5 匯出製品 artifact

根據生成的 manifest，執行下面的命令製作製品（artifact）。

export KKZONE=cn
./kk artifact export -m ksp-v3.4.1-v1.28-manifest.yaml -o ksp-v3.4.1-v1.28-artifact.tar.gz

正確執行後，輸出結果如下 :（受限於篇幅，僅展示最終結果）

....
06:05:28 CST success: [LocalHost]
06:05:28 CST [ChownOutputModule] Chown output file
06:05:28 CST success: [LocalHost]
06:05:28 CST [ChownWorkerModule] Chown ./kubekey dir
06:05:28 CST success: [LocalHost]
06:05:28 CST Pipeline[ArtifactExportPipeline] execute successfully

製品製作完成後，檢視製品大小（全映象，製品包居然達到了 13G，生產環境還是有選擇的裁剪吧）。

$ ls -lh ksp-v3.4.1-v1.28-artifact.tar.gz
-rw-r--r-- 1 root root 13G May 20 06:05 ksp-v3.4.1-v1.28-artifact.tar.gz

1.6 匯出 KubeKey 離線安裝包

把 KubeKey 工具也製作成壓縮包，便於複製到離線節點。

$ tar zcvf kubekey-offline-v3.4.1-v1.28.tar.gz kk kubekey-v3.1.1-linux-amd64.tar.gz

2. 準備離線部署 KubeSphere 和 Kubernetes 的前置資料

請注意，以下操作無特殊說明時需在離線環境部署（Registry）節點上執行。

2.1 上傳離線部署資源包到部署節點

將以下離線部署資源包，上傳至離線環境部署（Registry）節點的 /data/ 目錄（可根據實際情況修改）。

Kubekey：kubekey-offline-v3.4.1-v1.28.tar.gz
製品 artifact：ksp-v3.4.1-v1.28-artifact.tar.gz

執行以下命令，解壓 KubeKey：

# 創離線資源存放的資料目錄
mkdir /data/kubekey
tar xvf /data/kubekey-offline-v3.4.1-v1.28.tar.gz -C /data/kubekey
mv ksp-v3.4.1-v1.28-artifact.tar.gz /data/kubekey

2.2 建立離線叢集配置檔案

執行以下命令建立離線叢集配置檔案

cd /data/kubekey
./kk create config --with-kubesphere v3.4.1 --with-kubernetes v1.28.8 -f ksp-v341-v1228-offline.yaml

命令執行成功後，在當前目錄會生成檔名為 ksp-v341-v1228-offline.yaml 的配置檔案。

2.3 修改 Cluster 配置

在離線叢集配置檔案檔案中 kind: Cluster 小節的作用是部署 Kubernetes 叢集。本文示例採用 3 個節點同時作為 control-plane、etcd 節點和 worker 節點。

執行以下命令修改離線叢集配置檔案 ksp-v341-v1228-offline.yaml：

vi ksp-v341-v1228-offline.yaml

修改 kind: Cluster 小節中 hosts 和 roleGroups 等資訊，修改說明如下。

hosts：指定節點的 IP、ssh 使用者、ssh 密碼、ssh 埠。示例演示了 ssh 埠號的配置方法。同時，新增一個 Registry 節點的配置
roleGroups：指定 3 個 etcd、control-plane 節點，複用相同的機器作為 3 個 worker 節點
必須指定主機組 registry 作為倉庫部署節點（本文為了滿足讀者的需求使用了 KubeKey 自動部署 Harbor 映象倉庫。當然，也可以使用已有的 Harbor，使用已有 Harbor 時此配置可以不加）
internalLoadbalancer：啟用內建的 HAProxy 負載均衡器
system.rpms：新增配置，部署時安裝 rpm 包（openEuler 系統預設沒有安裝 tar 包，必須提前安裝）
domain：自定義了一個 opsxlab.cn，沒特殊需求的場景保留預設值即可
containerManager：使用了 containerd
storage.openebs.basePath：新增配置，指定 openebs 預設儲存路徑為 /data/openebs/local
registry：必須指定 type 型別為 harbor，否則預設安裝 docker registry

修改後的完整示例如下：

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: ksp-master-1, address: 192.168.9.91, internalAddress: 192.168.9.91, port:22, user: root, password: "OpsXlab@2024"}
  - {name: ksp-master-2, address: 192.168.9.92, internalAddress: 192.168.9.92, user: root, password: "OpsXlab@2024"}
  - {name: ksp-master-3, address: 192.168.9.93, internalAddress: 192.168.9.93, user: root, password: "OpsXlab@2024"}
  - {name: ksp-registry, address: 192.168.9.90, internalAddress: 192.168.9.90, user: root, password: "OpsXlab@2024"}
  roleGroups:
    etcd:
    - ksp-master-1
    - ksp-master-2
    - ksp-master-3
    control-plane: 
    - ksp-master-1
    - ksp-master-2
    - ksp-master-3
    worker:
    - ksp-master-1
    - ksp-master-2
    - ksp-master-3
    registry:
    - ksp-registry
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    internalLoadbalancer: haproxy

    domain: lb.opsxlab.cn
    address: ""
    port: 6443
  system:
    rpms:
      - tar
  kubernetes:
    version: v1.28.8
    clusterName: opsxlab.cn
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  storage:
    openebs:
      basePath: /data/openebs/local # 預設沒有的新增配置，base path of the local PV provisioner
  registry:
    # 如需使用 kk 部署 harbor, 可將該引數設定為 harbor，不設定該引數且需使用 kk 建立容器映象倉庫，將預設使用 docker registry。
    type: harbor
    # 注意：
    # 1、如需使用 kk 部署的 harbor 或其他自定義倉庫，可設定對應倉庫的 auths，如使用 kk 建立 docker registry 倉庫，則無需配置該引數。
    # 2、kk 部署的 harbor，預設地址為 dockerhub.kubekey.local，如要修改確保與 privateRegistry 欄位的值保持一致。
    auths:
      "registry.opsxlab.cn":
        username: admin
        password: Harbor12345
        certsPath: "/etc/docker/certs.d/registry.opsxlab.cn"
    # 設定叢集部署時使用的私有倉庫
    privateRegistry: "registry.opsxlab.cn"
    namespaceOverride: "kubesphereio"
    registryMirrors: []
    insecureRegistries: []
  addons: []

2.4 修改 ClusterConfiguration 配置

在離線叢集配置檔案中 kind: ClusterConfiguration 小節的作用是部署 KubeSphere 及相關元件。

本文為了驗證離線部署的完整性，啟用了除 KubeEdge 、GateKeeper 以外的所有外掛。

繼續編輯離線叢集配置檔案 ksp-v341-v1228-offline.yaml，修改 kind: ClusterConfiguration 部分來啟用可插拔元件，具體的修改說明如下。

啟用 etcd 監控

etcd:
    monitoring: true # 將 "false" 更改為 "true"
    endpointIps: localhost
    port: 2379
    tlsEnable: true

啟用 KubeSphere 告警系統

alerting:
  enabled: true # 將 "false" 更改為 "true"

啟用 KubeSphere 審計日誌

auditing:
  enabled: true # 將 "false" 更改為 "true"

啟用 KubeSphere DevOps 系統

devops:
  enabled: true # 將 "false" 更改為 "true"

啟用 KubeSphere 事件系統

events:
  enabled: true # 將 "false" 更改為 "true"

啟用 KubeSphere 日誌系統（v3.4.0 開始預設啟用 opensearch）

logging:
  enabled: true # 將 "false" 更改為 "true"

啟用 Metrics Server

metrics_server:
  enabled: true # 將 "false" 更改為 "true"

說明：KubeSphere 支援用於部署的容器組（Pod）彈性伸縮程式 (HPA)。在 KubeSphere 中，Metrics Server 控制著 HPA 是否啟用。

啟用網路策略、容器組 IP 池，服務拓撲圖（名字排序，對應配置引數排序）

network:
  networkpolicy:
    enabled: true # 將 "false" 更改為 "true"
  ippool:
    type: calico # 將 "none" 更改為 "calico"
  topology:
    type: weave-scope # 將 "none" 更改為 "weave-scope"

啟用應用商店

openpitrix:
  store:
    enabled: true # 將 "false" 更改為 "true"

啟用 KubeSphere 服務網格（Istio）

servicemesh:
  enabled: true # 將 "false" 更改為 "true"

修改上面的所有引數後，必須加入一個引數（ 2.x 版本的 KubeKey 沒這個問題，3.x 的到 v3.1.1 為止，都存在這個問題。不加的話，在部署 KubeSphere 的時候會有名稱空間不匹配的問題）

spec:
  namespace_override: kubesphereio
  ......

經過上述步驟，我們成功完成了對離線叢集配置檔案 ksp-v341-v1228-offline.yaml 的修改。由於篇幅限制，無法在此展示完整的檔案內容，請各位讀者根據上文提供的配置說明仔細核對。

3. 安裝配置 Harbor

為了驗證 KubeKey 離線部署 Harbor 服務的能力，本文采用 KubeKey 部署映象倉庫 Harbor。生產環境建議提前自建。

請注意，以下操作如無特殊說明，均在離線環境部署（Registry）節點上執行。

3.1 安裝 Harbor

執行以下命令安裝映象倉庫 Harbor：

./kk init registry -f ksp-v341-v1228-offline.yaml -a ksp-v3.4.1-v1.28-artifact.tar.gz

檢視安裝映象

$ docker images
REPOSITORY                      TAG       IMAGE ID       CREATED        SIZE
goharbor/harbor-exporter        v2.10.1   3b6d345aa4e6   2 months ago   106MB
goharbor/redis-photon           v2.10.1   9a994e1173fc   2 months ago   164MB
goharbor/trivy-adapter-photon   v2.10.1   108e2a78abf7   2 months ago   502MB
goharbor/harbor-registryctl     v2.10.1   da66353bc245   2 months ago   149MB
goharbor/registry-photon        v2.10.1   4cb6a644ceeb   2 months ago   83.5MB
goharbor/nginx-photon           v2.10.1   180accc8c6be   2 months ago   153MB
goharbor/harbor-log             v2.10.1   2215e7f0f2e7   2 months ago   163MB
goharbor/harbor-jobservice      v2.10.1   ab688ea50ad8   2 months ago   140MB
goharbor/harbor-core            v2.10.1   de73267578a3   2 months ago   169MB
goharbor/harbor-portal          v2.10.1   c75282ddf5df   2 months ago   162MB
goharbor/harbor-db              v2.10.1   db2ff40c7b27   2 months ago   269MB
goharbor/prepare                v2.10.1   92197f61701a   2 months ago   207MB

檢視 Harbor 服務狀態

$ cd /opt/harbor/
$ docker-compose ps -a
WARN[0000] /opt/harbor/docker-compose.yml: `version` is obsolete
NAME                IMAGE                                   COMMAND                  SERVICE         CREATED         STATUS                   PORTS
harbor-core         goharbor/harbor-core:v2.10.1            "/harbor/entrypoint.…"   core            4 minutes ago   Up 4 minutes (healthy)
harbor-db           goharbor/harbor-db:v2.10.1              "/docker-entrypoint.…"   postgresql      4 minutes ago   Up 4 minutes (healthy)
harbor-jobservice   goharbor/harbor-jobservice:v2.10.1      "/harbor/entrypoint.…"   jobservice      4 minutes ago   Up 4 minutes (healthy)
harbor-log          goharbor/harbor-log:v2.10.1             "/bin/sh -c /usr/loc…"   log             4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:1514->10514/tcp
harbor-portal       goharbor/harbor-portal:v2.10.1          "nginx -g 'daemon of…"   portal          4 minutes ago   Up 4 minutes (healthy)
nginx               goharbor/nginx-photon:v2.10.1           "nginx -g 'daemon of…"   proxy           4 minutes ago   Up 4 minutes (healthy)   0.0.0.0:80->8080/tcp, :::80->8080/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp
redis               goharbor/redis-photon:v2.10.1           "redis-server /etc/r…"   redis           4 minutes ago   Up 4 minutes (healthy)
registry            goharbor/registry-photon:v2.10.1        "/home/harbor/entryp…"   registry        4 minutes ago   Up 4 minutes (healthy)
registryctl         goharbor/harbor-registryctl:v2.10.1     "/home/harbor/start.…"   registryctl     4 minutes ago   Up 4 minutes (healthy)
trivy-adapter       goharbor/trivy-adapter-photon:v2.10.1   "/home/scanner/entry…"   trivy-adapter   4 minutes ago   Up 4 minutes (healthy)

檢視 hosts（確保使用了自定義域名）

$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# kubekey hosts BEGIN
192.168.9.90  ksp-registry.opsxlab.cn ksp-registry
192.168.9.91  ksp-master-1.opsxlab.cn ksp-master-1
192.168.9.92  ksp-master-2.opsxlab.cn ksp-master-2
192.168.9.93  ksp-master-3.opsxlab.cn ksp-master-3
192.168.9.90  registry.opsxlab.cn
192.168.9.91  lb.opsxlab.cn
# kubekey hosts END

檢視 Harbor 配置的域名（確保使用了自定義域名）

$ cat /opt/harbor/harbor.yml | grep hostname:
hostname: registry.opsxlab.cn

檢視 Docker 是否配置了私有證書（確保使用了自定義域名及證書）

$ ll /etc/docker/certs.d/registry.opsxlab.cn/
total 12
-rw-r--r--. 1 root root 1103 May 20 6:01 ca.crt
-rw-r--r--. 1 root root 1253 May 20 6:01 registry.opsxlab.cn.cert
-rw-------. 1 root root 1679 May 20 6:01 registry.opsxlab.cn.key

3.2 在 Harbor 中建立專案

使用 KubeKey 安裝的 Harbor 預設資訊如下：

登陸賬戶資訊：管理員賬號：admin，密碼：Harbor12345（生產環境必須修改）。

使用 Shell 指令碼建立專案，vim create_project_harbor.sh。

#!/usr/bin/env bash

# Harbor 倉庫地址（寫域名，預設配置為 https://dockerhub.kubekey.local）
url="https://registry.opsxlab.cn"

# 訪問 Harbor 倉庫的預設使用者和密碼（生產環境建議修改）
user="admin"
passwd="Harbor12345"

# 需要建立的專案名列表，按我們製作的離線包的映象命名規則，實際上只需要建立一個 kubesphereio 即可，這裡保留了所有可用值，各位可根據自己的離線倉庫映象名稱規則修改。
harbor_projects=(kubesphereio)

for project in "${harbor_projects[@]}"; do
    echo "creating $project"
    curl -k -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}"
done

執行指令碼建立專案

sh create_project_harbor.sh

3.3 推送離線映象到 Harbor 倉庫

將提前準備好的離線映象推送到 Harbor 倉庫，這一步為可選項，因為建立叢集的時候預設會推送映象（本文使用引數忽略了）。為了部署成功率，建議先推送。

推送離線映象

./kk artifact image push -f ksp-v341-v1228-offline.yaml -a ksp-v3.4.1-v1.28-artifact.tar.gz

正確的安裝結果如下（受限於篇幅，內容有刪減）：

......
7:32:04 CST success: [LocalHost]
7:32:04 CST [ChownWorkerModule] Chown ./kubekey dir
7:32:04 CST success: [LocalHost]
7:32:04 CST Pipeline[ArtifactImagesPushPipeline] execute successfully

Harbor 管理頁面檢視專案和映象倉庫（提前在自己電腦上做好域名解析配置）

ksp-offline-harbor-v210-projects

從專案管理頁面可以看到一共建立了 124 個映象倉庫。

4. 安裝 KubeSphere 和 Kubernetes 叢集

請注意，以下操作如無特殊說明，均在離線環境部署（Registry）節點上執行。

4.1 安裝 KubeSphere 和 Kubernetes 叢集

執行以下命令，安裝 KubeSphere 和 Kubernetes 叢集。

./kk create cluster -f ksp-v341-v1228-offline.yaml -a ksp-v3.4.1-v1.28-artifact.tar.gz --with-packages --skip-push-images

引數說明

--with-packages：安裝作業系統依賴

--skip-push-images： 忽略推送映象，前面已經完成了推送映象到私有倉庫的任務

特殊說明：

由於本文在安裝的過程中啟用了日誌外掛，因此在安裝的過程中必須按照「問題 2」的描述手工介入處理，否則安裝會失敗。
當出現熟悉的安裝捲軸 >>---> 後，可以另開一個終端使用命令 kubectl get pod -A 或是 kubectl get pod -A | grep -v Running 觀察進度，如出現異常可及時介入處理。
也可以透過下面的命令檢視詳細的部署過程日誌及報錯資訊。

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

部署完成需要大約 10-30 分鐘左右，具體看網速、機器配置、啟用多少外掛等。

部署完成後，您應該會在終端上看到類似於下面的輸出。提示部署完成的同時，輸出中還會顯示使用者登陸 KubeSphere 的預設管理員使用者和密碼。

#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://192.168.9.91:30880
Account: admin
Password: P@88w0rd
NOTES：
  1. After you log into the console, please check the
     monitoring status of service components in
     "Cluster Management". If any service is not
     ready, please wait patiently until all components
     are up and running.
  2. Please change the default password after login.

#####################################################
https://kubesphere.io             2024-05-19 14:49:40
#####################################################
14:49:42 CST skipped: [ksp-master-3]
14:49:42 CST skipped: [ksp-master-2]
14:49:42 CST success: [ksp-master-1]
14:49:42 CST Pipeline[CreateClusterPipeline] execute successfully
Installation is complete.

Please check the result using the command:

        kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

4.2 部署結果驗證

登入 Web 控制檯透過 http://{IP}:30880 使用預設帳戶和密碼 admin/P@88w0rd 訪問 KubeSphere 的 Web 控制檯，簡單的驗證一下部署結果。

檢視叢集節點狀態

檢視系統元件狀態

檢視系統監控狀態

5. 常見問題

5.1 問題 1

問題現象

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  18m                   default-scheduler  Successfully assigned kube-system/metrics-server-5d65c798b8-m9tbj to ksp-master-3
  Normal   Pulling    16m (x4 over 18m)     kubelet            Pulling image "registry.opsxlab.cn/kubesphere/metrics-server:v0.4.2"
  Warning  Failed     16m (x4 over 18m)     kubelet            Failed to pull image "registry.opsxlab.cn/kubesphere/metrics-server:v0.4.2": rpc error: code = NotFound desc = failed to pull and unpack image "registry.opsxlab.cn/kubesphere/metrics-server:v0.4.2": failed to resolve reference "registry.opsxlab.cn/kubesphere/metrics-server:v0.4.2": registry.opsxlab.cn/kubesphere/metrics-server:v0.4.2: not found
  Warning  Failed     16m (x4 over 18m)     kubelet            Error: ErrImagePull
  Warning  Failed     16m (x6 over 17m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m57s (x64 over 17m)  kubelet            Back-off pulling image "registry.opsxlab.cn/kubesphere/metrics-server:v0.4.2"

解決方案

在離線叢集配置檔案 kind: ClusterConfiguration 小節的 spec 部分加入引數 namespace_override: kubesphereio

5.2 問題 2

問題現象

# kubectl describe pod opensearch-cluster-data-0 -n kubesphere-logging-system
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  3m                   default-scheduler  Successfully assigned kubesphere-logging-system/opensearch-cluster-data-0 to ksp-master-2
  Warning  Failed     2m17s                kubelet            Failed to pull image "busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve reference "docker.io/library/busybox:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/latest": dial tcp: lookup registry-1.docker.io on 114.114.114.114:53: read udp 192.168.9.92:49491->114.114.114.114:53: i/o timeout
  Warning  Failed     85s                  kubelet            Failed to pull image "busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve reference "docker.io/library/busybox:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/latest": dial tcp: lookup registry-1.docker.io on 114.114.114.114:53: read udp 192.168.9.92:57815->114.114.114.114:53: i/o timeout
  Normal   Pulling    56s (x3 over 2m57s)  kubelet            Pulling image "busybox:latest"
  Warning  Failed     15s (x3 over 2m17s)  kubelet            Error: ErrImagePull
  Warning  Failed     15s                  kubelet            Failed to pull image "busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve reference "docker.io/library/busybox:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/latest": dial tcp: lookup registry-1.docker.io on 114.114.114.114:53: read udp 192.168.9.92:37639->114.114.114.114:53: i/o timeout
  Normal   BackOff    0s (x3 over 2m16s)   kubelet            Back-off pulling image "busybox:latest"
  Warning  Failed     0s (x3 over 2m16s)   kubelet            Error: ImagePullBackOff

解決方案

官方配置寫死了 busybox 的地址，暫時需要手動修改。

# 檢視 sts
$ kubectl get sts -n kubesphere-logging-system
NAME                        READY   AGE
opensearch-cluster-data     0/2     7m42s
opensearch-cluster-master   0/1     7m40s

# 修改 sts 使用的 busyboy 映象為本地映象（細節不展示）
kubectl edit sts opensearch-cluster-data -n kubesphere-logging-system
kubectl edit sts opensearch-cluster-master -n kubesphere-logging-system

# 本文修改後 image 內容（自己根據實際情況修改域名字首）
registry.opsxlab.cn/kubesphereio/busybox:1.31.1

本文雖然基於作業系統 openEuler 22.03 LTS SP3，但對於 CentOS、Ubuntu 等其他作業系統同樣具有借鑑意義。

免責宣告：

筆者水平有限，儘管經過多次驗證和檢查，盡力確保內容的準確性，但仍可能存在疏漏之處。敬請業界專家大佬不吝指教。
本文所述內容僅透過實戰環境驗證測試，讀者可學習、借鑑，但嚴禁直接用於生產環境。由此引發的任何問題，作者概不負責！

本文由部落格一文多發平臺 OpenWrite 釋出！

使用 KubeKey v3.1.1 離線部署原生 Kubernetes v1.28.8 實戰
2024-06-12
用 KubeKey 快速離線部署 K8s 與 KubeSphere
2022-03-11
K8S
使用 KubeKey 搭建 Kubernetes/KubeSphere 環境的"心路(累)歷程"
2022-06-13
KubeSphere 3.3.0 離線安裝教程
2022-07-19
一文搞定全場景K3s離線安裝
2021-01-28
Kubesphere中DevOps流水線無法部署/部署失敗
2023-04-23
dev
一文搞定！在 OpenStack 上部署 Hadoop
2021-09-09
Hadoop
KubeKey v3.1 釋出：快速自定義離線安裝包
2024-05-17
滴滴雲部署Hadoop3.1.1
2018-11-21
Hadoop
CentOS7離線部署JDK
2024-05-12
CentOSJDK
（1）開發部署離線版本
2024-06-15
一文搞定FastDFS分散式檔案系統配置與部署
2017-04-15
AST分散式
Kubernetes(K8S) kubesphere 安裝
2023-04-27
K8S
Kubernetes(K8S) kubesphere 介紹
2023-04-26
K8S
一文搞定Git！
2020-11-08
Git
Linux下離線部署Node環境和前端服務
2018-10-10
Linux前端
靠，終於搞定反離線瀏覽器設定
2003-11-06
瀏覽器
Kubernetes 升級不棄 Docker：KubeKey 的絲滑之道
2024-04-09
Docker
KubeKey 部署 K8s v1.28.8 實戰
2024-05-11
K8S
容器化 | 在 KubeSphere 中部署 MySQL 叢集
2022-02-25
MySql
招商銀行 KubeVela 離線部署實踐
2022-04-06
LM Studio 本地離線部署大模型
2024-05-12
大模型
一文搞定HashMap的實現原理和麵試
2019-06-20
HashMap
一文說清 KubeSphere 容器平臺的價值
2020-07-06
Ubuntu20.04離線部署Opencv4.5.5
2024-05-28
UbuntuOpenCV
一文搞定 Spring Data JPA
2020-07-07
Spring
基於linux離線部署nginx，使用rpm
2023-03-13
LinuxNginx
使用Kubernetes和Istio實現藍綠部署
2018-10-22
已完成的python專案-環境離線部署
2022-02-12
Python
一文替你全部搞定：MySQL命令。
2018-11-20
MySql
一文搞定防盜鏈設計
2024-04-02
Kubernetes 多叢集在開源專案 KubeSphere 的應用
2020-08-21
3.4.1 About Quiescing a Database
2020-03-14
UIDatabase
一文搞定：前端如何選擇Angular、React和Vue三大主流框架
2023-04-03
前端AngularReactVue框架
一文搞定規範化Git Commit
2020-09-28
GitMIT
K8S-1.23.17+Ceph+KubeSphere 一主二從部署攻略
2024-03-15
K8S
一文讀透HTTP的長連線和短連線
2020-09-16
HTTP
Django + react + 阿里雲ecs 前後端分離部署上線
2018-12-24
DjangoReact阿里後端