Kubernetes 是用於自動部署，擴充套件和管理容器化應用程式的開源系統。本文將介紹如何快速開始 K8s 的使用。
瞭解 K8s
搭建 K8s
本地開發測試，需要搭建一個 K8s 輕量服務。實際部署時，可以用雲廠商的 K8s 服務。
本文以 k3d 為例，於 macOS 搭建 K8s 服務。於 Ubuntu 則推薦 MicroK8s。其他可替代方案有：
k3d
k3s 是 Rancher 推出的 K8s 輕量版。而 k3d 即 k3s in docker，以 docker 容器管理 k3s 叢集。
以下搭建過程，是於 macOS 的筆記，供參考。其他平臺，請依照官方文件進行。
# 安裝 kubectl: 命令列工具
brew install kubectl
# 安裝 kubecm: 配置管理工具
brew install kubecm
# 安裝 k3d
brew install k3d
❯ k3d version
k3d version v4.4.8
k3s version latest (default)
建立叢集（1主2從）：
❯ k3d cluster create mycluster --api-port 6550 --servers 1 --agents 2 --port 8080:80@loadbalancer --wait
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-mycluster' (23dc5761582b1a4b74d9aa64d8dca2256b5bc510c4580b3228123c26e93f456e)
INFO[0000] Created volume 'k3d-mycluster-images'
INFO[0001] Creating node 'k3d-mycluster-server-0'
INFO[0001] Creating node 'k3d-mycluster-agent-0'
INFO[0001] Creating node 'k3d-mycluster-agent-1'
INFO[0001] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0001] Starting cluster 'mycluster'
INFO[0001] Starting servers...
INFO[0001] Starting Node 'k3d-mycluster-server-0'
INFO[0009] Starting agents...
INFO[0009] Starting Node 'k3d-mycluster-agent-0'
INFO[0022] Starting Node 'k3d-mycluster-agent-1'
INFO[0030] Starting helpers...
INFO[0030] Starting Node 'k3d-mycluster-serverlb'
INFO[0031] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0036] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
INFO[0036] Cluster 'mycluster' created successfully!
INFO[0036] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
INFO[0036] You can now use it like this:
kubectl config use-context k3d-mycluster
kubectl cluster-info
檢視叢集資訊：
❯ kubectl cluster-info
Kubernetes control plane is running at https://0.0.0.0:6550
CoreDNS is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://0.0.0.0:6550/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
檢視資源資訊：
# 檢視 Nodes
❯ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3d-mycluster-agent-0 Ready <none> 2m12s v1.20.10+k3s1
k3d-mycluster-server-0 Ready control-plane,master 2m23s v1.20.10+k3s1
k3d-mycluster-agent-1 Ready <none> 2m4s v1.20.10+k3s1
# 檢視 Pods
❯ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6488c6fcc6-5n7d9 1/1 Running 0 2m12s
kube-system metrics-server-86cbb8457f-dr7lh 1/1 Running 0 2m12s
kube-system local-path-provisioner-5ff76fc89d-zbxf4 1/1 Running 0 2m12s
kube-system helm-install-traefik-bfm4c 0/1 Completed 0 2m12s
kube-system svclb-traefik-zx98g 2/2 Running 0 68s
kube-system svclb-traefik-7bx2r 2/2 Running 0 68s
kube-system svclb-traefik-cmdrm 2/2 Running 0 68s
kube-system traefik-6f9cbd9bd4-2mxhk 1/1 Running 0 69s
測試 Nginx：
# 建立 Nginx Deployment
kubectl create deployment nginx --image=nginx
# 建立 ClusterIP Service，暴露 Nginx 埠
kubectl create service clusterip nginx --tcp=80:80
# 建立 Ingress Object
# k3s 以 traefik 為預設 ingress controller
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
EOF
# 訪問 Nginx Service
# kubectl get pods 確認 nginx STATUS=Running
open http://127.0.0.1:8080
測試 Dashboard：
# 建立 Dashboard
GITHUB_URL=https://github.com/kubernetes/dashboard/releases
VERSION_KUBE_DASHBOARD=$(curl -w '%{url_effective}' -I -L -s -S ${GITHUB_URL}/latest -o /dev/null | sed -e 's|.*/||')
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/${VERSION_KUBE_DASHBOARD}/aio/deploy/recommended.yaml
# 配置 RBAC
# admin user
cat <<EOF > dashboard.admin-user.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
EOF
# admin user role
cat <<EOF > dashboard.admin-user-role.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF
# 配置部署
kubectl create -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml
# 獲取 Bearer Token
kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token
# 訪問代理
kubectl proxy
# 訪問 Dashboard
# 輸入 Token 登入
open http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
刪除叢集：
k3d cluster delete mycluster
切換叢集：
kubecm s
參考：
MicroK8s
MicroK8s 是 Ubuntu 官方生態提供的 K8s 輕量版，適合用於開發工作站、IoT、Edge、CI/CD。
以下搭建過程，是於 Ubuntu 18/20 的筆記，供參考。其他平臺，請依照官方文件進行。
# 檢查 hostname
# 要求不含大寫字母和下劃線，不然依照後文修改
hostname
# 安裝 microk8s
sudo apt install snapd -y
snap info microk8s
sudo snap install microk8s --classic --channel=1.21/stable
# 新增使用者組
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s
id $USER
## 一些確保拉到映象的方法
# 配置代理（如果有）
# MicroK8s / Installing behind a proxy
# https://microk8s.io/docs/install-proxy
# Issue: Pull images from others than k8s.gcr.io
# https://github.com/ubuntu/microk8s/issues/472
sudo vi /var/snap/microk8s/current/args/containerd-env
HTTPS_PROXY=http://127.0.0.1:7890
NO_PROXY=10.1.0.0/16,10.152.183.0/24
# 新增映象（docker.io）
# 映象加速器
# https://yeasy.gitbook.io/docker_practice/install/mirror
# 還可改 args/ 裡不同模板的 sandbox_image
sudo vi /var/snap/microk8s/current/args/containerd-template.toml
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://x.mirror.aliyuncs.com", "https://registry-1.docker.io", ]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:32000"]
endpoint = ["http://localhost:32000"]
# 手動匯入，見後文啟用外掛那
# 重啟服務
microk8s stop
microk8s start
檢查狀態：
$ microk8s status
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
ha-cluster # Configure high availability on the current node
disabled:
ambassador # Ambassador API Gateway and Ingress
cilium # SDN, fast with full network policy
dashboard # The Kubernetes dashboard
dns # CoreDNS
fluentd # Elasticsearch-Fluentd-Kibana logging and monitoring
gpu # Automatic enablement of Nvidia CUDA
helm # Helm 2 - the package manager for Kubernetes
helm3 # Helm 3 - Kubernetes package manager
host-access # Allow Pods connecting to Host services smoothly
ingress # Ingress controller for external access
istio # Core Istio service mesh services
jaeger # Kubernetes Jaeger operator with its simple config
keda # Kubernetes-based Event Driven Autoscaling
knative # The Knative framework on Kubernetes.
kubeflow # Kubeflow for easy ML deployments
linkerd # Linkerd is a service mesh for Kubernetes and other frameworks
metallb # Loadbalancer for your Kubernetes cluster
metrics-server # K8s Metrics Server for API access to service metrics
multus # Multus CNI enables attaching multiple network interfaces to pods
openebs # OpenEBS is the open-source storage solution for Kubernetes
openfaas # openfaas serverless framework
portainer # Portainer UI for your Kubernetes cluster
prometheus # Prometheus operator for monitoring and logging
rbac # Role-Based Access Control for authorisation
registry # Private image registry exposed on localhost:32000
storage # Storage class; allocates storage from host directory
traefik # traefik Ingress controller for external access
如果 status 不正確時，可以如下排查錯誤：
microk8s inspect
grep -r error /var/snap/microk8s/2346/inspection-report
如果要修改 hostname：
# 改名稱
sudo hostnamectl set-hostname ubuntu-vm
# 改 host
sudo vi /etc/hosts
# 雲主機的話，還要改下配置
sudo vi /etc/cloud/cloud.cfg
preserve_hostname: true
# 如果只修改 preserve_hostname 不生效，那就直接註釋掉 set/update_hostname
cloud_init_modules:
# - set_hostname
# - update_hostname
# 重啟，驗證生效
sudo reboot
接著，啟用些基礎外掛：
microk8s enable dns dashboard
# 檢視 Pods ，確認 running
microk8s kubectl get pods --all-namespaces
# 不然，詳情裡看下錯誤原因
microk8s kubectl describe pod --all-namespaces
直到全部正常 running：
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kubernetes-dashboard-85fd7f45cb-snqrv 1/1 Running 1 15h
kube-system dashboard-metrics-scraper-78d7698477-tmb7k 1/1 Running 1 15h
kube-system metrics-server-8bbfb4bdb-wlf8g 1/1 Running 1 15h
kube-system calico-node-p97kh 1/1 Running 1 6m18s
kube-system coredns-7f9c69c78c-255fg 1/1 Running 1 15h
kube-system calico-kube-controllers-f7868dd95-st9p7 1/1 Running 1 16h
如果拉取映象失敗，可以 microk8s ctr image pull <mirror>。或者，docker pull 後匯入 containerd：
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker save k8s.gcr.io/pause:3.1 > pause:3.1.tar
microk8s ctr image import pause:3.1.tar
docker pull calico/cni:v3.13.2
docker save calico/cni:v3.13.2 > cni:v3.13.2.tar
microk8s ctr image import cni:v3.13.2.tar
docker pull calico/node:v3.13.2
docker save calico/node:v3.13.2 > node:v3.13.2.tar
microk8s ctr image import node:v3.13.2.tar
如果 calico-node CrashLoopBackOff，可能網路配置問題：
# 查具體日誌
microk8s kubectl logs -f -n kube-system calico-node-l5wl2 -c calico-node
# 如果有 Unable to auto-detect an IPv4 address，那麼 ip a 找出哪個網口有 IP 。修改：
sudo vi /var/snap/microk8s/current/args/cni-network/cni.yaml
- name: IP_AUTODETECTION_METHOD
value: "interface=wlo.*"
# 重啟服務
microk8s stop; microk8s start
## 參考
# Issue: Microk8s 1.19 not working on Ubuntu 20.04.1
# https://github.com/ubuntu/microk8s/issues/1554
# Issue: CrashLoopBackOff for calico-node pods
# https://github.com/projectcalico/calico/issues/3094
# Changing the pods CIDR in a MicroK8s cluster
# https://microk8s.io/docs/change-cidr
# MicroK8s IPv6 DualStack HOW-TO
# https://discuss.kubernetes.io/t/microk8s-ipv6-dualstack-how-to/14507
然後，可以開啟 Dashboard 看看：
# 獲取 Token （未啟用 RBAC 時）
token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token
# 轉發埠
microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443
# 開啟網頁，輸入 Token 登入
xdg-open https://127.0.0.1:10443
# 更多說明 https://microk8s.io/docs/addon-dashboard
# Issue: Your connection is not private
# https://github.com/kubernetes/dashboard/issues/3804
更多操作，請閱讀官方文件。本文之後仍以 k3d 為例。
準備 K8s 應用
Go 應用
http_server.go:
package main
import (
"fmt"
"log"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}
func main() {
http.HandleFunc("/", handler)
fmt.Println("HTTP Server running ...")
log.Fatal(http.ListenAndServe(":3000", nil))
}
構建映象
http_server.dockerfile:
FROM golang:1.17-alpine AS builder
WORKDIR /app
ADD ./http_server.go /app
RUN cd /app && go build http_server.go
FROM alpine:3.14
WORKDIR /app
COPY --from=builder /app/http_server /app/
EXPOSE 3000
ENTRYPOINT ./http_server
# 編譯映象
docker build -t http_server:1.0 -f http_server.dockerfile .
# 執行應用
docker run --rm -p 3000:3000 http_server:1.0
# 測試應用
❯ curl http://127.0.0.1:3000/go
Hi there, I love go!
部署 K8s 應用
瞭解概念
之後，參照官方教程，我們將使用 Deployment 執行 Go 應用（無狀態）。
匯入映象
首先，我們手動匯入映象進叢集：
docker save http_server:1.0 > http_server:1.0.tar
k3d image import http_server:1.0.tar -c mycluster
如果有自己的私有倉庫，參見 k3d / Registries 進行配置。
建立 Deployment
# 配置 Deployment （2個副本）
cat <<EOF > go-http-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-http
labels:
app: go-http
spec:
replicas: 2
selector:
matchLabels:
app: go-http
template:
metadata:
labels:
app: go-http
spec:
containers:
- name: go-http
image: http_server:1.0
ports:
- containerPort: 3000
EOF
# 應用 Deployment
# --record: 記錄命令
kubectl apply -f go-http-deployment.yaml --record
檢視 Deployment：
# 檢視 Deployment
❯ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 2d
go-http 2/2 2 2 22s
# 檢視 Deployment 資訊
kubectl describe deploy go-http
# 檢視 Deployment 建立的 ReplicaSet （2個）
kubectl get rs
# 檢視 Deployment 建立的 Pods （2個）
kubectl get po -l app=go-http -o wide --show-labels
# 檢視某一 Pod 資訊
kubectl describe po go-http-5848d49c7c-wzmxh
建立 Service
# 建立 Service，名為 go-http
# 將請求代理到 app=go-http, tcp=3000 的 Pod 上
kubectl expose deployment go-http --name=go-http
# 或
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
name: go-http
labels:
app: go-http
spec:
selector:
app: go-http
ports:
- protocol: TCP
port: 3000
targetPort: 3000
EOF
# 檢視 Service
kubectl get svc
# 檢視 Service 資訊
kubectl describe svc go-http
# 檢視 Endpoints 對比看看
# kubectl get ep go-http
# kubectl get po -l app=go-http -o wide
# 刪除 Service （如果）
kubectl delete svc go-http
訪問 Service (DNS)：
❯ kubectl run curl --image=radial/busyboxplus:curl -i --tty
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$ nslookup go-http
Server: 10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local
Name: go-http
Address 1: 10.43.102.17 go-http.default.svc.cluster.local
[ root@curl:/ ]$ curl http://go-http:3000/go
Hi there, I love go!
暴露 Service (Ingress)：
# 建立 Ingress Object
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: go-http
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /go
pathType: Prefix
backend:
service:
name: go-http
port:
number: 3000
EOF
# 檢視 Ingress
kubectl get ingress
# 檢視 Ingress 資訊
kubectl describe ingress go-http
# 刪除 Ingress （如果）
kubectl delete ingress go-http
訪問 Service (Ingress)：
❯ open http://127.0.0.1:8080/go
# 或，
❯ curl http://127.0.0.1:8080/go
Hi there, I love go!
# Nginx 是在 http://127.0.0.1:8080
更新
僅當 Deployment Pod 模板發生改變時，例如模板的標籤或容器映象被更新，才會觸發 Deployment 上線。其他更新（如對 Deployment 執行擴縮容的操作）不會觸發上線動作。
所以，我們準備 http_server:2.0 映象匯入叢集，然後更新：
❯ kubectl set image deployment/go-http go-http=http_server:2.0 --record
deployment.apps/go-http image updated
之後，可以檢視上線狀態：
# 檢視上線狀態
❯ kubectl rollout status deployment/go-http
deployment "go-http" successfully rolled out
# 檢視 ReplicaSet 狀態：新的擴容，舊的縮容，完成更新
❯ kubectl get rs
NAME DESIRED CURRENT READY AGE
go-http-586694b4f6 2 2 2 10s
go-http-5848d49c7c 0 0 0 6d
測試服務：
❯ curl http://127.0.0.1:8080/go
Hi there v2, I love go!
回滾
檢視 Deployment 修訂歷史：
❯ kubectl rollout history deployment.v1.apps/go-http
deployment.apps/go-http
REVISION CHANGE-CAUSE
1 kubectl apply --filename=go-http-deployment.yaml --record=true
2 kubectl set image deployment/go-http go-http=http_server:2.0 --record=true
# 檢視修訂資訊
kubectl rollout history deployment.v1.apps/go-http --revision=2
回滾到之前的修訂版本：
# 回滾到上一版
kubectl rollout undo deployment.v1.apps/go-http
# 回滾到指定版本
kubectl rollout undo deployment.v1.apps/go-http --to-revision=1
縮放
# 縮放 Deployment 的 ReplicaSet 數
kubectl scale deployment.v1.apps/go-http --replicas=10
# 如果叢集啟用了 Pod 的水平自動縮放，可以根據現有 Pods 的 CPU 利用率選擇上下限
# Horizontal Pod Autoscaler Walkthrough
# https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
kubectl autoscale deployment.v1.apps/nginx-deployment --min=10 --max=15 --cpu-percent=80
暫停、恢復
# 暫停 Deployment
kubectl rollout pause deployment.v1.apps/go-http
# 恢復 Deployment
kubectl rollout resume deployment.v1.apps/go-http
期間可以更新 Deployment ，但不會觸發上線。
刪除
kubectl delete deployment go-http
金絲雀部署
灰度部署，用多標籤區分多個部署，新舊版可同時執行。部署新版時，用少量流量驗證，沒問題再全量更新。
Helm 釋出
Helm 是 K8s 的包管理工具，包格式稱為 charts。現在來發布我們的 Go 服務吧。
安裝 Helm
# macOS
brew install helm
# Ubuntu
sudo snap install helm --classic
執行 helm 瞭解命令。
建立 Chart
helm create go-http
檢視內容：
❯ tree go-http -aF --dirsfirst
go-http
├── charts/ # 包依賴的 charts，稱 subcharts
├── templates/ # 包的 K8s 檔案模板，用的 Go 模板
│ ├── tests/
│ │ └── test-connection.yaml
│ ├── NOTES.txt # 包的幫助文字
│ ├── _helpers.tpl # 模板可重用的片段，模板裡 include 引用
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ └── serviceaccount.yaml
├── .helmignore # 打包忽略說明
├── Chart.yaml # 包的描述檔案
└── values.yaml # 變數預設值，可安裝時覆蓋，模板裡 .Values 引用
修改內容：
- 修改
Chart.yaml裡的描述 - 修改
values.yaml裡的變數- 修改
image為釋出的 Go 服務 - 修改
ingress為true，及一些配置 - 刪除
serviceAccountautoscaling，templates/裡也搜尋刪除相關內容
- 修改
- 修改
templates/裡的模板- 刪除
tests/，剩餘deployment.yamlservice.yamlingress.yaml有用
- 刪除
結果可見 start-k8s/helm/go-http。
檢查錯誤：
❯ helm lint --strict go-http
==> Linting go-http
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
渲染模板：
# --set 覆蓋預設配置，或者用 -f 選擇自定的 values.yaml
helm template go-http-helm ./go-http \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"
安裝 Chart：
helm install go-http-helm ./go-http \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"
# 或，打包後安裝
helm package go-http
helm install go-http-helm go-http-1.0.0.tgz \
--set replicaCount=2 \
--set "ingress.hosts[0].paths[0].path=/helm" \
--set "ingress.hosts[0].paths[0].pathType=Prefix"
# 檢視安裝列表
helm list
測試服務：
❯ kubectl get deploy go-http-helm
NAME READY UP-TO-DATE AVAILABLE AGE
go-http-helm 2/2 2 2 2m42s
❯ curl http://127.0.0.1:8080/helm
Hi there, I love helm!
解除安裝 Chart：
helm uninstall go-http-helm
釋出 Chart
官方倉庫 ArtifactHub 上有很多分享的 Helm charts 。可見 velkoz1108/helm-chart 把我們的 Go 服務釋出到 Hub 上。
最後
開始 K8s 吧！本文樣例在 ikuokuo/start-k8s。
GoCoding 個人實踐的經驗分享，可關注公眾號！