公眾號關注「WeiyiGeek」
設為「特別關注」,每天帶你玩轉網路安全運維、應用開發、物聯網IOT學習!
本章目錄:
-
0x00 前言簡述
-
0x01 環境準備
-
主機規劃
-
軟體版本
-
網路規劃
-
0x02 安裝部署
-
1.基礎主機環境準備配置
-
2.負載均衡管理工具安裝與核心載入
-
3.高可用HAproxy與Keepalived軟體安裝配置
-
4.配置部署etcd叢集與etcd證書籤發
-
5.Containerd 執行時安裝部署
溫馨提示: 由於實踐篇幅太長,此處分為上下兩節進行釋出。
0x00 前言簡述
描述: 在我部落格以及前面的文章之中講解Kubernetes相關叢集環境的搭建, 隨著K8S及其相關元件的迭代, 與讀者當前接觸的版本有所不同,所以在當前【2022年4月26日 10:08:29】時間節點,博主使用ubuntu 20.04 、haproxy、keepalive、containerd、etcd、kubeadm、kubectl 等相關工具外掛【最新或者穩定的版本】進行實踐高可用的kubernetes叢集的搭建,這裡不再對k8s等相關基礎知識做介紹,如有新入門的童鞋,請訪問如下【部落格文章】(https://blog.weiyigeek.top/tags/k8s/) 或者【B站專欄】(https://www.bilibili.com/read/readlist/rl520875?spm_id_from=333.999.0.0) 按照順序學習。
簡述
Kubernetes(後續簡稱k8s)是 Google(2014年6月) 開源的一個容器編排引擎,使用Go語言開發,它支援自動化部署、大規模可伸縮、以及雲平臺中多個主機上的容器化應用進行管理。其目標是讓部署容器化的應用更加簡單並且高效,提供了資源排程、部署管理、服務發現、擴容縮容、狀態 監控、維護等一整套功能, 努力成為跨主機叢集的自動化部署、自動化擴充套件以及執行應用程式容器的平臺,它支援一些列CNCF畢業專案,包括 Containerd、calico 等 。
0x01 環境準備
主機規劃
主機地址 | 主機名稱 | 主機配置 | 主機角色 | 軟體元件 |
---|---|---|---|---|
10.10.107.223 | master-223 | 4C/4G/ | 控制節點 | |
10.10.107.224 | master-224 | 4C/4G | 控制節點 | |
10.10.107.225 | master-225 | 4C/8G | 控制節點 | |
10.10.107.226 | node-1 | 4C/2G | 工作節點 | |
10.10.107.227 | node-2 | 4C/2G | 工作節點 | |
10.10.107.222 | weiyigeek.cluster.k8s | - | 虛擬VIP | 虛擬網路卡地址 |
溫馨提示: 此處使用的是 Ubuntu 20.04 作業系統, 該系統已做安全加固和核心優化符合等保2.0要求【SecOpsDev/Ubuntu-InitializeSecurity.sh at master · WeiyiGeek/SecOpsDev (github.com)】, 如你的Linux未進行相應配置環境可能與讀者有些許差異, 如需要進行(windows server 、Ubuntu、CentOS)安全加固請參照如下加固指令碼進行加固, 請大家瘋狂的 star 。
加固指令碼地址:【 https://github.com/WeiyiGeek/SecOpsDev/blob/master/OS-作業系統/Linux/Ubuntu/Ubuntu-InitializeSecurity.sh 】
軟體版本
作業系統
- Ubuntu 20.04 LTS - 5.4.0-107-generic
TLS證書籤發
-
cfssl - v1.6.1
-
cfssl-certinfo - v1.6.1
-
cfssljson - v1.6.1
高可用軟體
-
ipvsadm - 1:1.31-1
-
haproxy - 2.0.13-2
-
keepalived - 1:2.0.19-2
ETCD資料庫
- etcd - v3.5.4
容器執行時
- containerd.io - 1.6.4
Kubernetes
-
kubeadm - v1.23.6
-
kube-apiserver - v1.23.6
-
kube-controller-manager - v1.23.6
-
kubectl - v1.23.6
-
kubelet - v1.23.6
-
kube-proxy - v1.23.6
-
kube-scheduler - v1.23.6
網路外掛&輔助軟體
calico - v3.22
coredns - v1.9.1
kubernetes-dashboard - v2.5.1
k9s - v0.25.18
網路規劃
子網 Subnet | 網段 | 備註 |
---|---|---|
nodeSubnet | 10.10.107.0/24 | C1 |
ServiceSubnet | 10.96.0.0/16 | C2 |
PodSubnet | 10.128.0.0/16 | C3 |
溫馨提示: 上述環境所使用的到相關軟體及外掛我已打包, 方便大家進行下載,可訪問如下連結(訪問密碼請訪問 WeiyiGeek 公眾號回覆【k8s二進位制】獲取)。
下載地址: http://share.weiyigeek.top/f/36158960-578443238-a1a5fa (訪問密碼:點選訪問 WeiyiGeek 公眾號回覆【k8s二進位制】)
/kubernetes-cluster-binary-install# tree ..├── calico│ └── calico-v3.22.yaml├── certificate│ ├── admin-csr.json│ ├── apiserver-csr.json│ ├── ca-config.json│ ├── ca-csr.json│ ├── cfssl│ ├── cfssl-certinfo│ ├── cfssljson│ ├── controller-manager-csr.json│ ├── etcd-csr.json│ ├── kube-scheduler-csr.json│ ├── proxy-csr.json│ └── scheduler-csr.json├── containerd.io│ └── config.toml├── coredns│ ├── coredns.yaml│ ├── coredns.yaml.sed│ └── deploy.sh├── cri-containerd-cni-1.6.4-linux-amd64.tar.gz├── etcd-v3.5.4-linux-amd64.tar.gz├── k9s├── kubernetes-dashboard│ ├── kubernetes-dashboard.yaml│ └── rbac-dashboard-admin.yaml├── kubernetes-server-linux-amd64.tar.gz└── nginx.yaml
0x02 安裝部署
1.基礎主機環境準備配置
步驟 01.【所有主機】主機名設定按照上述主機規劃進行設定。
# 例如, 在10.10.107.223主機中執行。hostnamectl set-hostname master-223# 例如, 在10.10.107.227主機中執行。hostnamectl set-hostname node-2
步驟 02.【所有主機】將規劃中的主機名稱與IP地址進行硬解析。
sudo tee -a /etc/hosts <<'EOF'10.10.107.223 master-22310.10.107.224 master-22410.10.107.225 master-22510.10.107.226 node-110.10.107.227 node-210.10.107.222 weiyigeek.cluster.k8sEOF
步驟 03.驗證每個節點上IP、MAC 地址和 product_uuid 的唯一性,保證其能相互正常通訊
# 使用命令 ip link 或 ifconfig -a 來獲取網路介面的 MAC 地址ifconfig -a# 使用命令 檢視 product_uuid 校驗sudo cat /sys/class/dmi/id/product_uuid
步驟 04.【所有主機】系統時間同步與時區設定
date -Rsudo ntpdate ntp.aliyun.comsudo timedatectl set-timezone Asia/Shanghai# 或者# sudo dpkg-reconfigure tzdatasudo timedatectl set-local-rtc 0timedatectl
步驟 05.【所有主機】禁用系統交換分割槽
swapoff -a && sed -i 's|^/swap.img|#/swap.ing|g' /etc/fstab# 驗證交換分割槽是否被禁用free | grep "Swap:"
步驟 07.【所有主機】系統核心引數調整
# 禁用 swap 分割槽egrep -q "^(#)?vm.swappiness.*" /etc/sysctl.conf && sed -ri "s|^(#)?vm.swappiness.*|vm.swappiness = 0|g" /etc/sysctl.conf || echo "vm.swappiness = 0" >> /etc/sysctl.conf# 允許轉發egrep -q "^(#)?net.ipv4.ip_forward.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.ipv4.ip_forward.*|net.ipv4.ip_forward = 1|g" /etc/sysctl.conf || echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf# - 允許 iptables 檢查橋接流量egrep -q "^(#)?net.bridge.bridge-nf-call-iptables.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.bridge.bridge-nf-call-iptables.*|net.bridge.bridge-nf-call-iptables = 1|g" /etc/sysctl.conf || echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.confegrep -q "^(#)?net.bridge.bridge-nf-call-ip6tables.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.bridge.bridge-nf-call-ip6tables.*|net.bridge.bridge-nf-call-ip6tables = 1|g" /etc/sysctl.conf || echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
步驟 07.【所有主機】禁用系統防火牆
ufw disable && systemctl disable ufw && systemctl stop ufw
步驟 08.【master-225 主機】使用 master-225 主機的公鑰免賬號密碼登陸其它主機(可選)方便檔案在各主機上傳下載。
# 生成ed25519格式的公金鑰sh-keygen -t ed25519# 例如,在master-225 主機上使用金鑰登入到 master-223 設定 (其它主機同樣)ssh-copy-id -p 20211 weiyigeek@10.10.107.223 # /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_ed25519.pub" # Are you sure you want to continue connecting (yes/no/[fingerprint])? yes # 輸入yes # weiyigeek@10.10.107.223s password: # 輸入主機密碼 # Number of key(s) added: 1 # Now try logging into the machine, with: "ssh -p '20211' 'weiyigeek@10.10.107.223'" # and check to make sure that only the key(s) you wanted were added.ssh-copy-id -p 20211 weiyigeek@10.10.107.224ssh-copy-id -p 20211 weiyigeek@10.10.107.226ssh-copy-id -p 20211 weiyigeek@10.10.107.227
2.負載均衡管理工具安裝與核心載入
步驟 01.安裝ipvs模組以及負載均衡相關依賴。
# 檢視可用版本sudo apt-cache madison ipvsadm # ipvsadm | 1:1.31-1 | http://mirrors.aliyun.com/ubuntu focal/main amd64 Packages# 安裝sudo apt -y install ipvsadm ipset sysstat conntrack# 鎖定版本 apt-mark hold ipvsadm # ipvsadm set on hold.
步驟 02.將模組載入到核心中(開機自動設定-需要重啟機器生效)
tee /etc/modules-load.d/k8s.conf <<'EOF'# netfilterbr_netfilter# containerdoverlay# nf_conntracknf_conntrack# ipvsip_vsip_vs_lcip_vs_lblcip_vs_lblcrip_vs_rrip_vs_wrrip_vs_ship_vs_dhip_vs_foip_vs_nqip_vs_sedip_vs_ftpip_tablesip_setipt_setipt_rpfilteript_REJECTipipxt_setEOF
步驟 03.手動載入模組到核心中
mkdir -vp /etc/modules.d/tee /etc/modules.d/k8s.modules <<'EOF'#!/bin/bash# netfilter 模組 允許 iptables 檢查橋接流量modprobe -- br_netfilter# containerdmodprobe -- overlay# nf_conntrackmodprobe -- nf_conntrack# ipvsmodprobe -- ip_vsmodprobe -- ip_vs_lcmodprobe -- ip_vs_lblcmodprobe -- ip_vs_lblcrmodprobe -- ip_vs_rrmodprobe -- ip_vs_wrrmodprobe -- ip_vs_shmodprobe -- ip_vs_dhmodprobe -- ip_vs_fomodprobe -- ip_vs_nqmodprobe -- ip_vs_sedmodprobe -- ip_vs_ftpmodprobe -- ip_tablesmodprobe -- ip_setmodprobe -- ipt_setmodprobe -- ipt_rpfiltermodprobe -- ipt_REJECTmodprobe -- ipipmodprobe -- xt_setEOFchmod 755 /etc/modules.d/k8s.modules && bash /etc/modules.d/k8s.modules && lsmod | grep -e ip_vs -e nf_conntrack # ip_vs_sh 16384 0 # ip_vs_wrr 16384 0 # ip_vs_rr 16384 0 # ip_vs 155648 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr # nf_conntrack 139264 1 ip_vs # nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs # nf_defrag_ipv4 16384 1 nf_conntrack # libcrc32c 16384 5 nf_conntrack,btrfs,xfs,raid456,ip_vssysctl --system
溫馨提示: 在 kernel 4.19 版本及以上將使用 nf_conntrack 模組, 則在 4.18 版本以下則需使用nf_conntrack_ipv4 模組。
3.高可用HAproxy與Keepalived軟體安裝配置
描述: 由於是測試學習環境, 此處我未專門準備兩臺HA伺服器, 而是直接採用master節點機器,如果是正式環境建議獨立出來。
步驟 01.【Master節點機器】安裝下載 haproxy (HA代理健康檢測) 與 keepalived (虛擬路由協議-主從)。
# 檢視可用版本sudo apt-cache madison haproxy keepalived # haproxy | 2.0.13-2ubuntu0.5 | http://mirrors.aliyun.com/ubuntu focal-security/main amd64 Packages # keepalived | 1:2.0.19-2ubuntu0.2 | http://mirrors.aliyun.com/ubuntu focal-updates/main amd64 Packages# 安裝sudo apt -y install haproxy keepalived# 鎖定版本 apt-mark hold haproxy keepalived
步驟 02.【Master節點機器】進行 HAProxy 配置,其配置目錄為 /etc/haproxy/
,所有節點配置是一致的。
sudo cp /etc/haproxy/haproxy.cfg{,.bak}tee /etc/haproxy/haproxy.cfg<<'EOF'global user haproxy group haproxy maxconn 2000 daemon log /dev/log local0 log /dev/log local1 err chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-ticketsdefaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s # errorfile 400 /etc/haproxy/errors/400.http # errorfile 403 /etc/haproxy/errors/403.http # errorfile 408 /etc/haproxy/errors/408.http # errorfile 500 /etc/haproxy/errors/500.http # errorfile 502 /etc/haproxy/errors/502.http # errorfile 503 /etc/haproxy/errors/503.http # errorfile 504 /etc/haproxy/errors/504.http# 注意: 管理HAproxy (可選)# frontend monitor-in# bind *:33305# mode http# option httplog# monitor-uri /monitor# 注意: 基於四層代理, 1644 3為VIP的 ApiServer 控制平面埠, 由於是與master節點在一起所以不能使用6443埠.frontend k8s-master bind 0.0.0.0:16443 bind 127.0.0.1:16443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master# 注意: Master 節點的預設 Apiserver 是6443埠backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server master-223 10.10.107.223:6443 check server master-224 10.10.107.224:6443 check server master-225 10.10.107.225:6443 checkEOF
步驟 03.【Master節點機器】進行 置KeepAlived 配置 ,其配置目錄為 /etc/haproxy/
# 建立配置目錄,分別在各個master節點執行。mkdir -vp /etc/keepalived# __ROLE__ 角色: MASTER 或者 BACKUP# __NETINTERFACE__ 宿主機物理網路卡名稱 例如我的ens32# __IP__ 宿主機物理IP地址# __VIP__ 虛擬VIP地址sudo tee /etc/keepalived/keepalived.conf <<'EOF'! Configuration File for keepalivedglobal_defs { router_id LVS_DEVELscript_user root enable_script_security}vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1}vrrp_instance VI_1 { state __ROLE__ interface __NETINTERFACE__ mcast_src_ip __IP__ virtual_router_id 51 priority 101 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { __VIP__ } # HA 健康檢查 # track_script { # chk_apiserver # }}EOF# 此處 master-225 效能較好所以配置為Master (master-225 主機上執行)# master-225 10.10.107.225 => MASTERsed -i -e 's#__ROLE__#MASTER#g' \ -e 's#__NETINTERFACE__#ens32#g' \ -e 's#__IP__#10.10.107.225#g' \ -e 's#__VIP__#10.10.107.222#g' /etc/keepalived/keepalived.conf # master-224 10.10.107.224 => BACKUP (master-224 主機上執行)sed -i -e 's#__ROLE__#BACKUP#g' \ -e 's#__NETINTERFACE__#ens32#g' \ -e 's#__IP__#10.10.107.224#g' \ -e 's#__VIP__#10.10.107.222#g' /etc/keepalived/keepalived.conf # master-223 10.10.107.223 => BACKUP (master-223 主機上執行)sed -i -e 's#__ROLE__#BACKUP#g' \ -e 's#__NETINTERFACE__#ens32#g' \ -e 's#__IP__#10.10.107.223#g' \ -e 's#__VIP__#10.10.107.222#g' /etc/keepalived/keepalived.conf
溫馨提示: 注意上述的健康檢查是關閉註釋了的,你需要將K8S叢集建立完成後再開啟。
track_script { chk_apiserver}
步驟 04.【Master節點機器】進行配置 KeepAlived 健康檢查檔案。
sudo tee /etc/keepalived/check_apiserver.sh <<'EOF'#!/bin/basherr=0for k in $(seq 1 3)do check_code=$(pgrep haproxy) if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 1 continue else err=0 break fidoneif [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1else exit 0fiEOFsudo chmod +x /etc/keepalived/check_apiserver.sh
步驟 05.【Master節點機器】啟動 haproxy 、keepalived 相關服務及測試VIP漂移。
# 過載 Systemd 設定 haproxy 、keepalived 開機自啟以及立即啟動sudo systemctl daemon-reloadsudo systemctl enable --now haproxy && sudo systemctl enable --now keepalived# Synchronizing state of haproxy.service with SysV service script with /lib/systemd/systemd-sysv-install.# Executing: /lib/systemd/systemd-sysv-install enable haproxy# Synchronizing state of keepalived.service with SysV service script with /lib/systemd/systemd-sysv-install.# Executing: /lib/systemd/systemd-sysv-install enable keepalived# 在 master-223 主機中發現vip地址在其主機上。root@master-223:~$ ip addr # 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 # link/ether 00:0c:29:00:0f:8f brd ff:ff:ff:ff:ff:ff # inet 10.10.107.223/24 brd 10.10.107.255 scope global ens32 # valid_lft forever preferred_lft forever # inet 10.10.107.222/32 scope global ens32 # valid_lft forever preferred_lft forever# 其它兩臺主機上通訊驗證。root@master-224:~$ ping 10.10.107.222root@master-225:~$ ping 10.10.107.222
# 手動驗證VIP漂移,我們將該伺服器上keepalived停止掉。root@master-223:~$ pgrep haproxy # 6320 # 6321root@master-223:~$ /usr/bin/systemctl stop keepalived# 此時,發現VIP已經飄到master-225主機中root@master-225:~$ ip addr show ens32 # 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 # link/ether 00:0c:29:93:28:61 brd ff:ff:ff:ff:ff:ff # inet 10.10.107.225/24 brd 10.10.107.255 scope global ens32 # valid_lft forever preferred_lft forever # inet 10.10.107.222/32 scope global ens32 # valid_lft forever preferred_lft forever
至此,HAProxy 與 Keepalived 配置就告一段落了,下面將學習 ETCD 叢集配置與證書籤發。
4.配置部署etcd叢集與etcd證書籤發
描述: 建立一個高可用的ETCD叢集,此處我們在【master-225】機器中操作。
步驟 01.【master-225】建立一個配置與相關檔案存放的目錄, 以及下載獲取cfssl工具進行CA證書製作與簽發(cfssl工具往期文章參考地址: https://blog.weiyigeek.top/2019/10-21-12.html#3-CFSSL-生成 )。
# 工作目錄建立mkdir -vp /app/k8s-init-work && cd /app/k8s-init-work# cfssl 最新下載地址: https://github.com/cloudflare/cfssl/releases# cfssl 相關工具拉取 (如果拉取較慢,建議使用某雷下載,然後上傳到伺服器裡)curl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 -o /usr/local/bin/cfsslcurl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -o /usr/local/bin/cfssljsoncurl -L https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64 -o /usr/local/bin/cfssl-certinfo# 賦予執行許可權chmod +x /usr/local/bin/cfssl*/app# cfssl version# Version: 1.2.0# Revision: dev# Runtime: go1.6
溫馨提示:
-
cfssl : CFSSL 命令列工具
-
cfssljson : 用於從cfssl程式中獲取JSON輸出並將證書、金鑰、證書籤名請求檔案CSR和Bundle寫入到檔案中,
步驟 02.利用上述 cfssl 工具建立 CA 證書。
# - CA 證書籤名請求配置檔案fssl print-defaults csr > ca-csr.jsontee ca-csr.json <<'EOF'{ "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "ChongQing", "ST": "ChongQing", "O": "k8s", "OU": "System" } ], "ca": { "expiry": "87600h" }}EOF# 關鍵引數解析:CN: Common Name,瀏覽器使用該欄位驗證網站是否合法,一般寫的是域名,非常重要。瀏覽器使用該欄位驗證網站是否合法key:生成證書的演算法hosts:表示哪些主機名(域名)或者IP可以使用此csr申請的證書,為空或者""表示所有的都可以使用(本例中沒有`"hosts": [""]`欄位)names:常見屬性設定 * C: Country, 國家 * ST: State,州或者是省份 * L: Locality Name,地區,城市 * O: Organization Name,組織名稱,公司名稱(在k8s中常用於指定Group,進行RBAC繫結) * OU: Organization Unit Name,組織單位名稱,公司部門# - CA 證書策略配置檔案cfssl print-defaults config > ca-config.jsontee ca-config.json <<'EOF'{ "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] }, "etcd": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } }}EOF# 關鍵引數解析:default 預設策略,指定了證書的預設有效期是10年(87600h)profile 自定義策略配置 * kubernetes:表示該配置(profile)的用途是為kubernetes生成證書及相關的校驗工作 * signing:表示該證書可用於簽名其它證書;生成的 ca.pem 證書中 CA=TRUE * server auth:表示可以該CA 對 server 提供的證書進行驗證 * client auth:表示可以用該 CA 對 client 提供的證書進行驗證 * expiry:也表示過期時間,如果不寫以default中的為準# - 執行cfssl gencert 命令生成CA證書# 利用CA證書籤名請求配置檔案 ca-csr.json 生成CA證書和CA私鑰和CSR(證書籤名請求):cfssl gencert -initca ca-csr.json | cfssljson -bare ca # 2022/04/27 16:49:37 [INFO] generating a new CA key and certificate from CSR # 2022/04/27 16:49:37 [INFO] generate received request # 2022/04/27 16:49:37 [INFO] received CSR # 2022/04/27 16:49:37 [INFO] generating key: rsa-2048 # 2022/04/27 16:49:37 [INFO] encoded CSR # 2022/04/27 16:49:37 [INFO] signed certificate with serial number 245643466964695827922023924375276493244980966303$ ls # ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem$ openssl x509 -in ca.pem -text -noout | grep "Not" # Not Before: Apr 27 08:45:00 2022 GMT # Not After : Apr 24 08:45:00 2032 GMT
溫馨提示: 如果將 expiry 設定為87600h 表示證書過期時間為十年。
步驟 03.配置ETCD證書相關檔案以及生成其證書,
# etcd 證書請求檔案tee etcd-csr.json <<'EOF'{ "CN": "etcd", "hosts": [ "127.0.0.1", "10.10.107.223", "10.10.107.224", "10.10.107.225", "etcd1", "etcd2", "etcd3" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "ChongQing", "ST": "ChongQing", "O": "etcd", "OU": "System" } ]}EOF# 利用ca證書籤發生成etcd證書cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd$ ls etcd*etcd.csr etcd-csr.json etcd-key.pem etcd.pem$ openssl x509 -in etcd.pem -text -noout | grep "X509v3 Subject Alternative Name" -A 1 # X509v3 Subject Alternative Name: # DNS:etcd1, DNS:etcd2, DNS:etcd3, IP Address:127.0.0.1, IP Address:10.10.107.223, IP Address:10.10.107.224, IP Address:10.10.107.225
步驟 04.【所有Master節點主機】下載部署ETCD叢集, 首先我們需要下載etcd軟體包, 可以 Github release 找到最新版本的etcd下載路徑(https://github.com/etcd-io/etcd/releases/)。
# 下載wget -L https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gztar -zxvf etcd-v3.5.4-linux-amd64.tar.gzcp -a etcd* /usr/local/bin/# 版本 etcd --version # etcd Version: 3.5.4 # Git SHA: 08407ff76 # Go Version: go1.16.15 # Go OS/Arch: linux/amd64# 複製到其它master主機上scp -P 20211 ./etcd-v3.5.4-linux-amd64.tar.gz weiyigeek@master-223:~scp -P 20211 ./etcd-v3.5.4-linux-amd64.tar.gz weiyigeek@master-224:~# 分別在master-223與master-224執行, 解壓到 /usr/local/ 目錄同樣複製二進位制檔案到 /usr/local/bin/tar -zxvf /home/weiyigeek/etcd-v3.5.4-linux-amd64.tar.gz -C /usr/local/cp -a /usr/local/etcd-v3.5.4-linux-amd64/etcd* /usr/local/bin/
溫馨提示: etcd 官網地址 ( https://etcd.io/)
步驟 05.建立etcd叢集所需的配置檔案。
# 證書準備mkdir -vp /etc/etcd/pki/cp *.pem /etc/etcd/pki/ls /etc/etcd/pki/ # ca-key.pem ca.pem etcd-key.pem etcd.pem# 上傳到~家目錄,並需要將其複製到 /etc/etcd/pki/ 目錄中scp -P 20211 *.pem weiyigeek@master-224:~scp -P 20211 *.pem weiyigeek@master-223:~ # ****************** [ 安全登陸 (Security Login) ] ***************** # Authorized only. All activity will be monitored and reported.By Security Center. # ca-key.pem 100% 1675 3.5MB/s 00:00 # ca.pem 100% 1375 5.2MB/s 00:00 # etcd-key.pem 100% 1679 7.0MB/s 00:00 # etcd.pem 100% 1399 5.8MB/s 00:00# master-225 執行tee /etc/etcd/etcd.conf <<'EOF'# [成員配置]# member 名稱ETCD_NAME=etcd1# 儲存資料的目錄(注意需要建立)ETCD_DATA_DIR="/var/lib/etcd/data"# 用於監聽客戶端etcdctl或者curl連線ETCD_LISTEN_CLIENT_URLS="https://10.10.107.225:2379,https://127.0.0.1:2379"# 用於監聽叢集中其它member的連線ETCD_LISTEN_PEER_URLS="https://10.10.107.225:2380"# [證書配置]# ETCD_CERT_FILE=/etc/etcd/pki/etcd.pem# ETCD_KEY_FILE=/etc/etcd/pki/etcd-key.pem# ETCD_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.pem# ETCD_CLIENT_CERT_AUTH=true# ETCD_PEER_CLIENT_CERT_AUTH=true# ETCD_PEER_CERT_FILE=/etc/etcd/pki/etcd.pem# ETCD_PEER_KEY_FILE=/etc/etcd/pki/etcd-key.pem# ETCD_PEER_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.pem# [叢集配置]# 本機地址用於通知客戶端,客戶端通過此IPs與叢集通訊;ETCD_ADVERTISE_CLIENT_URLS="https://10.10.107.225:2379"# 本機地址用於通知叢集member與member通訊ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.107.225:2380"# 描述叢集中所有節點的資訊,本member根據此資訊去聯絡其他memberETCD_INITIAL_CLUSTER="etcd1=https://10.10.107.225:2380,etcd2=https://10.10.107.224:2380,etcd3=https://10.10.107.223:2380"# 叢集狀態新建叢集時候設定為new,若是想加入某個已經存在的叢集設定為existingETCD_INITIAL_CLUSTER_STATE=newEOF# master-224 執行tee /etc/etcd/etcd.conf <<'EOF'# [成員配置]# member 名稱ETCD_NAME=etcd2# 儲存資料的目錄(注意需要建立)ETCD_DATA_DIR="/var/lib/etcd/data"# 用於監聽客戶端etcdctl或者curl連線ETCD_LISTEN_CLIENT_URLS="https://10.10.107.224:2379,https://127.0.0.1:2379"# 用於監聽叢集中其它member的連線ETCD_LISTEN_PEER_URLS="https://10.10.107.224:2380"# [叢集配置]# 本機地址用於通知客戶端,客戶端通過此IPs與叢集通訊;ETCD_ADVERTISE_CLIENT_URLS="https://10.10.107.224:2379"# 本機地址用於通知叢集member與member通訊ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.107.224:2380"# 描述叢集中所有節點的資訊,本member根據此資訊去聯絡其他memberETCD_INITIAL_CLUSTER="etcd1=https://10.10.107.225:2380,etcd2=https://10.10.107.224:2380,etcd3=https://10.10.107.223:2380"# 叢集狀態新建叢集時候設定為new,若是想加入某個已經存在的叢集設定為existingETCD_INITIAL_CLUSTER_STATE=newEOF# master-223 執行tee /etc/etcd/etcd.conf <<'EOF'# [成員配置]# member 名稱ETCD_NAME=etcd3# 儲存資料的目錄(注意需要建立)ETCD_DATA_DIR="/var/lib/etcd/data"# 用於監聽客戶端etcdctl或者curl連線ETCD_LISTEN_CLIENT_URLS="https://10.10.107.223:2379,https://127.0.0.1:2379"# 用於監聽叢集中其它member的連線ETCD_LISTEN_PEER_URLS="https://10.10.107.223:2380"# [叢集配置]# 本機地址用於通知客戶端,客戶端通過此IPs與叢集通訊;ETCD_ADVERTISE_CLIENT_URLS="https://10.10.107.223:2379"# 本機地址用於通知叢集member與member通訊ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.107.223:2380"# 描述叢集中所有節點的資訊,本member根據此資訊去聯絡其他memberETCD_INITIAL_CLUSTER="etcd1=https://10.10.107.225:2380,etcd2=https://10.10.107.224:2380,etcd3=https://10.10.107.223:2380"# 叢集狀態新建叢集時候設定為new,若是想加入某個已經存在的叢集設定為existingETCD_INITIAL_CLUSTER_STATE=newEOF
步驟 06.【所有Master節點主機】建立配置 etcd 的 systemd 管理配置檔案,並啟動其服務。
mkdir -vp /var/lib/etcd/cat > /usr/lib/systemd/system/etcd.service <<EOF[Unit]Description=Etcd ServerDocumentation=https://github.com/etcd-io/etcdAfter=network.targetAfter=network-online.targetwants=network-online.target[Service]Type=notifyWorkingDirectory=/var/lib/etcd/EnvironmentFile=-/etc/etcd/etcd.confExecStart=/usr/local/bin/etcd \ --client-cert-auth \ --trusted-ca-file /etc/etcd/pki/ca.pem \ --cert-file /etc/etcd/pki/etcd.pem \ --key-file /etc/etcd/pki/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file /etc/etcd/pki/ca.pem \ --peer-cert-file /etc/etcd/pki/etcd.pem \ --peer-key-file /etc/etcd/pki/etcd-key.pemRestart=on-failureRestartSec=5LimitNOFILE=65535LimitNPROC=65535[Install]WantedBy=multi-user.targetEOF# 過載 systemd && 開機啟動與手動啟用etcd服務systemctl daemon-reload && systemctl enable --now etcd.service
步驟 07.【所有Master節點主機】檢視各個master節點的etcd叢集服務是否正常及其健康狀態。
# 服務檢視systemctl status etcd.service# 利用 etcdctl 工具檢視叢集成員資訊export ETCDCTL_API=3etcdctl --endpoints=https://10.10.107.225:2379,https://10.10.107.224:2379,https://10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=table member list # +------------------+---------+-------+----------------------------+----------------------------+------------+ # | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | # +------------------+---------+-------+----------------------------+----------------------------+------------+ # | 144934d02ad45ec7 | started | etcd3 | https://10.10.107.223:2380 | https://10.10.107.223:2379 | false | # | 2480d95a2df867a4 | started | etcd2 | https://10.10.107.224:2380 | https://10.10.107.224:2379 | false | # | 2e8fddd3366a3d88 | started | etcd1 | https://10.10.107.225:2380 | https://10.10.107.225:2379 | false | # +------------------+---------+-------+----------------------------+----------------------------+------------+# 叢集節點資訊etcdctl --endpoints=https://10.10.107.225:2379,https://10.10.107.224:2379,https://10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=table endpoint status # +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ # | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | # +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ # | https://10.10.107.225:2379 | 2e8fddd3366a3d88 | 3.5.4 | 20 kB | false | false | 3 | 12 | 12 | | # | https://10.10.107.224:2379 | 2480d95a2df867a4 | 3.5.4 | 20 kB | true | false | 3 | 12 | 12 | | # | https://10.10.107.223:2379 | 144934d02ad45ec7 | 3.5.4 | 20 kB | false | false | 3 | 12 | 12 | | # +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+# 叢集節點健康狀態etcdctl --endpoints=https://10.10.107.225:2379,https://10.10.107.224:2379,https://10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=table endpoint health # +----------------------------+--------+-------------+-------+ # | ENDPOINT | HEALTH | TOOK | ERROR | # +----------------------------+--------+-------------+-------+ # | https://10.10.107.225:2379 | true | 9.151813ms | | # | https://10.10.107.224:2379 | true | 10.965914ms | | # | https://10.10.107.223:2379 | true | 11.165228ms | | # +----------------------------+--------+-------------+-------+# 叢集節點效能測試etcdctl --endpoints=https://10.10.107.225:2379,https://10.10.107.224:2379,https://10.10.107.223:2379 \--cacert="/etc/etcd/pki/ca.pem" --cert="/etc/etcd/pki/etcd.pem" --key="/etc/etcd/pki/etcd-key.pem" \--write-out=tableendpoint check perf# 59 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom ! 98.33%PASS: Throughput is 148 writes/s# Slowest request took too long: 1.344053s# Stddev too high: 0.143059s# FAIL
5.Containerd 執行時安裝部署
步驟 01.【所有節點】在各主機中安裝二進位制版本的 containerd.io 執行時服務,Kubernertes 通過 CRI 外掛來連線 containerd 服務中, 控制容器的生命週期。
# 從 Github 中下載最新的版本的 cri-containerd-cni wget -L https://github.com/containerd/containerd/releases/download/v1.6.4/cri-containerd-cni-1.6.4-linux-amd64.tar.gz# 解壓到當前cri-containerd-cni目錄中。mkdir -vp cri-containerd-cnitar -zxvf cri-containerd-cni-1.6.4-linux-amd64.tar.gz -C cri-containerd-cni
步驟 02.檢視其檔案以及配置檔案路徑資訊。
$ tree ./cri-containerd-cni/.├── etc│ ├── cni│ │ └── net.d│ │ └── 10-containerd-net.conflist│ ├── crictl.yaml│ └── systemd│ └── system│ └── containerd.service├── opt│ ├── cni│ │ └── bin│ │ ├── bandwidth│ │ ├── bridge│ │ ├── dhcp│ │ ├── firewall│ │ ├── host-device│ │ ├── host-local│ │ ├── ipvlan│ │ ├── loopback│ │ ├── macvlan│ │ ├── portmap│ │ ├── ptp│ │ ├── sbr│ │ ├── static│ │ ├── tuning│ │ ├── vlan│ │ └── vrf│ └── containerd│ └── cluster│ ├── gce│ │ ├── cloud-init│ │ │ ├── master.yaml│ │ │ └── node.yaml│ │ ├── cni.template│ │ ├── configure.sh│ │ └── env│ └── version└── usr └── local ├── bin │ ├── containerd │ ├── containerd-shim │ ├── containerd-shim-runc-v1 │ ├── containerd-shim-runc-v2 │ ├── containerd-stress │ ├── crictl │ ├── critest │ ├── ctd-decoder │ └── ctr └── sbin └── runc# 然後在所有節點上覆制到上述資料夾到對應目錄中cd ./cri-containerd-cni/cp -r etc/ /cp -r opt/ /cp -r usr/ /
步驟 03.【所有節點】進行containerd 配置建立並修改 config.toml .
mkdir -vp /etc/containerd# 預設配置生成containerd config default >/etc/containerd/config.tomlls /etc/containerd/config.toml # /etc/containerd/config.toml# pause 映象源sed -i "s#k8s.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml# 使用 SystemdCgroupsed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml# docker.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."docker.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["https://xlx9erfu.mirror.aliyuncs.com","https://docker.mirrors.ustc.edu.cn"]' /etc/containerd/config.toml# gcr.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."gcr.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["https://gcr.mirrors.ustc.edu.cn"]' /etc/containerd/config.toml# k8s.gcr.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."k8s.gcr.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["https://gcr.mirrors.ustc.edu.cn/google-containers/","https://registry.cn-hangzhou.aliyuncs.com/google_containers/"]' /etc/containerd/config.toml# quay.io mirrorsed -i '/registry.mirrors]/a\ \ \ \ \ \ \ \ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."quay.io"]' /etc/containerd/config.tomlsed -i '/registry.mirrors."quay.io"]/a\ \ \ \ \ \ \ \ \ \ endpoint = ["https://quay.mirrors.ustc.edu.cn"]' /etc/containerd/config.toml
步驟 04.客戶端工具 runtime 與 映象 端點配置:
# 手動設定臨時生效# crictl config runtime-endpoint /run/containerd/containerd.sock# /run/containerd/containerd.sock # 配置檔案設定永久生效cat <<EOF > /etc/crictl.yamlruntime-endpoint: unix:///run/containerd/containerd.sockimage-endpoint: unix:///run/containerd/containerd.socktimeout: 10debug: falseEOF
步驟 05.過載 systemd自啟和啟動containerd.io服務。
systemctl daemon-reload && systemctl enable --now containerd.servicesystemctl status containerd.servicectr version # Client: # Version: 1.5.11 # Revision: 3df54a852345ae127d1fa3092b95168e4a88e2f8 # Go version: go1.17.8 # Server: # Version: 1.5.11 # Revision: 3df54a852345ae127d1fa3092b95168e4a88e2f8 # UUID: 71a28bbb-6ed6-408d-a873-e394d48b35d8
步驟 06.用於根據OCI規範生成和執行容器的CLI工具 runc 版本檢視
runc -v # runc version 1.1.1 # commit: v1.1.1-0-g52de29d7 # spec: 1.0.2-dev # go: go1.17.9 # libseccomp: 2.5.1
溫馨提示: 當預設 runc 執行提示 runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond
時,由於上述軟體包中包含的runc對系統依賴過多,所以建議單獨下載安裝 runc 二進位制專案(https://github.com/opencontainers/runc/)
wget https://github.com/opencontainers/runc/releases/download/v1.1.1/runc.amd64# 執行許可權賦予chmod +x runc.amd64# 替換掉 /usr/local/sbin/ 路徑原軟體包中的 runcmv runc.amd64 /usr/local/sbin/runc
本文至此完畢,更多技術文章,盡情期待下一章節!
歡迎各位志同道合的朋友一起學習交流,如文章有誤請在下方留下您寶貴的經驗知識,個人郵箱地址【master#weiyigeek.top】
或者個人公眾號【WeiyiGeek】
聯絡我。
更多文章來源於【WeiyiGeek Blog 個人部落格 - 為了能到遠方,腳下的每一步都不能少 】
個人主頁: 【 https://weiyigeek.top】
部落格地址: 【 https://blog.weiyigeek.top 】
專欄書寫不易,如果您覺得這個專欄還不錯的,請給這篇專欄 【點個贊、投個幣、收個藏、關個注,轉個發,留個言】(人間六大情),這將對我的肯定,謝謝!。
-
echo "【點個贊】,動動你那粗壯的拇指或者芊芊玉手,親!"
-
printf("%s", "【投個幣】,萬水千山總是情,投個硬幣行不行,親!")
-
fmt.Printf("【收個藏】,閱後即焚不吃灰,親!")
-
console.info("【轉個發】,讓更多的志同道合的朋友一起學習交流,親!")
-
System.out.println("【關個注】,後續瀏覽檢視不迷路喲,親!")
-
cout << "【留個言】,文章寫得好不好、有沒有錯誤,一定要留言喲,親! " << endl;
往期相關文章
記一次在k8s叢集搭建的Harbor私有倉庫無法提供服務之映象遷移恢復實踐
4.如何使用nerdctl工具並配合Containerd容器執行時來替代Docker容器環境
WeiyiGeek
Always keep a beginner's mind, don't forget the beginner's mind. Blog :【https://weiyigeek.top】
174篇原創內容
更多網路安全、系統運維、應用開發、全棧文章,盡在【個人部落格 - https://blog.weiyigeek.top】站點,謝謝支援!
↓↓↓ 更多文章,請點選下方閱讀原文。