kubernetes——二進位制多節點部署
Kubernetes二進位制部署
文章目錄
- Kubernetes 是用於自動部署，擴充套件和管理容器化應用程式的開源系統。
- Kubernetes 是一個可移植的、可擴充套件的開源平臺，用於管理容器化的工作負載和服務，可促進宣告式配置和自動化。Kubernetes 擁有一個龐大且快速增長的生態系統。Kubernetes 的服務、支援和工具廣泛可用。名稱 Kubernetes 源於希臘語，意為 "舵手"或 “飛行員”。Google 在 2014 年開源了 Kubernetes 專案。Kubernetes 建立在 Google在大規模執行生產工作負載方面擁有十幾年的經驗的基礎上，結合了社群中最好的想法和實踐。
Kubernetes 為您提供：
(1) 服務發現和負載均衡
- Kubernetes 可以使用 DNS 名稱或自己的 IP 地址公開容器，如果到容器的流量很大，Kubernetes
可以負載均衡並分配網路流量，從而使部署穩定。
(2) 儲存編排
- Kubernetes 允許您自動掛載您選擇的儲存系統，例如本地儲存、公共雲提供商等。
(3) 自動部署和回滾
- 您可以使用 Kubernetes 描述已部署容器的所需狀態，它可以以受控的速率將實際狀態更改為所需狀態。例如，您可以自動化Kubernetes 來為您的部署建立新容器，刪除現有容器並將它們的所有資源用於新容器。
(4) 自動二進位制打包
- Kubernetes 允許您指定每個容器所需 CPU 和記憶體（RAM）。當容器指定了資源請求時，Kubernetes
可以做出更好的決策來管理容器的資源。
(5) 自我修復
- Kubernetes 重新啟動失敗的容器、替換容器、殺死不響應使用者定義的執行狀況檢查的容器，並且在準備好服務之前不將其通告給客戶端。
(6) 金鑰與配置管理
- Kubernetes 允許您儲存和管理敏感資訊，例如密碼、OAuth 令牌和 ssh金鑰。您可以在不重建容器映象的情況下部署和更新金鑰和應用程式配置，也無需在堆疊配置中暴露金鑰。 批處理 。提供一次性任務，定時任務；滿足批量資料處理和分析的場景。
一、 實驗環境
二、環境部署
三、K8S單點部署
3.1 master01操作
（1）製作證書
[root@localhost ~] mkdir k8s
[root@localhost ~] cd k8s/
[root@localhost k8s] ls //從宿主機拖進來
etcd-cert.sh etcd.sh
[root@localhost k8s] mkdir etcd-cert
[root@localhost k8s] mv etcd-cert.sh etcd-cert
官方文件證書介紹https://kubernetes.io/zh/docs/concepts/cluster-administration/certificates/
//下載證書製作工具
[root@localhost k8s] vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
//下載cfssl官方包
[root@localhost k8s] bash cfssl.sh
[root@localhost k8s] ls /usr/local/bin/
cfssl cfssl-certinfo cfssljson
//加執行許可權
//開始製作證書
//cfssl 生成證書工具 cfssljson通過傳入json檔案生成證書
cfssl-certinfo檢視證書資訊
//定義ca證書
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
//實現證書籤名
cat > ca-csr.json <<EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
//生產證書，生成ca-key.pem ca.pem
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2020/01/13 16:32:56 [INFO] generating a new CA key and certificate from CSR
2020/01/13 16:32:56 [INFO] generate received request
2020/01/13 16:32:56 [INFO] received CSR
2020/01/13 16:32:56 [INFO] generating key: rsa-2048
2020/01/13 16:32:56 [INFO] encoded CSR
2020/01/13 16:32:56 [INFO] signed certificate with serial number 595395605361409801445623232629543954602649157326
//指定etcd三個節點之間的通訊驗證
cat > server-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"192.168.75.200",
"192.168.75.201",
"192.168.75.144"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
//生成ETCD證書 server-key.pem server.pem
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2020/01/13 17:01:30 [INFO] generate received request
2020/01/13 17:01:30 [INFO] received CSR
2020/01/13 17:01:30 [INFO] generating key: rsa-2048
2020/01/13 17:01:30 [INFO] encoded CSR
2020/01/13 17:01:30 [INFO] signed certificate with serial number 202782620910318985225034109831178600652439985681
2020/01/13 17:01:30 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
ETCD 二進位制包地址
https://github.com/etcd-io/etcd/releases
複製到centos7中
[root@localhost etcd-cert] ls
ca-config.json etcd-cert.sh server-csr.json
ca.csr etcd-v3.3.10-linux-amd64.tar.gz server-key.pem
ca-csr.json flannel-v0.10.0-linux-amd64.tar.gz server.pem
ca-key.pem kubernetes-server-linux-amd64.tar.gz
ca.pem server.csr
[root@localhost etcd-cert] mv *.tar.gz ../
[root@localhost k8s] ls
cfssl.sh etcd.sh flannel-v0.10.0-linux-amd64.tar.gz
etcd-cert etcd-v3.3.10-linux-amd64.tar.gz kubernetes-server-linux-amd64.tar.gz
[root@localhost k8s] tar zxvf etcd-v3.3.10-linux-amd64.tar.gz
[root@localhost k8s] ls etcd-v3.3.10-linux-amd64
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
[root@localhost k8s] mkdir /opt/etcd/{cfg,bin,ssl} -p //配置檔案，命令檔案，證書
[root@localhost k8s] mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
//證書拷貝
[root@localhost k8s] cp etcd-cert/*.pem /opt/etcd/ssl/
//進入卡住狀態等待其他節點加入
[root@localhost k8s] bash etcd.sh etcd01 192.168.75.200 etcd02=https://192.168.75.201:2380,etcd03=https://192.168.75.144:2380
//使用另外一個會話開啟，會發現etcd程式已經開啟
[root@localhost ~] ps -ef | grep etcd
//拷貝證書去其他節點
[root@localhost k8s] scp -r /opt/etcd/ root@192.168.75.201:/opt/
[root@localhost k8s] scp -r /opt/etcd/ root@192.168.75.144:/opt
//啟動指令碼拷貝其他節點
[root@localhost k8s] scp /usr/lib/systemd/system/etcd.service root@192.168.75.201:/usr/lib/systemd/system/
[root@localhost k8s] scp /usr/lib/systemd/system/etcd.service root@192.168.75.144:/usr/lib/systemd/system/
（2）在node01節點修改
[root@localhost ~] vim /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.75.201:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.75.201:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.75.201:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.75.201:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.75.200:2380,etcd02=https://192.168.75.201:2380,etcd03=https://192.168.75.144:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
//啟動
[root@localhost ssl] systemctl start etcd
[root@localhost ssl] systemctl status etcd
（3）node02節點修改
[root@localhost ~] vim /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.75.144:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.75.144:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.75.144:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.75.144:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.75.200:2380,etcd02=https://192.168.75.201:2380,etcd03=https://192.168.75.144:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
//啟動
[root@localhost ssl] systemctl start etcd
[root@localhost ssl]systemctl status etcd
//檢查群集狀態
[root@localhost etcd-cert]/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.75.200:2379,https://192.168.75.201:2379,https://192.168.75.144:2379" cluster-health
member 3eae9a550e2e3ec is healthy: got healthy result from https://192.168.75.144:2379
member 26cd4dcf17bc5cbd is healthy: got healthy result from https://192.168.75.201:2379
member 2fcd2df8a9411750 is healthy: got healthy result from https://192.168.75.200:2379
cluster is healthy
（4）docker引擎部署
- 所有node節點部署docker引擎
- 詳見docker安裝指令碼
- flannel網路配置
//寫入分配的子網段到ETCD中，供flannel使用
[root@localhost etcd-cert]/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.75.200:2379,https://192.168.75.201:2379,https://192.168.75.144:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}' { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
//檢視寫入的資訊
[root@localhost etcd-cert]/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.75.200:2379,https://192.168.75.201:2379,https://192.168.75.144:2379" get /coreos.com/network/config
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
//拷貝到所有node節點（只需要部署在node節點即可）
[root@localhost k8s] scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.75.201:/root
[root@localhost k8s] scp flannel-v0.10.0-linux-amd64.tar.gz root@192.168.75.144:/root
- 所有node節點操作解壓
[root@localhost ~] tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
flanneld
mk-docker-opts.sh
README.md
//k8s工作目錄
[root@localhost ~] mkdir /opt/kubernetes/{cfg,bin,ssl} -p
[root@localhost ~] mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/
[root@localhost ~] vim flannel.sh
#!/bin/bash
ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}
cat <<EOF >/opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF
cat <<EOF >/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
Bash flannel.sh
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld
//開啟flannel網路功能
[root@localhost~]bash flannel.sh https://192.168.75.200:2379,https://192.168.75.201:2379,https://192.168.75.144:2379 Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
//配置docker連線flannel
[root@localhost ~] vim /usr/lib/systemd/system/docker.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
[root@localhost ~] cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.64.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
//說明：bip指定啟動時的子網
DOCKER_NETWORK_OPTIONS=" --bip=172.17.64.1/24 --ip-masq=false --mtu=1450"
//重啟docker服務
[root@localhost ~] systemctl daemon-reload
[root@localhost ~] systemctl restart docker
//檢視flannel網路
[root@localhost ~] ifconfig
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.64.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::fc7c:e1ff:fe1d:224 prefixlen 64 scopeid 0x20<link>
ether fe:7c:e1:1d:02:24 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0
//測試ping通對方docker0網路卡 證明flannel起到路由作用
[root@localhost ssl] ping 172.17.20.1
PING 172.17.20.1 (172.17.20.1) 56(84) bytes of data.
64 bytes from 172.17.20.1: icmp_seq=1 ttl=64 time=3.04 ms
64 bytes from 172.17.20.1: icmp_seq=2 ttl=64 time=0.481 ms
^C
--- 172.17.20.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.481/1.763/3.046/1.283 ms
[root@localhost ssl]#
[root@localhost ~] docker run -it centos:7 /bin/bash
[root@5f9a65565b53 /] yum install net-tools -y
[root@5f9a65565b53 /] ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.64.2 netmask 255.255.255.0 broadcast 172.17.84.255
ether 02:42:ac:11:54:02 txqueuelen 0 (Ethernet)
RX packets 18192 bytes 13930229 (13.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6179 bytes 337037 (329.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
//再次測試ping通兩個node中的centos:7容器
（5）部署master元件
//在master上操作，api-server生成證書
[root@localhost k8s] unzip master.zip
[root@localhost k8s] mkdir /opt/kubernetes/{cfg,bin,ssl} -p
[root@localhost k8s] mkdir k8s-cert
[root@localhost k8s] cd k8s-cert/
[root@localhost k8s-cert]# ls
k8s-cert.sh
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
#-----------------------
cat > server-csr.json <<EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"192.168.75.200", //master1
"192.168.75.122", //master2
"192.168.75.166", //vip
"192.168.75.155", //lb （master）
"192.168.75.177", //lb （backup）
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
#-----------------------
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
#-----------------------
cat > kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
//生成k8s證書
[root@localhost k8s-cert] bash k8s-cert.sh
2020/01/15 23:31:03 [INFO] generating a new CA key and certificate from CSR
2020/01/15 23:31:03 [INFO] generate received request
2020/01/15 23:31:03 [INFO] received CSR
2020/01/15 23:31:03 [INFO] generating key: rsa-2048
2020/01/15 23:31:03 [INFO] encoded CSR
2020/01/15 23:31:03 [INFO] signed certificate with serial number 200957285008634365032949076461783766565292979186
2020/01/15 23:31:03 [INFO] generate received request
2020/01/15 23:31:03 [INFO] received CSR
2020/01/15 23:31:03 [INFO] generating key: rsa-2048
2020/01/15 23:31:03 [INFO] encoded CSR
2020/01/15 23:31:03 [INFO] signed certificate with serial number 531833477097469967316212525772159687029821034128
2020/01/15 23:31:03 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
2020/01/15 23:31:04 [INFO] generate received request
2020/01/15 23:31:04 [INFO] received CSR
2020/01/15 23:31:04 [INFO] generating key: rsa-2048
2020/01/15 23:31:04 [INFO] encoded CSR
2020/01/15 23:31:04 [INFO] signed certificate with serial number 684040931566157342098288079791465097738732990534
2020/01/15 23:31:04 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
2020/01/15 23:31:04 [INFO] generate received request
2020/01/15 23:31:04 [INFO] received CSR
2020/01/15 23:31:04 [INFO] generating key: rsa-2048
2020/01/15 23:31:04 [INFO] encoded CSR
2020/01/15 23:31:04 [INFO] signed certificate with serial number 681469506930419424853732902538890426797365900103
2020/01/15 23:31:04 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@localhost k8s-cert] ls *pem
admin-key.pem ca-key.pem kube-proxy-key.pem server-key.pem
admin.pem ca.pem kube-proxy.pem server.pem
[root@localhost k8s-cert] cp ca*pem server*pem /opt/kubernetes/ssl/
[root@localhost k8s-cert] cd ..
//解壓kubernetes壓縮包
[root@localhost k8s] tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@localhost k8s] cd /root/k8s/kubernetes/server/bin
//複製關鍵命令檔案
[root@localhost bin] cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/
[root@localhost k8s] cd /root/k8s
[root@localhost k8s] vim /opt/kubernetes/cfg/token.csv
0fb61c46f8991b718eb38d27b605b008,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
序列號，使用者名稱，id，角色
//使用 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 可以隨機生成序列號
//二進位制檔案，token，證書都準備好，開啟apiserver
[root@localhost k8s] bash apiserver.sh 192.168.75.200 https://192.168.75.200:2379,https://192.168.75.201:2379,https://192.168.75.144:2379
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
//檢查程式是否啟動成功
[root@localhost k8s] ps aux | grep kube
//檢視配置檔案
[root@localhost k8s] cat /opt/kubernetes/cfg/kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.75.200:2379,https://192.168.75.201:2379,https://192.168.75.144:2379 \
--bind-address=192.168.75.200 \
--secure-port=6443 \
--advertise-address=192.168.75.200 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
//監聽的https埠
[root@localhost k8s] netstat -ntap | grep 6443
tcp 0 0 192.168.75.200:6443 0.0.0.0:* LISTEN 46459/kube-apiserve
tcp 0 0 192.168.75.200:6443 192.168.75.200:36806 ESTABLISHED 46459/kube-apiserve
tcp 0 0 192.168.75.200:36806 192.168.75.200:6443 ESTABLISHED 46459/kube-apiserve
[root@localhost k8s] netstat -ntap | grep 8080
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 46459/kube-apiserve
//啟動scheduler服務
[root@localhost k8s] ./scheduler.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@localhost k8s] ps aux | grep ku
[root@localhost k8s] chmod +x controller-manager.sh
//啟動controller-manager
[root@localhost k8s] ./controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
//檢視master 節點狀態
[root@localhost k8s] /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
--------------------------------------------------node節點部署------------------------------------------
//master上操作
//把 kubelet、kube-proxy拷貝到node節點上去
[root@localhost bin] scp kubelet kube-proxy root@192.168.75.201:/opt/kubernetes/bin/
root@192.168.75.201's password:
kubelet 100% 168MB 27.9MB/s 00:06
kube-proxy 100% 48MB 31.5MB/s 00:01
[root@localhost bin]# scp kubelet kube-proxy root@192.168.75.144:/opt/kubernetes/bin/
root@192.168.75.144's password:
kubelet 100% 168MB 56.1MB/s 00:03
kube-proxy 100% 48MB 37.3MB/s 00:01
//nod01節點操作（複製node.zip到/root目錄下再解壓）
[root@localhost ~] ls
anaconda-ks.cfg flannel-v0.10.0-linux-amd64.tar.gz node.zip 公共 視訊 文件 音樂
flannel.sh initial-setup-ks.cfg README.md 模板 圖片 下載 桌面
//解壓node.zip，獲得kubelet.sh proxy.sh
[root@localhost ~] unzip node.zip
//在master上操作
[root@localhost k8s] mkdir kubeconfig
[root@localhost k8s] cd kubeconfig/
//拷貝kubeconfig.sh檔案進行重新命名
[root@localhost kubeconfig] mv kubeconfig.sh kubeconfig
[root@localhost kubeconfig] vim kubeconfig
----------------刪除以下部分----------------------------------------------------------------------
# 建立 TLS Bootstrapping Token
#BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
BOOTSTRAP_TOKEN=0fb61c46f8991b718eb38d27b605b008
cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
//獲取token資訊（紅色部分）
[root@localhost ~] cat /opt/kubernetes/cfg/token.csv
6351d652249951f79c33acdab329e4c4,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
//配置檔案修改為tokenID
# 設定客戶端認證引數
kubectl config set-credentials kubelet-bootstrap \
--token=6351d652249951f79c33acdab329e4c4 \
--kubeconfig=bootstrap.kubeconfig
//設定環境變數（可以寫入到/etc/profile中）
[root@localhost kubeconfig] export PATH=$PATH:/opt/kubernetes/bin/
[root@localhost kubeconfig] kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
//生成配置檔案
[root@localhost kubeconfig] bash kubeconfig 192.168.75.200 /opt/k8s-cert
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".
[root@localhost kubeconfig] ls
bootstrap.kubeconfig kubeconfig kube-proxy.kubeconfig
//拷貝配置檔案到node節點
[root@localhost kubeconfig] scp bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.75.201:/opt/kubernetes/cfg/
root@192.168.75.201's password:
bootstrap.kubeconfig 100% 2169 2.1MB/s 00:00
kube-proxy.kubeconfig 100% 6275 4.2MB/s 00:00
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.75.144:/opt/kubernetes/cfg/
root@192.168.75.144's password:
bootstrap.kubeconfig 100% 2169 1.8MB/s 00:00
kube-proxy.kubeconfig 100% 6275 5.3MB/s 00:00
//建立bootstrap角色賦予許可權用於連線apiserver請求籤名（關鍵）
[root@localhost kubeconfig] kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
（6）在node01節點上操作
[root@localhost ~] bash kubelet.sh 192.168.75.201
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
//檢查kubelet服務啟動
[root@localhost ~] ps aux | grep kube
root 106845 1.4 1.1 371744 44780 ? Ssl 00:34 0:01 /opt/kubernetes/bin/kubelet --logtostderr=true --v=4 --hostname-override=192.168.75.201 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfgkubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
root 106876 0.0 0.0 112676 984 pts/0 S+ 00:35 0:00 grep --color=auto kube
//master上操作
//檢查到node01節點的請求
[root@localhost kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 4m27s kubelet-bootstrap Pending（等待叢集給該節點頒發證書）
[root@localhost kubeconfig] kubectl certificate approve node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A
certificatesigningrequest.certificates.k8s.io/node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A approved
//繼續檢視證書狀態
[root@localhost kubeconfig] kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 8m56s kubelet-bootstrap Approved,Issued（已經被允許加入群集）
//檢視群集節點，成功加入node01節點
[root@localhost kubeconfig] kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.75.201 Ready <none> 118s v1.12.3
//在node01節點操作，啟動proxy服務
[root@localhost ~] bash proxy.sh 192.168.75.201
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[root@localhost ~] systemctl status kube-proxy.service
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2020-02-02 00:47:29 CST; 11s ago
Main PID: 108006 (kube-proxy)
Memory: 7.5M
CGroup: /system.slice/kube-proxy.service
‣ 108006 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=1...
2月 02 00:47:32 localhost.localdomain kube-proxy[108006]: I0202 00:47:32.040427 108006 config...te
2月 02 00:47:32 localhost.localdomain kube-proxy[108006]: I0202 00:47:32.057419 108006 config...te
2月 02 00:47:34 localhost.localdomain kube-proxy[108006]: I0202 00:47:34.059627 108006 config...te
2月 02 00:47:34 localhost.localdomain kube-proxy[108006]: I0202 00:47:34.076914 108006 config...te
2月 02 00:47:36 localhost.localdomain kube-proxy[108006]: I0202 00:47:36.091570 108006 config...te
2月 02 00:47:36 localhost.localdomain kube-proxy[108006]: I0202 00:47:36.105162 108006 config...te
2月 02 00:47:38 localhost.localdomain kube-proxy[108006]: I0202 00:47:38.103518 108006 config...te
2月 02 00:47:38 localhost.localdomain kube-proxy[108006]: I0202 00:47:38.115902 108006 config...te
2月 02 00:47:40 localhost.localdomain kube-proxy[108006]: I0202 00:47:40.113628 108006 config...te
2月 02 00:47:40 localhost.localdomain kube-proxy[108006]: I0202 00:47:40.125818 108006 config...te
Hint: Some lines were ellipsized, use -l to show in full.
（6）node02節點部署
//在node01節點操作
//把現成的/opt/kubernetes目錄複製到其他節點進行修改即可
[root@localhost ~] scp -r /opt/kubernetes/ root@192.168.75.144:/opt/
The authenticity of host '192.168.75.144 (192.168.75.144)' can't be established.
ECDSA key fingerprint is SHA256:HyV9L/xOcN5435t9zCPMCC63XiwMwgBLIa7L++Gea0k.
ECDSA key fingerprint is MD5:4b:01:0f:c3:cb:3e:3a:4c:f0:51:85:fc:c1:6b:c5:fe.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.75.144' (ECDSA) to the list of known hosts.
root@192.168.75.144's password:
flanneld 100% 241 471.1KB/s 00:00
bootstrap.kubeconfig 100% 2169 2.9MB/s 00:00
kube-proxy.kubeconfig 100% 6275 10.3MB/s 00:00
kubelet 100% 379 130.7KB/s 00:00
kubelet.config 100% 269 420.0KB/s 00:00
kubelet.kubeconfig 100% 2298 3.5MB/s 00:00
kube-proxy 100% 191 353.2KB/s 00:00
mk-docker-opts.sh 100% 2139 4.5MB/s 00:00
flanneld 100% 35MB 50.5MB/s 00:00
kubelet 100% 168MB 84.9MB/s 00:01
kube-proxy 100% 48MB 94.7MB/s 00:00
kubelet.crt 100% 2197 902.9KB/s 00:00
kubelet.key 100% 1679 2.3MB/s 00:00
kubelet-client-2020-02-02-00-42-27.pem 100% 1277 493.9KB/s 00:00
kubelet-client-current.pem 100% 1277 429.5KB/s 00:00
//把kubelet，kube-proxy的service檔案拷貝到node2中
[root@localhost ~] scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.75.144:/usr/lib/systemd/system/
root@192.168.75.144's password:
kubelet.service 100% 264 303.9KB/s 00:00
kube-proxy.service 100% 231 209.1KB/s 00:00
//在node02上操作，進行修改
//首先刪除複製過來的證書，等會node02會自行申請證書
[root@localhost ~] cd /opt/kubernetes/ssl/
[root@localhost ssl] rm -rf *
//修改配置檔案kubelet kubelet.config kube-proxy（三個配置檔案）
[root@localhost ssl] cd ../cfg/
[root@localhost cfg] vim kubelet
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.75.144 \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
[root@localhost cfg] vim kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.75.144
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
~
[root@localhost cfg] vim kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.75.144 \
--cluster-cidr=10.0.0.0/24 \
--proxy-mode=ipvs \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
//啟動服務
[root@localhost cfg] systemctl start kubelet.service
[root@localhost cfg] systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@localhost cfg] systemctl start kube-proxy.service
[root@localhost cfg] systemctl enable kube-proxy.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
//在master上操作檢視請求
[root@localhost k8s] kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU 15s kubelet-bootstrap Pending
//授權許可加入群集
[root@localhost k8s] kubectl certificate approve node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU
certificatesigningrequest.certificates.k8s.io/node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU approved
//檢視群集中的節點
[root@localhost k8s] kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.75.201 Ready <none> 21h v1.12.3
192.168.75.144 Ready <none> 37s v1.12.3
- 先具備單master節點部署環境
3.2 master02部署
//優先關閉防火牆和selinux服務
//在master01上操作
//複製kubernetes目錄到master02
[root@localhost k8s] scp -r /opt/kubernetes/ root@192.168.75.121:/opt
The authenticity of host '192.168.75.121 (192.168.75.121)' can't be established.
ECDSA key fingerprint is SHA256:IJ43xXlBWD7qPaL/uFG+4qW4qd7C8xBqUttHiYME8YE.
ECDSA key fingerprint is MD5:cf:3e:dc:e5:89:86:e9:43:38:ee:31:9d:8c:d4:75:9f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.75.121' (ECDSA) to the list of known hosts.
root@192.168.75.121's password:
token.csv 100% 84 86.1KB/s 00:00
kube-apiserver 100% 939 1.2MB/s 00:00
kube-scheduler 100% 94 52.0KB/s 00:00
kube-controller-manager 100% 483 446.5KB/s 00:00
kube-apiserver 100% 184MB 30.6MB/s 00:06
kubectl 100% 55MB 32.1MB/s 00:01
kube-controller-manager 100% 155MB 31.1MB/s 00:05
kube-scheduler 100% 55MB 30.7MB/s 00:01
ca-key.pem 100% 1679 741.3KB/s 00:00
ca.pem 100% 1359 1.5MB/s 00:00
server-key.pem 100% 1675 1.3MB/s 00:00
server.pem 100% 1643 1.6MB/s 00:00
//複製master中的三個元件啟動指令碼kube-apiserver.service kube-controller-manager.service kube-scheduler.service
[root@localhost k8s] scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.75.121:/usr/lib/systemd/system/
root@192.168.75.121's password:
kube-apiserver.service 100% 282 268.1KB/s 00:00
kube-controller-manager.service 100% 317 294.2KB/s 00:00
kube-scheduler.service 100% 281 257.5KB/s 00:00
//master02上操作
//修改配置檔案kube-apiserver中的IP
[root@localhost ~]# cd /opt/kubernetes/cfg/
[root@localhost cfg]# vim kube-apiserver
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.75.149:2379,https://192.168.75.150:2379,https://192.168.75.151:2379 \
--bind-address=192.168.75.121 \
--secure-port=6443 \
--advertise-address=192.168.75.121 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"
//特別注意：master02一定要有etcd證書
//需要拷貝master01上已有的etcd證書給master02使用
[root@localhost k8s]# scp -r /opt/etcd/ root@192.168.75.121:/opt/
root@192.168.75.121's password:
etcd 100% 523 415.0KB/s 00:00
etcd 100% 18MB 42.7MB/s 00:00
etcdctl 100% 15MB 35.2MB/s 00:00
ca-key.pem 100% 1675 612.1KB/s 00:00
ca.pem 100% 1265 1.0MB/s 00:00
server-key.pem 100% 1679 1.7MB/s 00:00
server.pem 100% 1338 1.7MB/s 00:00
//啟動master02中的三個元件服務
[root@localhost cfg] systemctl start kube-apiserver.service
[root@localhost cfg] systemctl start kube-controller-manager.service
[root@localhost cfg] systemctl start kube-scheduler.service
//增加環境變數
[root@localhost cfg] vim /etc/profile
#末尾新增
export PATH=$PATH:/opt/kubernetes/bin/
[root@localhost cfg] source /etc/profile
[root@localhost cfg] kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.75.150 Ready <none> 2d12h v1.12.3
192.168.75.151 Ready <none> 38h v1.12.3
3.3 負載均衡Nginx01和Nginx02配置
//Nginx01和 Nginx02操作
//安裝nginx服務，把nginx.sh和keepalived.conf指令碼拷貝到家目錄
[root@localhost ~] systemctl stop firewalld.service
[root@localhost ~] setenforce 0
[root@localhost ~] vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
[root@localhost ~] yum install nginx -y
//新增四層轉發
[root@localhost ~] vim /etc/nginx/nginx.conf
events {
worker_connections 1024;
}
stream {
log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
server 192.168.75.201:6443;
server 192.168.75.144:6443;
}
server {
listen 6443;
proxy_pass k8s-apiserver;
}
}
http {
[root@localhost ~] systemctl start nginx
//部署keepalived服務
[root@localhost ~] yum install keepalived -y
//修改配置檔案
[root@localhost ~] cp keepalived.conf /etc/keepalived/keepalived.conf
cp：是否覆蓋"/etc/keepalived/keepalived.conf"？ yes
//注意：lb01是Mster配置如下：
! Configuration File for keepalived
global_defs {
# 接收郵件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 郵件傳送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/usr/local/nginx/sbin/check_nginx.sh"
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51 # VRRP 路由 ID例項，每個例項是唯一的
priority 166 # 優先順序，備伺服器設定 90
advert_int 1 # 指定VRRP 心跳包通告間隔時間，預設1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.75.166/24
}
track_script {
check_nginx
}
}//注意：Nginx02是Backup配置如下：
! Configuration File for keepalived
global_defs {
# 接收郵件地址
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 郵件傳送地址
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script "/usr/local/nginx/sbin/check_nginx.sh"
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51 # VRRP 路由 ID例項，每個例項是唯一的
priority 90 # 優先順序，備伺服器設定 90
advert_int 1 # 指定VRRP 心跳包通告間隔時間，預設1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.75.166/24
}
track_script {
check_nginx
}
}
[root@localhost ~] vim /usr/local/nginx/sbin/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi
[root@localhost ~] chmod +x /usr/local/nginx/sbin/check_nginx.sh
[root@localhost ~] systemctl start keepalived
//檢視lb01地址資訊
[root@localhost ~] ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1660
link/ether 00:0c:29:eb:11:2a brd ff:ff:ff:ff:ff:ff
inet 192.168.75.155/24 brd 192.168.75.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.75.166/24 scope global secondary ens33 //漂移地址在lb01中
valid_lft forever preferred_lft forever
inet6 fe80::53ba:daab:3e22:e711/64 scope link
valid_lft forever preferred_lft forever
//檢視Nginx02地址資訊
[root@localhost nginx] ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1660
link/ether 00:0c:29:c9:9d:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.75.177/24 brd 192.168.75.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::55c0:6788:9feb:550d/64 scope link
valid_lft forever preferred_lft forever
//驗證地址漂移（lb01中使用pkill nginx，再在lb02中使用ip a 檢視）
//恢復操作（在lb01中先啟動nginx服務，再啟動keepalived服務）
//nginx站點/usr/share/nginx/html
//開始修改node節點配置檔案統一VIP（bootstrap.kubeconfig,kubelet.kubeconfig）
[root@localhost cfg] vim /opt/kubernetes/cfg/bootstrap.kubeconfig
[root@localhost cfg] vim /opt/kubernetes/cfg/kubelet.kubeconfig
[root@localhost cfg] vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
//統統修改為VIP
server: https://192.168.75.166:6443
[root@localhost cfg] systemctl restart kubelet.service
[root@localhost cfg] systemctl restart kube-proxy.service
//替換完成直接自檢
[root@localhost cfg] grep 166 *
bootstrap.kubeconfig: server: https://192.168.75.166:6443
kubelet.kubeconfig: server: https://192.168.75.166:6443
kube-proxy.kubeconfig: server: https://192.168.75.166:6443
//在lb01上檢視nginx的k8s日誌
[root@localhost ~] tail /var/log/nginx/k8s-access.log
192.168.75.201 192.168.75.200:6443 - [05/Feb/2020:12:43:50 +0800] 200 1121
192.168.75.201 192.168.75.122:6443 - [05/Feb/2020:12:43:50 +0800] 200 1120
192.168.75.144 192.168.75.200:6443 - [05/Feb/2020:12:45:38 +0800] 200 1121
192.168.75.144 192.168.75.122:6443 - [05/Feb/2020:12:45:38 +0800] 200 1121
//在master01上操作
//測試建立pod
[root@localhost ~] kubectl run nginx --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created
//檢視狀態
[root@localhost ~] kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-nf9sk 0/1 ContainerCreating 0 33s //正在建立中
[root@localhost ~] kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-dbddb74b8-nf9sk 1/1 Running 0 80s //建立完成，執行中
//注意日誌問題
[root@localhost ~] kubectl logs nginx-dbddb74b8-nf9sk
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-nf9sk)
[root@localhost ~] kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
clusterrolebinding.rbac.authorization.k8s.io/cluster-system-anonymous created
//檢視pod網路
[root@localhost ~] kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
nginx-dbddb74b8-nf9sk 1/1 Running 0 11m 172.17.31.3 192.168.75.150 <none>
//在對應網段的node節點上操作可以直接訪問
[root@localhost cfg] curl 172.17.31.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
//訪問就會產生日誌
//回到master01操作
[root@localhost ~] kubectl logs nginx-dbddb74b8-nf9sk
172.17.31.1 - - [05/Feb/2020:05:08:36 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
四、故障解決
問題一：
- 故障描述：node1向master請求證書。Master沒有下發證書。
- 故障原因：master在生成bootstrap.kubeconfig 、
kube-proxy.kubeconfig檔案時，指向的證書路徑錯誤，導致生成錯誤檔案然後傳送給node1和node2。從而之後node向master請求證書和master下發證書失敗 - 故障解決：重新生成正確的bootstrap.kubeconfig 、
kube-proxy.kubeconfig檔案下發給node1和node2.
問題二：
- 故障描述：node1和node2 docker0網路卡，無法互通。
- 故障原因：node1和node2無法互通。並且docker 0和flanneld 的地址不相同。
- 故障解決：檢視docker和flanneld的配置檔案,地址是否相符。然後重啟docker。之後再次檢視網路卡資訊。驗證node節點互通。
問題三：
- 故障描述：kubectl get node 發現node1 和node2 切換性is unhealthy.
- 故障原因：虛擬機器掛起服務之後，可能會有一些服務沒有正確啟動狀態。
- 故障解決：
檢查flanneld伺服器之間網路的互通性。如果發現問題重啟docker。
檢查ETCD是否running狀態。如果出現問題嘗試重啟ETCD。
- 故障總結：每次關機或者掛載之後，flanneld、ETCD服務可能出現停止狀態。需要手工再次啟動。或者開啟自啟動。
- 服務啟動
systemctl start etcd
systemctl status etcd
systemctl enable etcd
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
systemctl start kubelet.service
systemctl enable kubelet.service
systemctl start kube-proxy.service
systemctl enable kube-proxy.service
systemctl status kube-proxy.service
kubectl get cs //檢視群集健康狀態
kubectl get csr //檢視證書狀態
kubectl get node //檢視節點
vim /usr/lib/systemd/system/docker.service //docker 配置檔案路徑
- master節點的配置檔案目錄
- Master：192.168.75.200
[root@master kubernetes]# tree
.
├── bin
│?? ├── kube-apiserver
│?? ├── kube-controller-manager
│?? ├── kubectl
│?? └── kube-scheduler
├── cfg
│?? ├── kube-apiserver
│?? ├── kube-controller-manager
│?? ├── kube-scheduler
│?? └── token.csv
└── ssl
├── admin-key.pem
├── admin.pem
├── ca-key.pem
├── ca.pem
├── kube-proxy-key.pem
├── kube-proxy.pem
├── server-key.pem
└── server.pem
- Node2:182.168.75.144
[root@localhost opt]# tree kubernetes/
kubernetes/
├── bin
│ ├── flanneld
│ ├── kubelet
│ ├── kube-proxy
│ └── mk-docker-opts.sh
├── cfg
│ ├── bootstrap.kubeconfig
│ ├── flanneld
│ ├── kubelet
│ ├── kubelet.config
│ ├── kubelet.kubeconfig
│ ├── kube-proxy
│ └── kube-proxy.kubeconfig
└── ssl
├── kubelet-client-2020-09-30-08-48-04.pem
├── kubelet-client-current.pem -> /opt/kubernetes/ssl/kubelet-client-2020-09-30-08-48-04.pem
├── kubelet.crt
└── kubelet.key
- Node 1:192.168.75.201
[root@localhost kubernetes]# tree
.
├── bin
│ ├── flanneld
│ ├── kubelet
│ ├── kube-proxy
│ └── mk-docker-opts.sh
├── cfg
│ ├── bootstrap.kubeconfig
│ ├── flanneld
│ ├── kubelet
│ ├── kubelet.config
│ ├── kubelet.kubeconfig
│ ├── kube-proxy
│ └── kube-proxy.kubeconfig
└── ssl
├── kubelet-client-2020-09-29-19-11-00.pem
├── kubelet-client-current.pem -> /opt/kubernetes/ssl/kubelet-client-2020-09-29-19-11-00.pem
├── kubelet.crt
└── kubelet.key
相關文章
- Kubernetes部署-二進位制方式
- Kubernetes-高可用k8s叢集部署（多Master節點二進位制方式）K8SAST
- 03 . 二進位制部署kubernetes1.18.4
- 二進位制下載部署NginxNginx
- 二進位制部署1.23.4版本k8s叢集-5-部署Master節點服務K8SAST
- 二進位制部署1.23.4版本k8s叢集-6-部署Node節點服務K8S
- 知多一點二進位制中的負數
- k8s二進位制部署K8S
- 二進位制部署 Prometheus+Alertmanager+GrafanaPrometheusGrafana
- 二進位制與二進位制運算
- 進位制詳解：二進位制、八進位制和十六進位制
- JavaScript 二進位制、八進位制與十六進位制JavaScript
- （二進位制）
- 二進位制
- 十進位制——二 (八、十六 )進位制
- 二進位制，八進位制，十進位制，十六進位制的相互轉換
- 【進位制轉換】二進位制、十六進位制、十進位制、八進位制對應關係
- 二進位制、十進位制與十六進位制相互轉化
- java中二進位制、八進位制、十進位制、十六進位制的轉換Java
- 二進位制，八進位制，十進位制，十六進位制之間的轉換
- Python 進位制互相轉換（二進位制、十進位制和十六進位制）Python
- 計算機基礎進位制轉換（二進位制、八進位制、十進位制、十六進位制）計算機
- 二進位制轉十進位制快速方法
- JAVA 二進位制，八進位制，十六進位制，十進位制間進行相互轉換Java
- 什麼是二進位制？二進位制如何轉換？
- JavaScript二進位制陣列建立注意點JavaScript陣列
- 有趣的二進位制3—浮點數
- 04 二進位制
- 【原】二進位制部署 k8s 1.18.3K8S
- 大話二進位制，八進位制，十進位制，十六進位制之間的轉換
- JavaScript十進位制轉換為二進位制JavaScript
- Oracle二進位制與十進位制轉換Oracle
- 十進位制轉二進位制推導（草稿）
- [計算機基礎] 計算機進位制轉換：二進位制、八進位制、十進位制、十六進位制計算機
- 一看就懂二進位制、八進位制、十六進位制數轉換十進位制
- python進位制轉換（二進位制、十進位制和十六進位制）及注意事項Python
- Oracle中的二進位制、八進位制、十進位制、十六進位制相互轉換函式Oracle函式
- 進位制之間的轉換之“十六進位制 轉 十進位制 轉 二進位制 方案”