分散式kv儲存系統之Etcd叢集

1874發表於2021-01-30

  etcd是什麼?

  etcd是一個高可用的分散式鍵值資料庫,可用於服務發現,etcd採用 raft 一致性演算法,基於 Go 語言實現。其特點有簡單易用,所謂簡單易用是指安裝配置簡單,提供http/https介面;安全,安全是指etcd支援ssl證照認證,支援叢集各節點間使用對等證照認證;客戶端和服務端的雙向證照認證;可靠,可靠是指etcd使用raft協議實現分散式系統資料的可用性和一致性;etcd主要有兩個版本v2和v3;v2和v3的api是互不相容的,所以我們在同一伺服器上安裝多個版本的etcd時,我們需要用ETCDCTL_API這個環境變數指定;

  etcd叢集部署

  環境準備

主機名稱 ip地址
master01.k8s.org 192.168.0.41
master02.k8s.org 192.168.0.42
master03.k8s.org 192.168.0.43

 

 

 

 

 

  各主機hosts檔案解析

[root@master01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.99 time.test.org time-node
192.168.0.41  master01 master01.k8s.org etcd01 etcd01.k8s.org 
192.168.0.42  master02 master02.k8s.org etcd02 etcd02.k8s.org
192.168.0.43  master03 master03.k8s.org etcd03 etcd03.k8s.org
192.168.0.44  node01 node01.k8s.org
192.168.0.45  node02 node02.k8s.org
192.168.0.46  node03 node03.k8s.org
[root@master01 ~]# 

  關閉各主機的firewalld服務

[root@master01 ~]# systemctl stop firewalld
[root@master01 ~]# systemctl disable firewalld
[root@master01 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@master01 ~]# 

  各主機間時間同步

[root@master01 ~]# grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server time.test.org iburst
# Serve time even if not synchronized to any NTP server.
[root@master01 ~]# systemctl restart chronyd.service 
[root@master01 ~]# systemctl status chronyd.service
● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-01-30 15:41:25 CST; 11s ago
     Docs: man:chronyd(8)
           man:chrony.conf(5)
  Process: 1411 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
  Process: 1407 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1409 (chronyd)
   CGroup: /system.slice/chronyd.service
           └─1409 /usr/sbin/chronyd

Jan 30 15:41:25 master01.k8s.org systemd[1]: Stopped NTP client/server.
Jan 30 15:41:25 master01.k8s.org systemd[1]: Starting NTP client/server...
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SI...+DEBUG)
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: commandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: generatecommandkey directive is no longer supported
Jan 30 15:41:25 master01.k8s.org chronyd[1409]: Frequency -25.600 +/- 2.450 ppm read from /var/lib/chrony/drift
Jan 30 15:41:25 master01.k8s.org systemd[1]: Started NTP client/server.
Jan 30 15:41:29 master01.k8s.org chronyd[1409]: Selected source 192.168.0.99
Hint: Some lines were ellipsized, use -l to show in full.
[root@master01 ~]# 

  提示:叢集內部可以使用自己搭建的時間服務,把chrony.conf中的server 指向對應時間伺服器,然後重啟chronyd即可;當然也可以使用網際網路上公有的時間伺服器;總之一個服務以叢集方式工作,其時間同步是非常重要;

  各主機間ssh 互信

[root@master01 ~]# ssh master02
Last login: Sat Jan 30 15:34:33 2021 from master01
[root@master02 ~]# exit
logout
Connection to master02 closed.
[root@master01 ~]# ssh master03
Last login: Sat Jan 30 15:34:37 2021 from master01
[root@master03 ~]# exit
logout
Connection to master03 closed.
[root@master01 ~]# 

  提示:有關ssh互信的配置請參考本人部落格:https://www.cnblogs.com/qiuhom-1874/p/11783371.html;各主機間實現ssh互信,其主要目的是方便各元件同步檔案;做好以上準備以後,我們就可以下載etcd二進位制包進行etcd叢集部署;這裡需要說明一下,在centos7上的extras倉庫中有etcd的rpm包,我們可以使用yum來安裝;但是extras倉庫中的版本不是最新的,要想使用最新的就需要到官方github倉庫中下載最新版本的etcd二進位制包進行部署;兩種部署方式沒有什麼特別的不同;如果對版本要求不是特別新的環境中,建議使用yum安裝;

  下載etcd二進位制包

[root@master01 ~]#wget https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz
--2021-01-30 15:46:18--  https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz
Resolving github.com (github.com)... 52.192.72.89
Connecting to github.com (github.com)|52.192.72.89|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-01-30 15:46:19--  https://github-releases.githubusercontent.com/11225014/ad6a1d80-2f1a-11eb-8cb8-2f1ae35d5487?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210130%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210130T074619Z&X-Amz-Expires=300&X-Amz-Signature=47569782ddb8a1f70fbd28350433d3a045d22f040dd95b7de1055c96e7b4c359&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=11225014&response-content-disposition=attachment%3B%20filename%3Detcd-v3.4.14-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.111.154, 185.199.109.154, 185.199.108.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.111.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17373058 (17M) [application/octet-stream]
Saving to: ‘etcd-v3.4.14-linux-amd64.tar.gz’

100%[=================================================================================================>] 17,373,058  24.9MB/s   in 0.7s   

2021-01-30 15:46:20 (24.9 MB/s) - ‘etcd-v3.4.14-linux-amd64.tar.gz’ saved [17373058/17373058]
[root@master01 ~]#

  解壓etcd二進位制包

[root@master01 ~]# ls
etcd-v3.4.14-linux-amd64.tar.gz
[root@master01 ~]# tar xf etcd-v3.4.14-linux-amd64.tar.gz  -C /usr/local/src/
[root@master01 ~]# cd /usr/local/src/
[root@master01 src]# ls
etcd-v3.4.14-linux-amd64
[root@master01 src]# cd etcd-v3.4.14-linux-amd64/
[root@master01 etcd-v3.4.14-linux-amd64]# ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md
[root@master01 etcd-v3.4.14-linux-amd64]# 

  把etcd和etcdctl軟連線至path環境變數下

[root@master01 etcd-v3.4.14-linux-amd64]# ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md
[root@master01 etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcd /usr/bin/
[root@master01 etcd-v3.4.14-linux-amd64]# ln -s /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl /usr/bin/
[root@master01 etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcd
lrwxrwxrwx 1 root root 44 Jan 30 15:59 /usr/bin/etcd -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcd
[root@master01 etcd-v3.4.14-linux-amd64]# ll /usr/bin/etcdctl 
lrwxrwxrwx 1 root root 47 Jan 30 15:59 /usr/bin/etcdctl -> /usr/local/src/etcd-v3.4.14-linux-amd64/etcdctl
[root@master01 etcd-v3.4.14-linux-amd64]# 

  編寫etcd.service unit檔案

[root@master01 ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd"
Type=notify

[Install]
WantedBy=multi-user.target
[root@master01 ~]# 

  提供etcd環境變數載入檔案/etc/etcd/etcd.conf檔案

[root@master01 ~]# mkdir /etc/etcd/
[root@master01 ~]# cd /etc/etcd/
[root@master01 etcd]# vim etcd.conf
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
#ETCD_LISTEN_PEER_URLS="http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="default"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
#ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_INITIAL_CLUSTER="default=http://localhost:2380"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
#ETCD_CERT_FILE=""
#ETCD_KEY_FILE=""
#ETCD_CLIENT_CERT_AUTH="false"
#ETCD_TRUSTED_CA_FILE=""
#ETCD_AUTO_TLS="false"
#ETCD_PEER_CERT_FILE=""
#ETCD_PEER_KEY_FILE=""
#ETCD_PEER_CLIENT_CERT_AUTH="false"
#ETCD_PEER_TRUSTED_CA_FILE=""
#ETCD_PEER_AUTO_TLS="false"
#
#[Logging]
#ETCD_DEBUG="false"
#ETCD_LOG_PACKAGE_LEVELS=""
#ETCD_LOG_OUTPUT="default"
#
#[Unsafe]
#ETCD_FORCE_NEW_CLUSTER="false"
#
#[Version]
#ETCD_VERSION="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#
#[Profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[Auth]
#ETCD_AUTH_TOKEN="simple"
"etcd.conf" [New] 69L, 1686C written                                                                                     
[root@master01 etcd]# 

  更改etcd.conf檔案

  提示:ETCD_DATA_DIR用於指定etcd資料目錄;ETCD_LISTEN_PEER_URLS用於指定叢集節點通訊監聽的url地址;ETCD_LISTEN_CLIENT_URLS使用者指定客戶端連線使用時的url地址;ETCD_NAME使用者指定當前節點etcd例項的名稱;ETCD_INITIAL_ADVERTISE_PEER_URLS用於指定叢集事務通告的url地址;ETCD_ADVERTISE_CLIENT_URLS使用者指定客戶端事務通告url地址;ETCD_INITIAL_CLUSTER使用者指定叢集成員,一個成員由成員名稱=對應叢集間通訊的url地址,多個成員用逗號隔開;

  建立/var/lib/etcd目錄

[root@master01 etcd]# mkdir /var/lib/etcd/
[root@master01 etcd]# ll -d /var/lib/etcd/
drwxr-xr-x 2 root root 6 Jan 30 16:20 /var/lib/etcd/
[root@master01 etcd]# 

  複製master01上的/usr/bin/etcd和etcdctl二進位制檔案到master02和master03的/usr/bin/目錄下

[root@master01 etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master02:/usr/bin/
etcd                                                                                                     100%   23MB  43.6MB/s   00:00    
etcdctl                                                                                                  100%   17MB  49.1MB/s   00:00    
[root@master01 etcd]# scp /usr/bin/etcd /usr/bin/etcdctl master03:/usr/bin/
etcd                                                                                                     100%   23MB  42.2MB/s   00:00    
etcdctl                                                                                                  100%   17MB  56.8MB/s   00:00    
[root@master01 etcd]# 

  複製master01上的etcd.service到master02和master03的/usr/lib/systemd/system目錄下

[root@master01 etcd]# scp /usr/lib/systemd/system/etcd.service master02:/usr/lib/systemd/system/
etcd.service                                                                                             100%  417   165.1KB/s   00:00    
[root@master01 etcd]# scp /usr/lib/systemd/system/etcd.service master03:/usr/lib/systemd/system/
etcd.service                                                                                             100%  417   175.7KB/s   00:00    
[root@master01 etcd]# 

  在master02和master03上建立/etc/etcd/目錄和/var/lib/etcd/目錄

[root@master01 etcd]# ssh master02 'mkdir /etc/etcd/ && mkdir /var/lib/etcd'
[root@master01 etcd]# ssh master03 'mkdir /etc/etcd/ && mkdir /var/lib/etcd' 
[root@master01 etcd]# 

  複製master01上的etcd.conf檔案到master02和master03的/etc/etcd/目錄下

[root@master01 etcd]# scp /etc/etcd/etcd.conf master02:/etc/etcd/
etcd.conf                                                                                                100% 1749   743.3KB/s   00:00    
[root@master01 etcd]# scp /etc/etcd/etcd.conf master03:/etc/etcd/
etcd.conf                                                                                                100% 1749   824.2KB/s   00:00    
[root@master01 etcd]# 

  在master01/02/03上建立etcd使用者

[root@master01 etcd]# useradd -r etcd -s /sbin/nologin
[root@master01 etcd]# getent passwd etcd
etcd:x:997:995::/home/etcd:/sbin/nologin
[root@master01 etcd]# ssh master02 'useradd -r etcd -s /sbin/nologin && getent passwd etcd'
etcd:x:997:995::/home/etcd:/sbin/nologin
[root@master01 etcd]# ssh master03 'useradd -r etcd -s /sbin/nologin && getent passwd etcd' 
etcd:x:997:995::/home/etcd:/sbin/nologin
[root@master01 etcd]# 

  更改master01/02/03上的/var/lib/etcd目錄的屬主和屬組為etcd使用者

[root@master01 etcd]# chown etcd.etcd /var/lib/etcd/
[root@master01 etcd]# ll -d /var/lib/etcd/
drwxr-xr-x 2 etcd etcd 6 Jan 30 17:36 /var/lib/etcd/
[root@master01 etcd]# ssh master02 ''chown etcd.etcd /var/lib/etcd/ && ls -ld /var/lib/etcd/ 
drwxr-xr-x 2 etcd etcd 6 Jan 30 17:36 /var/lib/etcd/
[root@master01 etcd]# ssh master03 ''chown etcd.etcd /var/lib/etcd/ && ls -ld /var/lib/etcd/ 
drwxr-xr-x 2 etcd etcd 6 Jan 30 17:36 /var/lib/etcd/
[root@master01 etcd]# 

  修改master02上的/etc/etcd/etcd.conf檔案

  修改master03上的/etc/etcd/etcd.conf檔案

  到此三個節點的配置檔案和相關使用者以及目錄都準備就緒,接下來我們要過載systemd的配置檔案,載入etcd.service檔案

[root@master01 ~]# systemctl daemon-reload
[root@master01 ~]# ssh master02 'systemctl daemon-reload'
[root@master01 ~]# ssh master03 'systemctl daemon-reload' 
[root@master01 ~]# 

  啟動etcd

[root@master01 ~]# systemctl start etcd
[root@master01 ~]# 

  提示:在每個節點上執行上述命令啟動etcd;第一個啟動的節點將阻塞,原因是etcd以叢集方式工作,它必須要有足夠的得票才能正常工作,如果叢集節點有3個,那麼至少有兩個節點正常啟動etcd才能正常工作;

  驗證:檢視各節點的2379和2380埠是否都處於監聽?

[root@master01 ~]# ss -tnl
State      Recv-Q Send-Q                                Local Address:Port                                               Peer Address:Port              
LISTEN     0      128                                    192.168.0.41:2379                                                          *:*                  
LISTEN     0      128                                    192.168.0.41:2380                                                          *:*                  
LISTEN     0      128                                               *:22                                                            *:*                  
LISTEN     0      100                                       127.0.0.1:25                                                            *:*                  
LISTEN     0      128                                              :::22                                                           :::*                  
LISTEN     0      100                                             ::1:25                                                           :::*                  
[root@master01 ~]# ssh master02 'ss -tnl'
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.42:2379                     *:*                  
LISTEN     0      128    192.168.0.42:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[root@master01 ~]# ssh master03 'ss -tnl' 
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.43:2379                     *:*                  
LISTEN     0      128    192.168.0.43:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[root@master01 ~]# 

  驗證:使用etcdctl檢視叢集狀態

[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  endpoint status
192.168.0.41:2379, b8b747c74aaea686, 3.4.14, 4.5 MB, true, false, 13, 2163, 2163, 
192.168.0.42:2379, b3504381e8ba3cb, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163, 
192.168.0.43:2379, f572fdfc5cb68406, 3.4.14, 4.5 MB, false, false, 13, 2163, 2163, 
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  member list
b3504381e8ba3cb, started, etcd02, http://etcd02:2380, http://etcd02:2379, false
b8b747c74aaea686, started, etcd01, http://etcd01:2380, http://etcd01:2379, false
f572fdfc5cb68406, started, etcd03, http://etcd03:2380, http://etcd03:2379, false
[root@master01 ~]# 

  提示:能夠列出叢集成員和檢視叢集成員狀態,就表示etcd叢集工作已經正常;

  驗證:向etcd任意節點寫資料,看看是否能夠正常寫?

[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  put name "test"
OK
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  get name 
name
test
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  del name    
1
[root@master01 ~]# etcdctl --endpoints=192.168.0.41:2379,192.168.0.42:2379,192.168.0.43:2379  get name 
[root@master01 ~]# 

  提示:使用etcdctl工具可以正常向etcd叢集寫入資料;

  為etcd叢集生成證照

  在某一節點上安裝git工具

[root@master01 ~]# yum install git -y 

  克隆生成證照的指令碼工具

[root@master01 ~]# git clone https://github.com/iKubernetes/k8s-certs-generator.git
Cloning into 'k8s-certs-generator'...
remote: Enumerating objects: 58, done.
remote: Total 58 (delta 0), reused 0 (delta 0), pack-reused 58
Unpacking objects: 100% (58/58), done.
[root@master01 ~]# ls
etcd-v3.4.14-linux-amd64.tar.gz  k8s-certs-generator
[root@master01 ~]# cd k8s-certs-generator/
[root@master01 k8s-certs-generator]# ls
etcd-certs-gen.sh  gencerts.sh  k8s-certs-gen.sh  openssl.conf  README.md
[root@master01 k8s-certs-generator]# 

  使用gencerts.sh指令碼生成etcd所需證照

[root@master01 k8s-certs-generator]# sh gencerts.sh -h
Usage: ./gencerts.sh etcd|k8s
[root@master01 k8s-certs-generator]# sh gencerts.sh etcd
Enter Domain Name [ilinux.io]: k8s.org
Generating RSA private key, 4096 bit long modulus
.......++
.................................................................................................................................................................................................................................................................++
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
.................................................+++
.........................+++
e is 65537 (0x10001)
Generating etcd/pki/peer.csr
Generating RSA private key, 2048 bit long modulus
...........................................................................................................................................+++
...............+++
e is 65537 (0x10001)
Generating etcd/pki/server.csr
Generating RSA private key, 2048 bit long modulus
..............................................................+++
............................+++
e is 65537 (0x10001)
Generating etcd/pki/apiserver-etcd-client.csr
Generating RSA private key, 2048 bit long modulus
............+++
.................................+++
e is 65537 (0x10001)
Generating etcd/pki/client.csr
Generating etcd/pki/peer.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Jan 30 10:46:52 2021 GMT
            Not After : Jan 28 10:46:52 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Server
            Netscape Comment: 
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier: 
                FC:BA:D7:73:4E:C7:1D:9D:73:12:E3:60:96:5B:69:58:CE:4F:14:FD
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
                DirName:/CN=etcd-ca
                serial:BE:88:C0:B5:81:5D:6D:D6

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:52 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/server.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4097 (0x1001)
        Validity
            Not Before: Jan 30 10:46:53 2021 GMT
            Not After : Jan 28 10:46:53 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Server
            Netscape Comment: 
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier: 
                1C:BE:22:C0:B7:5F:03:39:5C:E0:FC:47:88:8D:3A:FC:27:FA:0E:BC
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE
                DirName:/CN=etcd-ca
                serial:BE:88:C0:B5:81:5D:6D:D6

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Subject Alternative Name: 
                DNS:*.k8s.org
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/apiserver-etcd-client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4098 (0x1002)
        Validity
            Not Before: Jan 30 10:46:53 2021 GMT
            Not After : Jan 28 10:46:53 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Client
            Netscape Comment: 
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier: 
                FD:52:EA:9F:84:72:35:46:9A:33:71:DE:D0:41:E6:8D:89:C0:62:AE
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
Generating etcd/pki/client.crt
Using configuration from openssl.conf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4099 (0x1003)
        Validity
            Not Before: Jan 30 10:46:53 2021 GMT
            Not After : Jan 28 10:46:53 2031 GMT
        Subject:
            commonName                = etcd
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            Netscape Cert Type: 
                SSL Client
            Netscape Comment: 
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier: 
                6B:31:50:84:00:9E:0F:6E:B8:56:7A:C1:57:82:F4:BB:12:57:52:B2
            X509v3 Authority Key Identifier: 
                keyid:9C:C0:85:32:DE:F7:78:C0:90:D5:E1:20:F9:14:A7:1A:F4:5B:C5:BE

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication
Certificate is to be certified until Jan 28 10:46:53 2031 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[root@master01 k8s-certs-generator]# ls
etcd  etcd-certs-gen.sh  gencerts.sh  k8s-certs-gen.sh  openssl.conf  README.md
[root@master01 k8s-certs-generator]# ls etcd
patches  pki
[root@master01 k8s-certs-generator]# ls etcd/pki/
apiserver-etcd-client.crt  apiserver-etcd-client.key  ca.crt  ca.key  client.crt  client.key  peer.crt  peer.key  server.crt  server.key
[root@master01 k8s-certs-generator]# 

  提示:server.crt和server.key用於etcd服務端的證照和金鑰;peer.crt和peer.key用於叢集內部各節間認證所需證照和金鑰;client.crt和client.key用於客戶端連線服務端所需的證照和金鑰;ca.crt和ca.key是用於叢集內部做認證和客戶端連線服務端所信任的ca的證照和金鑰;

  複製證照檔案到其他節點

[root@master01 k8s-certs-generator]# cp -a etcd/pki/ /etc/etcd/
[root@master01 k8s-certs-generator]# cd /etc/etcd/
[root@master01 etcd]# ls
etcd.conf  pki
[root@master01 etcd]# scp -r pki/ master02:/etc/etcd/
ca.key                                                                                                                  100% 3247     1.8MB/s   00:00    
ca.crt                                                                                                                  100% 1814     1.2MB/s   00:00    
peer.key                                                                                                                100% 1679     1.1MB/s   00:00    
server.key                                                                                                              100% 1679     1.2MB/s   00:00    
apiserver-etcd-client.key                                                                                               100% 1675     1.3MB/s   00:00    
client.key                                                                                                              100% 1675     1.1MB/s   00:00    
peer.crt                                                                                                                100% 1659    75.0KB/s   00:00    
server.crt                                                                                                              100% 1647   917.8KB/s   00:00    
apiserver-etcd-client.crt                                                                                               100% 1570     1.2MB/s   00:00    
client.crt                                                                                                              100% 1570   902.2KB/s   00:00    
[root@master01 etcd]# scp -r pki/ master03:/etc/etcd/
ca.key                                                                                                                  100% 3247     1.1MB/s   00:00    
ca.crt                                                                                                                  100% 1814   695.0KB/s   00:00    
peer.key                                                                                                                100% 1679   621.6KB/s   00:00    
server.key                                                                                                              100% 1679   657.1KB/s   00:00    
apiserver-etcd-client.key                                                                                               100% 1675   950.4KB/s   00:00    
client.key                                                                                                              100% 1675     1.0MB/s   00:00    
peer.crt                                                                                                                100% 1659   916.3KB/s   00:00    
server.crt                                                                                                              100% 1647     1.0MB/s   00:00    
apiserver-etcd-client.crt                                                                                               100% 1570   850.8KB/s   00:00    
client.crt                                                                                                              100% 1570   872.7KB/s   00:00    
[root@master01 etcd]# 

  配置etcd基於https協議提供服務

  配置master01上的etcd啟用證照認證

[root@master01 etcd]# cat etcd.conf 
#[Member]
#ETCD_CORS=""
ETCD_DATA_DIR="/var/lib/etcd/cluster.etcd"
#ETCD_WAL_DIR=""
ETCD_LISTEN_PEER_URLS="http://192.168.0.41:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.41:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
ETCD_NAME="etcd01"
#ETCD_SNAPSHOT_COUNT="100000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_QUOTA_BACKEND_BYTES="0"
#ETCD_MAX_REQUEST_BYTES="1572864"
#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"
#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"
#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"
#
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd01:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://etcd01:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_DISCOVERY_SRV=""
ETCD_INITIAL_CLUSTER="etcd01=http://etcd01:2380,etcd02=http://etcd02:2380,etcd03=http://etcd03:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_STRICT_RECONFIG_CHECK="true"
#ETCD_ENABLE_V2="true"
#
#[Proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[Security]
ETCD_CERT_FILE="/etc/etcd/pki/server.crt"
ETCD_KEY_FILE="/etc/etcd/pki/server.key"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt"
ETCD_AUTO_TLS="false"
ETCD_PEER_CERT_FILE="/etc/etcd/pki/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/pki/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/pki/ca.crt"
ETCD_PEER_AUTO_TLS="false"
#
#[Logging]
#ETCD_DEBUG="false"
#ETCD_LOG_PACKAGE_LEVELS=""
#ETCD_LOG_OUTPUT="default"
#
#[Unsafe]
#ETCD_FORCE_NEW_CLUSTER="false"
#
#[Version]
#ETCD_VERSION="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
#
#[Profiling]
#ETCD_ENABLE_PPROF="false"
#ETCD_METRICS="basic"
#
#[Auth]
#ETCD_AUTH_TOKEN="simple"
[root@master01 etcd]# 

  提示:ETCD_CERT_FILE用於指定etcd服務端證照檔案路徑;ETCD_KEY_FILE使用者指定服務端證照檔案所對應的金鑰檔案路徑;ETCD_CLIENT_CERT_AUTH使用者指定是否啟用客戶端證照認證;ETCD_TRUSTED_CA_FILE使用者指定客戶端認證信任的ca證照檔案;

ETCD_AUTO_TLS用於指定是否自動生成證照檔案;ETCD_PEER_CERT_FILE用於指定叢集間對等證照檔案路徑;ETCD_PEER_KEY_FILE用於指定叢集間對等證照對應的金鑰檔案;ETCD_PEER_CLIENT_CERT_AUTH用於指定是否啟用對等證照認證;ETCD_PEER_TRUSTED_CA_FILE用於指定對等證照認證所信賴的ca證照;ETCD_PEER_AUTO_TLS用於指定是否自動生成對等證照;

  修改/etc/etcd/etcd.conf檔案,將etcd01修改成etcd01.k8s.org,將etcd02修改成etcd02.k8s.org,將etcd03修改成etcd03.k8s.org,將http修改成https

  停止etcd服務,刪除/var/lib/etcd/目錄下的所有檔案

[root@master01 etcd]# systemctl stop etcd
[root@master01 etcd]# rm -rf /var/lib/etcd/*
[root@master01 etcd]# ll /var/lib/etcd/
total 0
[root@master01 etcd]# 

  配置master02啟用證照認證,並將對應http修改成很https,把對應短格式名稱修改為類似etcd01.k8s.org名稱

  停止etcd服務,刪除/var/lib/etcd/目錄下的所有檔案

[root@master02 ~]# systemctl stop etcd
[root@master02 ~]# rm -rf /var/lib/etcd/*
[root@master02 ~]# ll /var/lib/etcd/
total 0
[root@master02 ~]# 

  配置master03啟用證照認證,並將對應http修改成很https,把對應短格式名稱修改為長格式名稱

  停止etcd服務,刪除/var/lib/etcd下的所有檔案

[root@master03 ~]# systemctl stop etcd
[root@master03 ~]# rm -rf /var/lib/etcd/*
[root@master03 ~]# ll /var/lib/etcd/
total 0
[root@master03 ~]# 

  啟動各節點上的etcd

[root@master01 etcd]# systemctl start etcd
[root@master01 etcd]# 

  提示:如果三個節點上的etcd都能正常啟動,說明我們配置檔案沒有問題;

  驗證:檢視所有節點的etcd服務是否都正常啟動,並監聽對應的的埠?

[root@master01 etcd]# ss -tnl
State      Recv-Q Send-Q                                Local Address:Port                                               Peer Address:Port              
LISTEN     0      128                                    192.168.0.41:2379                                                          *:*                  
LISTEN     0      128                                    192.168.0.41:2380                                                          *:*                  
LISTEN     0      128                                               *:22                                                            *:*                  
LISTEN     0      100                                       127.0.0.1:25                                                            *:*                  
LISTEN     0      128                                              :::22                                                           :::*                  
LISTEN     0      100                                             ::1:25                                                           :::*                  
[root@master01 etcd]# ssh master02 'ss -tnl'
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.42:2379                     *:*                  
LISTEN     0      128    192.168.0.42:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[root@master01 etcd]# ssh master03 'ss -tnl' 
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    192.168.0.43:2379                     *:*                  
LISTEN     0      128    192.168.0.43:2380                     *:*                  
LISTEN     0      128          *:22                       *:*                  
LISTEN     0      100    127.0.0.1:25                       *:*                  
LISTEN     0      128         :::22                      :::*                  
LISTEN     0      100        ::1:25                      :::*                  
[root@master01 etcd]# 

  驗證:檢視叢集成員

[root@master01 etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379,https://etcd02.k8s.org:2379,https://etcd03.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" endpoint status
https://etcd01.k8s.org:2379, 61d91b7ed8f88f32, 3.4.14, 20 kB, true, false, 6, 9, 9, 
https://etcd02.k8s.org:2379, ef13441fdfe8af38, 3.4.14, 20 kB, false, false, 6, 9, 9, 
https://etcd03.k8s.org:2379, f11ed09b6567910f, 3.4.14, 20 kB, false, false, 6, 9, 9, 
[root@master01 etcd]# etcdctl --endpoints="https://etcd01.k8s.org:2379" --cacert="/etc/etcd/pki/ca.crt" --cert="/etc/etcd/pki/client.crt" --key="/etc/etcd/pki/client.key" member list
61d91b7ed8f88f32, started, etcd01.k8s.org, https://etcd01.k8s.org:2380, https://etcd01.k8s.org:2379, false
ef13441fdfe8af38, started, etcd02.k8s.org, https://etcd02.k8s.org:2380, https://etcd02.k8s.org:2379, false
f11ed09b6567910f, started, etcd03.k8s.org, https://etcd03.k8s.org:2380, https://etcd03.k8s.org:2379, false
[root@master01 etcd]# 

  提示:現在etcd啟用了ssl認證功能,客戶端訪問必須攜帶對應的客戶端證照和私鑰檔案以及對應認證所信任的ca證照,才可以正常訪問到etcd叢集;這裡需要注意指定的endpoints需要使用域名格式給出,給定ip地址是無法正常通過認證的;

相關文章