kafka
1. zookeeper叢集搭建
1.1 作用
什麼是zookeeper
zookeeper致力於維護開源伺服器，實現高度可靠的分散式協調
zookeeper是一個用於維護配置資訊，命名，提供分散式服務和提供組服務的集中式服務
說白了：zookeeper的作用就是為分散式叢集各節點提供資料共享的功能
1.2 應用場景
- kafka
- dubbo
- hdfs
一切需要用到分散式服務的中介軟體都需要使用zookeeper
zookeeper叢集推薦三臺為一個叢集
1.3 zookeeper單節點部署
(1) 上傳軟體包到伺服器並解壓
tar xf apache-zookeeper-3.8.4-bin.tar.gz -C /oldboy/softwares/ [root@zookeeper1 /softwares]#cd /oldboy/softwares/ [root@zookeeper1 /oldboy/softwares]#ll total 0 drwxr-xr-x 6 root root 133 Oct 14 23:08 apache-zookeeper-3.8.4-bin(2) 建立環境變數：將bin下的檔案配置即可
1. 建立軟連結 [root@zookeeper1 /oldboy/softwares]#ln -s apache-zookeeper-3.8.4-bin/ zk [root@zookeeper1 /oldboy/softwares]#ll total 0 drwxr-xr-x 6 root root 133 Oct 14 23:08 apache-zookeeper-3.8.4-bin lrwxrwxrwx 1 root root 27 Oct 14 23:12 zk -> apache-zookeeper-3.8.4-bin/ 2. 配置環境變數 [root@zookeeper1 /oldboy/softwares]#vim /etc/profile.d/zk.sh [root@zookeeper1 /oldboy/softwares]#cat /etc/profile.d/zk.sh #!/bin/bash export ZK_HOME=/oldboy/softwares/zk export PATH=$PATH:$ZK_HOME/bin source /etc/profile.d/zk.sh(3) 生成配置檔案
1. 複製出來一個zoo.cfg的配置檔案 [root@zookeeper1 /oldboy/softwares]#cp zk/conf/{zoo_sample,zoo}.cfg [root@zookeeper1 /oldboy/softwares]#ll total 0 drwxr-xr-x 6 root root 133 Oct 14 23:08 apache-zookeeper-3.8.4-bin lrwxrwxrwx 1 root root 27 Oct 14 23:12 zk -> apache-zookeeper-3.8.4-bin/ [root@zookeeper1 /oldboy/softwares]#ll zk/conf/ total 20 -rw-r--r-- 1 oldboy oldboy 535 Feb 13 2024 configuration.xsl -rw-r--r-- 1 oldboy oldboy 4559 Feb 13 2024 logback.xml -rw-r--r-- 1 root root 1183 Oct 14 23:21 zoo.cfg -rw-r--r-- 1 oldboy oldboy 1183 Feb 13 2024 zoo_sample.cfg(4) 啟動服務
[root@zookeeper1 /oldboy/softwares]#zkServer.sh start ZooKeeper JMX enabled by default Using config: /oldboy/softwares/zk/bin/../conf/zoo.cfg Starting zookeeper ... STARTED # 注意 需要安裝jdk(5) 檢視服務狀態
[root@zookeeper1 /oldboy/softwares]#zkServer.sh status ZooKeeper JMX enabled by default Using config: /oldboy/softwares/zk/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: standalone standalone : 單點(6) 驗證
[root@zookeeper1 /oldboy/softwares]# ss -ntl |grep 2181 LISTEN 0 50 [::]:2181 [::]:*(7) 連線測試
[root@zookeeper1 /oldboy/softwares]#zkCli.sh Connecting to localhost:2181 ...... [zk: localhost:2181(CONNECTED) 0] ls / [zookeeper](8) 停止服務
[root@zookeeper1 /oldboy/softwares]#zkServer.sh stop ZooKeeper JMX enabled by default Using config: /oldboy/softwares/zk/bin/../conf/zoo.cfg Stopping zookeeper ... STOPPED
1.4 zookeeper叢集部署
(1) 編寫配置檔案
[root@zookeeper1 /oldboy/softwares]#vim zk/conf/zoo.cfg [root@zookeeper1 /oldboy/softwares]#cat zk/conf/zoo.cfg # 定義最小單元的時間範圍tick。 tickTime=2000 # 啟動時最長等待tick數量。 initLimit=5 # 資料同步時最長等待的tick時間進行響應ACK syncLimit=2 # 指定資料目錄 dataDir=/oldboyedu/data/zk # 監聽埠 clientPort=2181 # 開啟四字命令允許所有的節點訪問。 4lw.commands.whitelist=* # server.ID=A:B:C[:D] # ID: # zk的唯一編號。 # A: # zk的主機地址。 # B: # leader的選舉埠，是誰leader角色，就會監聽該埠。 # C: # 資料通訊埠。 # D: # 可選配置，指定角色。 server.201=10.0.0.201:5888:6888 server.202=10.0.0.202:5888:6888 server.203=10.0.0.203:5888:6888 ## Metrics Providers # # https://prometheus.io Metrics Exporter #metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider #metricsProvider.httpHost=0.0.0.0 #metricsProvider.httpPort=7000 #metricsProvider.exportJvmInfo=true(2) 同步配置檔案到其他節點
(3) 建立資料目錄
mkdir -p /oldboyedu/data/zk(4) 建立myid檔案
for ((host_id=201;host_id<=203;host_id++)) do ssh 10.0.0.${host_id} "echo ${host_id} > /oldboyedu/data/zk/myid";done # 驗證 [root@zookeeper1 ~]#cat /oldboyedu/data/zk/myid 201 [root@zookeeper2 ~]#cat /oldboyedu/data/zk/myid 202 [root@zookeeper3 ~]#cat /oldboyedu/data/zk/myid 203(5) 啟動叢集
[root@zookeeper1 ~]#zkServer.sh start ZooKeeper JMX enabled by default Using config: /oldboy/softwares/zk/bin/../conf/zoo.cfg Starting zookeeper ... STARTED 三個節點同時傳送啟動指定(6) 檢視狀態及主
[root@zookeeper1 ~]#zkServer.sh status ZooKeeper JMX enabled by default Using config: /oldboy/softwares/zk/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower [root@zookeeper2 ~]#zkServer.sh status ZooKeeper JMX enabled by default Using config: /oldboy/softwares/zk/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower [root@zookeeper3 ~]#zkServer.sh status ZooKeeper JMX enabled by default Using config: /oldboy/softwares/zk/bin/../conf/zoo.cfg Client port found: 2181. Client address: localhost. Client SSL: false. Mode: leader(7) 使用叢集的方式工作
- 在各節點zookeeper服務執行時執行下面命令並無影響
zkCli.sh -server 10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181溫馨提示:
1.可以嘗試掛掉leader節點，驗證是否能夠正常使用。
2.當叢集半數以上節點存活，發現叢集可以正常提供服務，如果叢集容錯率是N，需要 2N+1臺機器;
3.zookeeper的leader選舉流程先比較zxid(zookeeper的事務ID)，如果比較不出來 再比較myid(zookeeper節點數字編號的唯一標識),誰大誰就是leader
- 半數以上機制，才能正常提供服務
1.5 zookeeper命令列的基本使用
(1) 進入客戶端
zkCli.sh(2) 建立zookeeper node
[zk: localhost:2181(CONNECTED) 0] create create [-s] [-e] [-c] [-t ttl] path [data] [acl] # 建立時不指定資料 [zk: localhost:2181(CONNECTED) 0] create /oldboy Created /oldboy # 建立時指定資料 [zk: localhost:2181(CONNECTED) 1] create /linux90 jiaoshi05 Created /linux90(3) 檢視zookeeper node列表
[zk: localhost:2181(CONNECTED) 2] ls / [linux90, oldboy, zookeeper] # # 檢視zookeeper node 儲存的資料 [zk: localhost:2181(CONNECTED) 3] get /linux90 jiaoshi05 [zk: localhost:2181(CONNECTED) 4] get /oldboy null(4) 修改資料
[zk: localhost:2181(CONNECTED) 4] get /oldboy null [zk: localhost:2181(CONNECTED) 5] set /oldboy elk [zk: localhost:2181(CONNECTED) 6] get /oldboy elk(5) 刪除zookeeper node
[zk: localhost:2181(CONNECTED) 7] delete /oldboy [zk: localhost:2181(CONNECTED) 8] ls / [linux90, zookeeper](6) 遞迴(巢狀)建立zookeeper node
- 建立多級zookeeper node 只能一級一級建立
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 16] create /oldboyedu Created /oldboyedu [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 17] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 17] create /oldboyedu/linux90 jiaoshi05 Created /oldboyedu/linux90 [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 18] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 18] create /oldboyedu/linux89 jiaoshi03 Created /oldboyedu/linux89 [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 19] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 19] ls /oldboyedu [linux89, linux90] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 20] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 20] get /oldboyedu/linux90 jiaoshi05 [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 21] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 21] get /oldboyedu/linux89 jiaoshi03 [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 22] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 22] get /oldboyedu null [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 23] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 23] set /oldboyedu laonanhai [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 24] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 24] get /oldboyedu laonanhai [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 25] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 25] deleteall /oldboyedu [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 26] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 26] ls / [linux90, zookeeper] [zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 27]
1.6 擴充套件
(1) 檢視zookeeper node的節點狀態
- zookeeper的擴充套件知識上篇（瞭解即可）
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 42] stat /linux90
cZxid = 0x60000000f
ctime = Tue Feb 27 07:11:33 UTC 2024
mZxid = 0x60000000f
mtime = Tue Feb 27 07:11:33 UTC 2024
pZxid = 0x60000000f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 9
numChildren = 0
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 43]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 43] ls -s /linux90
[]
cZxid = 0x60000000f
ctime = Tue Feb 27 07:11:33 UTC 2024
mZxid = 0x60000000f
mtime = Tue Feb 27 07:11:33 UTC 2024
pZxid = 0x60000000f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 9
numChildren = 0
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 44]
使用stat命令可以檢視znode的後設資料資訊。
cZxid:
資料節點建立時的事物ID。
ctime:
資料節點建立時的時間。
mZxid:
資料節點最後一次更新時的事物ID。
mtime:
資料節點最後一次更新時的時間。
pZxid:
資料節點的子節點最後一次被修改時的事務ID。
cversion:
子節點的更改次數。
dataVersion:
資料節點的更改次數，即維護的是一個資料版本號。
aclVersion:
節點的ACL的更改次數。
ephemeralOwner:
如果節點是臨時節點，則表示建立該節點的會話SessionID，如果節點是持久節點，則該屬性值為0。
dataLength:
資料內容的長度。
numChildren:
資料節點當前的子節點的數量。
(2) zookeeper的node型別
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 53] create -s /linux90/c2
Created /linux90/c20000000001
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 54]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 54] create -e /linux90/c3
Created /linux90/c3
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 55]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 55] ls /linux90
[c1, c20000000001, c3]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 56]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 56] stat /linux90/c1
cZxid = 0x60000001e
ctime = Tue Feb 27 07:25:18 UTC 2024
mZxid = 0x60000001e
mtime = Tue Feb 27 07:25:18 UTC 2024
pZxid = 0x60000001e
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 57]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 57] stat /linux90/c20000000001
cZxid = 0x60000001f
ctime = Tue Feb 27 07:25:29 UTC 2024
mZxid = 0x60000001f
mtime = Tue Feb 27 07:25:29 UTC 2024
pZxid = 0x60000001f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 58]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 58] stat /linux90/c3
cZxid = 0x600000020
ctime = Tue Feb 27 07:25:34 UTC 2024
mZxid = 0x600000020
mtime = Tue Feb 27 07:25:34 UTC 2024
pZxid = 0x600000020
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x5f0004831c010002
dataLength = 0
numChildren = 0
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 59]
溫馨提示:
-s ：
建立一個帶序號 的zookeeper node。
-e:
建立的是臨時的zookeeper node，其ephemeralOwner狀態預設為會話的session ID，比如:"0x5f0004831c010002"
(3) watch機制
3.1 終端1
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 3] ls -w /linux90
[c1, c20000000001]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 4]
WATCHER::
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/linux90
3.2 終端2
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 4] ls /linux90
[c1, c20000000001]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 5]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 5] create /linux90/c3
Created /linux90/c3
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 6]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 6] ls /linux90
[c1, c20000000001, c3]
[zk: 10.0.0.94:2181,10.0.0.95:2181,10.0.0.96:2181(CONNECTED) 7]
溫馨提示:
zookeeper node的watch機制是一次性的。
(4) zookeeper的acl許可權控制概述
zookeeper類似檔案系統，client可以建立節點，更新節點，刪除節點，那麼如何做到節點的許可權的控制呢？
zookeeper的Access Control List(訪問控制列表，簡稱ACL)可以做到這一點。
zookeeper的ACL許可權控制使用"scheme:id:permission"來標識，主要涵蓋如下三個方面:
許可權模式(scheme):
所謂的許可權模式指的是採用何種方式授權。
zookeeper支援以下幾種許可權模式:
world:
只有一個使用者，即"anyone"，代表登入zookeeper的所有人，這也是預設的許可權模式。
ip:
對客戶端使用IP地址認證。
auth:
使用已新增認證的使用者認證。
digest:
使用"使用者名稱:密碼"方式進行認證。
授權的物件(id):
所謂的授權物件指的是給誰授予許可權。授權物件ID是指許可權賦予的實體，例如: IP地址或使用者。
許可權(permission):
指的是授權的許可權。zookeeper支援以下幾種許可權:
create(簡寫"c"):
表示可以建立子節點。
delete(簡稱"d"):
可以刪除子節點。
read(簡稱"r"):
可以讀取節點資料及顯示子節點列表。
write(簡稱"w"):
可以修改節點資料。
admin(簡稱"a"):
可以設定節點訪問控制列表許可權。
zookeeper的ACL的特點如下:
(1)zookeeper的許可權控制是基於每個znode節點的，需要對每個節點設定許可權;
(2)每個znode支援設定多種許可權控制方案和多個許可權;
(3)子節點不會繼承父節點的許可權，也就是說，客戶端無權訪問某個znode，並不代表無法訪問它的子節點;
授權管理的相關命令:
getAcl:
讀取ACL許可權，使用語法:"getAcl [-s] path"。
setAcl:
設定ACL許可權，使用語法:"setAcl [-s] [-v version] [-R] path acl"。
addauth
新增認證使用者，使用語法:"addauth scheme auth"。
(5) 基於world許可權模式(scheme)認證案例
[zk: localhost:2181(CONNECTED) 20] create /oldboyedu-linux
Created /oldboyedu-linux
[zk: localhost:2181(CONNECTED) 21]
[zk: localhost:2181(CONNECTED) 21] create /oldboyedu-linux/c1
Created /oldboyedu-linux/c1
[zk: localhost:2181(CONNECTED) 22]
[zk: localhost:2181(CONNECTED) 22] getAcl /oldboyedu-linux
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 23]
[zk: localhost:2181(CONNECTED) 23] setAcl /oldboyedu-linux world:anyone:rwa
[zk: localhost:2181(CONNECTED) 24]
[zk: localhost:2181(CONNECTED) 24] getAcl /oldboyedu-linux
'world,'anyone
: rwa
[zk: localhost:2181(CONNECTED) 25]
[zk: localhost:2181(CONNECTED) 25] ls /oldboyedu-linux
[c1]
[zk: localhost:2181(CONNECTED) 26]
[zk: localhost:2181(CONNECTED) 26] delete /oldboyedu-linux/c1
delete deleteall
[zk: localhost:2181(CONNECTED) 26] delete /oldboyedu-linux/c1
Insufficient permission : /oldboyedu-linux/c1
[zk: localhost:2181(CONNECTED) 27]
[zk: localhost:2181(CONNECTED) 27] create /oldboyedu-linux/c2
Insufficient permission : /oldboyedu-linux/c2
[zk: localhost:2181(CONNECTED) 28]
[zk: localhost:2181(CONNECTED) 28] get /oldboyedu-linux
null
[zk: localhost:2181(CONNECTED) 29]
[zk: localhost:2181(CONNECTED) 29] set /oldboyedu-linux oldboy
[zk: localhost:2181(CONNECTED) 30]
[zk: localhost:2181(CONNECTED) 30] get /oldboyedu-linux
oldboy
[zk: localhost:2181(CONNECTED) 31]
[zk: localhost:2181(CONNECTED) 31] setAcl /oldboyedu-linux world:anyone:rwacd
[zk: localhost:2181(CONNECTED) 32]
[zk: localhost:2181(CONNECTED) 32] getAcl /oldboyedu-linux
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 33]
[zk: localhost:2181(CONNECTED) 33] create /oldboyedu-linux/c2
Created /oldboyedu-linux/c2
[zk: localhost:2181(CONNECTED) 34]
[zk: localhost:2181(CONNECTED) 34] delete /oldboyedu-linux/c1
[zk: localhost:2181(CONNECTED) 35]
[zk: localhost:2181(CONNECTED) 35] ls /oldboyedu-linux
[c2]
[zk: localhost:2181(CONNECTED) 36]
(6) 基於ip許可權模式(scheme)認證案例
3.1 終端1操作
[zk: localhost:2181(CONNECTED) 38] getAcl /oldboyedu-linux
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 39]
[zk: localhost:2181(CONNECTED) 39] setAcl /oldboyedu-linux ip:10.0.0.96:cra
[zk: localhost:2181(CONNECTED) 40]
[zk: localhost:2181(CONNECTED) 40] getAcl /oldboyedu-linux
Insufficient permission : /oldboyedu-linux
[zk: localhost:2181(CONNECTED) 41]
[zk: localhost:2181(CONNECTED) 41] whoami
Auth scheme: User
ip: 127.0.0.1
[zk: localhost:2181(CONNECTED) 42]
3.2 終端2操作
[root@elk96 ~]# zkCli.sh -server 10.0.0.94
...
[zk: 10.0.0.94(CONNECTED) 1] ls /oldboyedu-linux
[c2]
[zk: 10.0.0.94(CONNECTED) 2]
[zk: 10.0.0.94(CONNECTED) 2] getAcl /oldboyedu-linux
'ip,'10.0.0.96
: cra
[zk: 10.0.0.94(CONNECTED) 3]
[zk: 10.0.0.94(CONNECTED) 3] whoami
Auth scheme: User
ip: 10.0.0.96
[zk: 10.0.0.94(CONNECTED) 4]
[zk: 10.0.0.94(CONNECTED) 4] get /oldboyedu-linux
oldboy
[zk: 10.0.0.94(CONNECTED) 5]
[zk: 10.0.0.94(CONNECTED) 5] set /oldboyedu-linux oldboyedu
Insufficient permission : /oldboyedu-linux
[zk: 10.0.0.94(CONNECTED) 6]
...
[zk: 10.0.0.94(CONNECTED) 8] setAcl /oldboyedu-linux ip:10.0.0.96:crawd
[zk: 10.0.0.94(CONNECTED) 9]
[zk: 10.0.0.94(CONNECTED) 9] getAcl /oldboyedu-linux
'ip,'10.0.0.96
: cdrwa
[zk: 10.0.0.94(CONNECTED) 10]
[zk: 10.0.0.94(CONNECTED) 10] get /oldboyedu-linux
oldboy
[zk: 10.0.0.94(CONNECTED) 11]
[zk: 10.0.0.94(CONNECTED) 11] set /oldboyedu-linux oldboyedu
[zk: 10.0.0.94(CONNECTED) 12]
[zk: 10.0.0.94(CONNECTED) 12] get /oldboyedu-linux
oldboyedu
[zk: 10.0.0.94(CONNECTED) 13]
[zk: 10.0.0.94(CONNECTED) 13]
溫馨提示:
也支援網段的配置。
[zk: localhost:2181(CONNECTED) 12] setAcl /linux90 ip:10.0.0.0/24:cra
[zk: localhost:2181(CONNECTED) 13]
[zk: localhost:2181(CONNECTED) 13] getAcl /linux90
'ip,'10.0.0.0/24
: cra
[zk: localhost:2181(CONNECTED) 14]
4.基於auth許可權模式(scheme)認證案例
4.1 終端1
[zk: 10.0.0.94(CONNECTED) 13] addauth digest admin:oldboyedu
[zk: 10.0.0.94(CONNECTED) 14]
[zk: 10.0.0.94(CONNECTED) 14] setAcl /oldboyedu-linux auth:admin:cra
[zk: 10.0.0.94(CONNECTED) 15]
[zk: 10.0.0.94(CONNECTED) 15] getAcl /oldboyedu-linux
'digest,'admin:Tmbbt77KcTd1bAgjQaI+GqI0hjM=
: cra
[zk: 10.0.0.94(CONNECTED) 16]
[zk: 10.0.0.94(CONNECTED) 16] get /oldboyedu-linux
oldboyedu
[zk: 10.0.0.94(CONNECTED) 17]
[zk: 10.0.0.94(CONNECTED) 17] whoami
Auth scheme: User
ip: 10.0.0.96
digest: admin
[zk: 10.0.0.94(CONNECTED) 18]
4.2 終端2，輸入正確的密碼
[zk: localhost:2181(CONNECTED) 0] addauth digest admin:oldboyedu
[zk: localhost:2181(CONNECTED) 1]
[zk: localhost:2181(CONNECTED) 1] whoami
Auth scheme: User
ip: 127.0.0.1
digest: admin
[zk: localhost:2181(CONNECTED) 2] ls /oldboyedu-linux
[c2]
[zk: localhost:2181(CONNECTED) 3]
4.3 終端3，輸入錯誤的密碼
[zk: localhost:2181(CONNECTED) 44] addauth digest admin:123
[zk: localhost:2181(CONNECTED) 45]
[zk: localhost:2181(CONNECTED) 45] whoami
Auth scheme: User
ip: 127.0.0.1
digest: admin
[zk: localhost:2181(CONNECTED) 46] getAcl /oldboyedu-linux
Insufficient permission : /oldboyedu-linux
[zk: localhost:2181(CONNECTED) 47]
5.基於digest許可權模式(scheme)認證案例
5.1 使用sha1加密演算法及base64編碼來進行一個密碼的計算
[root@elk94 ~]# echo -n admin:oldboyedu | openssl dgst -binary -sha1 | base64
Tmbbt77KcTd1bAgjQaI+GqI0hjM=
[root@elk94 ~]#
5.2 終端1，建立znode時同時指定ACL
[zk: localhost:2181(CONNECTED) 3] create /oldboyedu-linux90 laonanhai digest:admin:Tmbbt77KcTd1bAgjQaI+GqI0hjM=:rwa
Created /oldboyedu-linux90
[zk: localhost:2181(CONNECTED) 4]
[zk: localhost:2181(CONNECTED) 4] getAcl /oldboyedu-linux90
Insufficient permission : /oldboyedu-linux90
[zk: localhost:2181(CONNECTED) 5]
[zk: localhost:2181(CONNECTED) 5] whoami
Auth scheme: User
ip: 127.0.0.1
[zk: localhost:2181(CONNECTED) 6]
5.3 終端2驗證
[zk: localhost:2181(CONNECTED) 1] whoami
Auth scheme: User
ip: 127.0.0.1
[zk: localhost:2181(CONNECTED) 2]
[zk: localhost:2181(CONNECTED) 2] addauth digest admin:oldboyedu
[zk: localhost:2181(CONNECTED) 3]
[zk: localhost:2181(CONNECTED) 3] whoami
Auth scheme: User
ip: 127.0.0.1
digest: admin
[zk: localhost:2181(CONNECTED) 4]
[zk: localhost:2181(CONNECTED) 4] getAcl /oldboyedu-linux90
'digest,'admin:Tmbbt77KcTd1bAgjQaI+GqI0hjM=
: rwa
[zk: localhost:2181(CONNECTED) 5]
[zk: localhost:2181(CONNECTED) 5] get /oldboyedu-linux90
laonanhai
[zk: localhost:2181(CONNECTED) 6]
[zk: localhost:2181(CONNECTED) 6] set /oldboyedu-linux90 yitiantian
[zk: localhost:2181(CONNECTED) 7]
[zk: localhost:2181(CONNECTED) 7] get /oldboyedu-linux90
yitiantian
[zk: localhost:2181(CONNECTED) 8]
[zk: localhost:2181(CONNECTED) 8] create /oldboyedu-linux90/c1
Insufficient permission : /oldboyedu-linux90/c1
[zk: localhost:2181(CONNECTED) 9]
(7) 解決ACL許可權不足的兩種方案
6.解決ACL許可權不足的兩種方案
6.1 問題復現
[zk: 10.0.0.94(CONNECTED) 64] create /oldboyedu
Created /oldboyedu
[zk: 10.0.0.94(CONNECTED) 65]
[zk: 10.0.0.94(CONNECTED) 65] create /oldboyedu/linux90
Created /oldboyedu/linux90
[zk: 10.0.0.94(CONNECTED) 66]
[zk: 10.0.0.94(CONNECTED) 66] setAcl /oldboyedu world:anyone:rw
[zk: 10.0.0.94(CONNECTED) 67]
[zk: 10.0.0.94(CONNECTED) 67] ls /oldboyedu
[linux90]
[zk: 10.0.0.94(CONNECTED) 68] create /oldboyedu/linux91
Insufficient permission : /oldboyedu/linux91
[zk: 10.0.0.94(CONNECTED) 69]
[zk: 10.0.0.94(CONNECTED) 69] delete /oldboyedu l
listquota ls
[zk: 10.0.0.94(CONNECTED) 69] delete /oldboyedu/linux90
Insufficient permission : /oldboyedu/linux90
[zk: 10.0.0.94(CONNECTED) 70]
[zk: 10.0.0.94(CONNECTED) 70] setAcl /oldboyedu world:anyone:rwdac
Insufficient permission : /oldboyedu
[zk: 10.0.0.94(CONNECTED) 71]
[zk: 10.0.0.94(CONNECTED) 71] deleteall /oldboyedu
Failed to delete some node(s) in the subtree!
[zk: 10.0.0.94(CONNECTED) 72]
溫馨提示:
出現上述情況，是新手可能會遇到的問題，可以嘗試下面兩種方法解決該問題。
6.2 zookeeper跳過許可權檢查實戰
6.2.1 所有節點修改zoo.cfg配置檔案，跳過許可權檢查
vim /oldboyedu/softwares/zk/conf/zoo.cfg
...
skipACL=yes
6.2.2 將配置同步到其他節點
[root@elk94 ~]# data_rsync.sh /oldboyedu/softwares/zk/conf/zoo.cfg
6.2.3 重啟叢集
[root@elk94 ~]# zkServer.sh restart
[root@elk95 ~]# zkServer.sh restart
[root@elk96 ~]# zkServer.sh restart
6.2.4 修改許可權驗證，發現沒有許可權依舊是可以建立資料的(原因是跳過了ACL檢查)
[zk: localhost:2181(CONNECTED) 4] getAcl /oldboyedu
'world,'anyone
: rw
[zk: localhost:2181(CONNECTED) 5]
[zk: localhost:2181(CONNECTED) 5] ls /oldboyedu
[linux90]
[zk: localhost:2181(CONNECTED) 6]
[zk: localhost:2181(CONNECTED) 6] create /oldboyedu/linux91
Created /oldboyedu/linux91
[zk: localhost:2181(CONNECTED) 7]
[zk: localhost:2181(CONNECTED) 7] setAcl /oldboyedu world:anyone:rwcda
[zk: localhost:2181(CONNECTED) 8]
[zk: localhost:2181(CONNECTED) 8] getAcl /oldboyedu
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 9]
[zk: localhost:2181(CONNECTED) 9]
6.2.5 所有節點修改zoo.cfg配置檔案，跳過許可權檢查
vim /oldboyedu/softwares/zk/conf/zoo.cfg
...
skipACL=no
6.2.6 將配置同步到其他節點
[root@elk94 ~]# data_rsync.sh /oldboyedu/softwares/zk/conf/zoo.cfg
6.2.7 重啟叢集
[root@elk94 ~]# zkServer.sh restart
[root@elk95 ~]# zkServer.sh restart
[root@elk96 ~]# zkServer.sh restart
6.2.8 再次驗證許可權
[zk: localhost:2181(CONNECTED) 3] getAcl /oldboyedu
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 4]
[zk: localhost:2181(CONNECTED) 4] setAcl /oldboyedu world:anyone:arc
[zk: localhost:2181(CONNECTED) 5]
[zk: localhost:2181(CONNECTED) 5] getAcl /oldboyedu
'world,'anyone
: cra
[zk: localhost:2181(CONNECTED) 6]
[zk: localhost:2181(CONNECTED) 6] create /oldboyedu/linux9
linux90 linux91
[zk: localhost:2181(CONNECTED) 6] create /oldboyedu/linux92
Created /oldboyedu/linux92
[zk: localhost:2181(CONNECTED) 7]
[zk: localhost:2181(CONNECTED) 7] delete /oldboyedu/linux92
Insufficient permission : /oldboyedu/linux92
[zk: localhost:2181(CONNECTED) 8]
溫馨提示:
實際工作中，應該滾動重啟服務，不應該批次重啟。
(8) 配置超級使用者-推薦部署叢集的時候使用
6.3.1 使用sha1加密演算法及base64編碼來進行一個密碼的計算
[root@elk94 ~]# echo -n admin:oldboyedu | openssl dgst -binary -sha1 | base64
Tmbbt77KcTd1bAgjQaI+GqI0hjM=
[root@elk94 ~]#
6.3.2 修改zkEnv.sh指令碼，配置超級使用者
[root@elk94 ~]# vim /oldboyedu/softwares/zk/bin/zkEnv.sh
...
export SERVER_JVMFLAGS="-Dzookeeper.DigestAuthenticationProvider.superDigest=admin:Tmbbt77KcTd1bAgjQaI+GqI0hjM= -Xmx${ZK_SERVER_HEAP}m $SERVER_JVMFLAGS"
6.3.3 同步指令碼叫其他zk節點
[root@elk94 ~]# data_rsync.sh /oldboyedu/softwares/zk/bin/zkEnv.sh
6.3.4 重啟叢集
[root@elk94 ~]# zkServer.sh restart
[root@elk95 ~]# zkServer.sh restart
[root@elk96 ~]# zkServer.sh restart
6.3.5驗證超級管理員使用者許可權
[zk: localhost:2181(CONNECTED) 3] getAcl /oldboyedu
'world,'anyone
: cra
[zk: localhost:2181(CONNECTED) 4]
[zk: localhost:2181(CONNECTED) 4] setAcl /oldboyedu world:anyone:rw
[zk: localhost:2181(CONNECTED) 5]
[zk: localhost:2181(CONNECTED) 5] getAcl /oldboyedu
'world,'anyone
: rw
[zk: localhost:2181(CONNECTED) 6]
[zk: localhost:2181(CONNECTED) 6] create /oldboyedu/linux93
Insufficient permission : /oldboyedu/linux93
[zk: localhost:2181(CONNECTED) 7]
[zk: localhost:2181(CONNECTED) 7]
[zk: localhost:2181(CONNECTED) 7] setAcl /oldboyedu world:anyone:rwc
Insufficient permission : /oldboyedu
[zk: localhost:2181(CONNECTED) 8]
[zk: localhost:2181(CONNECTED) 8] whoami
Auth scheme: User
ip: 127.0.0.1
[zk: localhost:2181(CONNECTED) 9]
[zk: localhost:2181(CONNECTED) 9] addauth digest admin:oldboyedu
[zk: localhost:2181(CONNECTED) 10]
[zk: localhost:2181(CONNECTED) 10] whoami
Auth scheme: User
super:
ip: 127.0.0.1
digest: admin
[zk: localhost:2181(CONNECTED) 11]
[zk: localhost:2181(CONNECTED) 11] setAcl /oldboyedu world:anyone:rwc
[zk: localhost:2181(CONNECTED) 12]
[zk: localhost:2181(CONNECTED) 12] getAcl /oldboyedu
'world,'anyone
: crw
[zk: localhost:2181(CONNECTED) 13]
[zk: localhost:2181(CONNECTED) 13] create /oldboyedu/linux93
Created /oldboyedu/linux93
[zk: localhost:2181(CONNECTED) 14]
[zk: localhost:2181(CONNECTED) 14] ls /oldboyedu
[linux90, linux91, linux92, linux93]
[zk: localhost:2181(CONNECTED) 15]
溫馨提示:
目前使用zookeeper 3.8版本貌似無法替換使用者名稱，超級管理員使用者名稱稱暫時推薦使用admin。
1.6.1 圖形化介面
1. 啟動服務
java -jar zkWeb-v1.2.1.jar
# 該工具適配的是jdk8版本
# 使用絕對路徑的jdk 使用與多個jdk版本 並且該版本為配置為系統環境變數
3.啟動服務
[root@elk94 ~]# /oldboyedu/softwares/jdk1.8.0_291/bin/java -jar zkWeb-v1.2.1.jar
溫馨提示:
不要使用我們高版本的JDK
[root@elk94 ~]# java --version
java 17.0.8 2023-07-18 LTS
Java(TM) SE Runtime Environment (build 17.0.8+9-LTS-211)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.8+9-LTS-211, mixed mode, sharing)
[root@elk94 ~]#
[root@elk94 ~]# which java
/oldboyedu/softwares/jdk//bin/java
[root@elk94 ~]#
4.檢查埠是否監聽
[root@elk94 ~]# ss -nlt | grep 8099
LISTEN 0 100 *:8099 *:*
[root@elk94 ~]#
5.訪問zkWeb UI
http://10.0.0.94:8099/#
1.7 zookeeper 啟動指令碼編寫
1.停止zookeeper叢集
[root@elk94 ~]# zkServer.sh stop
[root@elk95 ~]# zkServer.sh stop
[root@elk96 ~]# zkServer.sh stop
2.所有節點編寫啟動指令碼
cat > /lib/systemd/system/zk.service << EOF
[Unit]
Description=oldboyedu linux zookeeper server daemon
Documentation=www.oldboyedu.com
After=network.target
[Service]
Type=forking
Environment=JAVA_HOME=/oldboyedu/softwares/jdk
ExecStart=/oldboyedu/softwares/zk/bin/zkServer.sh start
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now zk
systemctl status zk
溫馨提示:
注意，此處的型別一定要設定為"forking"，因為咱們案例中ExecStart對應的命令執行完成後就結束了，不會阻塞。
1.8 jvm 調優
- zookeeper的JVM調優
1.檢視預設的堆記憶體為1GB
[root@elk94 ~]# ps -ef | grep zookeeper | grep Xmx
root 4793 1 0 18:01 ? 00:00:03 /oldboyedu/softwares/jdk/bin/java ... -Xmx1000m ...
[root@elk94 ~]#
2.配置ZK的堆記憶體
[root@elk94 ~]# cat > /oldboyedu/softwares/zk/conf/java.env <<'EOF'
#!/bin/bash
# 指定JDK的按住路徑
export JAVA_HOME=/oldboyedu/softwares/jdk
# 指定zookeeper的堆記憶體大小
export JVMFLAGS="-Xms128m -Xmx128m $JVMFLAGS"
EOF
3.同步檔案到其他節點
[root@elk94 ~]# data_rsync.sh /oldboyedu/softwares/zk/conf/java.env
4.重啟叢集
[root@elk94 ~]# systemctl restart zk
[root@elk95 ~]# systemctl restart zk
[root@elk96 ~]# systemctl restart zk
5.驗證配置
[root@elk94 ~]# ps -ef | grep zookeeper | grep Xmx
root 4793 1 0 18:01 ? 00:00:03 /oldboyedu/softwares/jdk/bin/java ...-Xmx1000m -Xms128m -Xmx128m ...
[root@elk94 ~]#
溫馨提示:
- 此處堆記憶體大小是128MB，但是會發現預設會有一個"-Xmx 1000m"，理論上後面的配置會覆蓋前面的配置。
如果擔心衝突，可以考慮修改指令碼，將預設的1000註釋掉。
[root@elk94 ~]# vim /oldboyedu/softwares/apache-zookeeper-3.8.3-bin/bin/zkEnv.sh
...
# 註釋zk的堆記憶體預設配置為1000MB。大概在138行。
# ZK_SERVER_HEAP="${ZK_SERVER_HEAP:-1000}"
ZK_SERVER_HEAP="${ZK_SERVER_HEAP:-128}"
[root@elk94 ~]#
[root@elk94 ~]# data_rsync.sh /oldboyedu/softwares/apache-zookeeper-3.8.3-bin/bin/zkEnv.sh
- 生產環境堆記憶體修改也不需要設定過大，基本上2~4GB足以。
1.9 zookeeper 監控
- zookeeper的監控
1.啟動zookeeper的JMX埠
1.1 修改配置檔案
vim /oldboyedu/softwares/zk/conf/zoo.cfg
# 新增下面的一行，啟動zk的4字監控命令
4lw.commands.whitelist=*
1.2 修改zk的啟動指令碼
vim /oldboyedu/softwares/zk/bin/zkServer.sh
...
# 如果修改上面的方式不生效，則需修改zkServer.sh指令碼中77行之後ZOOMAIN的值即可。
ZOOMAIN="-Dzookeeper.4lw.commands.whitelist=* ${ZOOMAIN}"
1.3 可選步驟，開啟zk節點的JMX
[root@elk94 ~]# vim /oldboyedu/softwares/zk/bin/zkEnv.sh
...
# Add by yinzhengjie for enable zookeeper JMX
JMXLOCALONLY=false
JMXHOSTNAME=10.0.0.94
JMXPORT=9999
JMXSSL=false
JMXLOG4J=false
參考連結:
http://www.kafka-eagle.org/articles/docs/quickstart/metrics.html
2.驗證四字命令
2.1 校驗服務是否存活
[root@elk96 ~]# echo ruok | nc 10.0.0.96 2181
imok
2.2 檢視zookeeper叢集的配置
[root@elk96 ~]# echo conf | nc 10.0.0.94 2181
clientPort=2181
secureClientPort=-1
dataDir=/oldboyedu/data/zk/version-2
dataDirSize=536873745
dataLogDir=/oldboyedu/data/zk/version-2
dataLogSize=536873745
tickTime=2000
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
clientPortListenBacklog=-1
serverId=94
initLimit=5
syncLimit=2
electionAlg=3
electionPort=6888
quorumPort=5888
peerType=0
membership:
server.96=10.0.0.96:5888:6888:participant
server.94=10.0.0.94:5888:6888:participant
server.95=10.0.0.95:5888:6888:participant
version=0
2.3 檢查叢集的模式，比如誰是leader和follower
[root@elk96 ~]# echo srvr | nc 10.0.0.94 2181
Zookeeper version: 3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC
Latency min/avg/max: 0/0.0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0xe0000000c
Mode: follower
Node count: 10
[root@elk96 ~]#
[root@elk96 ~]#
[root@elk96 ~]# echo srvr | nc 10.0.0.95 2181
Zookeeper version: 3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC
Latency min/avg/max: 0/0.0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x1000000000
Mode: leader
Node count: 10
Proposal sizes last/min/max: -1/-1/-1
[root@elk96 ~]#
[root@elk96 ~]#
[root@elk96 ~]#
[root@elk96 ~]# echo srvr | nc 10.0.0.96 2181
Zookeeper version: 3.8.3-6ad6d364c7c0bcf0de452d54ebefa3058098ab56, built on 2023-10-05 10:34 UTC
Latency min/avg/max: 0/0.0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0xe0000000c
Mode: follower
Node count: 10
[root@elk96 ~]#
3.驗證JMX埠是否監聽成功
3.1 服務端檢查埠
[root@elk94 ~]# ss -ntl | grep 9999
LISTEN 0 50 *:9999 *:*
[root@elk94 ~]#
3.2 windows安裝JDK環境並切換到安裝目錄，先敲擊cmd
C:\Users\yinzhengjie>cd C:\Program Files\Java\jdk-1.8\bin\
C:\Program Files\Java\jdk-1.8\bin>
C:\Program Files\Java\jdk-1.8\bin>jconsole.exe
2. kafka
2.1 MQ
2.1.1 MQ概述
什麼是mq
MQ的全稱是“Message Queue” 即 訊息佇列
訊息佇列是在訊息傳輸過程中儲存訊息的容器，多用於分散式系統之間通訊
類似與一個水管 能將水源抽往目的地
2.1.2 MQ的優勢
(1) 應用解耦
(2) 非同步提速
(3) 削峰填谷
將訊息全部儲存到訊息佇列中，慢慢的消費，這樣就抗住了資料量高峰，但是可能導致延遲
2.1.3 MQ的劣勢
- 系統的可用性降低
- 系統引入的瓦斯不依賴越多，系統的穩定性越差，一旦MQ當機，就會對業務造成影響，如果MQ能實現高可用，那問題就自然也就解決了
- 系統的複雜性提升
- 對使用者而言，提升了學習成本，也存在資料丟失的風險
綜上所述：我們希望MQ有以下幾點特徵
- 吞吐量高
- 支援高可用叢集
- 資料安全：資料訪問安全，資料儲存安全
2.1.4 常見的MQ產品
2.1.5 MQ的工作模式
(1) MQ的點對點模式(1對1模式，消費者主動拉取資料，訊息收到後MQ會將訊息清除)
(2) MQ釋出/訂閱模式(1對多，消費者消費資料後不會清除訊息)
一對多，消費者消費資料後不會清除訊息。
如下圖所示，消費者可以主動去Broker伺服器去拉取資料，當然，也可以是Broker主動推送資料。
拉取(pull):
優點:
消費者程式可以根據自身的硬體配置去broker消費。
缺點:
消費者需要長期執行一個程序來詢問broker是否有資料。推(push):
優點:
無需客戶端主動拉去資料，而是由服務端主動傳送資料，典型的應用場景就是我們的"公眾號"等業務。
缺點:
(1)broker推送資料給消費者是，可能因為消費者消費能力不足，直接導致客戶端程式崩潰掉;
(2)broker內部需要維護一個訂閱者列表，當訂閱者較多時，可能會很佔用記憶體喲;值得注意的是，kafka broker僅支援拉取(pull)等工作方式。
2.2 kafka單點搭建
(1) 部署zookeeper叢集
(2) 解壓軟體包
tar xf kafka_2.13-3.7.0.tgz -C /oldboy/softwares/(3) 修改配置檔案
[root@zookeeper3 /softwares]#cd /oldboy/softwares/kafka_2.13-3.7.0/ [root@zookeeper3 /oldboy/softwares/kafka_2.13-3.7.0]#vim config/server.properties ... # 指定kafka例項的唯一編號，叢集中該編號要唯一。 # broker.id=0 broker.id=203 ... # 指定資料的儲存路徑 #log.dirs=/tmp/kafka-logs log.dirs=/oldboyedu/data/kafka ... # 指定zookeeper叢集地址 # zookeeper.connect=localhost:2181 zookeeper.connect=10.0.0.201:2181,10.0.0.202:2181,10.0.0.203:2181/oldboy-kafka370(4) 配置環境變數
[root@zookeeper3 ~]#vim /etc/profile.d/kafka.sh [root@zookeeper3 ~]#cat /etc/profile.d/kafka.sh #!/bin/bash export KAFKA_HOME=/oldboy/softwares/kafka_2.13-3.7.0 export PATH=$PATH:$KAFKA_HOME/bin [root@zookeeper3 ~]#source /etc/profile.d/kafka.sh(5) 啟動kafka
- 必須啟動zookeeper叢集
[root@zookeeper3 ~]#kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties [root@zookeeper3 ~]#jps 2225 Kafka 2258 Jps 1806 QuorumPeerMain [root@zookeeper3 /oldboyedu/data/kafka]#ss -ntl |grep 9092 LISTEN 0 50 [::]:9092 [::]:* # 檢視zookeeper的資料 [root@zookeeper3 /oldboyedu/data/kafka]#zkCli.sh 。。。 [zk: localhost:2181(CONNECTED) 0] ls / [linux90, oldboy-kafka370, zookeeper]
2.3 jvm調優（調記憶體）
- kafka的JVM調優及JMX埠開啟 1.停止kafka服務 [root@elk94 ~]# kafka-server-stop.sh [root@elk94 ~]# [root@elk94 ~]# jps | grep Kafka [root@elk94 ~]# 2.修改啟動指令碼 [root@elk94 ~]# vim `which kafka-server-start.sh ` +28 ... if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then ... # 將原有的註釋掉,更改的是if fi裡面的export # export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" export KAFKA_HEAP_OPTS="-server -Xmx256m -Xms256m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70" # windows使用jconsole驗證JMX埠是否啟用,可以不使用 export JMX_PORT="8888" fi 溫馨提示: 生成環境建議大家將其設定為6GB即可。實體記憶體建議32GB+，磁碟根據實際資料資料週期而定。 3.啟動服務 [root@elk94 ~]# kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties 溫馨提示： -daemon是放在後臺執行，如果發現kafka啟動失敗且沒有日誌生成，則考慮去掉該引數，觀察報錯資訊即可。 4.檢查堆記憶體大小 [root@elk94 ~]# ps -ef | grep kafka | grep Xms root 11938 1653 20 10:44 pts/0 00:00:04 /oldboyedu/softwares/jdk//bin/java -server -Xmx256m -Xms256m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 ... 5.檢查JMX埠 [root@elk94 ~]# ss -ntl | grep 8888 LISTEN 0 50 *:8888 *:* [root@elk94 ~]# windows使用jconsole驗證JMX埠是否啟用 C:\Program Files\Java\jdk-1.8\bin>jconsole
2.4 kafka啟動指令碼
1. 停止kafka
[root@zookeeper3 /oldboyedu/data/kafka]#kafka-server-stop.sh
2. 編寫指令碼
cat > /lib/systemd/system/kafka.service <<EOF
[Unit]
Description=oldboyedu linux kafka server daemon
Documentation=www.oldboyedu.com
After=network.target
[Service]
Type=forking
Environment=JAVA_HOME=/oldboy/softwares/jdk8
ExecStart=/oldboy/softwares/kafka_2.13-3.7.0/bin/kafka-server-start.sh -daemon /oldboy/softwares/kafka_2.13-3.7.0/config/server.properties
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now kafka
systemctl status kafka
2.5 kafka叢集部署
與單點搭建一樣即可，逐個單點操作即可
- kafka叢集搭建 1.同步配置檔案，軟體包，環境變數，啟動指令碼到其他節點 [root@elk94 ~]# data_rsync.sh /oldboyedu/softwares/kafka_2.13-3.7.0/ [root@elk94 ~]# data_rsync.sh /etc/profile.d/kafka.sh [root@elk94 ~]# data_rsync.sh /lib/systemd/system/kafka.service 2.其他節點修改配置檔案 [root@elk95 ~]# source /etc/profile.d/kafka.sh [root@elk95 ~]# [root@elk95 ~]# vim $KAFKA_HOME/config/server.properties [root@elk95 ~]# [root@elk95 ~]# grep ^broker.id $KAFKA_HOME/config/server.properties broker.id=95 [root@elk95 ~]# [root@elk96 ~]# source /etc/profile.d/kafka.sh [root@elk96 ~]# [root@elk96 ~]# vim $KAFKA_HOME/config/server.properties [root@elk96 ~]# [root@elk96 ~]# grep ^broker.id $KAFKA_HOME/config/server.properties broker.id=96 [root@elk96 ~]# 3.其他節點啟動kafka服務 [root@elk95 ~]# systemctl daemon-reload [root@elk95 ~]# [root@elk95 ~]# systemctl enable --now kafka [root@elk95 ~]# [root@elk95 ~]# systemctl status kafka [root@elk96 ~]# systemctl daemon-reload [root@elk96 ~]# [root@elk96 ~]# systemctl enable --now kafka [root@elk96 ~]# [root@elk96 ~]# systemctl status kafka 4.zookeeper檢查broker節點是否上線 [zk: localhost:2181(CONNECTED) 13] ls /oldboyedu-kafka370/brokers/ids [94, 95, 96] [zk: localhost:2181(CONNECTED) 14] 5.驗證停止kafka服務是否會導致臨時zookeeper node自動刪除 溫馨提示: broker節點啟動時會連線zookeeper叢集，並註冊一個臨時zookeeper node，名稱為對應broker.id。 5.1 檢查zookeeper環境 [zk: localhost:2181(CONNECTED) 14] stat /oldboyedu-kafka370/brokers/ids/94 cZxid = 0x1000000061 ctime = Wed Feb 28 02:57:45 UTC 2024 mZxid = 0x1000000061 mtime = Wed Feb 28 02:57:45 UTC 2024 pZxid = 0x1000000061 cversion = 0 dataVersion = 1 aclVersion = 0 ephemeralOwner = 0x6000007437480003 dataLength = 196 numChildren = 0 [zk: localhost:2181(CONNECTED) 15] [zk: localhost:2181(CONNECTED) 15] [zk: localhost:2181(CONNECTED) 15] stat /oldboyedu-kafka370/brokers/ids/95 cZxid = 0x1000000088 ctime = Wed Feb 28 03:36:02 UTC 2024 mZxid = 0x1000000088 mtime = Wed Feb 28 03:36:02 UTC 2024 pZxid = 0x1000000088 cversion = 0 dataVersion = 1 aclVersion = 0 ephemeralOwner = 0x5f0000743cca0002 dataLength = 196 numChildren = 0 [zk: localhost:2181(CONNECTED) 16] [zk: localhost:2181(CONNECTED) 16] stat /oldboyedu-kafka370/brokers/ids/96 cZxid = 0x100000009b ctime = Wed Feb 28 03:37:47 UTC 2024 mZxid = 0x100000009b mtime = Wed Feb 28 03:37:47 UTC 2024 pZxid = 0x100000009b cversion = 0 dataVersion = 1 aclVersion = 0 ephemeralOwner = 0x5f0000743cca0003 dataLength = 196 numChildren = 0 [zk: localhost:2181(CONNECTED) 17] 5.2 讓96節點下線 [root@elk96 ~]# systemctl stop kafka 5.3 觀察zookeeper node的變化 [zk: localhost:2181(CONNECTED) 17] ls /oldboyedu-kafka370/brokers/ids [94, 95] [zk: localhost:2181(CONNECTED) 18] 5.4 讓96節點上線 [root@elk96 ~]# systemctl start kafka 5.5 再次觀察zookeeper node的變化 [zk: localhost:2181(CONNECTED) 22] ls /oldboyedu-kafka370/brokers/ids [94, 95, 96] [zk: localhost:2181(CONNECTED) 23] [zk: localhost:2181(CONNECTED) 23] stat /oldboyedu-kafka370/brokers/ids/96 cZxid = 0x10000000ae ctime = Wed Feb 28 03:40:48 UTC 2024 mZxid = 0x10000000ae mtime = Wed Feb 28 03:40:48 UTC 2024 pZxid = 0x10000000ae cversion = 0 dataVersion = 1 aclVersion = 0 ephemeralOwner = 0x5f0000743cca0004 dataLength = 196 numChildren = 0 [zk: localhost:2181(CONNECTED) 24] [zk: localhost:2181(CONNECTED) 24] - 使用systemctl管理zkweb服務 1.移動jar包到指定目錄 [root@elk94 ~]# mv zkWeb-v1.2.1.jar /oldboyedu/softwares/ 2.編寫啟動指令碼 cat > /lib/systemd/system/zkWeb.service <<EOF [Unit] Description=oldboyedu linux zkWeb server daemon Documentation=www.oldboyedu.com After=network.target [Service] ExecStart=/oldboyedu/softwares/jdk1.8.0_291/bin/java -Xms128m -Xmx128m -jar /oldboyedu/softwares/zkWeb-v1.2.1.jar [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable --now zkWeb systemctl status zkWeb 3.檢查服務是否正常啟動 [root@elk94 ~]# ss -ntl | grep 8099 LISTEN 0 100 *:8099 *:* [root@elk94 ~]# [root@elk94 ~]# systemctl status zkWeb ● zkWeb.service - oldboyedu linux zkWeb server daemon Loaded: loaded (/lib/systemd/system/zkWeb.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2024-02-28 11:54:55 CST; 57s ago Main PID: 16975 (java) Tasks: 38 (limit: 4515) Memory: 205.6M CPU: 10.772s CGroup: /system.slice/zkWeb.service └─16975 /oldboyedu/softwares/jdk1.8.0_291/bin/java -Xms128m -Xmx128m -jar /oldboyedu/softwares/zkWeb-v1.2.1.jar Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main MBeanExporter.java:433] o.s.j.e.a.AnnotationMBeanExporter --> Registering beans for JMX > Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main DirectJDKLog.java:180] o.a.coyote.http11.Http11NioProtocol --> Starting ProtocolHandler > Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main DirectJDKLog.java:180] o.a.tomcat.util.net.NioSelectorPool --> Using a shared selector f> Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main DirectJDKLog.java:180] o.a.c.c.C.[Tomcat].[localhost].[/] --> Initializing Spring Framew> Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main FrameworkServlet.java:494] o.s.web.servlet.DispatcherServlet --> FrameworkServlet 'dispa> Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main FrameworkServlet.java:509] o.s.web.servlet.DispatcherServlet --> FrameworkServlet 'dispa> Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main TomcatWebServer.java:206] o.s.b.w.e.tomcat.TomcatWebServer --> Tomcat started on port(s)> Feb 28 11:55:00 elk94 java[16975]: [2024-02-28 03:55:00 INFO main StartupInfoLogger.java:59] c.y.zkweb.ZkWebSpringBootApplication --> Started ZkWebSpringB> Feb 28 11:55:38 elk94 java[16975]: [2024-02-28 03:55:38 WARN main-SendThread(elk95:2181) ClientCnxn.java:1108] org.apache.zookeeper.ClientCnxn --> Client > Feb 28 11:55:50 elk94 java[16975]: [2024-02-28 03:55:50 WARN main-SendThread(elk96:2181) ClientCnxn.java:1108] org.apache.zookeeper.ClientCnxn --> Client > [root@elk94 ~]# [root@elk94 ~]# /oldboyedu/softwares/jdk1.8.0_291/bin/jmap -heap 16975 | grep MaxHeapSize MaxHeapSize = 134217728 (128.0MB) [root@elk94 ~]#
2.6 kafka 工作原理補充
副本 分割槽
2.7 topic的基礎管理
(1) 建立topic
方式1 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --topic oldboyedu-linux --partitions 3 --replication-factor 1 --create 建立一個3分割槽1副本對應的topic名稱為"oldboyedu-linux"。 方式2 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092 --topic oldboyedu-linux --partitions 3 --replication-factor 1 --create [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --topic oldboyedu-linux-001 --partitions 3 --replication-factor 3 --create 建立一個3分割槽3副本對應的topic名稱為"oldboyedu-linux-001"。 溫馨提示: 建立的副本數量不得大於存活的broker數量。(2) 檢視topic列表
檢視topic的名稱列表 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --list oldboyedu-linux oldboyedu-linux-001 [root@elk94 ~]# 單節點檢視 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092 --list oldboyedu-linux oldboyedu-linux-001 [root@elk94 ~]# 2.2 檢視指定topic的詳細列表 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --topic oldboyedu-linux --describe Topic: oldboyedu-linux TopicId: eVLUPmeFTGOapg9W-U5aow PartitionCount: 3 ReplicationFactor: 1 Configs: Topic: oldboyedu-linux Partition: 0 Leader: 94 Replicas: 94 Isr: 94 Topic: oldboyedu-linux Partition: 1 Leader: 95 Replicas: 95 Isr: 95 Topic: oldboyedu-linux Partition: 2 Leader: 96 Replicas: 96 Isr: 96 [root@elk94 ~]# 2.3 檢視所有topic的詳細資訊 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --describe Topic: oldboyedu-linux-001 TopicId: 3YcpgAgSTgy77eu15OGHLg PartitionCount: 3 ReplicationFactor: 3 Configs: Topic: oldboyedu-linux-001 Partition: 0 Leader: 96 Replicas: 96,94,95 Isr: 96,94,95 Topic: oldboyedu-linux-001 Partition: 1 Leader: 94 Replicas: 94,95,96 Isr: 94,95,96 Topic: oldboyedu-linux-001 Partition: 2 Leader: 95 Replicas: 95,96,94 Isr: 95,96,94 Topic: oldboyedu-linux TopicId: eVLUPmeFTGOapg9W-U5aow PartitionCount: 3 ReplicationFactor: 1 Configs: Topic: oldboyedu-linux Partition: 0 Leader: 94 Replicas: 94 Isr: 94 Topic: oldboyedu-linux Partition: 1 Leader: 95 Replicas: 95 Isr: 95 Topic: oldboyedu-linux Partition: 2 Leader: 96 Replicas: 96 Isr: 96 [root@elk94 ~]#(3) 修改topic
3.1 修改topic的分割槽數 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.96:9092 --topic oldboyedu-linux --alter --partitions 5 3.2 檢視修改後的分割槽數量 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --describe --topic oldboyedu-linux Topic: oldboyedu-linux TopicId: eVLUPmeFTGOapg9W-U5aow PartitionCount: 5 ReplicationFactor: 1 Configs: Topic: oldboyedu-linux Partition: 0 Leader: 94 Replicas: 94 Isr: 94 Topic: oldboyedu-linux Partition: 1 Leader: 95 Replicas: 95 Isr: 95 Topic: oldboyedu-linux Partition: 2 Leader: 96 Replicas: 96 Isr: 96 Topic: oldboyedu-linux Partition: 3 Leader: 94 Replicas: 94 Isr: 94 Topic: oldboyedu-linux Partition: 4 Leader: 95 Replicas: 95 Isr: 95 [root@elk94 ~]# 溫馨提示： 分割槽數只能調大，不能調小。 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.96:9092 --topic oldboyedu-linux --alter --partitions 2 Error while executing topic command : Topic currently has 5 partitions, which is higher than the requested 2. [2024-02-28 07:09:04,590] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: Topic currently has 5 partitions, which is higher than the requested 2. (org.apache.kafka.tools.TopicCommand) [root@elk94 ~]# 3.3 修改topic的副本數量[瞭解即可] https://www.cnblogs.com/yinzhengjie/p/9808125.html(4) 刪除topic
4.1 檢視現有的topic列表 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --list oldboyedu-linux oldboyedu-linux-001 [root@elk94 ~]# 4.2 刪除指定的topic [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.96:9092 --topic oldboyedu-linux --delete 4.3 再次檢視現有的topic列表 [root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092,10.0.0.95:9092,10.0.0.96:9092 --list oldboyedu-linux-001 [root@elk94 ~]# 常見的錯誤分析: 1.Replication factor: 4 larger than available brokers: 3. 錯誤原因: 建立的副本數量大於存活的節點數量。 解決方案: 將副本數量小於存活的節點數量或者加節點。 2.Topic currently has 5 partitions, which is higher than the requested 2 錯誤原因: 分割槽數只能調大，不能調小。 解決方案: 分割槽數只能比現有的分割槽數大，不能比現有的分割槽數小。
2.8 生產者和消費者
- 啟動生產者
[root@zookeeper1 ~]#kafka-console-producer.sh --bootstrap-server 10.0.0.201:9092 --topic oldboyedu-linux-001 >11111111111111 >22222222222222 >33333333333333 > 溫馨提示: 此時不會產生任何的新的topic，因為此時寫入的topic我們已經提前建立過。
- 啟動消費者
[root@zookeeper2 /oldboy/softwares/kafka_2.13-3.7.0]#kafka-console-consumer.sh --bootstrap-server 10.0.0.202:9092 --topic oldboyedu-linux-001 --from-beginning 11111111111111 22222222222222 33333333333333 55555555555 6666666666 --from-beginning 從開頭開始消費 不加 就是消費最新的 該消費者啟動時，如果生產者 生產了資料才會消費 馨提示: 此時啟動消費者後，會多出來一個新的topic，名稱為"__consumer_offsets"的topic。 該topic由系統維護，預設有50個分割槽，1個副本。該topic裡面儲存了每個消費者offset相關消費記錄。 [root@zookeeper2 /oldboy/softwares/kafka_2.13-3.7.0]#kafka-topics.sh --bootstrap-server 10.0.0.201:9092,10.0.0.202:9092,10.0.0.203:9092 --list __consumer_offsets oldboyedu-linux-001 # 檢視該主題 [root@zookeeper2 /oldboy/softwares/kafka_2.13-3.7.0]#kafka-topics.sh --bootstrap-server 10.0.0.201:9092,10.0.0.202:9092,10.0.0.203:9092 --describe --topic __consumer_offsets Topic: __consumer_offsets TopicId: ySdEquYYToSxjoGm4wT9aw PartitionCount: 50 ReplicationFactor: 1 Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600 Topic: __consumer_offsets Partition: 0 Leader: 203 Replicas: 203 Isr: 203 Topic: __consumer_offsets Partition: 1 Leader: 202 Replicas: 202 Isr: 202 Topic: __consumer_offsets Partition: 2 Leader: 201 Replicas: 201 Isr: 201 Topic: __consumer_offsets Partition: 3 Leader: 203 Replicas: 203 Isr: 203 Topic: __consumer_offsets Partition: 4 Leader: 202 Replicas: 202 Isr: 202 Topic: __consumer_offsets Partition: 5 Leader: 201 Replicas: 201 Isr: 201
- 消費者組概念(consumer groups)
(1) 消費者組包含一個或多個消費者
(2) 消費者：負責從kafka拉取資料的角色
(3) offset: 偏移量，記錄了消費者組消費的陣列位置
- 這個偏移量儲存在kafka叢集內建的topic，我們稱之為"__consumer_offsets"。
早期kafka 0.9.0-版本將這個偏移量儲存在zookeeper叢集。(4) rebalance: 重平衡，指的是當一個消費組的消費者數量發生變化時，消費者重新分配partitons的動作。
3.1 在消費資料的時候新增消費者組
1. 啟動生產者 此時不存在oldboyedu-linux 主題 ，直接指定會自動建立 溫馨提示: 寫入資料到kafka叢集時，可能會出現以上的報錯資訊，但是其會自動建立topic，我們忽略即可。 生產環境中，我強烈建議大家將其關閉。 [2024-10-20 18:29:15,651] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {oldboyedu-linux=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [root@zookeeper1 ~]#kafka-console-producer.sh --bootstrap-server 10.0.0.201:9092 --topic oldboyedu-linux >AAA [2024-10-20 18:29:15,651] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {oldboyedu-linux=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) 2. 啟動消費者 (1) 基於配置檔案指定消費者組 ①：修改配置檔案，linux90 指定消費者組名稱為linux90 [root@zookeeper2 /oldboy/softwares/kafka_2.13-3.7.0/config]#vim consumer.properties [root@zookeeper2 /oldboy/softwares/kafka_2.13-3.7.0/config]#grep ^group.id consumer.properties group.id=linux90 ②：啟動消費者時指定啊配置檔案 [root@zookeeper2 /oldboy/softwares/kafka_2.13-3.7.0/config]#kafka-console-consumer.sh --bootstrap-server 10.0.0.203:9092 --consumer.config $KAFKA_HOME/config/consumer.properties --topic oldboyedu-linux --from-beginning AAA ^CProcessed a total of 1 messages --bootstrap-server 10.0.0.203:9092 隨機指定或全部指定都可以 (2) 基於配置屬性指定消費者組啟動消費者 kafka-console-consumer.sh --bootstrap-server 10.0.0.95:9092 --topic oldboyedu-linux --consumer-property group.id=linux90 3. 檢視消費者組的詳細資訊 [root@zookeeper1 ~]#kafka-consumer-groups.sh --bootstrap-server 10.0.0.203:9092 --describe --group linux90 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID linux90 oldboyedu-linux 0 1 1 0 console-consumer-d7b34f12-6ebc-4a24-b2c4-5d00ea16497c /10.0.0.202 console-consumer # 檢視消費者組 [root@zookeeper1 ~]#kafka-consumer-groups.sh --bootstrap-server 10.0.0.203:9092 --list linux90 4. 停止消費者，並往生產者寫入兩條資料 # 停止消費者 4.1 停止消費者 將95節點的kafka-console-consumer.sh終止執行。 4.2 94節點傳送2條測試資料 [root@elk94 ~]# kafka-console-producer.sh --bootstrap-server 10.0.0.94:9092 --topic oldboyedu-linux ... >123456 >78910jqk > 4.3 再次檢視linux90這個消費者的資訊，發現有2條資料延遲。 [root@elk94 ~]# kafka-consumer-groups.sh --bootstrap-server 10.0.0.95:9092 --describe --group linux90 Consumer group 'linux90' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID linux90 oldboyedu-linux 0 2 4 2 - - - [root@elk94 ~]# 溫馨提示: 上面的“Consumer group 'linux90' has no active members”提示，說明當前環境的linux90這個組沒有消費者拿資料。注意，此時有2條資料延遲未消費。 5.再次基於配置屬性啟動消費者進行資料消費 5.1 在命令列中指定消費者去kafka節點拉取資料 [root@elk96 ~]# kafka-console-consumer.sh --bootstrap-server 10.0.0.95:9092 --topic oldboyedu-linux --group linux90 --from-beginning 123456 78910jqk 溫馨提示: 儘管我使用"--from-beginning"，也不會直接從頭開始拿資料，因為此時在kafka叢集中記錄了linux90這個消費者組最新的CURRENT-OFFSET為2。 5.2 再次檢視消費組資訊，發現資料被消費了。 [root@elk94 ~]# kafka-consumer-groups.sh --bootstrap-server 10.0.0.95:9092 --describe --group linux90 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID linux90 oldboyedu-linux 0 4 4 0 console-consumer-7d073e4e-c7be-48b9-b114-803f3e5377f2 /10.0.0.96 console-consumer [root@elk94 ~]# 6.停止消費者，並往生產者寫入2條測試資料 6.1 停止消費者 將96節點的kafka-console-consumer.sh終止執行。 6.2 94節點傳送2條測試資料 [root@elk94 ~]# kafka-console-producer.sh --bootstrap-server 10.0.0.94:9092 --topic oldboyedu-linux ... >ABCDEFG >abcdefg > 6.3 再次檢視linux90這個消費者的資訊，發現有2條資料延遲。 [root@elk94 ~]# kafka-consumer-groups.sh --bootstrap-server 10.0.0.95:9092 --describe --group linux90 Consumer group 'linux90' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID linux90 oldboyedu-linux 0 4 6 2 - - - [root@elk94 ~]# 7.再次啟動消費者並執行消費者組 7.1 基於配置屬性啟動消費者 [root@elk96 ~]# kafka-console-consumer.sh --bootstrap-server 10.0.0.95:9092 --topic oldboyedu-linux --consumer-property group.id=linux90 ABCDEFG abcdefg 7.2 再次檢視消費者的資訊 [root@elk94 ~]# kafka-consumer-groups.sh --bootstrap-server 10.0.0.95:9092 --describe --group linux90 GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID linux90 oldboyedu-linux 0 6 6 0 console-consumer-9bdf881c-1c1e-481f-846d-51ceefa533d6 /10.0.0.96 console-consumer [root@elk94 ~]#
2.9 kafka消費者rebalance案例
- kafka消費者rebalance案例
1.建立topic
[root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092 --topic oldboyedu-linux90 --partitions 3 --replication-factor 1 --create
Created topic oldboyedu-linux90.
[root@elk94 ~]#
[root@elk94 ~]# kafka-topics.sh --bootstrap-server 10.0.0.94:9092 --topic oldboyedu-linux90 --describe
Topic: oldboyedu-linux90 TopicId: 1ORIw6K4R--Te4AhxTOCpQ PartitionCount: 3 ReplicationFactor: 1 Configs:
Topic: oldboyedu-linux90 Partition: 0 Leader: 95 Replicas: 95 Isr: 95
Topic: oldboyedu-linux90 Partition: 1 Leader: 96 Replicas: 96 Isr: 96
Topic: oldboyedu-linux90 Partition: 2 Leader: 94 Replicas: 94 Isr: 94
[root@elk94 ~]#
2.啟動生產者並寫入測試資料
[root@elk94 ~]# kafka-console-producer.sh --bootstrap-server 10.0.0.94:9092 --topic oldboyedu-linux90
>123
>456
>
3.啟動第一個消費者
[root@elk95 ~]# kafka-console-consumer.sh --bootstrap-server 10.0.0.95:9092 --topic oldboyedu-linux90 --group jasonyin --from-beginning
123
456
4.觀察消費者組資訊
[root@elk94 ~]# kafka-consumer-groups.sh --bootstrap-server 10.0.0.95:9092 --describe --group jasonyin
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
jasonyin oldboyedu-linux90 0 0 0 0 console-consumer-62560ece-0fbf-474d-a3e4-919064661433 /10.0.0.95 console-consumer
jasonyin oldboyedu-linux90 1 2 2 0 console-consumer-62560ece-0fbf-474d-a3e4-919064661433 /10.0.0.95 console-consumer
jasonyin oldboyedu-linux90 2 0 0 0 console-consumer-62560ece-0fbf-474d-a3e4-919064661433 /10.0.0.95 console-consumer
[root@elk94 ~]#
溫馨提示:
不難發現，此時"jasonyin"這個消費者組僅有一個ID為"console-consumer-62560ece-0fbf-474d-a3e4-919064661433"的消費者。
5.啟動第二個消費者
[root@elk96 ~]# kafka-console-consumer.sh --bootstrap-server 10.0.0.96:9092 --topic oldboyedu-linux90 --group jasonyin
6.再次觀察消費者組資訊
[root@elk94 ~]# kafka-consumer-groups.sh --bootstrap-server 10.0.0.95:9092 --describe --group jasonyin
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
jasonyin oldboyedu-linux90 0 0 0 0 console-consumer-62560ece-0fbf-474d-a3e4-919064661433 /10.0.0.95 console-consumer
jasonyin oldboyedu-linux90 1 2 2 0 console-consumer-62560ece-0fbf-474d-a3e4-919064661433 /10.0.0.95 console-consumer
jasonyin oldboyedu-linux90 2 0 0 0 console-consumer-b0f4f974-57d2-4597-bd88-110e80f2a1ef /10.0.0.96 console-consumer
[root@elk94 ~]#
溫馨提示:
不難發現，此時對於"jasonyin"這個消費者組而言，有2個消費者，消費者數量發生了變化，因此會出發rebalance，也就是重新分配partition。
其中"console-consumer-62560ece-0fbf-474d-a3e4-919064661433"佔用0和1對應的partition。
而"console-consumer-b0f4f974-57d2-4597-bd88-110e80f2a1ef"佔用2對應的partition
7.再次啟動多個生產者寫入測試資料
[root@elk94 ~]# kafka-console-producer.sh --bootstrap-server 10.0.0.94:9092 --topic oldboyedu-linux90
>11111111111111111111111
>2222222222222222222222222
>
溫馨提示:
可能啟動2個生產者就能看到效果，也可以需要多啟動幾個生產者才能在96節點看到輸出的效果喲~
[root@elk96 ~]# kafka-console-consumer.sh --bootstrap-server 10.0.0.96:9092 --topic oldboyedu-linux90 --group jasonyin
11111111111111111111111
2222222222222222222222222
8.再次觀察消費者組資訊
[root@elk94 ~]# kafka-consumer-groups.sh --bootstrap-server 10.0.0.95:9092 --describe --group jasonyin
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
jasonyin oldboyedu-linux90 0 0 0 0 console-consumer-62560ece-0fbf-474d-a3e4-919064661433 /10.0.0.95 console-consumer
jasonyin oldboyedu-linux90 1 20 20 0 console-consumer-62560ece-0fbf-474d-a3e4-919064661433 /10.0.0.95 console-consumer
jasonyin oldboyedu-linux90 2 2 2 0 console-consumer-b0f4f974-57d2-4597-bd88-110e80f2a1ef /10.0.0.96 console-consumer
[root@elk94 ~]#