zookeeper 叢集安裝(單點與分散式成功安裝)

破棉襖發表於2014-08-07

ZooKeeper是一個分散式開源框架,提供了協調分散式應用的基本服務,它向外部應用暴露一組通用服務——分散式同步(Distributed Synchronization)、命名服務(Naming Service)、叢集維護(Group Maintenance)等,簡化分散式應用協調及其管理的難度,提供高效能的分散式服務。ZooKeeper本身可以以Standalone模式安裝執行,不過它的長處在於透過分散式ZooKeeper叢集(一個Leader,多個Follower),基於一定的策略來保證ZooKeeper叢集的穩定性和可用性,從而實現分散式應用的可靠性。

有關ZooKeeper的介紹,網上很多,也可以參考文章後面,我整理的一些相關連結。

下面,我們簡單說明一下ZooKeeper的配置。


ZooKeeper Standalone模式


從Apache網站上(zookeeper.apache.org)下載ZooKeeper軟體包,我選擇了3.3.4版本的(zookeeper-3.3.4.tar.gz),在一臺Linux機器上安裝非常容易,只需要解壓縮後,簡單配置一下即可以啟動ZooKeeper伺服器程式。

將zookeeper-3.3.4/conf目錄下面的 zoo_sample.cfg修改為zoo.cfg,配置檔案內容如下所示:


[plain] view plaincopy
  1. tickTime=2000  
  2. dataDir=/home/hadoop/storage/zookeeper  
  3. clientPort=2181  
  4. initLimit=5  
  5. syncLimit=2  
上面各個配置引數的含義也非常簡單,引用如下所示:



[plain] view plaincopy
  1. tickTime —— the basic time unit in milliseconds used by ZooKeeper. It is used to do heartbeats and the minimum session timeout will be twice the tickTime.  
  2. dataDir —— the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database.  
  3. clientPort —— the port to listen for client connections  
下面啟動ZooKeeper伺服器程式:



[plain] view plaincopy
  1. cd zookeeper-3.3.4/  
  2. bin/zkServer.sh start  
透過jps命令可以檢視ZooKeeper伺服器程式,名稱為QuorumPeerMain。


在客戶端連線ZooKeeper伺服器,執行如下命令:


[plain] view plaincopy
  1. bin/zkCli.sh -server dynamic:2181  
上面dynamic是我的主機名,如果在本機執行,則執行如下命令即可:
[plain] view plaincopy
  1. bin/zkCli.sh  
客戶端連線資訊如下所示:



接著,可以使用help檢視Zookeeper客戶端可以使用的基本操作命令。



ZooKeeper Distributed模式


ZooKeeper分散式模式安裝(ZooKeeper叢集)也比較容易,這裡說明一下基本要點。

首先要明確的是,ZooKeeper叢集是一個獨立的分散式協調服務叢集,“獨立”的含義就是說,如果想使用ZooKeeper實現分散式應用的協調與管理,簡化協調與管理,任何分散式應用都可以使用,這就要歸功於Zookeeper的資料模型(Data Model)和層次名稱空間(Hierarchical Namespace)結構,詳細可以參考。在設計你的分散式應用協調服務時,首要的就是考慮如何組織層次名稱空間。

下面說明分散式模式的安裝配置,過程如下所示:

第一步:主機名稱到IP地址對映配置

ZooKeeper叢集中具有兩個關鍵的角色:Leader和Follower。叢集中所有的結點作為一個整體對分散式應用提供服務,叢集中每個結點之間都互相連線,所以,在配置的ZooKeeper叢集的時候,每一個結點的host到IP地址的對映都要配置上叢集中其它結點的對映資訊。

例如,我的ZooKeeper叢集中每個結點的配置,以slave-01為例,/etc/hosts內容如下所示:


[plain] view plaincopy
  1. 192.168.0.179   slave-01  
  2. 192.168.0.178   slave-02  
  3. 192.168.0.177   slave-03  


ZooKeeper採用一種稱為Leader election的選舉演算法。在整個叢集執行過程中,只有一個Leader,其他的都是Follower,如果ZooKeeper叢集在執行過程中Leader出了問題,系統會採用該演算法重新選出一個Leader。因此,各個結點之間要能夠保證互相連線,必須配置上述對映。

ZooKeeper叢集啟動的時候,會首先選出一個Leader,在Leader election過程中,某一個滿足選舉算的結點就能成為Leader。整個叢集的架構可以參考。

第二步:修改ZooKeeper配置檔案

在其中一臺機器(slave-01)上,解壓縮zookeeper-3.3.4.tar.gz,修改配置檔案conf/zoo.cfg,內容如下所示:


[plain] view plaincopy
  1. tickTime=2000  
  2. dataDir=/home/hadoop/storage/zookeeper  
  3. clientPort=2181  
  4. initLimit=5  
  5. syncLimit=2  
  6. server.1=slave-01:2888:3888  
  7. server.2=slave-02:2888:3888  
  8. server.3=slave-03:2888:3888  
上述配置內容說明,可以參考。


第三步:遠端複製分發安裝檔案

上面已經在一臺機器slave-01上配置完成ZooKeeper,現在可以將該配置好的安裝檔案遠端複製到叢集中的各個結點對應的目錄下:


[plain] view plaincopy
  1. cd /home/hadoop/installation/  
  2. scp -r zookeeper-3.3.4/ hadoop@slave-02:/home/hadoop/installation/  
  3. scp -r zookeeper-3.3.4/ hadoop@slave-03:/home/hadoop/installation/  
第四步:設定myid


在我們配置的dataDir指定的目錄下面,建立一個myid檔案,裡面內容為一個數字,用來標識當前主機,conf/zoo.cfg檔案中配置的server.X中X為什麼數字,則myid檔案中就輸入這個數字,例如:

[plain] view plaincopy
  1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ echo "1" > /home/hadoop/storage/zookeeper/myid  
  2. hadoop@slave-02:~/installation/zookeeper-3.3.4$ echo "2" > /home/hadoop/storage/zookeeper/myid  
  3. hadoop@slave-03:~/installation/zookeeper-3.3.4$ echo "3" > /home/hadoop/storage/zookeeper/myid  
按照上述進行配置即可。


第五步:啟動ZooKeeper叢集

在ZooKeeper叢集的每個結點上,執行啟動ZooKeeper服務的指令碼,如下所示:

[plain] view plaincopy
  1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ bin/zkServer.sh start  
  2. hadoop@slave-02:~/installation/zookeeper-3.3.4$ bin/zkServer.sh start  
  3. hadoop@slave-03:~/installation/zookeeper-3.3.4$ bin/zkServer.sh start  
以結點slave-01為例,日誌如下所示:
[plain] view plaincopy
  1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ tail -500f zookeeper.out   
  2. 2012-01-08 06:51:19,117 - INFO  [main:QuorumPeerConfig@90] - Reading configuration from: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
  3. 2012-01-08 06:51:19,133 - INFO  [main:QuorumPeerConfig@310] - Defaulting to majority quorums  
  4. 2012-01-08 06:51:19,167 - INFO  [main:QuorumPeerMain@119] - Starting quorum peer  
  5. 2012-01-08 06:51:19,227 - INFO  [main:NIOServerCnxn$Factory@143] - binding to port 0.0.0.0/0.0.0.0:2181  
  6. 2012-01-08 06:51:19,277 - INFO  [main:QuorumPeer@819] - tickTime set to 2000  
  7. 2012-01-08 06:51:19,278 - INFO  [main:QuorumPeer@830] - minSessionTimeout set to -1  
  8. 2012-01-08 06:51:19,279 - INFO  [main:QuorumPeer@841] - maxSessionTimeout set to -1  
  9. 2012-01-08 06:51:19,281 - INFO  [main:QuorumPeer@856] - initLimit set to 5  
  10. 2012-01-08 06:51:19,347 - INFO  [Thread-1:QuorumCnxManager$Listener@473] - My election bind port: 3888  
  11. 2012-01-08 06:51:19,393 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@621] - LOOKING  
  12. 2012-01-08 06:51:19,396 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@663] - New election. My id =  1, Proposed zxid = 0  
  13. 2012-01-08 06:51:19,400 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 1 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)  
  14. 2012-01-08 06:51:19,416 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
  15. java.net.ConnectException: Connection refused  
  16.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  17.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  18.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  19.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  20.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)  
  21.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)  
  22.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)  
  23.         at java.lang.Thread.run(Thread.java:662)  
  24. 2012-01-08 06:51:19,420 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
  25. java.net.ConnectException: Connection refused  
  26.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  27.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  28.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  29.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  30.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)  
  31.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)  
  32.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)  
  33.         at java.lang.Thread.run(Thread.java:662)  
  34. 2012-01-08 06:51:19,612 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
  35. java.net.ConnectException: Connection refused  
  36.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  37.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  38.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  39.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  40.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  41.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  42.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  43. 2012-01-08 06:51:19,615 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
  44. java.net.ConnectException: Connection refused  
  45.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  46.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  47.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  48.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  49.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  50.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  51.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  52. 2012-01-08 06:51:19,616 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 400  
  53. 2012-01-08 06:51:20,019 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
  54. java.net.ConnectException: Connection refused  
  55.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  56.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  57.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  58.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  59.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  60.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  61.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  62. 2012-01-08 06:51:20,021 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
  63. java.net.ConnectException: Connection refused  
  64.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  65.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  66.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  67.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  68.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  69.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  70.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  71. 2012-01-08 06:51:20,022 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 800  
  72. 2012-01-08 06:51:20,825 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
  73. java.net.ConnectException: Connection refused  
  74.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  75.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  76.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  77.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  78.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  79.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  80.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  81. 2012-01-08 06:51:20,827 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
  82. java.net.ConnectException: Connection refused  
  83.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  84.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  85.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  86.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  87.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  88.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  89.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  90. 2012-01-08 06:51:20,828 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 1600  
  91. 2012-01-08 06:51:22,435 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/192.168.0.178:3888  
  92. java.net.ConnectException: Connection refused  
  93.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  94.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  95.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  96.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  97.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  98.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  99.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  100. 2012-01-08 06:51:22,439 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
  101. java.net.ConnectException: Connection refused  
  102.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  103.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  104.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  105.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  106.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  107.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  108.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  109. 2012-01-08 06:51:22,441 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 3200  
  110. 2012-01-08 06:51:22,945 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 2 (n.sid), LOOKING (my state)  
  111. 2012-01-08 06:51:22,946 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@721] - Updating proposal  
  112. 2012-01-08 06:51:22,949 - WARN  [WorkerSender Thread:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/192.168.0.177:3888  
  113. java.net.ConnectException: Connection refused  
  114.         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)  
  115.         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)  
  116.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)  
  117.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  118.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:340)  
  119.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:360)  
  120.         at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:333)  
  121.         at java.lang.Thread.run(Thread.java:662)  
  122. 2012-01-08 06:51:22,951 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 2 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 1 (n.sid), LOOKING (my state)  
  123. 2012-01-08 06:51:23,156 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumPeer@643] - FOLLOWING  
  124. 2012-01-08 06:51:23,170 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@80] - TCP NoDelay set to: true  
  125. 2012-01-08 06:51:23,206 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT  
  126. 2012-01-08 06:51:23,207 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:host.name=slave-01  
  127. 2012-01-08 06:51:23,207 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.version=1.6.0_30  
  128. 2012-01-08 06:51:23,208 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.vendor=Sun Microsystems Inc.  
  129. 2012-01-08 06:51:23,208 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.home=/home/hadoop/installation/jdk1.6.0_30/jre  
  130. 2012-01-08 06:51:23,209 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.class.path=/home/hadoop/installation/zookeeper-3.3.4/bin/../build/classes:/home/hadoop/installation/zookeeper-3.3.4/bin/../build/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../zookeeper-3.3.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/log4j-1.2.15.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/jline-0.9.94.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-lang-2.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-collections-3.2.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-cli-1.1.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-tasks-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-core-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../src/java/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../conf:/home/hadoop/installation/jdk1.6.0_30/lib/*.jar:/home/hadoop/installation/jdk1.6.0_30/jre/lib/*.jar  
  131. 2012-01-08 06:51:23,210 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.library.path=/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386/client:/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386:/home/hadoop/installation/jdk1.6.0_30/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib  
  132. 2012-01-08 06:51:23,210 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.io.tmpdir=/tmp  
  133. 2012-01-08 06:51:23,212 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:java.compiler=  
  134. 2012-01-08 06:51:23,212 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.name=Linux  
  135. 2012-01-08 06:51:23,212 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.arch=i386  
  136. 2012-01-08 06:51:23,213 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:os.version=3.0.0-14-generic  
  137. 2012-01-08 06:51:23,213 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.name=hadoop  
  138. 2012-01-08 06:51:23,214 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.home=/home/hadoop  
  139. 2012-01-08 06:51:23,214 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Environment@97] - Server environment:user.dir=/home/hadoop/installation/zookeeper-3.3.4  
  140. 2012-01-08 06:51:23,223 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@151] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /home/hadoop/storage/zookeeper/version-2 snapdir /home/hadoop/storage/zookeeper/version-2  
  141. 2012-01-08 06:51:23,339 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@294] - Getting a snapshot from leader  
  142. 2012-01-08 06:51:23,358 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:Learner@325] - Setting leader epoch 1  
  143. 2012-01-08 06:51:23,358 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FileTxnSnapLog@254] - Snapshotting: 0  
  144. 2012-01-08 06:51:25,511 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 1 (n.round), LOOKING (n.state), 3 (n.sid), FOLLOWING (my state)  
  145. 2012-01-08 06:51:42,584 - INFO  [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 0 (n.zxid), 2 (n.round), LOOKING (n.state), 3 (n.sid), FOLLOWING (my state)  
我啟動的順序是slave-01>slave-02>slave-03,由於ZooKeeper叢集啟動的時候,每個結點都試圖去連線叢集中的其它結點,先啟動的肯定連不上後面還沒啟動的,所以上面日誌前面部分的異常是可以忽略的。透過後面部分可以看到,叢集在選出一個Leader後,最後穩定了。


其他結點可能也出現類似問題,屬於正常。

第六步:安裝驗證

可以透過ZooKeeper的指令碼來檢視啟動狀態,包括叢集中各個結點的角色(或是Leader,或是Follower),如下所示,是在ZooKeeper叢集中的每個結點上查詢的結果:


[plain] view plaincopy
  1. hadoop@slave-01:~/installation/zookeeper-3.3.4$ bin/zkServer.sh status  
  2. JMX enabled by default  
  3. Using config: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
  4. Mode: follower  
  5.   
  6. hadoop@slave-02:~/installation/zookeeper-3.3.4$  bin/zkServer.sh status  
  7. JMX enabled by default  
  8. Using config: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
  9. Mode: leader  
  10.   
  11. hadoop@slave-03:~/installation/zookeeper-3.3.4$  bin/zkServer.sh status  
  12. JMX enabled by default  
  13. Using config: /home/hadoop/installation/zookeeper-3.3.4/bin/../conf/zoo.cfg  
  14. Mode: follower  
透過上面狀態查詢結果可見,slave-02是叢集的Leader,其餘的兩個結點是Follower。

另外,可以透過客戶端指令碼,連線到ZooKeeper叢集上。對於客戶端來說,ZooKeeper是一個整體(ensemble),連線到ZooKeeper叢集實際上感覺在獨享整個叢集的服務,所以,你可以在任何一個結點上建立到服務叢集的連線,例如:

[plain] view plaincopy
  1. hadoop@slave-03:~/installation/zookeeper-3.3.4$ bin/zkCli.sh -server slave-01:2181  
  2. Connecting to slave-01:2181  
  3. 2012-01-08 07:14:21,068 - INFO  [main:Environment@97] - Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT  
  4. 2012-01-08 07:14:21,080 - INFO  [main:Environment@97] - Client environment:host.name=slave-03  
  5. 2012-01-08 07:14:21,085 - INFO  [main:Environment@97] - Client environment:java.version=1.6.0_30  
  6. 2012-01-08 07:14:21,089 - INFO  [main:Environment@97] - Client environment:java.vendor=Sun Microsystems Inc.  
  7. 2012-01-08 07:14:21,095 - INFO  [main:Environment@97] - Client environment:java.home=/home/hadoop/installation/jdk1.6.0_30/jre  
  8. 2012-01-08 07:14:21,104 - INFO  [main:Environment@97] - Client environment:java.class.path=/home/hadoop/installation/zookeeper-3.3.4/bin/../build/classes:/home/hadoop/installation/zookeeper-3.3.4/bin/../build/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../zookeeper-3.3.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/log4j-1.2.15.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/jline-0.9.94.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-lang-2.4.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-collections-3.2.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/commons-cli-1.1.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-tasks-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../lib/apache-rat-core-0.6.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../src/java/lib/*.jar:/home/hadoop/installation/zookeeper-3.3.4/bin/../conf:/home/hadoop/installation/jdk1.6.0_30/lib/*.jar:/home/hadoop/installation/jdk1.6.0_30/jre/lib/*.jar  
  9. 2012-01-08 07:14:21,111 - INFO  [main:Environment@97] - Client environment:java.library.path=/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386/client:/home/hadoop/installation/jdk1.6.0_30/jre/lib/i386:/home/hadoop/installation/jdk1.6.0_30/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib  
  10. 2012-01-08 07:14:21,116 - INFO  [main:Environment@97] - Client environment:java.io.tmpdir=/tmp  
  11. 2012-01-08 07:14:21,124 - INFO  [main:Environment@97] - Client environment:java.compiler=  
  12. 2012-01-08 07:14:21,169 - INFO  [main:Environment@97] - Client environment:os.name=Linux  
  13. 2012-01-08 07:14:21,175 - INFO  [main:Environment@97] - Client environment:os.arch=i386  
  14. 2012-01-08 07:14:21,177 - INFO  [main:Environment@97] - Client environment:os.version=3.0.0-14-generic  
  15. 2012-01-08 07:14:21,185 - INFO  [main:Environment@97] - Client environment:user.name=hadoop  
  16. 2012-01-08 07:14:21,188 - INFO  [main:Environment@97] - Client environment:user.home=/home/hadoop  
  17. 2012-01-08 07:14:21,190 - INFO  [main:Environment@97] - Client environment:user.dir=/home/hadoop/installation/zookeeper-3.3.4  
  18. 2012-01-08 07:14:21,197 - INFO  [main:ZooKeeper@379] - Initiating client connection, connectString=slave-01:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@bf32c  
  19. 2012-01-08 07:14:21,305 - INFO  [main-SendThread():ClientCnxn$SendThread@1061] - Opening socket connection to server slave-01/192.168.0.179:2181  
  20. Welcome to ZooKeeper!  
  21. 2012-01-08 07:14:21,376 - INFO  [main-SendThread(slave-01:2181):ClientCnxn$SendThread@950] - Socket connection established to slave-01/192.168.0.179:2181, initiating session  
  22. JLine support is enabled  
  23. [zk: slave-01:2181(CONNECTING) 0] 2012-01-08 07:14:21,872 - INFO  [main-SendThread(slave-01:2181):ClientCnxn$SendThread@739] - Session establishment complete on server slave-01/192.168.0.179:2181, sessionid = 0x134bdcd6b730000, negotiated timeout = 30000  
  24.   
  25. WATCHER::  
  26.   
  27. WatchedEvent state:SyncConnected type:None path:null  
  28.   
  29. [zk: slave-01:2181(CONNECTED) 0] ls /  
  30. [zookeeper]  
當前根路徑為/zookeeper。


總結說明



主機名與IP地址對映配置問題

啟動ZooKeeper叢集時,如果ZooKeeper叢集中slave-01結點的日誌出現如下錯誤:


[plain] view plaincopy
  1. java.net.SocketTimeoutException  
  2.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:109)  
  3.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  4.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  5.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  6.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  7. 2012-01-08 06:37:46,026 - INFO  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:FastLeaderElection@697] - Notification time out: 6400  
  8. 2012-01-08 06:37:57,431 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 2 at election address slave-02/202.106.199.35:3888  
  9. java.net.SocketTimeoutException  
  10.         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:109)  
  11.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:371)  
  12.         at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404)  
  13.         at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:688)  
  14.         at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:622)  
  15. 2012-01-08 06:38:02,442 - WARN  [QuorumPeer:/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@384] - Cannot open channel to 3 at election address slave-03/202.106.199.35:3888  

很顯然,slave-01在啟動時連線叢集中其他結點(slave-02、slave-03)時,主機名對映的IP與我們實際配置的不一致,所以叢集中各個結點之間無法建立鏈路,整個ZooKeeper叢集啟動是失敗的。

上面錯誤日誌中slave-02/202.106.199.35:3888實際應該是slave-02/202.192.168.0.178:3888就對了,但是在進行域名解析的時候對映有問題,修改每個結點的/etc/hosts檔案,將ZooKeeper叢集中所有結點主機名到IP地址的對映配置上。


原文地址:http://www.blogjava.net/hello-yun/archive/2012/05/03/377250.html

來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/29754888/viewspace-1247258/,如需轉載,請註明出處,否則將追究法律責任。

相關文章