問題描述；

叢集 start-all.sh 啟動後，發現少了一個namenode。

單節點namenode啟動成功，但是啟動其他節點後，jps發現namenode消失了

格式化namenode報錯

問題排查與解析

解決思路

1、單節點啟動排查。

整體關閉stop-all.sh，

單節點啟動 hadoop-daemon.sh start namenode；jps namenode單個節點啟動成功，

再次啟動下一個hadoop-daemon.sh start datanode;jps後，發現namenode 消失了

說明namenode啟動失敗

2、透過logs檢視問題

2022-06-02 06:30:39,722 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.io.IOException: Failed to load FSImage file, see error(s) above for more info.

at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)

at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)

at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)

報錯的原因：fsimage的映象檔案載入失敗

基本定位到問題

3、解決方案

試圖透過格式化 hadoop namenode -format解決，結果仍舊報錯

通namenode(active)的節點，複製 dfs的檔案到掛掉的節點。scp -r dfs/ hadoop001:`pwd`

[root@hadoop002 data]# scp -r dfs/ hadoop001:`pwd`

root@hadoop001's password: (輸入密碼)

再次start-all.sh

[root@hadoop001 logs]# start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [hadoop001 hadoop002]

hadoop001: starting namenode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-namenode-hadoop001.out

hadoop002: starting namenode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-namenode-hadoop002.out

hadoop001: starting datanode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop001.out

hadoop003: starting datanode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop003.out

hadoop002: starting datanode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop002.out

Starting journal nodes [hadoop001 hadoop002 hadoop003]

hadoop003: starting journalnode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-journalnode-hadoop003.out

hadoop001: starting journalnode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-journalnode-hadoop001.out

hadoop002: starting journalnode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-journalnode-hadoop002.out

Starting ZK Failover Controllers on NN hosts [hadoop001 hadoop002]

hadoop001: starting zkfc, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-zkfc-hadoop001.out

hadoop002: starting zkfc, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-zkfc-hadoop002.out

starting yarn daemons

starting resourcemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-resourcemanager-hadoop001.out

hadoop002: starting nodemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop002.out

hadoop001: starting nodemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop001.out

hadoop003: starting nodemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop003.out

透過jps檢視

[root@hadoop001 logs]# jps

106723 DFSZKFailoverController

105860 NameNode

106052 DataNode

107065 NodeManager

106427 JournalNode

106939 ResourceManager

12126 QuorumPeerMain

107182 Jps

問題解決！

namenode單節點啟動成功後自動消失/格式化失敗/fsimage載入失敗

相關文章