org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:6

偶然一次停電，導致機房機器重啟，於是很多機器靜態IP衝突失效。好不容易把網路調整好，Hbase叢集卻無法正常啟動，Hadoop卻可以正常使用。大致情況如下：
正常啟動Hadoop和zookeeper，
然後啟動Hbase，jps檢視一切正常。
當開啟web頁面（60010）檢視的時候卻出現500錯誤，錯誤大致如下：


HTTP ERROR: 500
Trying to contact region server null for region , row ", but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1


RequestURI=/master.jsp


Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row ", but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1


at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
 at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
 at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
 at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
 at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
 at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:324)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
 at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)






檢視Hbase   master的日誌，大致錯誤如下：  


 
2011-12-08 14:34:54,305 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.
2011-12-08 14:34:54,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2011-12-08 14:34:55,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2011-12-08 14:34:55,307 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.10.11.184:60020
2011-12-08 14:34:55,308 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.
2011-12-08 14:35:15,404 INFO org.apache.hadoop.hbase.master.ServerManager: 4 region servers, 0 dead, average load 0.3333333333333333
2011-12-08 14:35:15,484 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.10.11.184:60020, regionname: -ROOT-,,0, startKey: <>}
2011-12-08 14:35:15,485 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.
2011-12-08 14:35:15,485 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:429)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:918)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:934)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:173)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)


日誌呈現結果大致是無法連線到  /10.10.11.184:60020 ,不能連線到regionServer
於是檢視regionServer的日誌，初次一看感覺沒什麼錯誤，就沒怎麼注意，然後繼續糾結在其他地方……後來多次失敗後重新審視日誌的時候才發現了一些端倪，如下


2011-12-08 22:45:02,365 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Created ZNode /hbase/rs/1323355502342 with data 0:0:0:0:0:0:0:1:60020
2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.regionserver.address=10.10.11.184
2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: fs.default.name=hdfs://hadoop5-virtual-machine:9000/hbase
2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://hadoop5-virtual-machine:9000/hbase
2011-12-08 22:45:02,395 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us address to use. Was=0:0:0:0:0:0:0:1:60020, Now=10.10.11.184
2011-12-08 22:45:02,510 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir hdfs://hadoop5-virtual-machine:9000/hbase/.logs/hadoop4-virtual-machine,60020,1323355502342
2011-12-08 22:45:02,521 INFO org.apache.hadoop.hbase.regionserver.HLog: HLog configuration: blocksize=67108864, rollsize=63753420, enabled=true, flushlogentries=100, optionallogflushinternal=10000ms
2011-12-08 22:45:02,561 INFO org.apache.hadoop.hbase.regionserver.HLog: New hlog /hbase/.logs/hadoop4-virtual-machine,60020,1323355502342/hlog.dat.1323355502522
2011-12-08 22:45:02,566 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=RegionServer, sessionId=regionserver/0:0:0:0:0:0:0:1:60020   


 然後我在slave節點下jps檢視，發現hbase相關的regionServer的程式掛掉了，而master的程式還在。


由上藍色標記字型看出：為什麼會出現IP6的地址呢？檢查配置檔案也沒有出錯，因為所有的配置檔案都是用主機名替代了IP地址。於是就想到了是不是hosts檔案的對映出錯了。
開啟/etc/hosts檔案，果不其然，regionServer的主機名對映的是 ::1，


我記得這個主機名和當前IP的對映應該是網路卡初始化的時候由NetworkManager自動加上去的，


把 ::1改成 regionServer的實際地址，重啟Hbase，訪問WEB站點，成功啟動！
回顧整個hadoop叢集啟動失敗這個過程，先是斷電，然後是靜態IP失效，問題應該就是在這個時候產生的，在實效的同試hosts檔案被修改。

其實HBASE叢集配置相當簡單，但由於一些小小的失誤，往往是我們自以為不會出錯的地方，檢視log的時候也不細心，然後以至於浪費我們很多的時間。至於IP與主機名對映的問題是我們配置叢集的時候最容易忽視的地方，大家在下次碰到這類相關問題的時候不妨去看看hosts檔案。

轉自此連結