一.場景:
spark透過phoenix 讀取hbase表，其實說白了先要去Zookeeper建立connection。

二.程式碼:

點選(此處)摺疊或開啟

val zkUrl = "192.168.100.39,192.168.100.40,192.168.100.41:2181"

val formatStr = "org.apache.phoenix.spark"

val oms_orderinfoDF = spark.read.format(formatStr)

  .options(Map("table" -> "oms_orderinfo", "zkUrl" -> zkUrl))

  .load

三.檢視SparkJob日誌:

點選(此處)摺疊或開啟

17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop40/192.168.100.40:2181. Will not attempt to authenticate using SASL (unknown error)

17/10/24 03:25:25 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.100.48:35952, server: hadoop40/192.168.100.40:2181

17/10/24 03:25:25 WARN zookeeper.ClientCnxn: Session 0x0 for server hadoop40/192.168.100.40:2181, unexpected error, closing socket connection and attempting reconnect

java.io.IOException: Connection reset by peer

    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)

    at sun.nio.ch.IOUtil.read(IOUtil.java:192)

    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)

    at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)

    at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)

    at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

17/10/24 03:25:25 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://nameservice1/user/hdfs/.sparkStaging/application_1507703377455_4854

17/10/24 03:25:25 INFO util.ShutdownHookManager: Shutdown hook called

主要錯誤是“java.io.IOException: Connection reset by peer”，其實意思很明白了Zookeeper將connection closing.
至於為什麼會被關閉connection？那麼去檢查一下hadoop40機器的zookeeper的日誌就會發現一些有用的資訊。

四.檢視Zookeeper日誌:

點選(此處)摺疊或開啟

2017-10-24 03:25:22,498 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

2017-10-24 03:25:25,588 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

2017-10-24 03:25:25,819 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: Too many connections from /192.168.100.40 - max is 500

2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception

EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee09897d, likely client has closed socket

    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)

    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

    at java.lang.Thread.run(Thread.java:745)

2017-10-24 03:25:26,092 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47084 which had sessionid 0x15ed091ee09897d

2017-10-24 03:25:26,092 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception

EndOfStreamException: Unable to read additional data from client sessionid 0x15ed091ee098981, likely client has closed socket

    at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:231)

    at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)

    at java.lang.Thread.run(Thread.java:745)

2017-10-24 03:25:26,093 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.100.40:47304 which had sessionid 0x15ed091ee098981

2017-10-24 03:25:26,093 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception

這個時候我們發現"Too many connections from /192.168.100.48 - max is 500",
那就說明這臺hadoop40機器的zookeeper的當前connection數已經超過配置的500了，所以新的connection必然被closed，

五.解決方法:
那麼我們只需去zookeeper的配置修改一下引數maxClientCnxns調大到1000，重啟zookeeper服務生效配置即可。

java.io.IOException: Connection reset by peer

相關文章