weblogic連線池重置（Connection reset）問題解決方案

neverinit發表於2017-08-17
Web
業務系統在高峰時期報錯，需要對weblogic的日誌進行分析，檢視問題出現的原因以及解決方案。
在weblogic的AdminServer.log中的報錯資訊如下：
  1. ####<2017-8-8 下午07時29分43秒 CST> <Warning> <JDBC> <scyy1> <AdminServer> <pool-4514-thread-165> <<anonymous>> <> <> <1502191783426> <BEA-001129> <Received exception while creating connection for pool "CAMSCORE": IO 錯誤: Connection reset.>
  2. ####<2017-8-8 下午07時29分43秒 CST> <Warning> <JDBC> <scyy1> <AdminServer> <pool-4514-thread-252> <<anonymous>> <> <> <1502191783427> <BEA-001129> <Received exception while creating connection for pool "CAMSCORE": IO 錯誤: Connection reset.>
  3. ####<2017-8-8 下午07時29分43秒 CST> <Warning> <JDBC> <scyy1> <AdminServer> <pool-4514-thread-243> <<anonymous>> <> <> <1502191783428> <BEA-001129> <Received exception while creating connection for pool "CAMSCORE": IO 錯誤: Connection reset.>
  4. ####<2017-8-8 下午07時29分43秒 CST> <Warning> <JDBC> <scyy1> <AdminServer> <pool-4514-thread-55> <<anonymous>> <> <> <1502191783428> <BEA-001129> <Received exception while creating connection for pool "CAMSCORE": IO 錯誤: Connection reset.>
  5. ####<2017-8-8 下午07時29分43秒 CST> <Info> <JDBC> <scyy1> <AdminServer> <pool-4514-thread-55> <<anonymous>> <> <> <1502191783429> <BEA-001156> <Stack trace associated with message 001129 follows:

  7. java.sql.SQLRecoverableException: IO 錯誤: Connection reset
  8.     at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:467)
  9.     at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
  10.     at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
  11.     at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  12.     at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
  13.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:357)
  14.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:239)
  15.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)
  16.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1227)
  17.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:451)
  18.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:343)
  19.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
  20.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
  21.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
  22.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
  23.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:158)
  24.     at weblogic.jdbc.pool.Driver.connect(Driver.java:132)
  25.     at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  26.     at weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  27.     at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  28.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  29.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  30.     at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  31.     at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  32.     at com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  33.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  34.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  35.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.update(SourceFile:167)
  36.     at com.bocom.cams.core.base.commons.BaseDao.update(BaseDao.java:37)
  37.     at com.bocom.cams.core.base.commons.PersistableDao.update(PersistableDao.java:46)
  38.     at com.bocom.cams.core.base.commons.PersistableService.modify(PersistableService.java:101)
  39.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:499)
  40.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  41.     at com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  42.     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  43.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  44.     at java.lang.Thread.run(Thread.java:662)
  45. Caused By: java.net.SocketException: Connection reset
  46.     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
  47.     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
  48.     at oracle.net.ns.DataPacket.send(DataPacket.java:210)
  49.     at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:230)
  50.     at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:312)
  51.     at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
  52.     at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
  53.     at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
  54.     at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
  55.     at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
  56.     at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
  57.     at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
  58.     at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
  59.     at oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:404)
  60.     at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:385)
  61.     at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
  62.     at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
  63.     at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  64.     at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
  65.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:357)
  66.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:239)
  67.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)
  68.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1227)
  69.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:451)
  70.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:343)
  71.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
  72.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
  73.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
  74.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
  75.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:158)
  76.     at weblogic.jdbc.pool.Driver.connect(Driver.java:132)
  77.     at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  78.     at weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  79.     at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  80.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  81.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  82.     at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  83.     at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  84.     at com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  85.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  86.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  87.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.update(SourceFile:167)
  88.     at com.bocom.cams.core.base.commons.BaseDao.update(BaseDao.java:37)
  89.     at com.bocom.cams.core.base.commons.PersistableDao.update(PersistableDao.java:46)
  90.     at com.bocom.cams.core.base.commons.PersistableService.modify(PersistableService.java:101)
  91.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:499)
  92.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  93.     at com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  94.     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  95.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  96.     at java.lang.Thread.run(Thread.java:662)
  97. >
  98. ####<2017-8-8 下午07時29分43秒 CST> <Info> <JDBC> <scyy1> <AdminServer> <pool-4514-thread-165> <<anonymous>> <> <> <1502191783427> <BEA-001156> <Stack trace associated with message 001129 follows:

  100. java.sql.SQLRecoverableException: IO 錯誤: Connection reset
  101.     at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:467)
  102.     at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
  103.     at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
  104.     at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  105.     at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
  106.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:357)
  107.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:239)
  108.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)
  109.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1227)
  110.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:451)
  111.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:343)
  112.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
  113.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
  114.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
  115.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
  116.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:158)
  117.     at weblogic.jdbc.pool.Driver.connect(Driver.java:132)
  118.     at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  119.     at weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  120.     at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  121.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  122.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  123.     at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  124.     at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  125.     at com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  126.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  127.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  128.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.queryForList(SourceFile:108)
  129.     at com.bocom.cams.core.base.commons.BaseDao.find(BaseDao.java:60)
  130.     at com.bocom.cams.core.base.commons.PersistableDao.findByBizKeys(PersistableDao.java:76)
  131.     at com.bocom.cams.core.base.commons.PersistableService.queryByBizKeys(PersistableService.java:192)
  132.     at com.bocom.cams.core.base.commons.PersistableService.getByBizKeysFromDataBase(PersistableService.java:321)
  133.     at com.bocom.cams.core.base.commons.PersistableService.getByBizKeys(PersistableService.java:223)
  134.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.calcTypeOfFixedtimeProductInts(InterestCalculateService.java:1388)
  135.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:371)
  136.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  137.     at com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  138.     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  139.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  140.     at java.lang.Thread.run(Thread.java:662)
  141. Caused By: java.net.SocketException: Connection reset
  142.     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
  143.     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
  144.     at oracle.net.ns.DataPacket.send(DataPacket.java:210)
  145.     at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:230)
  146.     at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:312)
  147.     at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
  148.     at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
  149.     at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
  150.     at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
  151.     at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
  152.     at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
  153.     at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
  154.     at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
  155.     at oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:404)
  156.     at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:385)
  157.     at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
  158.     at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
  159.     at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
  160.     at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
  161.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(ConnectionEnvFactory.java:357)
  162.     at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:239)
  163.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)
  164.     at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1227)
  165.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:451)
  166.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:343)
  167.     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:330)
  168.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:469)
  169.     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:363)
  170.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:125)
  171.     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:158)
  172.     at weblogic.jdbc.pool.Driver.connect(Driver.java:132)
  173.     at weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  174.     at weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  175.     at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  176.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  177.     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  178.     at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  179.     at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  180.     at com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  181.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  182.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  183.     at com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.queryForList(SourceFile:108)
  184.     at com.bocom.cams.core.base.commons.BaseDao.find(BaseDao.java:60)
  185.     at com.bocom.cams.core.base.commons.PersistableDao.findByBizKeys(PersistableDao.java:76)
  186.     at com.bocom.cams.core.base.commons.PersistableService.queryByBizKeys(PersistableService.java:192)
  187.     at com.bocom.cams.core.base.commons.PersistableService.getByBizKeysFromDataBase(PersistableService.java:321)
  188.     at com.bocom.cams.core.base.commons.PersistableService.getByBizKeys(PersistableService.java:223)
  189.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.calcTypeOfFixedtimeProductInts(InterestCalculateService.java:1388)
  190.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:371)
  191.     at com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  192.     at com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  193.     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  194.     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  195.     at java.lang.Thread.run(Thread.java:662)
  196. > 
在應用日誌中的報錯資訊如下：

  1. 19:31:43.633 [hostName:scyy1] [pool-4514-thread-243] [2017080900000020] ERROR 報錯資訊如下：weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO 錯誤: Connection reset
  2. weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:250)
  3. weblogic.jdbc.pool.Driver.connect(Driver.java:142)
  4. weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  5. weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  6. weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  7. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  8. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  9. org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  10. org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  11. com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  12. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  13. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  14. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.queryForList(SourceFile:108)
  15. com.bocom.cams.core.base.commons.BaseDao.find(BaseDao.java:60)
  16. com.bocom.cams.core.base.commons.PersistableDao.findByBizKeys(PersistableDao.java:76)
  17. com.bocom.cams.core.base.commons.PersistableService.queryByBizKeys(PersistableService.java:192)
  18. com.bocom.cams.core.base.commons.PersistableService.getByBizKeysFromDataBase(PersistableService.java:321)
  19. com.bocom.cams.core.base.commons.PersistableService.getByBizKeys(PersistableService.java:223)
  20. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:248)
  21. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  22. com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  23. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  24. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  25. java.lang.Thread.run(Thread.java:662)

  27. 19:31:43.633 [hostName:scyy1] [pool-4514-thread-260] [2017080900000020] ERROR 報錯資訊如下：weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool CAMSCORE is Suspended, cannot allocate resources to applications..
  28. weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:251)
  29. weblogic.jdbc.pool.Driver.connect(Driver.java:142)
  30. weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  31. weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  32. weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  33. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  34. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  35. org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  36. org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  37. com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  38. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  39. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  40. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.update(SourceFile:167)
  41. com.bocom.cams.core.base.commons.BaseDao.update(BaseDao.java:37)
  42. com.bocom.cams.core.base.commons.PersistableDao.update(PersistableDao.java:46)
  43. com.bocom.cams.core.base.commons.PersistableService.modify(PersistableService.java:101)
  44. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:499)
  45. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  46. com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  47. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  48. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  49. java.lang.Thread.run(Thread.java:662)

  51. 19:31:43.633 [hostName:scyy1] [pool-4514-thread-233] [2017080900000020] ERROR 報錯資訊如下：weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool CAMSCORE is Suspended, cannot allocate resources to applications..
  52. weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:251)
  53. weblogic.jdbc.pool.Driver.connect(Driver.java:142)
  54. weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  55. weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  56. weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  57. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  58. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  59. org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  60. org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  61. com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  62. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  63. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  64. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.insert(SourceFile:155)
  65. com.bocom.cams.core.base.commons.BaseDao.save(BaseDao.java:26)
  66. com.bocom.cams.core.base.commons.PersistableDao.insert(PersistableDao.java:36)
  67. com.bocom.cams.core.base.commons.PersistableService.add(PersistableService.java:73)
  68. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:521)
  69. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  70. com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  71. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  72. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  73. java.lang.Thread.run(Thread.java:662)

  75. 19:31:43.634 [hostName:scyy1] [pool-4514-thread-281] [2017080900000020] ERROR 報錯資訊如下：weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool CAMSCORE is Suspended, cannot allocate resources to applications..
  76. weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:251)
  77. weblogic.jdbc.pool.Driver.connect(Driver.java:142)
  78. weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:658)
  79. weblogic.jdbc.jts.Driver.connect(Driver.java:127)
  80. weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:533)
  81. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:498)
  82. weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:491)
  83. org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
  84. org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
  85. com.bocom.jump.bp.service.sqlmap.support.SqlSessionUtils.getSqlSession(SourceFile:116)
  86. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:56)
  87. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.a(SourceFile:67)
  88. com.bocom.jump.bp.service.sqlmap.impl.SqlMapImpl.update(SourceFile:167)
  89. com.bocom.cams.core.base.commons.BaseDao.update(BaseDao.java:37)
  90. com.bocom.cams.core.base.commons.PersistableDao.update(PersistableDao.java:46)
  91. com.bocom.cams.core.base.commons.PersistableService.modify(PersistableService.java:101)
  92. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService.interestCalculate(InterestCalculateService.java:499)
  93. com.bocom.cams.core.cd.calcandclearinterest.service.InterestCalculateService$3.toProcess(InterestCalculateService.java:2241)
  94. com.bocom.cams.core.base.commons.support.AsynchronousProcessor.run(AsynchronousProcessor.java:65)
  95. java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  96. java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  97. java.lang.Thread.run(Thread.java:662)

在MOS上輸入關鍵字“Connection reset”，找到相關的文章：

單擊此項可新增到收藏夾 Connection Reset When Creating a JDBC Pool Connection (文件 ID 1532894.1)

In this Document


APPLIES TO:

Oracle WebLogic Server - Version 10.3.6 to 10.3.6
Information in this document applies to any platform.

SYMPTOMS

Once a new connection pool is configured and testing is also successful, the managed server is restarted but it comes up in Admin State and shows JDBC error.

In the standard error, the following sequence of exceptions are shown:

<Feb 22, 2013 11:24:32 AM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.> 
<Feb 22, 2013 11:24:32 AM CST> <Info> <SAFService> <BEA-281003> <The SAF service has been initialized.> 
<Feb 22, 2013 11:24:32 AM CST> <Info> <SAFService> <BEA-281002> <The SAF service has been started.> 
<Feb 22, 2013 11:24:32 AM CST> <Info> <WseeCore> <BEA-220502> <The Wsee Service is starting> 
<Feb 22, 2013 11:24:32 AM CST> <Info> <Deployer> <BEA-149209> <The Slave Deployer is resuming.> 
<Feb 22, 2013 11:24:33 AM CST> <Info> <JDBC> <BEA-001177> <Creating data source connection pool named GTEM_DS, URL = jdbc:oracle:thin:@<URL>:<PORT>:<DB>, Properties = user=<USER>;.> 
<Feb 22, 2013 11:24:35 AM CST> <Info> <Common> <BEA-000626> <Free resources in pool "GTEM_DS" will be tested every "120" seconds.> 
<Feb 22, 2013 11:25:06 AM CST> <Info> <JDBC> <BEA-001508> <Destroying data source connection pool GTEM_DS.> 
<Feb 22, 2013 11:25:06 AM CST> <Info> <JDBC> <BEA-001155> <The following exception has occurred: 

weblogic.common.ResourceException: Unknown Data Source GTEM_DS
at weblogic.jdbc.common.internal.ConnectionPoolManager.shutdownConnectionPool(ConnectionPoolManager.java:630)
at weblogic.jdbc.common.internal.ConnectionPoolManager.shutdownAndDestroyPool(ConnectionPoolManager.java:602)
at weblogic.jdbc.common.internal.ConnectionPoolManager.shutdownAndDestroyPool(ConnectionPoolManager.java:533)
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:330)
at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:100)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:173)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:167)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:35)
at weblogic.application.internal.flow.ModuleStateDriver.prepare(ModuleStateDriver.java:38)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:139)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:57)
at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:707)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:35)
at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:237)
at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:48)
at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
at weblogic.deploy.internal.targetserver.SystemResourceDeployment.prepare(SystemResourceDeployment.java:66)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:40)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:191)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:22)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:166)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:187)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:95)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
> 
<Feb 22, 2013 11:25:06 AM CST> <Error> <Deployer> <BEA-149205> <Failed to initialize the application "GTEM_DS" due to error weblogic.application.ModuleException: 
weblogic.application.ModuleException: 
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:337)
at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:100)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:173)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:167)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:35)
at weblogic.application.internal.flow.ModuleStateDriver.prepare(ModuleStateDriver.java:38)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:139)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:57)
at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:707)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:35)
at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:237)
at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:48)
at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
at weblogic.deploy.internal.targetserver.SystemResourceDeployment.prepare(SystemResourceDeployment.java:66)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:40)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:191)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:22)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:166)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:187)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:95)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: weblogic.common.ResourceException: weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO Error: Connection reset
at weblogic.jdbc.common.internal.ConnectionEnvFactory.createResource(ConnectionEnvFactory.java:288)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1310)
at weblogic.common.resourcepool.ResourcePoolImpl.makeResources(ResourcePoolImpl.java:1227)
at weblogic.common.resourcepool.ResourcePoolImpl.start(ResourcePoolImpl.java:250)
at weblogic.jdbc.common.internal.ConnectionPool.doStart(ConnectionPool.java:1397)
at weblogic.jdbc.common.internal.ConnectionPool.start(ConnectionPool.java:187)
at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:521)
at weblogic.jdbc.common.internal.ConnectionPoolManager.createAndStartPool(ConnectionPoolManager.java:442)
at weblogic.jdbc.module.JDBCModule.prepare(JDBCModule.java:288)
at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:100)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:173)
at weblogic.application.internal.flow.ModuleStateDriver$1.next(ModuleStateDriver.java:167)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:35)
at weblogic.application.internal.flow.ModuleStateDriver.prepare(ModuleStateDriver.java:38)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:139)
at weblogic.application.internal.flow.DeploymentCallbackFlow.prepare(DeploymentCallbackFlow.java:57)
at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:707)
at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:35)
at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:237)
at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:48)
at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:158)
at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
at weblogic.deploy.internal.targetserver.SystemResourceDeployment.prepare(SystemResourceDeployment.java:66)
at weblogic.management.deploy.internal.DeploymentAdapter$1.doPrepare(DeploymentAdapter.java:40)
at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:191)
at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:22)
at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:166)
at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:187)
at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:95)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>

CHANGES

It is a brand new DS. Default DS configuration level was applied:

<?xml version='1.0' encoding='UTF-8'?>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source" xmlns:sec="http://xmlns.oracle.com/weblogic/security" xmlns:wls="http://xmlns.oracle.com/weblogic/security/wls" xmlns:xsi="" xsi:schemaLocation="http://xmlns.oracle.com/weblogic/jdbc-data-source http://xmlns.oracle.com/weblogic/jdbc-data-source/1.2/jdbc-data-source.xsd">
  <name>GTEM_DS</name>
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@[URL]:[PORT]:[DB]</url>
    <driver-name>oracle.jdbc.OracleDriver</driver-name>
    <properties>
      <property>
        <name>user</name>
        <value>[USER]</value>
      </property>
    </properties>
    <password-encrypted>[ENCLIPTED VALUE]</password-encrypted>
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name>
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>GTEM_DS</jndi-name>
    <global-transactions-protocol>OnePhaseCommit</global-transactions-protocol>
  </jdbc-data-source-params>
</jdbc-data-source>

CAUSE

The issue is caused by the database not being able to respond to a large number of connections at the same time and hence throwing a connection reset.

SOLUTION

Please follow these steps to resolve the issue:

  1.       1.From the WLS admin console, select the datasource.
  2.       2.Under the connection pool tab, click on Advanced.
  3.       3.Check the "Enable Test Connections On Reserve" option.
  4.       4.Make "Login Delay" 1.
  5.       5.Make "Connection Creation Retry Frequency" 2.
  6.       6.Save the configuration.
  7.       7.Restart the Cluster.

REFERENCES

NOTE:1280894.1 - WebLogic Server (WLS) Support Pattern: Investigating JDBC Issues



該問題是由資料庫無法同時響應大量連線引起的，因此引起連線重置。這時，建議按照文中給出的SOLUTION來解決Connection reset的問題。

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/31394774/viewspace-2143743/，如需轉載，請註明出處，否則將追究法律責任。

相關文章