OS臨時埠耗盡導致無法建立資料庫連線

westzq1984發表於2013-01-16
資料庫
800x600

1.   報錯資訊

zhjs3$[/home/oracle]lsnrctl status

 

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 16-JAN-2013 16:12:43

 

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

 

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=133.37.253.82)(PORT=1521)))

TNS-12547: TNS:lost contact

 TNS-12560: TNS:protocol adapter error

  TNS-00517: Lost contact

   IBM/AIX RISC System/6000 Error: 32: Broken pipe

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC)))

TNS-12547: TNS:lost contact

 TNS-12560: TNS:protocol adapter error

  TNS-00517: Lost contact

   IBM/AIX RISC System/6000 Error: 32: Broken pipe

 

zhjs3$[/home/oracle]tnsping `hostname`

 

TNS Ping Utility for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 16-JAN-2013 16:12:55

 

Copyright (c) 1997, 2011, Oracle.  All rights reserved.

 

Used parameter files:

 

Used HOSTNAME adapter to resolve the alias

Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=133.37.253.82)(PORT=1521)))

TNS-12547: TNS:lost contact

 

 

2.   連線只是在本機存在異常，如果在第三方伺服器上請求連線，一些OK

 

3.   對lsnrctl命令做TRUSS，沒有發現太多異常，只是最後返回一個異常

1200408: 1413557: thread_setmystate(0x0FFFFFFFFFFF7630, 0x0000000000000000) = 0

1200408: 1413557: kwrite(5, "\0 肻0\00104\0\001 :01 ,".., 195) = -1

1200408:     Received signal #13, SIGPIPE [ignored]

1200408: 1413557: close(5)                              = 0

1200408: 1413557: lseek(4, 40448, 0)                    = 40448

 

4.   從報錯來看，應該是資源不足造成的，但是檢查也未發現什麼異常。

zhjs3$[/home/oracle]ulimit -a

time(seconds)        unlimited

file(blocks)         unlimited

data(kbytes)         unlimited

stack(kbytes)        32768

memory(kbytes)       32768

coredump(blocks)     2097151

nofiles(descriptors) 2000

 

5.   topas檢查發現幾個sftp CPU消耗極高。瞭解到這個伺服器上同時部署了資料庫與應用。檢查一下網路使用情況，發現存在異常的本機聯機到133.37.116.121的FTP連線，高達3W+個

zhjs3$[/home/oracle]netstat -an

tcp4       0      0  133.37.253.82.50436    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50437    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50438    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50439    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50440    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50441    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50442    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50443    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50444    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50445    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50446    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50447    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50448    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50449    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50450    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50451    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50452    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50453    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50454    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50455    133.37.116.121.21      CLOSE_WAIT

tcp4       0      0  133.37.253.82.50456    133.37.116.121.21      CLOSE_WAIT

 

zhjs3$[/home/oracle]netstat -an|grep 133.37.116.121.21|wc -l

   32341

 

6.   當資料庫客戶端與監聽建立連線時，需要在本機分配一個臨時的埠來建立socket連線。如果沒有臨時的埠可用，則無法建立連線。32341個指向133.37.116.121.21的連線，這個數目和常見的限制值32768很接近。懷疑由於FTP客戶端程式耗盡了埠，導致本機連線資料庫請求不到埠報錯。

 

7.   開發商停止應用程式後，埠釋放出來，資料庫連線恢復正常

 

8.   查詢資料得知，AIX平臺下，臨時埠通過tcp_ephemeral_low,tcp_ephemeral_high 引數指定，在這臺伺服器上，其定義的臨時埠範圍確實為32768個。

Normal 0 7.8 磅 0 2 false false false EN-US ZH-CN X-NONE

相關文章