問題處理

問題描述

2017-05-25 日 xcrmdb1 ADG 發生報障， SSC 在之前郵件及下面 SR 中進行了跟蹤與分析；本報告對這起故障進行綜合彙報，並提供原因及建議。

故障分析

1. Instance2 的 lmsc 程式收到錯誤序列的 UDP 包，這個重要程式退出，導致例項被逼終止。

>>>alert_crmdb12.log

Thu May 25 05:52:38 2017

Archived Log entry 48804 added for thread 1 sequence 77393 ID 0x3d8c0bae dest 1:

Thu May 25 05:58:06 2017

Errors in file /oracle/app/oracle/diag/rdbms/xcrmdb1/crmdb12/trace/crmdb12_lmsc_4967.trc (incident=384241):

ORA-00600: internal error code, arguments: [ kjctr_pbmsg:badseq ], [32], [0], [16777216], [], [], [], [], [], [], [], []

<<<lmsc 程式收到錯誤的資訊包，下面需要重啟 Instance 來保證 DB 的完整性

Incident details in: /oracle/app/oracle/diag/rdbms/xcrmdb1/crmdb12/incident/incdir_384241/crmdb12_lmsc_4967_i384241.trc

Thu May 25 05:58:10 2017

Dumping diagnostic data in directory=[cdmp_20170525055810], requested by (instance=2, osid=680322 (LMSC)), summary=[incident=384241].

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Thu May 25 05:58:12 2017

Errors in file /oracle/app/oracle/diag/rdbms/xcrmdb1/crmdb12/trace/crmdb12_lmsc_4967.trc:

ORA-00600: internal error code, arguments: [ kjctr_pbmsg:badseq ], [32], [0], [16777216], [], [], [], [], [], [], [], []

Thu May 25 05:58:12 2017

USER (ospid: 4967): terminating the instance due to error 484

>>>lms 程式異常， Instance 需要中斷

Thu May 25 05:58:14 2017

License high water mark = 866

Thu May 25 05:58:17 2017

Instance terminated by USER, pid = 4967

Thu May 25 05:58:18 2017

USER (ospid: 17292): terminating the instance

Thu May 25 05:58:18 2017

Instance terminated by USER, pid = 17292

Thu May 25 05:59:11 2017

Starting ORACLE instance (normal) (OS id: 17791)

>>>alert_crmdb11.log

Thu May 25 05:58:14 2017

Reconfiguration started (old inc 6, new inc 8)

List of instances (total 1) :

1

Dead instances (total 1) :

2

My inst 1

>>>Instance1 感知 Instance2 退出，開始 Reconfiguration

05:58:06 Node1:[kjctr_pbmsg:badseq]

05:58:12 Node1:[kjctr_pbmsg:badseq]

05:58:12 Node1:terminating the instance

05:58:14 Node2:Reconfiguration started

05:58:18 Node1:terminating the instance

05:59:11 Node1:Starting ORACLE instance

2. lmsc 於 kjctr_pbmsg:badseq 處收到錯誤的包而 abort

>>>crmdb12_lmsc_4967.trc

*** 2017-05-25 05:58:06.823

ORA-00600: internal error code, arguments: [kjctr_pbmsg:badseq], [32], [0], [16777216], [], [], [], [], [], [], [], []

kjmsm: caught non-fatal error 600

lms abort after exception 600

<<<lmsc 出現 600 異常而終止 (error 484)

KJC Communication Dump:

-------------------8<-------------------

>>>crmdb12_lmsc_4967_i384241.trc

Error Stack: ORA-600[kjctr_pbmsg:badseq]

Main Stack:

kjctr_pbmsg <- kjctr_watq <- kjctr_rksxp <- kjctrcv <- kjcsrmg <- kjmsm <- ksbrdp <- opirip

<- opidrv <- sou2o <- opimai_real <- ssthrdmain <- main <- main_opd_entry

----- Incident Context Dump -----

Address: 0x9fffffffffff59f0

Incident ID: 384241

Problem Key: ORA 600 [kjctr_pbmsg:badseq]

Error: ORA-600 [kjctr_pbmsg:badseq] [32] [0] [16777216] [] [] [] [] [] [] [] []

[00]: dbgexProcessError [diag_dde]

[01]: dbgeExecuteForError [diag_dde]

[02]: dbgePostErrorKGE [diag_dde]

[03]: dbkePostKGE_kgsf [rdbms_dde]

[04]: kgeadse []

[05]: kgerinv_internal []

[06]: kgerinv []

[07]: kgeasnmierr []

[08]: kjctr_pbmsg []<-- Signaling

[09]: kjctr_watq []

[10]: kjctr_rksxp []

[11]: kjctrcv []

[12]: kjcsrmg []

[13]: kjmsm [RAC_MLMDS]

[14]: ksbrdp [background_proc]

[15]: opirip [OPI]

[16]: opidrv [OPI]

[17]: sou2o []

[18]: opimai_real [OPI]

[19]: ssthrdmain []

[20]: main []

[21]: main_opd_entry []

>>> 出錯的 function 在 kjctr_pbmsg

3. 在 05:16:43~05:21:45/05:42:51~05:45:52/05:57:56~05:58:26 這三個波段有突發性的分片流量，導致分片超時丟棄和佇列溢位以及 UDP 校驗錯。

最後一波雖然最小持續最短，但是按機率導致 lmsc 程式收到錯誤的包而 abort ，最終 Instance2 中斷。

4. 放大一下區域性細節： ( 第二張圖裡 : 溢位是紅色，校驗錯是綠色，超時丟棄是藍色 )

5. 相關的統計項如下

udp:

20067 incomplete headers

9124 bad checksums <<<UDP 校驗錯

ip:

872460826 fragments received

16823 fragments dropped (dup or out of space) <<< 分片溢位

3712 fragments dropped after timeout <<< 分片超時丟棄

6. 分片流量突發，引起分片佇列溢位，進而出現分片超時丟棄和 UDP校驗錯。在這種情形下，上層 DB程式存在一定機率收到錯誤的包，引發例項故障。建議調整如下 OS Kernel引數：

ip_fragment_timeout 這個調整為 1s

ip_reass_mem_limit 這個調整為 10M

問題總結

問題描述：

例項程式 lms 出現 ”ORA-00600: internal error code, arguments: [kjctr_pbmsg:badseq], [32], [0], [16777216], [], [], [], [], [], [], [], []” 而 abort ，導致 Instance 終止

適用範圍：

所有多 Node RAC 結構，故障主要發生在 lms 程式；不限於 DB 版本。

問題現象：

lms 程式收到錯誤的資訊包，需要重啟 Instance

>>>alert_crmdb12.log

Thu May 25 05:52:38 2017

Archived Log entry 48804 added for thread 1 sequence 77393 ID 0x3d8c0bae dest 1:

Thu May 25 05:58:06 2017

Errors in file /oracle/app/oracle/diag/rdbms/xcrmdb1/crmdb12/trace/crmdb12_lmsc_4967.trc (incident=384241):

ORA-00600: internal error code, arguments: [ kjctr_pbmsg:badseq ], [32], [0], [16777216], [], [], [], [], [], [], [], []

<<<lmsc 程式收到錯誤的資訊包，下面需要重啟 Instance 來保證 DB 的完整性

Incident details in: /oracle/app/oracle/diag/rdbms/xcrmdb1/crmdb12/incident/incdir_384241/crmdb12_lmsc_4967_i384241.trc

Thu May 25 05:58:10 2017

Dumping diagnostic data in directory=[cdmp_20170525055810], requested by (instance=2, osid=680322 (LMSC)), summary=[incident=384241].

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Thu May 25 05:58:12 2017

Errors in file /oracle/app/oracle/diag/rdbms/xcrmdb1/crmdb12/trace/crmdb12_lmsc_4967.trc:

ORA-00600: internal error code, arguments: [ kjctr_pbmsg:badseq ], [32], [0], [16777216], [], [], [], [], [], [], [], []

Thu May 25 05:58:12 2017

USER (ospid: 4967): terminating the instance due to error 484

>>>lms 程式異常， Instance 需要中斷

Thu May 25 05:58:14 2017

License high water mark = 866

Thu May 25 05:58:17 2017

Instance terminated by USER, pid = 4967

Thu May 25 05:58:18 2017

USER (ospid: 17292): terminating the instance

Thu May 25 05:58:18 2017

Instance terminated by USER, pid = 17292

Thu May 25 05:59:11 2017

Starting ORACLE instance (normal) (OS id: 17791)

OSW netstat 統計

udp:

20067 incomplete headers

9124 bad checksums <<<UDP 校驗錯

ip:

872460826 fragments received

16823 fragments dropped (dup or out of space) <<< 分片溢位

3712 fragments dropped after timeout <<< 分片超時丟棄

問題原因：

分片流量突發，引起分片佇列溢位，進而出現分片超時丟棄和 UDP 校驗錯。在這種情形下，上層 DB 程式存在一定機率收到錯誤的包，引發例項故障

解決辦法：

對於 HP OS ，在 Kernel 調整相關的分片佇列引數，一般是如下兩個引數。

ip_fragment_timeout 這個調整為 1s

ip_reass_mem_limit 這個調整為 10M

ADG 例項異常終止故障分析報告

問題處理

問題描述

故障分析

相關文章