在執行表空間的遷移操作後，RAC資料庫出現了上面的錯誤。

在節點1也就是執行表空間遷移的節點上，alert檔案中出現了大量的ORA-600(kjbrchkpkeywait:timeout)錯誤，同時還出現了ORA-7445(kjxmgmb_breq_header)以及ORA-603錯誤。

詳細資訊如下：

Thu Sep 17 14:47:40 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_27466.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [792720], [61114], [], [], [], []
Thu Sep 17 14:47:41 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_27466.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [792720], [61114], [], [], [], []
Thu Sep 17 14:47:41 2009
Trace dumping is performing id=[cdmp_20090917144741]
Thu Sep 17 14:48:23 2009
Trace dumping is performing id=[cdmp_20090917144823]
Thu Sep 17 14:49:05 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_3484.trc:
ORA-00600: 內部錯誤程式碼, 引數: [kjbrchkpkeywait:timeout], [791819], [792805], [61114], [], [], [], []
Thu Sep 17 14:49:06 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_3484.trc:
ORA-00600: 內部錯誤程式碼, 引數: [600], [], [], [], [], [], [], []
Thu Sep 17 14:49:06 2009
Trace dumping is performing id=[cdmp_20090917144906]
Thu Sep 17 14:49:48 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_3484.trc:
ORA-00603: ORACLE 伺服器會話因致命錯誤而終止
ORA-00600: 內部錯誤程式碼, 引數: [600], [], [], [], [], [], [], []
Thu Sep 17 14:49:48 2009
Trace dumping is performing id=[cdmp_20090917144948]
Thu Sep 17 14:50:29 2009
Trace dumping is performing id=[cdmp_20090917145029]
Thu Sep 17 14:59:16 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_5801.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793416], [61114], [], [], [], []
Thu Sep 17 14:59:17 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_5801.trc:
ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], []
Thu Sep 17 14:59:17 2009
Trace dumping is performing id=[cdmp_20090917145917]
Thu Sep 17 14:59:18 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_5801.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [600], [], [], [], [], [], [], []
Thu Sep 17 14:59:20 2009
Trace dumping is performing id=[cdmp_20090917145920]
Thu Sep 17 14:59:23 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_17753.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793423], [61114], [], [], [], []
Thu Sep 17 14:59:24 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_17753.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793423], [61114], [], [], [], []
Thu Sep 17 14:59:24 2009
Trace dumping is performing id=[cdmp_20090917145924]
Thu Sep 17 14:59:35 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_20760.trc:
ORA-00600: internal error code, arguments: [kclcls_8], [], [], [], [], [], [], []
Thu Sep 17 15:00:19 2009
Trace dumping is performing id=[cdmp_20090917150019]
Thu Sep 17 15:00:20 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_23218.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793480], [61114], [], [], [], []
Thu Sep 17 15:00:21 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_23218.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793480], [61114], [], [], [], []
.
.
.
Thu Sep 17 15:08:43 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_17033.trc:
ORA-07445: exception encountered: core dump [kjxmgmb_breq_header()+136] [SIGSEGV] [Address not mapped to object] [0x000000004] [] []
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [793982], [61114], [], [], [], []
Thu Sep 17 15:09:21 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc:
ORA-00600: 內部錯誤程式碼, 引數: [kjbrchkpkeywait:timeout], [793120], [794021], [139374], [], [], [], []
Thu Sep 17 15:09:23 2009
Trace dumping is performing id=[cdmp_20090917150923]
Thu Sep 17 15:09:23 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc:
ORA-00600: 內部錯誤程式碼, 引數: [600], [], [], [], [], [], [], []
Thu Sep 17 15:09:25 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc:
ORA-00603: ORACLE 伺服器會話因致命錯誤而終止
ORA-00600: 內部錯誤程式碼, 引數: [600], [], [], [], [], [], [], []
Thu Sep 17 15:09:26 2009
Thread 1 advanced to log sequence 49
Current log# 4 seq# 49 mem# 0: +MEMBER/tradedb/tradedb_redo1_4_1_1g
Current log# 4 seq# 49 mem# 1: +MEMBER/tradedb/tradedb_redo1_4_2_1g
Thu Sep 17 15:09:26 2009
Trace dumping is performing id=[cdmp_20090917150926]
Thu Sep 17 15:09:28 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_1851.trc:
ORA-00600: internal error code, arguments: [kjbrchkpkeywait:timeout], [791819], [794028], [61114], [], [], [], []
.
.
.
Thu Sep 17 15:18:25 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb1_ora_18739.trc:
ORA-00603: ORACLE 伺服器會話因致命錯誤而終止
ORA-00600: 內部錯誤程式碼, 引數: [600], [], [], [], [], [], [], []

其中ORA-603的錯誤也是ORA-600錯誤所造成的：

SQL> exec dbms_stats.gather_schema_stats(user, cascade => true)
BEGIN dbms_stats.gather_schema_stats(user, cascade => true); END;

*
第 1 行出現錯誤:
ORA-00603: ORACLE 伺服器會話因致命錯誤而終止

這是收集統計資訊報錯對應的trace檔案資訊：

bash-3.00$ more /data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc
/data/oracle/admin/tradedb/udump/tradedb1_ora_7778.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /data/oracle/product/10.2/database
System name:    SunOS
Node name:      ser1
Release:        5.10
Version:        Generic_139555-08
Machine:        sun4u
Instance name: tradedb1
Redo thread mounted by this instance: 1
Oracle process number: 34
Unix process pid: 7778, image: oracle@ser1 (TNS V1-V3)

*** ACTION NAME:() 2009-09-17 14:51:45.397
*** MODULE NAME:(SQL*Plus) 2009-09-17 14:51:45.397
*** SERVICE NAME:(SYS$USERS) 2009-09-17 14:51:45.397
*** SESSION ID:(147.6274) 2009-09-17 14:51:45.397
*** 2009-09-17 14:51:45.397
Cleanout Start
Cleanout End
*** 2009-09-17 15:09:21.300
GCS RESOURCE 50f3f92f0 hashq [51057d560,51fb74638] name[0x2d00b.170000] pkey 62260
   grant 50c8ba7b8 cvt 0 send 0,0 write 0,0@65535
   flag 0x0 mdrole 0x0 mode 0 scan 0 role LOCAL
   disk: 0x0103.0706ae1d write: 0x0000.00000000 cnt 0x0 hist 0x7d
   xid 0x0000.000.00000000 sid 0 pkwait 901s
   pkey 62260
   hv 32 [stat 0x0, 0->0, wm 32767, RMno 0, reminc 4, dom 0]
   kjga st 0x4, step 0.0.0, cinc 4, rmno 10, flags 0x0
   lb 0, hb 0, myb 14849, drmb 14849, apifrz 0
   GCS SHADOW 50c8ba7b8,2 sq[50f3f9310,50f3f9310] resp[50f3f92f0,0x2d00b.170000] pkey 62260
     grant 0 cvt 0 mdrole 0x20 st 0x40 GRANTQ rl LOCAL
     master 0 owner 1 sid 0 remote[526fcda58,3] hist 0xa6752557
     history 0x17.0x15.0x12.0x1d.0x26.0x2. cflag 0x0 sender 0 flags 0x0 replay# 0
     disk: 0x0000.00000000 write request: 0x0000.00000000
     pi scn: 0x0000.00000000
     msgseq 0x2 updseq 0x0 reqids[3,0,0] infop 0x0
*** 2009-09-17 15:09:21.300
ksedmp: internal or fatal error
ORA-00600: 內部錯誤程式碼, 引數: [kjbrchkpkeywait:timeout], [793120], [794021], [139374], [], [], [], []
Current SQL statement for this session:
select /*+ no_parallel(t) no_parallel_index(t) dbms_stats cursor_sharing_exact use_weak_name_resl dynamic_sampling(0) no_monitoring
*/ count(*),count("REMOTE_USER_CODE"),count(distinct "REMOTE_USER_CODE"),sum(sys_op_opnsize("REMOTE_USER_CODE")),substrb(dump(min(su
bstrb("REMOTE_USER_CODE",1,32)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE_USER_CODE",1,32)),16,0,32),1,120),count("REMOTE_USE
R_PASS"),count(distinct "REMOTE_USER_PASS"),sum(sys_op_opnsize("REMOTE_USER_PASS")),substrb(dump(min(substrb("REMOTE_USER_PASS",1,32
)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE_USER_PASS",1,32)),16,0,32),1,120),count("REMOTE_USER_BROWSER"),count(distinct "R
EMOTE_USER_BROWSER"),sum(sys_op_opnsize("REMOTE_USER_BROWSER")),substrb(dump(min(substrb("REMOTE_USER_BROWSER",1,32)),16,0,32),1,120
),substrb(dump(max(substrb("REMOTE_USER_BROWSER",1,32)),16,0,32),1,120),count("REMOTE_USER_OS"),count(distinct "REMOTE_USER_OS"),sum
(sys_op_opnsize("REMOTE_USER_OS")),substrb(dump(min(substrb("REMOTE_USER_OS",1,32)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE
_USER_OS",1,32)),16,0,32),1,120),count("REMOTE_USER_IP"),count(distinct "REMOTE_USER_IP"),sum(sys_op_opnsize("REMOTE_USER_IP")),subs
trb(dump(min(substrb("REMOTE_USER_IP",1,32)),16,0,32),1,120),substrb(dump(max(substrb("REMOTE_USER_IP",1,32)),16,0,32),1,120),count(
"SERVER_FIRST_MENU"),count(distinct "SERVER_FIRST_MENU"),sum(sys_op_opnsize("SERVER_FIRST_MENU")),substrb(dump(min(substrb("SERVER_F
IRST_MENU",1,32)),16,0,32),1,120),substrb(dump(max(substrb("SERVER_FIRST_MENU",1,32)),16,0,32),1,120),count("SERVER_SECOND_MENU"),co
unt(distinct "SERVER_SECOND_MENU"),sum(sys_op_opnsize("SERVER_SECOND_MENU")),substrb(dump(min(substrb("SERVER_SECOND_MENU",1,32)),16
,0,32),1,120),substrb(dump(max(substrb("SERVER_SECOND_MENU",1,32)),16,0,32),1,120),count("SERVER_HANDLER"),count(distinct "SERVER_HA
NDLER"),sum(sys_op_opnsize("SERVER_HANDLER")),substrb(dump(min(substrb("SERVER_HANDLER",1,32)),16,0,32),1,120),substrb(dump(max(subs
trb("SERVER_HANDLER",1,32)),16,0,32),1,120),count("SERVER_OPERATE"),count(distinct "SERVER_OPERATE"),sum(sys_op_opnsize("SERVER_OPER
ATE")),substrb(dump(min(substrb("SERVER_OPERATE",1,32)),16,0,32),1,120),substrb(dump(max(substrb("SERVER_OPERATE",1,32)),16,0,32),1,
120),count("SERVER_ERROR_MSG"),count(distinct "SERVER_ERROR_MSG"),sum(sys_op_opnsize("SERVER_ERROR_MSG")),substrb(dump(min(substrb("
SERVER_ERROR_MSG",1,32)),16,0,32),1,120),substrb(dump(max(substrb("SERVER_ERROR_MSG",1,32)),16,0,32),1,120),count("WEB_EXPEND_TIME")
,count(distinct "WEB_EXPEND_TIME"),sum(sys_op_opnsize("WEB_EXPEND_TIME")),substrb(dump(min("WEB_EXPEND_TIME"),16,0,32),1,120),substr
b(dump(max("WEB_EXPEND_TIME"),16,0,32),1,120),count(distinct "ERROR_CREATE_DATE"),substrb(dump(min("ERROR_CREATE_DATE"),16,0,32),1,1
20),substrb(dump(max("ERROR_CREATE_DATE"),16,0,32),1,120),sum(sys_op_opnsize("ID")),substrb(dump(min("ID"),16,0,32),1,120),substrb(d
ump(max("ID"),16,0,32),1,120) from "ZHEJIANG"."EMED_WEB_LOG" sample (   .4800581069) t where TBL$OR$IDX$PART$NUM("ZHEJIANG"."EMED_W
EB_LOG",0,4,0,"ROWID") = :objn
----- PL/SQL Call Stack -----
object      line object
handle    number name
52083e9d0      1218 package body SYS.DBMS_SYS_SQL
52083ef30       334 package body SYS.DBMS_SQL
520974208     10750 package body SYS.DBMS_STATS
520974208     12215 package body SYS.DBMS_STATS
520974208     12494 package body SYS.DBMS_STATS
520974208     12623 package body SYS.DBMS_STATS
520974208     13090 package body SYS.DBMS_STATS
520974208     13291 package body SYS.DBMS_STATS
520974208     13682 package body SYS.DBMS_STATS
520974208     13760 package body SYS.DBMS_STATS
520974208     13719 package body SYS.DBMS_STATS
4d271a018         1 anonymous block
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+744         CALL     ksedst()             000000840 ?
                                                   FFFFFFFF7FFEF34C ?
                                                   000000000 ?
                                                   FFFFFFFF7FFEBE40 ?
                                                   FFFFFFFF7FFEABA8 ?
                                                   FFFFFFFF7FFEB5A8 ?
kgerinv()+200        PTR_CALL 0000000000000000     000106400 ? 10652D264 ?
                                                   10652D000 ? 00010652D ?
                                                   000106400 ? 10652D264 ?
kgeasnmierr()+28     CALL     kgerinv()            106527C18 ? 000000000 ?
                                                   105FFA068 ? 000000003 ?
                                                   FFFFFFFF7FFEF7F0 ?
                                                   000001430 ?
kjbcropen()+4892     CALL     kgeasnmierr()        106527C18 ?
                                                   FFFFFFFF7C026270 ?
                                                   105FFA068 ? 000000003 ?
                                                   000000000 ? 0000C1A20 ?
kclscrs()+1200       CALL     kjbcropen()          10652D000 ? 1065278D8 ?
                                                   000106400 ? 105FFA000 ?
                                                   000106400 ? 0000C1DA5 ?
kclgclk()+9936       CALL     kclscrs()            000000000 ? 000004000 ?
                                                   000000000 ?
                                                   FFFFFFFF7FFEFBB0 ?
                                                   000000000 ? 000000000 ?

而在另外一個節點，則錯誤主要以ORA-600()為主：

Thu Sep 17 14:50:10 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb2_ora_18519.trc:
ORA-00600: internal error code, arguments: [kclcls_8], [], [], [], [], [], [], []
Thu Sep 17 14:50:46 2009
Trace dumping is performing id=[cdmp_20090917145046]
Thu Sep 17 14:52:21 2009
Trace dumping is performing id=[cdmp_20090917144741]
Thu Sep 17 14:53:03 2009
Trace dumping is performing id=[cdmp_20090917144823]
Thu Sep 17 14:53:46 2009
Trace dumping is performing id=[cdmp_20090917144906]
Thu Sep 17 14:54:28 2009
Trace dumping is performing id=[cdmp_20090917144948]
Thu Sep 17 14:55:09 2009
Trace dumping is performing id=[cdmp_20090917145029]
Thu Sep 17 15:03:57 2009
Trace dumping is performing id=[cdmp_20090917145917]
Thu Sep 17 15:04:00 2009
Trace dumping is performing id=[cdmp_20090917145920]
Thu Sep 17 15:04:04 2009
Trace dumping is performing id=[cdmp_20090917145924]
Thu Sep 17 15:04:59 2009
Trace dumping is performing id=[cdmp_20090917150019]
Thu Sep 17 15:05:01 2009
Trace dumping is performing id=[cdmp_20090917150021]
Thu Sep 17 15:05:54 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb2_ora_20097.trc:
ORA-00600: internal error code, arguments: [kclcls_8], [], [], [], [], [], [], []
Thu Sep 17 15:05:56 2009
Trace dumping is performing id=[cdmp_20090917150556]
Thu Sep 17 15:05:58 2009
Trace dumping is performing id=[cdmp_20090917150118]
.
.
.
Thu Sep 17 15:51:16 2009
Errors in file /data/oracle/admin/tradedb/udump/tradedb2_ora_25090.trc:
ORA-00600: 內部錯誤程式碼, 引數: [kclcls_8], [], [], [], [], [], [], []

查詢metalink，發現bug描述Doc ID: 5165885.8。雖然在這個bug中，並沒有說明是什麼導致了這個問題的產生，但是環境和現象的描述都與當前完全一致。這個bug影響10.2.0.3，也就是當前的版本。雖然Oracle提供了單獨的補丁集，但是似乎安裝這個補丁後有可能導致其他的ORA-600錯誤的產生，甚至是系統CRASH。

雖然Oracle沒有說明是什麼導致了問題，但是根據執行的操作可以推斷出大致是什麼原因。由於在執行遷移表空間的過程中，只是刪除了表空間，而沒有刪除使用者，因此應用程式並沒有斷開連線。由於RAC的特殊性，導致兩個節點之間在處理不正確的CURSOR出現了問題，最終造成了兩個節點的狀態不一致，產生了大量的ORA-600錯誤。

重啟整個資料庫後，這個錯誤不再出現。

ORA-600(kjbrchkpkeywait:timeout)和ORA-600(kclcls_8)錯誤

相關文章