Oracle錯誤處理思路(一)

思路一:
1,發現錯誤(從alert檔案以及錯誤對應的trace檔案)
2,到metalink上查詢該錯誤號
3,根據metalink的描述開始猜測,把可能性羅列出來
4,通過重現問題來驗證3中列出的可能性
5,找到原因後解決問題.

下面針對Ora-600(17069)錯誤的解決過程詳細說明以上思路,該例來自楊廷坤大師.

詳細的錯誤資訊為：

Fri Feb 20 08:16:44 2009
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_j015_5099.trc:
ORA-00600: internal error code, arguments: [17069], [0x6A5DEE1E0], [], [], [], [], [], []
Fri Feb 20 08:16:47 2009
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_j015_5099.trc:
ORA-00600: internal error code, arguments: [17069], [0x6A5DEE1E0], [], [], [], [], [], []

進一步檢查對應的trace檔案：

bash-2.03$ more /u1/oracle/admin/repdb01/bdump/repdb01_j015_5099.trc
/u1/oracle/admin/repdb01/bdump/repdb01_j015_5099.trc
Oracle9i Enterprise Edition Release 9.2.0.4.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.4.0 - Production
ORACLE_HOME = /data/oracle/product/920
System name: SunOS
Node name: newreport
Release: 5.8
Version: Generic_117350-26
Machine: sun4u
Instance name: repdb01
Redo thread mounted by this instance: 1
Oracle process number: 35
Unix process pid: 5099, image: oracle@newreport (J015)

*** SESSION ID:(12.28191) 2009-02-20 08:16:44.060
*** 2009-02-20 08:16:44.060
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [17069], [0x6A5DEE1E0], [], [], [], [], [], []
Current SQL statement for this session:
DECLARE job BINARY_INTEGER := :job; next_date DATE := :mydate; broken BOOLEAN := FALSE; BEGIN P_GENERATE_REPDATA('FR20T000002000000
0000032'); :mydate := next_date; IF broken THEN :b := 1; ELSE :b := 0; END IF; END;
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+328 CALL ksedst()+0 FFFFFFFF7FFF6430 ?
000000000 ? 000000000 ?
00000003E ?
FFFFFFFF7FFF6CC8 ?
1031D56C8 ?
kgeriv()+208 PTR_CALL 0000000000000000 000000000 ? 000103400 ?
0001035D9 ? 000102C00 ?
1035D9000 ? 1035D9C28 ?
kgesiv()+108 CALL kgeriv()+0 1035D9E88 ? 1036C7148 ?
000000258 ? 0000013C8 ?
FFFFFFFF7FFF7608 ?
1035DB258 ?
kgesic1()+32 CALL kgesiv()+0 1035D9E88 ? 1036C7148 ?
0000042AD ? 000000001 ?
FFFFFFFF7FFF7608 ?
004000000 ?
kglgob()+1972 CALL kgesic1()+0 1035D9E88 ? 1036C7148 ?
0000042AD ? 000000002 ?
6A5DEE1E0 ? 0000010A0 ?
kgldpo()+524 CALL kglgob()+0 000000000 ? 000000000 ?
6A5DEE1E0 ?
FFFFFFFF7FFF77A8 ?
000080000 ? 000000010 ?
kgldon()+248 CALL kgldpo()+0 000000000 ? 000000000 ?
69A79DA60 ? 000000001 ?
000000002 ?
FFFFFFFF7FFF7BEE ?
pkldon()+108 CALL kgldon()+0 1035D9E88 ?
FFFFFFFF7FFF7DE0 ?
69A79DA60 ? 000000001 ?
000000000 ?
FFFFFFFF7FFF7D8E ?
pkloud()+204 CALL pkldon()+0 FFFFFFFF7FFFA1A0 ?
FFFFFFFF7FFF7DE0 ?
69A79DA60 ? 000000001 ?
000000000 ?
FFFFFFFF7FFF7D8E ?
phnnrl_name_resolve CALL pkloud()+0 1033FCA90 ?
_by_loading()+280 FFFFFFFF7FFF7E3C ?
000000000 ? 000000000 ?
000030000 ? 6A475CE18 ?
phngdl_get_defining CALL phnnrl_name_resolve 000000000 ? 000020015 ?
_libunit()+124 _by_loading()+0 FFFFFFFF7FFF9830 ?
FFFFFFFF7FFF8160 ?
000020015 ? 000000000 ?
phnrpls_resolve_pre CALL phngdl_get_defining FFFFFFFF7FFF9830 ?
fix_libscope()+12 _libunit()+0 FFFFFFFF7FFF85A8 ?
FFFFFFFF7FFF843C ?
000000000 ? 000000000 ?
000000000 ?

無論是從trace檔案對應的名稱，還是從trace檔案中對應的語句都可以確定，引起問題的是一個JOB。

檢查metalink，Oracle在文件Doc ID: 39616.1中對這個錯誤的已知bug，進行了彙總，不過這些bug的描述似乎沒有和當前十分相符的。

檢視文件的描述，發現ORA-600錯誤的第二個引數，這裡是0x6A5DEE1E0，代表Library Cache Object Handle。

看來問題可能和LATCH有關。

但是根據資訊在V$LATCH和V$LATCH_CHILDREN檢視中，沒有找到有價值的資訊。

這個JOB由於失敗會自動再次執行，檢查JOB執行時的V$LOCK資訊：

SQL> SELECT ADDR, TYPE, ID1, ID2, LMODE, REQUEST, BLOCK
2 FROM V$LOCK
3 WHERE SID = 75;

ADDR TY ID1 ID2 LMODE REQUEST BLOCK
---------------- -- ---------- ---------- ---------- ---------- ----------
0000000690342780 CU -1.703E+09 6 6 0 0
00000006903426F8 JQ 0 63 6 0 0

從V$LOCK中看不到什麼特別有價值的資訊，接著檢查V$SESSION_WAIT，看看這個JOB在等待什麼：

SQL> SELECT EVENT, P1TEXT, P1RAW, P2TEXT, P2RAW, STATE
2 FROM V$SESSION_WAIT
3 WHERE SID = 75;

EVENT P1TEXT P1RAW P2TEXT P2RAW STATE
----------------- --------------- ---------------- ------------ ---------------- -------
library cache pin handle address 00000006A5DEE1E0 pin address 00000006B1A971A8 WAITING

這次的資訊就明顯了，ORA-600錯誤的第二個引數就是V$SESSION_WAIT檢視的P1RAW的值，而且從等待事件上也可以看到，問題就是出現在LIBRARY CACHE PIN的過程中。

重新檢視METALINK的資訊，這個錯誤可能發生在一個長時間執行的程式，在其執行過程中，所依賴的物件被編譯或者刪除了。

檢查JOB呼叫的過程的狀態：

SQL> SELECT OWNER, OBJECT_NAME, OBJECT_TYPE, STATUS
2 FROM DBA_OBJECTS
3 WHERE WNER = 'FUJIANREP'
4 AND OBJECT_NAME = 'P_GENERATE_REPDATA';

OWNER OBJECT_NAME OBJECT_TYPE STATUS
------------------------------ ------------------------------ ------------------ -------
FUJIANREP P_GENERATE_REPDATA PROCEDURE INVALID

果然問題過程處於不正常的狀態。

將JOB至於BROKEN狀態，避免JOB再次執行：

SQL> EXEC DBMS_JOB.BROKEN(63, TRUE)

PL/SQL procedure successfully completed.

SQL> COMMIT;

Commit complete.

殺掉JOB對應的PROCESS：

SQL> SELECT SPID FROM V$PROCESS WHERE ADDR IN (SELECT PADDR FROM V$SESSION WHERE SID = 75);

SPID
------------
14927

SQL> HOST kill -9 14927

下面用重新編譯該過程：

SQL> ALTER PROCEDURE P_GENERATE_REPDATA COMPILE;
ALTER PROCEDURE P_GENERATE_REPDATA COMPILE
*
ERROR at line 1:
ORA-04021: timeout occurred while waiting to lock object FUJIANREP.P_GENERATE_REPDATA

由於從V$LOCK和V$LATCH無法得到資訊，只能看看有沒有其他人當前在訪問P_GENERATE_REPDATA所依賴的物件：

SQL> SELECT * FROM V$ACCESS
2 WHERE (OWNER, OBJECT) IN
3 (SELECT REFERENCED_OWNER, REFERENCED_NAME
4 FROM DBA_DEPENDENCIES
5 WHERE WNER = 'FUJIANREP'
6 AND NAME = 'P_GENERATE_REPDATA');

SID OWNER OBJECT TYPE
---------- ------------------------------ ------------------------------ ------------------------
54 FUJIANREP CAT_BUYER SYNONYM
54 FUJIANREP CAT_CATEGORY SYNONYM
54 FUJIANREP CAT_DOSEAGE_FORM. SYNONYM
54 FUJIANREP CAT_DRUG SYNONYM
54 FUJIANREP CAT_ENTERPRISE SYNONYM
54 FUJIANREP CAT_METRIC SYNONYM
54 FUJIANREP CAT_ORG SYNONYM
54 FUJIANREP CAT_PRODUCT SYNONYM
54 FUJIANREP CAT_QUALITY_DEFINE SYNONYM
54 FUJIANREP GOV_CAT_BUYER TABLE
54 FUJIANREP GOV_CAT_ENTERPRISE TABLE
54 FUJIANREP GOV_S_MO_BU TABLE
54 FUJIANREP GOV_S_MO_BU_EN TABLE
54 FUJIANREP GOV_S_MO_BU_PR TABLE
54 FUJIANREP GOV_S_MO_EN TABLE
54 FUJIANREP GOV_S_MO_ME TABLE
54 FUJIANREP GOV_S_MO_ME_CA TABLE
54 FUJIANREP GOV_S_MO_ME_PR TABLE
54 FUJIANREP GOV_S_MO_ORDER TABLE
54 FUJIANREP GOV_S_YE_ORDER TABLE
54 FUJIANREP GRP_HOSPITAL TABLE
54 FUJIANREP GRP_LEVEL TABLE
54 FUJIANREP ORD_ORDER TABLE
54 FUJIANREP ORD_ORDER_ITEM TABLE
54 FUJIANREP ORD_ORDER_ITEM_REP CURSOR
54 FUJIANREP ORD_ORDER_RECEIVE TABLE
54 FUJIANREP ORD_ORDER_RECEIVE_REP SYNONYM
54 FUJIANREP ORD_ORDER_REP CURSOR
54 FUJIANREP ORD_ORDER_RETURN TABLE
54 FUJIANREP ORD_ORDER_RETURN_REP CURSOR
54 FUJIANREP PLT_PLAT CURSOR
54 FUJIANREP USER_TAB_PARTITIONS CURSOR
54 NDMAIN CAT_BUYER TABLE
54 NDMAIN CAT_CATEGORY TABLE
54 NDMAIN CAT_DOSEAGE_FORM. TABLE
54 NDMAIN CAT_DRUG TABLE
54 NDMAIN CAT_ENTERPRISE TABLE
54 NDMAIN CAT_METRIC TABLE
54 NDMAIN CAT_ORG TABLE
54 NDMAIN CAT_PRODUCT TABLE
54 NDMAIN CAT_QUALITY_DEFINE TABLE
54 NDMAIN ORD_ORDER VIEW
54 NDMAIN ORD_ORDER_ITEM VIEW
54 NDMAIN ORD_ORDER_RECEIVE VIEW
54 NDMAIN ORD_ORDER_RETURN VIEW
54 NDMAIN PLT_PLAT TABLE
54 PUBLIC USER_TAB_PARTITIONS SYNONYM
54 SYS STANDARD PACKAGE
145 SYS STANDARD PACKAGE
54 SYS SYS_STUB_FOR_PURITY_ANALYSIS PACKAGE
54 SYS USER_TAB_PARTITIONS VIEW

51 rows selected.

物件果然被其他人所訪問，看看這個會話在做什麼：

SQL> SELECT SID, SERIAL#, USERNAME, PROGRAM, TERMINAL
2 FROM V$SESSION
3 WHERE SID = 54;

SID SERIAL# USERNAME PROGRAM TERMINAL
---------- ---------- ------------------------------ ------------ ----------
54 26216 FUJIANREP PlSqlDev.exe LIBY

沒想到是同事的連線的會話，看看他在幹什麼：

SQL> SELECT SQL_TEXT FROM V$SQL
2 WHERE ADDRESS IN
3 (SELECT SQL_ADDRESS FROM V$SESSION
4 WHERE SID = 54);

SQL_TEXT
---------------------------------------------------------------------
ALTER TABLE GOV_S_MO_EN TRUNCATE PARTITION P200901

居然是TRUNCATE分割槽操作，難怪會導致過程處於INVALID狀態，不過這個操作應該不會持續很長時間的，難道這個操作一直沒有完成嗎：

SQL> SELECT EVENT, P1TEXT, P1, P2TEXT, P2, P3TEXT, P3, SECONDS_IN_WAIT
2 FROM V$SESSION_WAIT WHERE SID = 54;

EVENT P1TEXT P1 P2TEXT P2 P3TEXT P3 SECONDS_IN_WAIT
------------------------- ------- ---- -------- -------- -------- ---- ---------------
db file sequential read file# 1 block# 170158 blocks 1 3995643

這個等待已經發生了幾十天了，顯然這是一個僵死的會話。

從後臺kill掉對應的程式：

SQL> SELECT SPID FROM V$PROCESS
2 WHERE ADDR IN (SELECT PADDR FROM V$SESSION WHERE SID = 54);

SPID
------------
12974

SQL> HOST kill -9 12974

切換為FUJIANREP使用者，再次編譯過程：

SQL> ALTER PROCEDURE P_GENERATE_REPDATA COMPILE;

Procedure altered.

至此，問題解決。將JOB重新設定BROKEN即可。

SQL> EXEC DBMS_JOB.BROKEN(63, FALSE)

PL/SQL procedure successfully completed.

SQL> COMMIT;

Commit complete.

列出Metalink:Note:146580.1

ORA-600 [17069] "Failed to pin a library cache object after 50 attempts" [ID 39616.1]
		修改時間 08-JUL-2011 型別 REFERENCE 狀態 PUBLISHED

Note: For additional ORA-600 related information please read Note:146580.1

PURPOSE:            
  This article discusses the internal error "ORA-600 [17069]", what 
  it means and possible actions. The information here is only applicable 
  to the versions listed and is provided only for guidance.
 
ERROR:              

#helpWin {position:absolute; width:400; visibility:hidden; background-color:#f7f7e7; padding:20px; border-width:2px; border-left-color:#336699; border-top-color:#336699; border-right-color:#336699; border-bottom-color:#336699; border-style.:solid; }