修改資料字典表導致ORA-600(ktecgsc:kcbz_objdchk)錯誤

公司的測試服務環境被整崩潰了，簡單記錄一下錯誤。

同事發現一個測試資料庫無法登入，對資料庫執行SHUTDOWN ABORT後，重新啟動bootstrap報錯：

[ora11g@hpserver2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Sun Mar 4 00:37:58 2012

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 2555744256 bytes
Fixed Size 2230912 bytes
Variable Size 637535616 bytes
Database Buffers 1862270976 bytes
Redo Buffers 53706752 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-08103: object no longer exists
Process ID: 20964
Session ID: 125 Serial number: 5

資料庫怎麼會突然出現這麼嚴重的問題，檢查告警日誌發現瞭如下的錯誤：

Errors in file /u02/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_31804.trc (incident=127465):
ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], []
Incident details in: /u02/app/oracle/diag/rdbms/orcl11g/orcl11g/incident/incdir_127465/orcl11g_ora_31804_i127465.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Dumping diagnostic data in directory=[cdmp_20120302133841], requested by (instance=1, sid=31804), summary=[incident=127465].
Sweep [inc][127465]: completed
Sweep [inc2][127465]: completed
Errors in file /u02/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_31900.trc (incident=127449):
ORA-00600: internal error code, arguments: [kkdlGetBaseUser2:authIdType], [0], [27], [GLOBAL_AQ_USER_ROLE], [], [], [], [], [], [], [], []
Incident details in: /u02/app/oracle/diag/rdbms/orcl11g/orcl11g/incident/incdir_127449/orcl11g_ora_31900_i127449.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Dumping diagnostic data in directory=[cdmp_20120302134009], requested by (instance=1, sid=31900), summary=[incident=127449].
Sweep [inc][127449]: completed
Sweep [inc2][127449]: completed

檢查第一個ORA-600對應的TRACE，發現出現錯誤的會話在執行匯入：

[ora11g@hpserver2 ~]$ more /u02/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_31804.trc
Trace file /u02/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_31804.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Oracle Label Security and Real Application Testing options
ORACLE_HOME = /u02/app/oracle/product/11.2.0/db_1
System name: Linux
Node name: hpserver2.enmotech.com
Release: 2.6.32-100.28.5.el6.x86_64
Version: #1 SMP Wed Feb 2 18:40:23 EST 2011
Machine: x86_64
Instance name: orcl11g
Redo thread mounted by this instance: 1
Oracle process number: 29
Unix process pid: 31804, image: oracle@hpserver2.enmotech.com (TNS V1-V3)

*** 2012-03-02 13:38:34.271
*** SESSION ID:(141.3959) 2012-03-02 13:38:34.271
*** CLIENT ID:() 2012-03-02 13:38:34.271
*** SERVICE NAME:(SYS$USERS) 2012-03-02 13:38:34.271
*** MODULE NAME:(imp@hpserver2.enmotech.com (TNS V1-V3)) 2012-03-02 13:38:34.271
*** ACTION NAME:() 2012-03-02 13:38:34.271

*** SESSION ID:(141.3959) 2012-03-02 13:38:34.270
DATA seg.obj=-2, on-disk bj=27, dsflg=0, dsobj=74, cls=4
Incident 127465 created, dump file: /u02/app/oracle/diag/rdbms/orcl11g/orcl11g/incident/incdir_127465/orcl11g_ora_31804_i127465.trc
ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], []

*** SESSION ID:(141.3959) 2012-03-02 13:38:40.815
DATA seg.obj=0, on-disk bj=27, dsflg=0, dsobj=74, cls=4

和同事確認了一下，確實有個同事在匯入資料字典表，而且本來應該匯入到單獨的使用者下，但是匯入時忘了新增TOUSER，導致資料被匯入到SYS使用者下，造成了資料字典的不一致，從而引發了這個問題。

這裡並不是要解決這個錯誤，事實上如果沒有完整的備份，資料庫已經很難恢復了。這裡主要是記錄這個問題，希望給其他人引以為戒。

DBA總要做一些高風險的操作，而DBA又有足夠的知識和能力導致問題一般情況下難以產生的前提要求得以實現。

比如當前這次事故，如果不是DBA，沒有一定專業的知識，無法使用IMP方式來匯入資料字典表。而對於缺少了TOUSER引數的情況下，表會匯入到SYS使用者下，但是這會出現表已經存在的錯誤，而使得匯入失敗。而對於DBA來說，這種常見錯誤並不會引發DBA的足夠重視，一個IGNORE=Y引數的使用，最終釀成了這次事故。