一次無法mount資料庫的經歷

本文系原創:

8月27日凌晨為使用者troubleshooting資料庫問題時需要重新啟動資料庫，當資料庫啟動步驟進行mount時，系統不斷出現ARC1: Thread not mounted，擷取部分alert.log資訊如下:
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=11, OS id=28762
ARC0: Archival started
ARC1 started with pid=12, OS id=28764
Sat Aug 27 00:54:31 2011
ARCH: STARTING ARCH PROCESSES COMPLETE
Sat Aug 27 00:54:31 2011
ARC1: Archival started
Sat Aug 27 00:54:31 2011
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no FAL' ARCHARC0: Thread not mounted
Sat Aug 27 00:54:31 2011
ARC1: Becoming the heartbeat ARCH
ARC1: Becoming the heartbeat ARCHARC1: Thread not mounted
Sat Aug 27 00:54:31 2011
ALTER DATABASE   MOUNT
Sat Aug 27 00:55:31 2011
ARC1: Thread not mounted
Sat Aug 27 00:56:31 2011
ARC1: Thread not mounted
Sat Aug 27 00:57:31 2011
ARC1: Thread not mounted
Sat Aug 27 00:58:31 2011
ARC1: Thread not mounted
Sat Aug 27 00:59:31 2011
ARC0: Thread not mounted
Sat Aug 27 00:59:31 2011
ARC1: Thread not mounted
在持續了20分鐘後，還在出現ARC1: Thread not mounted時,透過在網上和metalink上搜尋相關資訊，採取措施：
1、重新起一個sqlplus的session
2、執行shutdown abort後，啟動資料庫,啟動過程的問題現象依舊，反覆出現“ARC0: Thread not mounted
Sat Aug 27 01:31:55 2011
ARC1: Thread not mounted”後，再次出現ORA-00600錯誤,部分alert.log如下:
Sat Aug 27 01:30:55 2011
ARC1: Thread not mounted
Sat Aug 27 01:31:55 2011
ARC0: Thread not mounted
Sat Aug 27 01:31:55 2011
ARC1: Thread not mounted
Sat Aug 27 01:32:00 2011
Errors in file /oracle/ora92/admin/ctais/udump/ctais_ora_1324.trc:
ORA-00600: internal error code, arguments: [2116], [900], [], [], [], [], [], []
Sat Aug 27 01:32:55 2011
ARC1: Thread not mounted
Sat Aug 27 01:33:55 2011
ARC1: Thread not mounted
我採取一直等待措施，一直等到20分鐘後，資料庫session因ORA-00600異常中斷為止。
啟動session中顯示如下:
SQL> shutdown abort
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 7520358264 bytes
Fixed Size                   745336 bytes
Variable Size            3221225472 bytes
Database Buffers         4294967296 bytes
Redo Buffers                3420160 bytes
ORA-00600: internal error code, arguments: [2116], [900], [], [], [], [], [],
[]

在網上查詢很久，沒有一個很好的關於ORA-00600[2116], [900]的解釋，據國外一位網友介紹，一般在資料庫進行I/0 CALL時才出現這種有關儲存的錯誤。

我決定再幹淨的關閉一次資料庫，試一把
 shutdown immediate,不一會兒，資料庫乾淨的關閉，檢查主機(HP-UX  B.11.11 U 9000/800 )層面的程式:
 ps -ef|grep ora,沒有發現oracle相關程式,執行ipcs,沒有ORACLE的共享記憶體，說明資料庫關閉的比較乾淨。

由於兩次啟動資料庫，均在mount時出現問題，網上的分析說，有可能和歸檔程式的爭用有關，可以忽略提示。所以我準備這次分三個階段啟動資料庫，
先做startup nomount  結果成功啟動
再做alter database mount;
日誌顯示:
alter database mount
Sat Aug 27 01:59:04 2011
Successful mount of redo thread 1, with mount id 1142904611
Sat Aug 27 01:59:04 2011
Database mounted in Exclusive Mode.
Completed: alter database mount
終於成功mount了

最後做:alter database open
日誌顯示:
alter database open
Sat Aug 27 01:59:28 2011
LGWR: Primary database is in CLUSTER CONSISTENT mode
Thread 1 opened at log sequence 8034
  Current log# 2 seq# 8034 mem# 0: /dev/vgora/rlv_c01g02
Successful open of redo thread 1
Sat Aug 27 01:59:28 2011
SMON: enabling cache recovery
Sat Aug 27 01:59:45 2011
Successfully onlined Undo Tablespace 1.
Sat Aug 27 01:59:45 2011
SMON: enabling tx recovery
Sat Aug 27 01:59:45 2011
Database Characterset is US7ASCII
replication_dependency_tracking turned off (no async multimaster replication found)
Completed: alter database open
成功啟動.

在前臺操作業務也沒有問題。
這時，再做一次乾淨的資料庫關閉和啟動，結果一些OK!
只是最後一次關閉資料庫的時間比較長,等了5分鐘,日誌如下:
Shutting down instance (immediate)
License high water mark = 7
Sat Aug 27 02:02:28 2011
ALTER DATABASE CLOSE NORMAL
Sat Aug 27 02:07:33 2011
Waiting for smon to disable tx recovery.
Sat Aug 27 02:12:42 2011
SMON: disabling tx recovery
SMON: disabling cache recovery
Sat Aug 27 02:12:42 2011
Shutting down archive processes
Archiving is disabled
Sat Aug 27 02:12:42 2011
ARCH shutting down
Sat Aug 27 02:12:42 2011
ARCH shutting down
Sat Aug 27 02:12:42 2011
ARC0: Archival stopped
Sat Aug 27 02:12:42 2011
ARC1: Archival stopped
Sat Aug 27 02:12:42 2011
Thread 1 closed at log sequence 8034
Successful close of redo thread 1
Sat Aug 27 02:12:42 2011
Completed: ALTER DATABASE CLOSE NORMAL
Sat Aug 27 02:12:42 2011
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active