1.問題現象
資料庫無法啟動，錯誤號ORA-03113
RACDB1@rac1 /home/oracle$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Sat Nov 6 15:52:49 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 167772160 bytes
Fixed Size                  1218292 bytes
Variable Size              88082700 bytes
Database Buffers           71303168 bytes
Redo Buffers                7168000 bytes
Database mounted.
ORA-03113: end-of-file on communication channel

2.問題分析
1）alert日誌檔案中記錄瞭如下錯誤資訊
alert中記錄的資訊
ARC2 started with pid=28, OS id=11242
Sat Nov 6 15:54:31 2010
Errors in file /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc:
ORA-19504: failed to create file "+DATA/racdb/1_21_733447664.dbf"
ORA-17502: ksfdcre:4 Failed to create file +DATA/racdb/1_21_733447664.dbf
ORA-15041: diskgroup space exhausted
Sat Nov 6 15:54:31 2010
ARC1: Error 19504 Creating archive log file to '+DATA/racdb/1_21_733447664.dbf'
ARC1: Failed to archive thread 1 sequence 21 (19504)
ARCH: Archival stopped, error occurred. Will continue retrying
Sat Nov 6 15:54:33 2010
ORACLE Instance RACDB1 - Archival Error
Sat Nov 6 15:54:33 2010
ORA-16038: log 2 sequence# 21 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'
ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'
Sat Nov 6 15:54:33 2010
Errors in file /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc:
ORA-16038: log 2 sequence# 21 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'
ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'
Sat Nov 6 15:54:34 2010

2）trace檔案中記錄的資訊如下
+ASM1@rac1 /home/oracle$ cat /u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc
/u01/app/oracle/admin/RACDB/bdump/racdb1_arc1_11177.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1
System name:    Linux
Node name:      rac1
Release:        2.6.18-53.el5xen
Version:        #1 SMP Wed Oct 10 17:06:12 EDT 2007
Machine:        i686
Instance name: RACDB1
Redo thread mounted by this instance: 1
Oracle process number: 26
Unix process pid: 11177, image: oracle@rac1 (ARC1)

*** SERVICE NAME:() 2010-11-06 15:54:30.927
*** SESSION ID:(144.1) 2010-11-06 15:54:30.927
kcrrwkx: work to do 0x1 (start)
Failed to create file '+DATA/racdb/1_21_733447664.dbf' (file not accessible?)
ORA-19504: failed to create file "+DATA/racdb/1_21_733447664.dbf"
ORA-17502: ksfdcre:4 Failed to create file +DATA/racdb/1_21_733447664.dbf
ORA-15041: diskgroup space exhausted
*** 2010-11-06 15:54:31.432 60679 kcrr.c
ARC1: Error 19504 Creating archive log file to '+DATA/racdb/1_21_733447664.dbf'
*** 2010-11-06 15:54:31.432 58941 kcrr.c
kcrrfail: dest:1 err:19504 force:0 blast:1
ARCH: Connecting to console port...
ARCH: Connecting to console port...
*** 2010-11-06 15:54:33.979 20145 kcrr.c
ORA-16038: log 2 sequence# 21 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 2 thread 1: '+DATA/racdb/onlinelog/group_2.262.733447669'
ORA-00312: online log 2 thread 1: '+FRA/racdb/onlinelog/group_2.258.733447671'

3.問題原因
導致資料庫例項無法啟動的原因是Oracle ASM存放歸檔日誌的磁碟組已被耗盡。因無法完成歸檔，導致資料庫無法啟動。

4.處理方法
1）第一種處理方法
如果可以將資料庫調整為非歸檔模式，可以保證資料庫例項正常啟動。
如果是生產環境不建議這樣操作。

2）第二種處理方法
使用asmcmd工具清理歸檔日誌，釋放磁碟組上的空間
具體操作過程如下：
RACDB1@rac1 /home/oracle$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
RACDB1@rac1 /home/oracle$ export ORACLE_SID=+ASM1
ASMCMD [+] > lsdg
State    Type    Rebal Unbal Sector Block       AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name
MOUNTED EXTERN N      N         512   4096 1048576      2047        7                0               7              0 DATA/
MOUNTED EXTERN N      N         512   4096 1048576      2047     1825                0            1825              0 FRA/
ASMCMD [+] > cd DATA
ASMCMD [+DATA] > cd RACDB
ASMCMD [+DATA/RACDB] > cd ARCHIVELOG
ASMCMD [+DATA/RACDB/ARCHIVELOG] > ls
2010_10_26/
2010_10_27/
2010_10_28/
2010_10_29/
2010_10_30/
2010_11_01/
2010_11_04/
ASMCMD [+DATA/RACDB/ARCHIVELOG] > du
Used_MB      Mirror_used_MB
    841                 841
ASMCMD [+DATA/RACDB/ARCHIVELOG] > rm -rf *
ASMCMD [+data/racdb] > lsdg
State    Type    Rebal Unbal Sector Block       AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name
MOUNTED EXTERN N      N         512   4096 1048576      2047      848                0             848              0 DATA/
MOUNTED EXTERN N      N         512   4096 1048576      2047     1825                0            1825              0 FRA/

此時資料庫例項便可正常啟動。

3）第三種處理方法
調整歸檔路徑，或增加調整磁碟組大小。

5.小結
在資料庫系統部署之前我們要對磁碟組大小充分的評估，以避免出現因空間不足導致系統故障，以及分配過多的空間造成磁碟和系統資源的浪費。

Good luck.

secooler
10.11.07

-- The End --

【問題處理】因ASM磁碟組空間不足導致資料庫例項無法啟動的故障處理

相關文章