ASM DISK Group載入ORA-15183錯誤一例
進入Oracle 11gR2，ASM（Automatic Storage Management）從Database元件中剝離出來，作為獨立元件Component進入Grid管理範疇。
本篇主要介紹筆者遇到的一個資料庫啟動載入過程中出現的問題。同官方MOS推薦的策略相比，有一些不同之處。記錄下來，留待需要的朋友待查使用。
1、問題說明
筆者環境是Oracle 單例項+Grid Infrastructure，版本號為11.2.0.4。由於安全原因，從MOS上下載了最新的安全補丁和升級補丁。升級之後的版本為11.2.0.4.6。
但是，在升級最後步驟——執行SQL指令碼環節，出現了一些問題。
SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:08:57 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2087780352 bytes
Fixed Size 2254824 bytes
Variable Size 553650200 bytes
Database Buffers 1526726656 bytes
Redo Buffers 5148672 bytes
ORA-00205: error in identifying control file, check alert log for more info
從提示資訊角度看，Oracle在經歷啟動nomount階段之後，在定位control file的過程中出現了問題。
老實說，雖然是測試環境，但是筆者還是比較驚慌的。於是嘗試使用srvctl叢集件啟動策略。
[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb
PRCC-1014 : sicsstb was already running
PRCR-1004 : Resource ora.sicsstb.db is already running
PRCR-1079 : Failed to start resource ora.sicsstb.db
CRS-5702: Resource 'ora.sicsstb.db' is already running on 'ncr-standby-asm'
2、問題分析
首先確認系統是否可以使用srvctl啟動，判斷一下GI上面各種資源resource狀態。
[grid@NCR-Standby-Asm ~]$ srvctl stop database -d sicsstb
[grid@NCR-Standby-Asm ~]$ srvctl status asm
ASM is running on ncr-standby-asm
[grid@NCR-Standby-Asm ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ncr-standby-asm
ora.LISTENER.lsnr
ONLINE ONLINE ncr-standby-asm
ora.RECO.dg
ONLINE ONLINE ncr-standby-asm
ora.asm
ONLINE ONLINE ncr-standby-asm Started
ora.ons
OFFLINE OFFLINE ncr-standby-asm
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE ncr-standby-asm
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE ncr-standby-asm
ora.sicsstb.db
1 OFFLINE OFFLINE Instance Shutdown
[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb
[grid@NCR-Standby-Asm ~]$
[oracle@NCR-Standby-Asm ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@NCR-Standby-Asm admin]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:14:00 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected.
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ WRITE
筆者猜測，這個故障和ASM相關。按照逐步抽絲剝繭的思路，先從資料庫日誌入手（找到失敗啟動的那次動作）。
Mon May 25 16:09:28 2015
MMON started with pid=17, OS id=4151
Mon May 25 16:09:28 2015
MMNL started with pid=18, OS id=4153
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
NOTE: initiating MARK startup
Starting background process MARK
ORACLE_BASE from environment = /u02/app/oracle
Mon May 25 16:09:28 2015
MARK started with pid=21, OS id=4161
NOTE: MARK has subscribed
Mon May 25 16:09:28 2015
ALTER DATABASE MOUNT
Mon May 25 16:09:28 2015
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))' SCOPE=MEMORY SID='sicsstb';
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup RECO was dismounted
ERROR: diskgroup RECO was not mounted
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+RECO/sicsstb/controlfile/current.256.878897845'
ORA-17503: ksfdopn:2 Failed to open file +RECO/sicsstb/controlfile/current.256.878897845
ORA-15001: diskgroup "RECO" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/sicsstb/controlfile/current.260.878897845'
ORA-17503: ksfdopn:2 Failed to open file +DATA/sicsstb/controlfile/current.260.878897845
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-205 signalled during: ALTER DATABASE MOUNT...
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Mon May 25 16:09:31 2015
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup RECO was dismounted
ERROR: diskgroup RECO was not mounted
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
從提示資訊看，Oracle在mount階段時候，利用spfile中指定的control file位置去訪問+DATA和+RECO磁碟組，但是兩個磁碟組沒有mount，所以才開始報錯。
引數中，control file以映象冗餘方式存在在ASM Diskgroup中。
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA/sicsstb/spfilesicsstb.ora
SQL> show parameter control
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
control_files string +DATA/sicsstb/controlfile/curr
ent.260.878897845, +RECO/sicsstb/controlfile/current.256.878
897845
control_management_pack_access string DIAGNOSTIC+TUNING
注意：此處的ASM無法啟動，並不是筆者沒有啟動ASM元件。如果是簡單因為ASM元件沒有開啟，先啟動資料庫服務的話，錯誤資訊如下：
[oracle@NCR-Standby-Asm ~]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.4.0 Production on Mon Jun 1 08:39:11 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/sicsstb/spfilesicsstb.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/sicsstb/spfilesicsstb.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
nomount階段要訪問spfile，我們的SPFILE是在+DATA裡面，如果ASM真的不可用的話，連nomount階段都不能進入。
提示資訊上，似乎是筆者的ASM驅動有問題。筆者作業系統環境是Red Hat Linux 6.5，使用kmod作為ASM驅動程式。
[root@NCR-Standby-Asm ~]# rpm -qa | grep asm
libatasmart-0.17-4.el6_2.x86_64
oracleasmlib-2.0.4-1.el6.x86_64
oracleasm-support-2.1.8-1.el6.x86_64
kmod-oracleasm-2.0.6.rh1-3.el6_5.x86_64
查詢對應生成的trace檔案，可以看到問題的更詳細描述。
[root@NCR-Standby-Asm trace]# tail -n 200 sicsstb_rbal_4147.trc
Trace file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u02/app/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: NCR-Standby-Asm
Release: 2.6.32-431.el6.x86_64
Version: #1 SMP Sun Nov 10 22:19:54 EST 2013
Machine: x86_64
VM name: VMWare Version: 6
Instance name: sicsstb
Redo thread mounted by this instance: 0
Oracle process number: 15
Unix process pid: 4147, image: oracle@NCR-Standby-Asm (RBAL)
*** 2015-05-25 16:09:31.634
*** SESSION ID:(190.1) 2015-05-25 16:09:31.634
*** CLIENT ID:() 2015-05-25 16:09:31.634
*** SERVICE NAME:() 2015-05-25 16:09:31.634
*** MODULE NAME:() 2015-05-25 16:09:31.634
*** ACTION NAME:() 2015-05-25 16:09:31.634
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Incident 9721 created, dump file: /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
error 488 detected in background process
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+465
----- End of Abridged Call Stack Trace -----
*** 2015-05-25 16:09:32.865
RBAL (ospid: 4147): terminating the instance due to error 488
ksuitm: waiting up to [5] seconds before killing DIAG(4129)
終止程式操作，檢視alert log的進一步詳細資訊。
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc (incident=9721):
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
RBAL (ospid: 4147): terminating the instance due to error 488
System state dump requested by (instance=1, osid=4147 (RBAL)), summary=[abnormal instance termination].
System State dumped to trace file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_diag_4129_20150525160933.trc
Dumping diagnostic data in directory=[cdmp_20150525160933], requested by (instance=1, osid=4147 (RBAL)), summary=[abnormal instance termination].
Instance terminated by RBAL, pid = 4147
Mon May 25 16:09:40 2015
Adjusting the default value of parameter parallel_max_servers
from 160 to 120 due to the value of parameter processes (150)
提示資訊中出現了ora-600錯誤，並且在最後有一個半提示半建議的資訊，要求提升引數parallel_max_servers的數量引數。GI和ASM要伴隨著多個並行工作程式，筆者猜測一種可能是不是程式數量過多，引數設定較小而引起的故障。
嘗試將引數進行調整。
SQL> show parameter parallel_max_servers
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
parallel_max_servers integer 120
SQL> alter system set parallel_max_servers=150 scope=both;
System altered.
SQL> show parameter parallel_max
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
parallel_max_servers integer 150
在MOS上，筆者也進行了檢查，Oracle一些文章認為是許可權問題。但是似乎沒有過多問題。
[oracle@NCR-Standby-Asm ~]$ cd $ORACLE_HOME/bin
[oracle@NCR-Standby-Asm bin]$ ls -l grep oracle
ls: cannot access grep: No such file or directory
-rwsr-s--x 1 oracle asmadmin 239882127 May 25 17:06 oracle
之後，重啟Database，服務正常。
SQL> startup
ORACLE instance started.
Total System Global Area 2087780352 bytes
Fixed Size 2254824 bytes
Variable Size 553650200 bytes
Database Buffers 1526726656 bytes
Redo Buffers 5148672 bytes
Database mounted.
Database opened.
故障解決。
3、結論
老實說，筆者對這個故障的解決還是有一些不明白的地方。從直觀看，在進行補丁操作之後，Oracle例項對程式數目要求是增加的，所以需要進行一些調整。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/17203031/viewspace-1687952/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- ASM Disk Group TemplateASM
- Oracle ASM spfile in a disk groupOracleASM
- Oracle ASM ACFS disk group rebalanceOracleASM
- Oracle ASM Disk Group AttributesOracleASM
- ASM disk group mount fails with ORA-15036ASMAI
- ORA-15260: permission denied on ASM disk groupASM
- How to move ASM spfile to a different disk group [ID 1082943.1]ASM
- ASM disk group mount fails with ORA-15036: disk is truncated [ID 1077175.1]ASMAI
- Oracle ASM disk誤被格式化OracleASM
- 移動ASM的spfile到一個新的disk groupASM
- oracle crs voting disk損壞一例（asm+rac）OracleASM
- drop asm disk、撤銷drop asm diskASM
- 規劃ASM DISK GROUP、檢視asm 磁碟當前狀態、mount or dismount 磁碟組ASM
- 手工建立ASM Disk Groups、為 ASM Disk Groups 新增 diskASM
- Asm disk managerASM
- asm報錯一例-support 報錯rename failedASMAI
- 執行SQL發生錯誤！錯誤：disk I/O errorSQLError
- Oracle ASM 配置錯誤OracleASM
- Oracle ASM Disk DirectoryOracleASM
- Oracle ASM Disk PartnerOracleASM
- ASM無法自動載入磁碟組問題解決一例ASM
- ASM: Device is already labeled for ASM diskASMdev
- Oracle ASM Disk HeaderOracleASMHeader
- 解決ORA-15036:disk is truncated錯誤
- [排錯]ORA-15024: discovered duplicately numbered ASM disk 0ASM
- Oracle ASM Disk Used Space DirectoryOracleASM
- ASM DISK HEADER CORRUPTION & REPAIRASMHeaderAI
- ASM Normal需要3個disk ?ASMORM
- asm disk 的結構（1）ASM
- SQL server 7安裝錯誤一例SQLServer
- oracleasm createdisk ASM: Device is already labeled for ASM diskOracleASMdev
- oracle 10.2.0.4 rac asm磁碟組載入時遭遇 ORA-00600 kfgFinalize_2 錯誤OracleASM
- ORA-15183 Unable to Create Database on Server using 11.2 ASMDatabaseServerASM
- ASM 翻譯系列第十彈：ASM Internal ASM DISK headerASMHeader
- PBOOTCMS網站程式提示“執行SQL發生錯誤！錯誤：DISK I/O ERROR”boot網站SQLError
- ASM裝載磁碟組時ORA-15063錯誤處理ASM
- MySQL sql_mode=only_full_group_by 錯誤MySql
- Using ASMLIB Management ASM DiskASM