ASM DISK Group載入ORA-15183錯誤一例

realkid4發表於2015-06-05

 

進入Oracle 11gR2ASMAutomatic Storage Management)從Database元件中剝離出來,作為獨立元件Component進入Grid管理範疇。

本篇主要介紹筆者遇到的一個資料庫啟動載入過程中出現的問題。同官方MOS推薦的策略相比,有一些不同之處。記錄下來,留待需要的朋友待查使用。

 

1、問題說明

 

筆者環境是Oracle 單例項+Grid Infrastructure,版本號為11.2.0.4。由於安全原因,從MOS上下載了最新的安全補丁和升級補丁。升級之後的版本為11.2.0.4.6

但是,在升級最後步驟——執行SQL指令碼環節,出現了一些問題。

 

 

SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:08:57 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

 

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup

ORACLE instance started.

 

Total System Global Area 2087780352 bytes

Fixed Size                  2254824 bytes

Variable Size             553650200 bytes

Database Buffers         1526726656 bytes

Redo Buffers                5148672 bytes

ORA-00205: error in identifying control file, check alert log for more info

 

 

從提示資訊角度看,Oracle在經歷啟動nomount階段之後,在定位control file的過程中出現了問題。

老實說,雖然是測試環境,但是筆者還是比較驚慌的。於是嘗試使用srvctl叢集件啟動策略。

 

 

[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb

PRCC-1014 : sicsstb was already running

PRCR-1004 : Resource ora.sicsstb.db is already running

PRCR-1079 : Failed to start resource ora.sicsstb.db

CRS-5702: Resource 'ora.sicsstb.db' is already running on 'ncr-standby-asm'

 

 

2、問題分析

 

首先確認系統是否可以使用srvctl啟動,判斷一下GI上面各種資源resource狀態。

 

 

[grid@NCR-Standby-Asm ~]$ srvctl stop database -d sicsstb

[grid@NCR-Standby-Asm ~]$ srvctl status asm

ASM is running on ncr-standby-asm

[grid@NCR-Standby-Asm ~]$ crsctl stat res -t -init

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS      

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

               ONLINE  ONLINE       ncr-standby-asm                             

ora.LISTENER.lsnr

               ONLINE  ONLINE       ncr-standby-asm                              

ora.RECO.dg

               ONLINE  ONLINE       ncr-standby-asm                             

ora.asm

               ONLINE  ONLINE       ncr-standby-asm          Started            

ora.ons

               OFFLINE OFFLINE      ncr-standby-asm                             

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.cssd

      1        ONLINE  ONLINE       ncr-standby-asm                             

ora.diskmon

      1        OFFLINE OFFLINE                                                  

ora.evmd

      1        ONLINE  ONLINE       ncr-standby-asm                             

ora.sicsstb.db

      1        OFFLINE OFFLINE                               Instance Shutdown  

 

 

[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb

[grid@NCR-Standby-Asm ~]$

 

 

[oracle@NCR-Standby-Asm ~]$ cd $ORACLE_HOME/rdbms/admin

[oracle@NCR-Standby-Asm admin]$ sqlplus /nolog

 

SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:14:00 2015

 

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

 

SQL> conn / as sysdba

Connected.

SQL> select open_mode from v$database;

 

OPEN_MODE

--------------------

READ WRITE

 

 

筆者猜測,這個故障和ASM相關。按照逐步抽絲剝繭的思路,先從資料庫日誌入手(找到失敗啟動的那次動作)。

 

 

Mon May 25 16:09:28 2015

MMON started with pid=17, OS id=4151

Mon May 25 16:09:28 2015

MMNL started with pid=18, OS id=4153

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

starting up 1 shared server(s) ...

NOTE: initiating MARK startup

Starting background process MARK

ORACLE_BASE from environment = /u02/app/oracle

Mon May 25 16:09:28 2015

MARK started with pid=21, OS id=4161

NOTE: MARK has subscribed

Mon May 25 16:09:28 2015

ALTER DATABASE   MOUNT

Mon May 25 16:09:28 2015

ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))' SCOPE=MEMORY SID='sicsstb';

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

SUCCESS: diskgroup DATA was dismounted

ERROR: diskgroup DATA was not mounted

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

SUCCESS: diskgroup RECO was dismounted

ERROR: diskgroup RECO was not mounted

ORA-00210: cannot open the specified control file

ORA-00202: control file: '+RECO/sicsstb/controlfile/current.256.878897845'

ORA-17503: ksfdopn:2 Failed to open file +RECO/sicsstb/controlfile/current.256.878897845

ORA-15001: diskgroup "RECO" does not exist or is not mounted

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-00210: cannot open the specified control file

ORA-00202: control file: '+DATA/sicsstb/controlfile/current.260.878897845'

ORA-17503: ksfdopn:2 Failed to open file +DATA/sicsstb/controlfile/current.260.878897845

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-205 signalled during: ALTER DATABASE   MOUNT...

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

Mon May 25 16:09:31 2015

SUCCESS: diskgroup DATA was dismounted

ERROR: diskgroup DATA was not mounted

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

SUCCESS: diskgroup RECO was dismounted

ERROR: diskgroup RECO was not mounted

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-15183: ASMLIB initialization error [driver/agent not installed]

 

 

從提示資訊看,Oraclemount階段時候,利用spfile中指定的control file位置去訪問+DATA+RECO磁碟組,但是兩個磁碟組沒有mount,所以才開始報錯。

引數中,control file以映象冗餘方式存在在ASM Diskgroup中。

 

 

SQL> show parameter spfile

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

spfile   string      +DATA/sicsstb/spfilesicsstb.ora

 

SQL> show parameter control

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

control_file_record_keep_time        integer     7

control_files                        string      +DATA/sicsstb/controlfile/curr

                                                 ent.260.878897845, +RECO/sicsstb/controlfile/current.256.878

                                                 897845

control_management_pack_access       string      DIAGNOSTIC+TUNING

 

 

注意:此處的ASM無法啟動,並不是筆者沒有啟動ASM元件。如果是簡單因為ASM元件沒有開啟,先啟動資料庫服務的話,錯誤資訊如下:

 

 

[oracle@NCR-Standby-Asm ~]$ sqlplus /nolog

 

SQL*Plus: Release 11.2.0.4.0 Production on Mon Jun 1 08:39:11 2015

 

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

 

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATA/sicsstb/spfilesicsstb.ora'

ORA-17503: ksfdopn:10 Failed to open file +DATA/sicsstb/spfilesicsstb.ora

ORA-15077: could not locate ASM instance serving a required diskgroup

 

 

nomount階段要訪問spfile,我們的SPFILE是在+DATA裡面,如果ASM真的不可用的話,連nomount階段都不能進入。

提示資訊上,似乎是筆者的ASM驅動有問題。筆者作業系統環境是Red Hat Linux 6.5,使用kmod作為ASM驅動程式。

 

 

[root@NCR-Standby-Asm ~]# rpm -qa | grep asm

libatasmart-0.17-4.el6_2.x86_64

oracleasmlib-2.0.4-1.el6.x86_64

oracleasm-support-2.1.8-1.el6.x86_64

kmod-oracleasm-2.0.6.rh1-3.el6_5.x86_64

 

 

查詢對應生成的trace檔案,可以看到問題的更詳細描述。

 

 

[root@NCR-Standby-Asm trace]# tail -n 200 sicsstb_rbal_4147.trc

Trace file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Data Mining

and Real Application Testing options

ORACLE_HOME = /u02/app/oracle/product/11.2.0/dbhome_1

System name:    Linux

Node name:      NCR-Standby-Asm

Release:        2.6.32-431.el6.x86_64

Version:        #1 SMP Sun Nov 10 22:19:54 EST 2013

Machine:        x86_64

VM name:        VMWare Version: 6

Instance name: sicsstb

Redo thread mounted by this instance: 0

Oracle process number: 15

Unix process pid: 4147, image: oracle@NCR-Standby-Asm (RBAL)

 

 

*** 2015-05-25 16:09:31.634

*** SESSION ID:(190.1) 2015-05-25 16:09:31.634

*** CLIENT ID:() 2015-05-25 16:09:31.634

*** SERVICE NAME:() 2015-05-25 16:09:31.634

*** MODULE NAME:() 2015-05-25 16:09:31.634

*** ACTION NAME:() 2015-05-25 16:09:31.634

 

ERROR: asm_version error. err: driver/agent not installed rc:2

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ERROR: asm_version error. err: driver/agent not installed rc:2

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ERROR: asm_version error. err: driver/agent not installed rc:2

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ERROR: asm_version error. err: driver/agent not installed rc:2

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ERROR: asm_version error. err: driver/agent not installed rc:2

ORA-15183: ASMLIB initialization error [driver/agent not installed]

ORA-15183: ASMLIB initialization error [driver/agent not installed]

Incident 9721 created, dump file: /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc

ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []

 

error 488 detected in background process

ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []

kjzduptcctx: Notifying DIAG for crash event

----- Abridged Call Stack Trace -----

ksedsts()+465

----- End of Abridged Call Stack Trace -----

 

*** 2015-05-25 16:09:32.865

RBAL (ospid: 4147): terminating the instance due to error 488

ksuitm: waiting up to [5] seconds before killing DIAG(4129)

 

 

終止程式操作,檢視alert log的進一步詳細資訊。

 

 

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc  (incident=9721):

ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []

Incident details in: /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:

ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []

RBAL (ospid: 4147): terminating the instance due to error 488

System state dump requested by (instance=1, osid=4147 (RBAL)), summary=[abnormal instance termination].

System State dumped to trace file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_diag_4129_20150525160933.trc

Dumping diagnostic data in directory=[cdmp_20150525160933], requested by (instance=1, osid=4147 (RBAL)), summary=[abnormal instance termination].

Instance terminated by RBAL, pid = 4147

Mon May 25 16:09:40 2015

Adjusting the default value of parameter parallel_max_servers

from 160 to 120 due to the value of parameter processes (150)

 

 

提示資訊中出現了ora-600錯誤,並且在最後有一個半提示半建議的資訊,要求提升引數parallel_max_servers的數量引數。GIASM要伴隨著多個並行工作程式,筆者猜測一種可能是不是程式數量過多,引數設定較小而引起的故障。

嘗試將引數進行調整。

 

 

SQL> show parameter parallel_max_servers

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

parallel_max_servers                 integer     120

 

SQL> alter system set parallel_max_servers=150 scope=both;

 

System altered.

 

SQL> show parameter parallel_max

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

parallel_max_servers                 integer     150

 

 

MOS上,筆者也進行了檢查,Oracle一些文章認為是許可權問題。但是似乎沒有過多問題。

 

 

[oracle@NCR-Standby-Asm ~]$ cd $ORACLE_HOME/bin

[oracle@NCR-Standby-Asm bin]$ ls -l grep oracle

ls: cannot access grep: No such file or directory

-rwsr-s--x 1 oracle asmadmin 239882127 May 25 17:06 oracle

 

 

之後,重啟Database,服務正常。

 

 

SQL> startup

ORACLE instance started.

 

Total System Global Area 2087780352 bytes

Fixed Size                  2254824 bytes

Variable Size             553650200 bytes

Database Buffers         1526726656 bytes

Redo Buffers                5148672 bytes

Database mounted.

Database opened.

 

 

故障解決。

 

3、結論

 

老實說,筆者對這個故障的解決還是有一些不明白的地方。從直觀看,在進行補丁操作之後,Oracle例項對程式數目要求是增加的,所以需要進行一些調整。


來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/17203031/viewspace-1687952/,如需轉載,請註明出處,否則將追究法律責任。

相關文章