解決因ocr資源狀態為unknown導致CRS-0223

OS: Linux 2.6.9-89
DB: 2節點RAC  10205
症狀：第一個instance無法啟動，
justin5077.$ srvctl start instance -d PRE030 -i PRE030N3
PRKP-1001 : Error starting instance PRE030N3 on node justin5077
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.PRE030.PRE030N3.inst' has placement error.

登陸該節點檢視
$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora....N3.inst application    ONLINE    UNKNOWN   seml...5077
ora....N4.inst application    ONLINE    ONLINE    seml...5078
ora.PRE030.db  application    ONLINE    ONLINE    seml...5077
ora....SM3.asm application    ONLINE    ONLINE    seml...5077
ora....77.lsnr application    ONLINE    UNKNOWN   seml...5077
ora....077.gsd application    ONLINE    ONLINE    seml...5077
ora....077.ons application    ONLINE    ONLINE    seml...5077
ora....077.vip application    ONLINE    ONLINE    seml...5077
ora....SM4.asm application    ONLINE    ONLINE    seml...5078
ora....78.lsnr application    ONLINE    UNKNOWN   seml...5078
ora....078.gsd application    ONLINE    ONLINE    seml...5078
ora....078.ons application    ONLINE    ONLINE    seml...5078
ora....078.vip application    ONLINE    ONLINE    seml...5078
嘗試srvctl啟動報告上述錯誤，改用sqlplus
$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Fri May 4 08:11:05 2012

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup nomount
;ORA-03113: end-of-file on communication channel

alterlog顯示錯誤ora-29702

檢視MOS，有篇文章與現有情況類似
OCR由於不正確的使用crs_stop/crs_start出現紊亂，導致ora開頭的資源狀態變為unkown；
解決方法是以毒攻毒，即使有crs_stop停止unknown的資源，然後在嘗試srvctl啟動；
操作之前必須確保OCR有最新的備份

$ crs_stop ora.PRE030.PRE030N3.inst
Attempting to stop `ora.PRE030.PRE030N3.inst` on member `justin5077`
Stop of `ora.PRE030.PRE030N3.inst` on member `justin5077` succeeded.

$ srvctl start instance -d PRE030 -i PRE030N3
此時可以成功啟動instance

Note: crs_stop and crs_start are NOT supported for use on ora.* resources, unless you are explicitly instructed to use them by an Oracle support analyst, or following this note's steps EXACTLY. 

Improper use of crs_stop and crs_start can corrupt your OCR.

Any CRS resource can be in an UNKNOWN state, including vip, gsd, database, or instances.  The following uses an instance resource as an example.

1. Find the proper name for your instance resource.  To find it, do "crs_stat -u | grep inst", this will give the names of all instance resources in the OCR.

$ crs_stat -u | grep inst
NAME=ora.V102.V1021.inst
NAME=ora.V102.V1022.inst
NAME=ora.prod.prod1.inst
NAME=ora.prod.prod2.inst
Pick the instance associated with the node and database in question.
For example, for a database named prod and instance named prod1, the resource name will be "ora.prod.prod1.inst".

2.  Before you go any further, make sure that you have a recent backup of your OCR.
You can check on your OCR automatic backups with the command "ocrconfig -showbackup":

$ ocrconfig -showbackup

oradb1 2009/06/19 07:35:28 /u01/app/oracle/product/10gR2/crs/cdata/temp

oradb1 2009/06/19 03:35:26 /u01/app/oracle/product/10gR2/crs/cdata/temp

oradb1 2009/06/18 23:35:25 /u01/app/oracle/product/10gR2/crs/cdata/temp

oradb1 2009/06/18 03:35:20 /u01/app/oracle/product/10gR2/crs/cdata/temp

oradb1 2009/06/02 07:01:26 /u01/app/oracle/product/10gR2/crs/cdata/temp
If no backup exists, then create one using ocrconfig -export, following the instructions in the documentation:

Oracle� Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 10g Release 2 (10.2)

3. Restore the instance resource to OFFLINE with this command:

crs_stop [instance resource name]

eg:
$ crs_stop ora.prod.prod1.inst

Note: if the instance is online then this command will stop it.

 

4. Check that instance status is now OFFLINE instead of UNKNOWN

$ crs_stat -u ora.prod.prod1.inst
NAME=ora.prod.prod1.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

5. If the instance status is OFFLINE, then you can try again to start it with srvctl.

6. If the startup fails again, then open a SR and provide Oracle Support with the following information:

a) Gather a trace by using the following script.

script. /tmp/trace1.out
env
crs_stat -u ora.prod.prod1.inst
crs_stop ora.prod.prod1.inst
crs_stat -u ora.prod.prod1.inst
export SRVM_TRACE=TRUE
srvctl start instance -d prod -i prod1
export SRVM_TRACE=
crs_stat -u ora.prod.prod1.inst
exit
(Replace "ora.prod.prod1.inst" with the name of your actual instance resource; replace prod with the name of your database and prod1 with the name of your instance).

The trace log will be found in /tmp/trace1.out.

b) latest crsd.log from this node

c) the resource log for this instance from $CRS_HOME/log/<>/racg

d) ls -l of $ORACLE_HOME/log/<>/racg