接續：RAC安裝配置和使用過程的問題解決方法總結二

在RAC安裝配置和使用過程的問題解決方法總結二中我們提過因為RAC節點虛擬機器重啟導致的crs daemon啟動失敗並進行了簡單的解決，
再次測試後發現，每次重啟/dev/raw下的raw1/2/3/4/5的許可權都會變為root.root，因此把下面兩句話加入/etc/rc.d/rc.local：
chown -R oracle.oinstall /dev/raw
chmod -R 777 /dev/raw

然後作如下測試：
[root@10grac1 init.d]# ./init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources 
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.


[root@10grac1 init.d]# ./init.crs start
Startup will be queued to init within 90 seconds.


[oracle@10grac1 bin]$ crs_stat -t
名稱         型別         目標    狀態    主機      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    UNKNOWN   10grac2     
ora....ac2.ons application    ONLINE    UNKNOWN   10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2 

--對UNKNOWN的服務進行手工stop手工start
[oracle@10grac1 bin]$ crs_stop -f ora.10grac2.gsd
Attempting to stop `ora.10grac2.gsd` on member `10grac2`
Stop of `ora.10grac2.gsd` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_start ora.10grac2.gsd
Attempting to start `ora.10grac2.gsd` on member `10grac2`
Start of `ora.10grac2.gsd` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_stat -t
名稱         型別         目標    狀態    主機      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    UNKNOWN   10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2     
[oracle@10grac1 bin]$ crs_stop -f ora.10grac2.ons
Attempting to stop `ora.10grac2.ons` on member `10grac2`
Stop of `ora.10grac2.ons` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_start ora.10grac2.ons
Attempting to start `ora.10grac2.ons` on member `10grac2`
Start of `ora.10grac2.ons` on member `10grac2` succeeded.
[oracle@10grac1 bin]$ crs_stat -t
名稱         型別         目標    狀態    主機      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2 


之後從頭再執行一遍：
[root@10grac1 init.d]# ./init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources 
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.


[root@10grac1 init.d]# ./init.crs start
Startup will be queued to init within 90 seconds.


[oracle@10grac1 bin]$ crs_stat -t
名稱         型別         目標    狀態    主機      
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2 
全部OK!

接下來再次重啟兩臺虛擬機器看看是什麼狀態：
重啟完成後，進入其中一個節點：
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    OFFLINE               
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2     
發現第二個節點的listener offline，嘗試解決：
NAME=ora.10grac2.LISTENER_10GRAC2.lsnr
TYPE=application
TARGET=ONLINE
STATE=OFFLINE

[oracle@10grac1 ~]$ crsctl start resources ora.10grac2.LISTENER_10GRAC2.lsnr
Starting resources.
Successfully started CRS resources 
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    UNKNOWN   10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2     

[oracle@10grac1 ~]$ crs_stop ora.10grac2.LISTENER_10GRAC2.lsnr
Attempting to stop `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2`
`ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-0216: Could not stop resource 'ora.10grac2.LISTENER_10GRAC2.lsnr'.


[oracle@10grac1 ~]$ crs_start ora.10grac2.LISTENER_10GRAC2.lsnr
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.10grac2.LISTENER_10GRAC2.lsnr'


CRS-0223: Resource 'ora.10grac2.LISTENER_10GRAC2.lsnr' has placement error.


[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    UNKNOWN   10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2  

等了一會兒，再執行，居然就對了。。。
[oracle@10grac1 ~]$ crs_stop ora.10grac2.LISTENER_10GRAC2.lsnr
Attempting to stop `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2`
Stop of `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_start ora.10grac2.LISTENER_10GRAC2.lsnr
Attempting to start `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2`
Start of `ora.10grac2.LISTENER_10GRAC2.lsnr` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    UNKNOWN   10grac2     

神奇啊，crs啊，你究竟在後面搗鼓啥呢！我讓你搗鼓個夠兒哈！
[oracle@10grac1 ~]$ crs_stop ora.racnan.racnan2.inst
Attempting to stop `ora.racnan.racnan2.inst` on member `10grac2`
Stop of `ora.racnan.racnan2.inst` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_start ora.racnan.racnan2.inst
Attempting to start `ora.racnan.racnan2.inst` on member `10grac2`
Start of `ora.racnan.racnan2.inst` on member `10grac2` succeeded.
[oracle@10grac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    10grac1     
ora....C1.lsnr application    ONLINE    ONLINE    10grac1     
ora....ac1.gsd application    ONLINE    ONLINE    10grac1     
ora....ac1.ons application    ONLINE    ONLINE    10grac1     
ora....ac1.vip application    ONLINE    ONLINE    10grac1     
ora....SM2.asm application    ONLINE    ONLINE    10grac2     
ora....C2.lsnr application    ONLINE    ONLINE    10grac2     
ora....ac2.gsd application    ONLINE    ONLINE    10grac2     
ora....ac2.ons application    ONLINE    ONLINE    10grac2     
ora....ac2.vip application    ONLINE    ONLINE    10grac2     
ora.racnan.db  application    ONLINE    ONLINE    10grac2     
ora....vice.cs application    ONLINE    ONLINE    10grac1     
ora....an1.srv application    ONLINE    ONLINE    10grac1     
ora....n1.inst application    ONLINE    ONLINE    10grac1     
ora....n2.inst application    ONLINE    ONLINE    10grac2     

---以後再重啟再遇到新情況就再來彙報，今天就到這裡啦，俺還要去搗鼓完NOSQL的作業呢

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26521853/viewspace-1085480/，如需轉載，請註明出處，否則將追究法律責任。