CRS-4639: Could not contact Oracle High Availability Services解決方案集

 [grid@vm11gr2] /home/grid> sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.1.0 Production on Sun Oct 25 10:16:21 2009

    Copyright (c) 1982, 2009, Oracle. All rights reserved.

    Connected to an idle instance.
    SQL> startup ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service SQL> 
    無法連線到CSS服務上.到作業系統上檢查一下看看

    [grid@vm11gr2] /home/grid> crsctl check css CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    [grid@vm11gr2] /home/grid> 
    [grid@vm11gr2] /home/grid> ps -ef|grep cssd
    果然沒有CSS的服務daemon程式,再看一下HAS(High Availability Service)的狀態

    [grid@vm11gr2] /home/grid> crsctl check has CRS-4638: Oracle High Availability Services is online [grid@vm11gr2] /home/grid> ps -ef|grep d.bin grid 5886 1 0 10:06 ? 00:00:01 /u01/app/grid/product/11.2/grid/bin/ohasd.bin reboot
    [grid@vm11gr2] /home/grid> 
    發現HAS的服務確實啟動了的,而ora.cssd和ora.diskmon這2個服務是依賴於HAS維護的. 進一步檢視各資源的狀態 [grid@vm11gr2] /home/grid> crs_stat -t Name                Type                 Target    State Host 
    --------------------------------------------------------------
    ora.FLASH_DATA.dg   ora.diskgroup.type   OFFLINE OFFLINE vm11gr2 
    ora.SYS_DATA.dg     ora.diskgroup.type   OFFLINE OFFLINE vm11gr2 
    ora.asm             ora.asm.type         OFFLINE OFFLINE vm11gr2 
    ora.cssd            ora.cssd.type        OFFLINE OFFLINE vm11gr2 
    ora.diskmon         ora.diskmon.type     OFFLINE OFFLINE vm11gr2 
    [grid@vm11gr2] /home/grid> 
    
    [grid@vm11gr2] /home/grid> crsctl status resource -t --------------------------------------------------------------------------------
    NAME               TARGET          STATE        SERVER     STATE_      DETAILS 
    --------------------------------------------------------------------------------
    Local Resources
    --------------------------------------------------------------------------------
    ora.FLASH_DATA.dg
                      OFFLINE          OFFLINE     vm11gr2 
    ora.SYS_DATA.dg   

                      OFFLINE          OFFLINE     vm11gr2 
    ora.asm           

                      OFFLINE          OFFLINE     vm11gr2 
    --------------------------------------------------------------------------------
    Cluster Resources
    --------------------------------------------------------------------------------
    ora.cssd         

            1 OFFLINE          OFFLINE  ora.diskmon      

            1 OFFLINE          OFFLINE  再看一下ora.cssd和ora.diskmon的屬性 [grid@vm11gr2] /home/grid> crs_stat -p ora.cssd
    NAME=ora.cssd
    TYPE=ora.cssd.type
    ACTION_SCRIPT=
    ACTIVE_PLACEMENT=0 AUTO_START=never CHECK_INTERVAL=30
    DESCRIPTION="Resource type for CSSD"
    FAILOVER_DELAY=0
    FAILURE_INTERVAL=3
    FAILURE_THRESHOLD=5
    HOSTING_MEMBERS=
    PLACEMENT=balanced
    RESTART_ATTEMPTS=5
    SCRIPT_TIMEOUT=600
    START_TIMEOUT=600
    STOP_TIMEOUT=900
    UPTIME_THRESHOLD=1m

    [grid@vm11gr2] /home/grid> crs_stat -p ora.diskmon NAME=ora.diskmon
    TYPE=ora.diskmon.type
    ACTION_SCRIPT=
    ACTIVE_PLACEMENT=0 AUTO_START=never CHECK_INTERVAL=20
    DESCRIPTION="Resource type for Diskmon"
    FAILOVER_DELAY=0
    FAILURE_INTERVAL=3
    FAILURE_THRESHOLD=5
    HOSTING_MEMBERS=
    PLACEMENT=balanced
    RESTART_ATTEMPTS=10
    SCRIPT_TIMEOUT=60
    START_TIMEOUT=60
    STOP_TIMEOUT=60
    UPTIME_THRESHOLD=5s
    [grid@vm11gr2] /home/grid> 
    到這裡基本就找到了原因了,可以看到這兩個資源的AUTO_START屬性預設都設定為never,也就是說他們不會隨著HAS服務的啟動而自動啟動的, 儘管預設情況下HAS服務是開機自動啟動的.好了，那我們就手動啟動一下吧:  [grid@vm11gr2] /home/grid> crsctl start resource ora.cssd CRS-2672: Attempting to start 'ora.cssd' on 'vm11gr2'
    CRS-2679: Attempting to clean 'ora.diskmon' on 'vm11gr2'
    CRS-2681: Clean of 'ora.diskmon' on 'vm11gr2' succeeded
    CRS-2672: Attempting to start 'ora.diskmon' on 'vm11gr2'
    CRS-2676: Start of 'ora.diskmon' on 'vm11gr2' succeeded
    CRS-2676: Start of 'ora.cssd' on 'vm11gr2' succeeded
    [grid@vm11gr2] /home/grid> 注:ora.cssd和ora.diskmon這兩個服務是有依賴關係的，啟動哪個都會把兩個都起來.

    [grid@vm11gr2] /home/grid> crs_stat -t Name                Type                 Target    State Host 
    --------------------------------------------------------------
    ora.FLASH_DATA.dg   ora.diskgroup.type   OFFLINE OFFLINE vm11gr2 
    ora.SYS_DATA.dg     ora.diskgroup.type   OFFLINE OFFLINE vm11gr2 
    ora.asm             ora.asm.type         OFFLINE OFFLINE vm11gr2 
    ora.cssd            ora.cssd.type        ONLINE  ONLINE  vm11gr2 
    ora.diskmon         ora.diskmon.type     ONLINE  ONLINE  vm11gr2 
    [grid@vm11gr2] /home/grid> CSS服務起來了，重啟動asm instance [grid@vm11gr2] /home/grid> sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.1.0 Production on Sun Oct 25 10:30:03 2009

    Copyright (c) 1982, 2009, Oracle. All rights reserved.

    Connected to an idle instance.

    SQL> startup ASM instance started

    Total System Global Area 284565504 bytes
    Fixed Size 1336036 bytes
    Variable Size 258063644 bytes
    ASM Cache 25165824 bytes
    ASM diskgroups mounted
    SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
    With the Automatic Storage Management option
    [grid@vm11gr2] /home/grid> crs_stat -t  Name                Type                 Target    State Host 
    --------------------------------------------------------------
    ora.FLASH_DATA.dg   ora.diskgroup.type   ONLINE  ONLINE  vm11gr2 
    ora.SYS_DATA.dg     ora.diskgroup.type   ONLINE  ONLINE  vm11gr2 
    ora.asm             ora.asm.type         ONLINE  ONLINE  vm11gr2 
    ora.cssd            ora.cssd.type        ONLINE  ONLINE  vm11gr2 
    ora.diskmon         ora.diskmon.type     ONLINE  ONLINE  vm11gr2 
    [grid@vm11gr2] /home/grid> tips 1)預設情況下HAS(High Availability Service)是自動啟動的.透過如下命令可以取消和啟用自動啟動
        crsctl disable has
        crsctl enable has
      2)HAS手動啟動和停止
        crsctl start has
        crsctl stop has
      3)檢視HAS的狀態
        crsctl check has
      4)如果想讓ora.css和ora.diskmon服務隨著HAS的啟動而自動啟動，那麼你可以這兩個服務的AUTO_START屬性
        crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
        or 
        crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"
      5)如果想取消ora.css和ora.diskmon的Auto start 
        crsctl modify resource "ora.cssd" -attr "AUTO_START=never"
        crsctl modify resource "ora.diskmon" -attr "AUTO_START=never" ==================================================================================
自己遇到的情況是：
 [grid@vm11gr2] /home/grid> sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.1.0 Production on Sun Oct 25 10:16:21 2009

    Copyright (c) 1982, 2009, Oracle. All rights reserved.

    Connected to an idle instance.


SQL> startup

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+RAC_DISK/RACDB/spfileRACDB.ora'

ORA-17503: ksfdopn:2 Failed to open file +RAC_DISK/RACDB/spfileRACDB.ora

ORA-15077: could not locate ASM instance serving a required diskgroup 

ORA-29701: unable to connect to Cluster Synchronization Service
----&gt>感同身受篇:rac02資料庫異常停機報錯:

Error: received an ABORT event from the Cluster Synchronization Service

Errors in file /u01/app/oracle/diag/rdbms/sun/sun/trace/sun_lmon_25888.trc:

ORA-29701: unable to connect to Cluster Synchronization Service

LMON (ospid: 25888): terminating the instance due to error 29701

Wed Nov 19 16:17:51 2014

System state dump is made for local instance

System State dumped to trace file /u01/app/oracle/diag/rdbms/sun/sun/trace/sun_diag_25864.trc

Instance terminated by LMON, pid = 25888

Wed Nov 19 23:38:23 2014

Starting ORACLE instance (normal)

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Interface type 1 eth1 172.16.2.0 configured from GPnP Profile for use as a cluster interconnect

Interface type 1 eth0 192.168.2.0 configured from GPnP Profile for use as a public interface

Picked latch-free SCN scheme 3

Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST

Autotune of undo retention is turned on.

IMODE=BR

ILAT =88

LICENSE_MAX_USERS = 0

SYS auditing is disabled

NOTE: Volume support  enabled

Starting up:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP, Data Mining

and Real Application Testing options.



--sun_lmon_25888.trc內容:

[oracle@rac02 trace]$ cat sun_lmon_25888.trc

Trace file /u01/app/oracle/diag/rdbms/bus/bus/trace/sun_lmon_25888.trc

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1

System name:    Linux

Node name:      rac02

Release:        2.6.18-308.el5

Version:        #1 SMP Fri Jan 27 17:17:51 EST 2012

Machine:        x86_64

Instance name: sun

Redo thread mounted by this instance: 1

Oracle process number: 11

Unix process pid: 25888, image: oracle@rac02 (LMON)



*** 2014-11-19 16:17:51.745

*** SESSION ID:(276.1) 2014-11-19 16:17:51.745

*** CLIENT ID:() 2014-11-19 16:17:51.745

*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-19 16:17:51.745

*** MODULE NAME:() 2014-11-19 16:17:51.745

*** ACTION NAME:() 2014-11-19 16:17:51.745

*** TRACE FILE RECREATED AFTER BEING REMOVED ***

2014-11-19 16:17:51.744: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1

2014-11-19 16:17:51.745: [ CSSCLNT]clssgsGroupGetStatus:  communications failed (0/3/-1)

2014-11-19 16:17:51.745: [ CSSCLNT]clssgsGroupGetStatus: returning 8

kgxgnpstat: received ABORT event from CLSS

kjxggpoll: received an abort event from DBALL_DB

Error: Cluster Synchronization Service aborts the instance (error 6)

LMON caught an error 29701 in the main loop

error 29701 detected in background process

ORA-29701: unable to connect to Cluster Synchronization Service



*** 2014-11-19 16:17:51.754

LMON (ospid: 25888): terminating the instance due to error 29701

ksuitm: waiting up to [5] seconds before killing DIAG(25864)