Clusterware Fails to Start With 'clssnmLocalJoinEvent: takeover aborted due to ALIVE node on Disk' in Ocssd.log (Doc ID 845573.1)

In this Document

Cause

Solution

References

Applies to:

Oracle Server - Enterprise Edition - Version 10.2.0.1 to 11.1.0.7 [Release 10.2 to 11.1]
Information in this document applies to any platform.
Oracle Clusterware with Vendor Clusterware

Symptoms

After a reboot, node two of a two-node cluster fails to start. 'Crsctl check crs' shows all daemons running and 'Cluvfy stage -post crsinst -n -verbose' reports:

Liveness of all the daemons
Node Name CRS daemon CSS daemon EVM daemon
------------ ------------------------ ------------------------ ----------
node02 yes yes yes
node01 yes yes yes

Checking CRS health...

Check: Health of CRS
Node Name CRS OK?
------------------------------------ ------------------------
node02 unknown
node01 yes

The ocssd.log on node02 reports the following repeatedly:

[ CSSD]2009-06-05 02:53:24.746 [6] >TRACE: clssnmReadDskHeartbeat: node(0) is down.
rcfg(4) wrtcnt(22410347) LATS(644748) Disk lastSeqNo(22410347)
[ CSSD]2009-06-05 02:53:25.927 [16] >TRACE: clssnmRcfgMgrThread: Local Join
[ CSSD]2009-06-05 02:53:25.927 [16] >WARNING: clssnmLocalJoinEvent: takeover aborted due to ALIVE node on Disk

Cause

Node02 is running the Oracle Clusterware stack on top of Veritas but node01 is running a
completely Oracle stack.

The ocssd.log for node02, on startup of the clusterware, shows:

[ CSSD]2009-06-05 02:52:59.553 [5] >TRACE: clssnm_skgxninit: initialized skgxn version
(2/0/Veritas Cluster Server MM

The ocssd.log for node01 shows:

[ CSSD]2008-09-10 05:14:09.268 [5] >TRACE: clssnm_skgxninit: Compatible vendor
clusterware not in use

ls -l of $CRS_HOME/lib/libskgxn2.so on both nodes shows:

node01:/oracle/product/crs/OPatch>ls -l $CRS_HOME/lib/libskgxn*
lrwxrwxrwx 1 root other 12 Jul 22 2008 /oracle/product/crs/lib/libskgxn2.so -> libskgxns.so
-rw-rw-r-- 1 oracle oinstall 12528 Apr 4 2008 /oracle/product/crs/lib/libskgxns.so

mtmtpd02:/oracle/product/crs/OPatch>ls -l $CRS_HOME/lib/libskgxn*
lrwxrwxrwx 1 oracle other 33 Mar 21 2006 /oracle/product/crs/lib/libskgxn2.so -> /opt/ORCLcluster/lib/libskgxn2.so
-rw-rw-r-- 1 oracle oinstall 12528 Apr 4 2008 /oracle/product/crs/lib/libskgxns.so

so we can see that libskgxn2.so on the first node is not pointing to the Veritas library.

Solution

-- To implement the solution, please execute the following steps::
1. Shutdown the Oracle Clusterware stack on node01
2. Change the link to point to /opt/ORCLcluster/lib/libskgxn2.so
3. Restart the Oracle Clusterware stack on node01
4. Shutdown and restart the Oracle Clusterware stack on node02

Database - RAC/Scalability Community
To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Database - RAC/Scalability Community