Oracle Enterprise Linux 5.6下安裝Oracle 10g RAC執行root.sh報錯問題解決

安裝clusterware執行到最後一步，要求分別在節點1和節點2上執行/u01/oraInventory/orainstRoot.sh和/u01/crs/root.sh

節點1執行/u01/oraInventory/orainstRoot.sh
[root@node1 logs]# cd /u01/oraInventory/
[root@node1 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/oraInventory to 770.
Changing groupname of /u01/oraInventory to oinstall.
The execution of the script is complete

節點2執行/u01/oraInventory/orainstRoot.sh
[root@node2 u01]# cd /u01/oraInventory/
[root@node2 oraInventory]# ./orainstRoot.sh
Changing permissions of /u01/oraInventory to 770.
Changing groupname of /u01/oraInventory to oinstall.
The execution of the script is complete

節點1執行/u01/crs/root.sh
[root@node1 oraInventory]# cd /u01/crs
[root@node1 crs]# ./root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01' is not owned by root
/u01/crs/bin/crsctl.bin: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory
Failure initializing entries in /etc/oracle/scls_scr/node1.
--提示缺少libstdc++.so.5


節點2執行/u01/crs/root.sh

[root@node2 oraInventory]# cd /u01/crs
[root@node2 crs]# ./root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory
Setting up NS directories
Failed to upgrade Oracle Cluster Registry configuration

在節點1上重新執行了次
[root@node1 node1]# cd /u01/crs/
[root@node1 crs]# ./root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Failed to upgrade Oracle Cluster Registry configuration

看下節點1上相關的日誌
cd /u01/crs/log/node1
[root@node1 node1]# cat alertnode1.log
2013-12-18 11:36:50.236
[client(6285)]CRS-1006:The OCR location /dev/raw/raw2 is inaccessible. Details in /u01/crs/log/node1/client/ocrconfig_6285.log.
2013-12-18 11:36:50.248
[client(6285)]CRS-1006:The OCR location /dev/raw/raw2 is inaccessible. Details in /u01/crs/log/node1/client/ocrconfig_6285.log.
2013-12-18 11:36:50.257
[client(6285)]CRS-1006:The OCR location /dev/raw/raw2 is inaccessible. Details in /u01/crs/log/node1/client/ocrconfig_6285.log.
2013-12-18 11:36:50.281
[client(6285)]CRS-1001:The OCR was formatted using version 2.

都是提示ocrconfig_6285.log，是跟OCR有關

[root@node1 node1]# cd client/
[root@node1 client]# ll
total 8
-rw-r----- 1 root root 2049 Dec 18 11:36 ocrconfig_6285.log
-rw-r----- 1 root root  533 Dec 18 11:46 ocrconfig_6580.log
[root@node1 client]# cat ocrconfig_6285.log
Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle.  All rights reserved.
2013-12-18 11:36:50.205: [ OCRCONF][3790528]ocrconfig starts...
2013-12-18 11:36:50.205: [ OCRCONF][3790528]Upgrading OCR data
2013-12-18 11:36:50.232: [  OCRRAW][3790528]propriogid:1: INVALID FORMAT
2013-12-18 11:36:50.236: [  OCRRAW][3790528]ibctx:1:ERROR: INVALID FORMAT
2013-12-18 11:36:50.236: [  OCRRAW][3790528]proprinit:problem reading the bootblock or superbloc 22

2013-12-18 11:36:50.237: [ default][3790528]a_init:7!: Backend init unsuccessful : [22]
2013-12-18 11:36:50.237: [ OCRCONF][3790528]Exporting OCR data to [OCRUPGRADEFILE]
2013-12-18 11:36:50.237: [  OCRAPI][3790528]a_init:7!: Backend init unsuccessful : [33]
2013-12-18 11:36:50.237: [ OCRCONF][3790528]There was no previous version of OCR. error:[PROC-33: Oracle Cluster Registry is not configured]
2013-12-18 11:36:50.247: [  OCRRAW][3790528]propriogid:1: INVALID FORMAT
2013-12-18 11:36:50.248: [  OCRRAW][3790528]ibctx:1:ERROR: INVALID FORMAT
2013-12-18 11:36:50.248: [  OCRRAW][3790528]proprinit:problem reading the bootblock or superbloc 22

2013-12-18 11:36:50.248: [ default][3790528]a_init:7!: Backend init unsuccessful : [22]
2013-12-18 11:36:50.256: [  OCRRAW][3790528]propriogid:1: INVALID FORMAT
2013-12-18 11:36:50.257: [  OCRRAW][3790528]ibctx:1:ERROR: INVALID FORMAT
2013-12-18 11:36:50.257: [  OCRRAW][3790528]proprinit:problem reading the bootblock or superbloc 22

2013-12-18 11:36:50.265: [  OCRRAW][3790528]propriogid:1: INVALID FORMAT
2013-12-18 11:36:50.275: [  OCRRAW][3790528]propriowv: Vote information on disk 0 [/dev/raw/raw2] is adjusted from [0/0] to [2/2]
2013-12-18 11:36:50.281: [  OCRRAW][3790528]propriniconfig:No 92 configuration
2013-12-18 11:36:50.281: [  OCRAPI][3790528]a_init:6a: Backend init successful
2013-12-18 11:36:50.313: [ OCRCONF][3790528]Initialized DATABASE keys in OCR
2013-12-18 11:36:50.347: [ OCRCONF][3790528]Successfully set skgfr block 0
2013-12-18 11:36:50.347: [ OCRCONF][3790528]Exiting [status=success]...

---這個錯誤參考了以下兩篇文章：

http://cnhtm.itpub.net:8080/22049049/viewspace-1033180/

透過清除OCR磁碟資訊，重新執行root.sh就可以解決

但是節點1第一次執行root.sh的那個錯誤，有點奇怪：
/u01/crs/bin/crsctl.bin: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory
Failure initializing entries in /etc/oracle/scls_scr/node1.

使用ldd命令檢視crsctl.bin使用的動態函式庫：
[root@node1 raw]# ldd /u01/crs/bin/crsctl.bin
        linux-gate.so.1 =>  (0x00570000)
        libclntsh.so.10.1 => not found
        libhasgen10.so => not found
        libocr10.so => not found
        libocrb10.so => not found
        libocrutl10.so => not found
        libskgxn2.so => not found
        libdl.so.2 => /lib/libdl.so.2 (0x00110000)
        libm.so.6 => /lib/libm.so.6 (0x00e6e000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x00231000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x00e9a000)
        libstdc++.so.5 => not found   ---確實沒有
        libc.so.6 => /lib/libc.so.6 (0x0024a000)
        /lib/ld-linux.so.2 (0x00c32000)

網上查了資料，都說是安裝了compat-libstdc++包就可以了，但是我係統裡明明有安裝這個包的，有點費解

[root@node1 raw]# rpm -qa | grep libstdc++
compat-libstdc++-296-2.96-138
libstdc++-4.1.2-50.el5
libstdc++-devel-4.1.2-50.el5
包明明都已安裝

[root@node1 raw]# cd /usr/lib
[root@node1 lib]# ll libstdc*
-rwxr-xr-x 1 root root 262092 Jun  5  2007 libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     31 Dec 12 16:36 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     18 Dec 10 21:18 libstdc++.so.6 -> libstdc++.so.6.0.8
-rwxr-xr-x 1 root root 925520 Nov 12  2010 libstdc++.so.6.0.8

自己建立個軟連線
[root@node1 lib]# ln -s /usr/lib/libstdc++.so.6 /usr/lib/libstdc++.so.5
[root@node1 lib]# ll libstdc*
-rwxr-xr-x 1 root root 262092 Jun  5  2007 libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     31 Dec 12 16:36 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     23 Dec 18 13:58 libstdc++.so.5 -> /usr/lib/libstdc++.so.6
lrwxrwxrwx 1 root root     18 Dec 10 21:18 libstdc++.so.6 -> libstdc++.so.6.0.8
-rwxr-xr-x 1 root root 925520 Nov 12  2010 libstdc++.so.6.0.8

先清空OCR磁碟/dev/raw/raw2
[root@node1 lib]# dd if=/dev/zero of=/dev/raw/raw2 bs=10240k
dd: writing `/dev/raw/raw2': No space left on device
11+0 records in
10+0 records out
113246208 bytes (113 MB) copied, 0.349732 seconds, 324 MB/s

重新執行root.sh
[root@node1 lib]# cd /u01/crs
[root@node1 crs]# ./root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01' is not owned by root
/u01/crs/bin/crsctl.bin: /usr/lib/libstdc++.so.5: version `CXXABI_1.2' not found (required by /u01/crs/bin/crsctl.bin)
/u01/crs/bin/crsctl.bin: /usr/lib/libstdc++.so.5: version `GLIBCPP_3.2' not found (required by /u01/crs/bin/crsctl.bin)
Failure initializing entries in /etc/oracle/scls_scr/node1.

--這樣不行，刪除剛建的軟連線
[root@node1 lib]# ll libstdc*
-rwxr-xr-x 1 root root 262092 Jun  5  2007 libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     31 Dec 12 16:36 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     23 Dec 18 13:58 libstdc++.so.5 -> /usr/lib/libstdc++.so.6
lrwxrwxrwx 1 root root     18 Dec 10 21:18 libstdc++.so.6 -> libstdc++.so.6.0.8
-rwxr-xr-x 1 root root 925520 Nov 12  2010 libstdc++.so.6.0.8
[root@node1 lib]# rm -rf libstdc++.so.5
[root@node1 lib]# ll libstdc*
-rwxr-xr-x 1 root root 262092 Jun  5  2007 libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     31 Dec 12 16:36 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     18 Dec 10 21:18 libstdc++.so.6 -> libstdc++.so.6.0.8
-rwxr-xr-x 1 root root 925520 Nov 12  2010 libstdc++.so.6.0.8
[root@node1 lib]#

[root@node1 lib]# cd /root/linux/Server/   --Oracle Enterprise Linux的ISO光碟掛載點
[root@node1 Server]# ll compat-libstdc++*
-rw-r--r-- 1 root root  92219 Nov 17  2010 compat-libstdc++-296-2.96-138.i386.rpm
-rw-r--r-- 1 root root 237144 Nov 17  2010 compat-libstdc++-33-3.2.3-61.i386.rpm
[root@node1 Server]# rpm -qa | grep compat-libstdc++
compat-libstdc++-296-2.96-138 

安裝compat-libstdc++-33-3.2.3-61.i386.rpm試試
[root@node1 Server]# rpm -ivh compat-libstdc++-33-3.2.3-61.i386.rpm
Preparing...                ########################################### [100%]
   1:compat-libstdc++-33    ########################################### [100%]

安裝完之後，libstdc++.so.5有了，估計之前是安裝錯版本了
[root@node1 Server]# cd /usr/lib
[root@node1 lib]# ll libstdc*
-rwxr-xr-x 1 root root 262092 Jun  5  2007 libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     31 Dec 12 16:36 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     18 Dec 18 14:00 libstdc++.so.5 -> libstdc++.so.5.0.7
-rwxr-xr-x 1 root root 733168 Jun  5  2007 libstdc++.so.5.0.7
lrwxrwxrwx 1 root root     18 Dec 10 21:18 libstdc++.so.6 -> libstdc++.so.6.0.8
-rwxr-xr-x 1 root root 925520 Nov 12  2010 libstdc++.so.6.0.8

[root@node1 ~]# ldd /u01/crs/bin/crsctl.bin
        linux-gate.so.1 =>  (0x008ba000)
        libclntsh.so.10.1 => not found
        libhasgen10.so => not found
        libocr10.so => not found
        libocrb10.so => not found
        libocrutl10.so => not found
        libskgxn2.so => not found
        libdl.so.2 => /lib/libdl.so.2 (0x00660000)
        libm.so.6 => /lib/libm.so.6 (0x00f60000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x004f1000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x002fe000)
        libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x00110000)
        libc.so.6 => /lib/libc.so.6 (0x00b94000)
        /lib/ld-linux.so.2 (0x001d3000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0093a000)

節點1重新執行/u01/crs/root.sh
[root@node1 lib]# dd if=/dev/zero of=/dev/raw/raw2 bs=10240k
dd: writing `/dev/raw/raw2': No space left on device
11+0 records in
10+0 records out
113246208 bytes (113 MB) copied, 0.338571 seconds, 334 MB/s
[root@node1 lib]# cd /u01/crs
[root@node1 crs]# ./root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01' is not owned by root
assigning default hostname node1 for node 1.
assigning default hostname node2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: node1 node1-priv node1
node 2: node2 node2-priv node2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw3
Format of 1 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        node1
CSS is inactive on these nodes.
        node2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
執行成功

節點2：
[root@node2 lib]# rm -rf libstdc++.so.5
[root@node2 lib]# ll libstdc*
-rwxr-xr-x 1 root root 262092 Jun  5  2007 libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     31 Dec 12 16:36 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     18 Dec 10 21:18 libstdc++.so.6 -> libstdc++.so.6.0.8
-rwxr-xr-x 1 root root 925520 Nov 12  2010 libstdc++.so.6.0.8
[root@node2 lib]# rpm -ivh /root/linux/Server/compat-libstdc++-33-3.2.3-61.i386.rpm
Preparing...                ########################################### [100%]
   1:compat-libstdc++-33    ########################################### [100%]
[root@node2 lib]# ll libstdc*
-rwxr-xr-x 1 root root 262092 Jun  5  2007 libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     31 Dec 12 16:36 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
lrwxrwxrwx 1 root root     18 Dec 18 14:58 libstdc++.so.5 -> libstdc++.so.5.0.7
-rwxr-xr-x 1 root root 733168 Jun  5  2007 libstdc++.so.5.0.7
lrwxrwxrwx 1 root root     18 Dec 10 21:18 libstdc++.so.6 -> libstdc++.so.6.0.8
-rwxr-xr-x 1 root root 925520 Nov 12  2010 libstdc++.so.6.0.8

重新執行：
[root@node2 lib]# cd /u01/crs
[root@node2 crs]# ./root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname node1 for node 1.
assigning default hostname node2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: node1 node1-priv node1
node 2: node2 node2-priv node2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        node1
        node2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
/u01/crs/jdk/jre//bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

節點2執行報錯，這個報錯，參考三思筆記，解決方法如下：
需要修改/u01/crs/bin下的vipca和srvctl:

[root@node2 crs]# pwd
/u01/crs
[root@node2 crs]# vi bin/vipca
在
#Remove this workaround when the bug 3937317 is fixed
       arch=`uname -m`
       if [ "$arch" = "i686" -o "$arch" = "ia64" ]
       then
            LD_ASSUME_KERNEL=2.4.19
            export LD_ASSUME_KERNEL
       fi
後面加一行
       unset LD_ASSUME_KERNEL

[root@node2 crs]# vi bin/srvctl
在
LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL
後面加一行
unset LD_ASSUME_KERNEL

在節點2重新執行root.sh
[root@node2 crs]# ./root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
Oracle CRS stack is already configured and will be running under init(1M)

這個時候，CRS上那些資源ONS、GSD、VIP都還沒有註冊到CRS上，在執行指令碼那個介面點選OK，再在另一個節點執行VIPCA就可以了

參考資料：
三思筆記：手把手教你用VMware在linux下安裝oracle10g RAC

http://cnhtm.itpub.net:8080/22049049/viewspace-1033180/
http://www.itpub.net/forum.php?mod=viewthread&tid=526976
http://blog.itpub.net/23937368/viewspace-1043824/