10G RAC安裝時Failed to upgrade Oracle Cluster Registry configuration

10G RAC安裝時遇到Failed to upgrade Oracle Cluster Registry configuration，原因為使用DMM多路徑軟體device-mapper-multipath，觸發此BUG。

關於DMM多路徑軟體device-mapper-multipath上安裝RAC，參考文件Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OL5 (文件 ID 564580.1)，使用多路徑時，需要手動建立裸裝置-10G中OCR and voting disks需要用裸裝置，11g開始支援DMM mapping出來的檔案。

3. Create Raw Devices

During the installation of Oracle Clusterware 10g Release 2 (10.2.0), the Universal Installer (OUI) is unable to verify the sharedness of block devices, therefore requires the use of raw devices (whether to singlepath or multipath devices) to be specified for OCR and voting disks. As mentioned earlier, this is no longer the case from Oracle11g R1 (11.1.0) that can use multipathed block devices directly.

Manually create raw devices to bind against multipathed device partitions (/dev/mapper/*pN). Disregard device permissions for now - this will be addressed later.For example: [root@315rac01 ~]# sh /oracle/app/oracle/product/10.2.0/crs/root.sh
WARNING: directory '/oracle/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/app/oracle/product' is not owned by root
WARNING: directory '/oracle/app/oracle' is not owned by root
WARNING: directory '/oracle/app' is not owned by root
WARNING: directory '/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory
Setting up NS directories
Failed to upgrade Oracle Cluster Registry configuration

這一點在參考文件Configuring raw devices (multipath) for Oracle Clusterware 10g Release 2 (10.2.0) on RHEL5/OL5 (文件 ID 564580.1)中也有描述，

7. Install Oracle 10gR2 Clusterware

Proceed to install Oracle Clusterware 10g Release 2 (10.2.0), ensuring to specify the appropriate raw devices (/dev/raw/rawN) for OCR and voting disks. OCR devices are initialised (formatted) as part of running the root.sh script. Before running root.sh, be aware that several known issues exist that will cause the Clusterware installation to fail, namely:

• Bug.4679769 FAILED TO FORMAT OCR DISK USING CLSFMT
• Note.414163.1 10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures)

Due to Bug.4679769, initialisation of multipathed OCR devices will fail. Therefore, before runningroot.sh, download and apply patch for Bug.4679769. If root.sh was already run without first having applied patch forBug.4679769, remove (null) the failed, partially initialised OCR structures from all OCR devices, for example:

也就是說在執行root.sh指令碼前要先打補丁Bug.4679769，再將OCR盤DD一下，再用clsfmt.bin驗證一下是否能FORMAT OCR盤

大致如下：

[root@315rac01 bin]# cd /home/oracle/install/4679769/
[root@315rac01 4679769]# ls
clsfmt.bin  README.txt
[root@315rac01 4679769]# ls -al
total 692
drwxrwxr-x 2 oracle oinstall   4096 Nov  9  2005 .
drwxr-xr-x 6 oracle oinstall   4096 Dec 29 16:51 ..
-rw-r--r-- 1 oracle oinstall 687320 Nov  9  2005 clsfmt.bin
-rw-rw-r-- 1 oracle oinstall   4266 Nov  9  2005 README.txt
[root@315rac01 4679769]# cp clsfmt.bin /oracle/app/oracle/product/10.2.0/crs/bin/
[root@315rac01 4679769]# cd -
/oracle/app/oracle/product/10.2.0/crs/bin
[root@315rac01 bin]# chmod 755 clsfmt.bin
[root@315rac01 bin]# dd if=/dev/null of=/dev/raw/raw9 bs=1024k count=1000
0+0 records in
0+0 records out
0 bytes (0 B) copied, 3.9e-05 seconds, 0.0 kB/s
[root@315rac01 bin]# dd if=/dev/zero of=/dev/raw/raw9 bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 2.00336 seconds, 523 MB/s
[root@315rac01 bin]# dd if=/dev/zero of=/dev/raw/raw10 bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.99393 seconds, 526 MB/s
[root@315rac01 bin]# ./clsfmtbin ocr /dev/raw/raw9
bash: ./clsfmtbin: No such file or directory
[root@315rac01 bin]# ./clsfmt.bin ocr /dev/raw/raw9
clsfmt: successfully initialized file /dev/raw/raw9
[root@315rac01 bin]# ./clsfmt.bin ocr /dev/raw/raw10
clsfmt: successfully initialized file /dev/raw/raw10

再次執行root.sh就可以了

[root@315rac01 bin]# sh /oracle/app/oracle/product/10.2.0/crs/root.sh
WARNING: directory '/oracle/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/app/oracle/product' is not owned by root
WARNING: directory '/oracle/app/oracle' is not owned by root
WARNING: directory '/oracle/app' is not owned by root
WARNING: directory '/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/app/oracle/product' is not owned by root
WARNING: directory '/oracle/app/oracle' is not owned by root
WARNING: directory '/oracle/app' is not owned by root
WARNING: directory '/oracle' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: 315rac01 priv_rac01 315rac01
node 2: 315rac02 priv_rac02 315rac02
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw3
Now formatting voting device: /dev/raw/raw4
Now formatting voting device: /dev/raw/raw5
Format of 3 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        315rac01
CSS is inactive on these nodes.
        315rac02
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.