Solaris8上安裝RAC10202環境（二）

前一陣一直在測試ORACLE 10R2的RAC環境在Solaris上的安裝。碰到了很多的問題，不過最後總算成功了，這裡簡單總結一下安裝步驟，以及碰到的問題和解決方法。

這一篇主要討論ORACLE的CLUSTERWARE的安裝。

作業系統準備工作可以參考：Solaris8上安裝RAC10202環境（一）：http://yangtingkun.itpub.net/post/468/271797

在上一篇文章中已經將作業系統準備完畢。下面需要安裝CLUSTER軟體，Solaris下的Oracle10gRAC環境可供選擇的CLUSTER軟體包括：

SunCluster 3.1

Oracle Clusterware

Fujitsu-Siemens PrimeCluster 4.2

Veritas Storage Foundation for Oracle RAC 5.0

這篇文章介紹使用ORACLE的CLUSTERWARE來安裝RAC環境。不過只有10G以上的版本才支援Oracle Clusterware。

將Oracle的cluster安裝檔案解壓，利用cpio –idmv < 10gr2_cluster_sol.cpio命令展開。然後進入展開目錄，進入cluvfy目錄執行下面的檢測命令：

$ cd cluster_disk
$ cd cluvfy

$ ./runcluvfy.sh stage -pre crsinst -n racnode1,racnode2

Performing pre-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "racnode1".

Checking user equivalence...
User equivalence check passed for user "oracle".

Checking administrative privileges...
User existence check passed for "oracle".
Group existence check passed for "oinstall".
Membership check for user "oracle" in group "oinstall" [as Primary] passed.

Administrative privileges check passed.

Checking node connectivity...

Node connectivity check passed for subnet "172.25.0.0" with node(s) racnode2,racnode1.
Node connectivity check passed for subnet "10.0.0.0" with node(s) racnode2,racnode1.

Suitable interfaces for the private interconnect on subnet "172.25.0.0":
racnode2 ce0:172.25.198.223
racnode1 ce0:172.25.198.222

Suitable interfaces for the private interconnect on subnet "10.0.0.0":
racnode2 ce1:10.0.0.2
racnode1 ce1:10.0.0.1

ERROR:
Could not find a suitable set of interfaces for VIPs.

Node connectivity check failed.

Checking system requirements for 'crs'...
Total memory check passed.
Free disk space check passed.
Swap space check failed.
Check failed on nodes:
racnode2,racnode1
System architecture check passed.
Operating system version check passed.
Package existence check passed for "SUNWarc".
Package existence check passed for "SUNWbtool".
Package existence check passed for "SUNWhea".
Package existence check passed for "SUNWlibm".
Package existence check passed for "SUNWlibms".
Package existence check passed for "SUNWsprot".
Package existence check passed for "SUNWsprox".
Package existence check passed for "SUNWtoo".
Package existence check passed for "SUNWi1of".
Package existence check passed for "SUNWi1cs".
Package existence check passed for "SUNWi15cs".
Package existence check passed for "SUNWxwfnt".
Package existence check passed for "SUNWlibC".
Package existence check failed for "SUNWscucm:3.1".
Check failed on nodes:
racnode2,racnode1
Package existence check failed for "SUNWudlmr:3.1".
Check failed on nodes:
racnode2,racnode1
Package existence check failed for "SUNWudlm:3.1".
Check failed on nodes:
racnode2,racnode1
Package existence check failed for "ORCLudlm:Dev_Release_06/11/04,_64bit_3.3.4.8_reentrant".
Check failed on nodes:
racnode2,racnode1
Package existence check failed for "SUNWscr:3.1".
Check failed on nodes:
racnode2,racnode1
Package existence check failed for "SUNWscu:3.1".
Check failed on nodes:
racnode2,racnode1
Group existence check passed for "dba".
Group existence check passed for "oinstall".
User existence check passed for "oracle".
User existence check passed for "nobody".

System requirement failed for 'crs'

Pre-check for cluster services setup was unsuccessful on all the nodes.

上一篇文章已經提到了VIP的錯誤原因。其中swap空間不足的錯誤可以忽略，在上一篇文章中已經進行了檢查，系統中有足夠的swap空間。下面在對一些系統包進行檢查時失敗，這些包是和SunCluster有關的包，由於安裝RAC準備使用Oracle的Clusterware，因此這些錯誤也可以忽略。

在安裝之前，首先配置Clusterware所需的共享儲存。這篇文件暫時沒有涉及儲存部分內容。這裡假定儲存已經安裝到兩臺伺服器上了，而且對於兩個伺服器都是可見的。而且儲存已經劃分出了一定的裸裝置供兩個伺服器使用。

由於兩臺測試伺服器的光纖卡不同，因此載入的裸裝置名稱也不相同，在racnoce1上裝置的名稱是：

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfd99114,0
1. c1t1d0
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w2100000c50acf424,0
2. c2t0d0
/pci@8,700000/QLGC,qla@3/sd@0,0
3. c2t0d1
/pci@8,700000/QLGC,qla@3/sd@0,1
4. c2t0d2
/pci@8,700000/QLGC,qla@3/sd@0,2
5. c2t0d3
/pci@8,700000/QLGC,qla@3/sd@0,3
6. c2t0d4
/pci@8,700000/QLGC,qla@3/sd@0,4
7. c2t0d5
/pci@8,700000/QLGC,qla@3/sd@0,5
8. c2t0d6
/pci@8,700000/QLGC,qla@3/sd@0,6
9. c2t0d7
/pci@8,700000/QLGC,qla@3/sd@0,7

而在racnode2上，載入的裝置名稱為：

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfd9e4b8,0
1. c1t1d0
/pci@9,600000/SUNW,qlc@2/fp@0,0/ssd@w21000004cfd9ead5,0
2. c2t500601603022E66Ad0
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,0
3. c2t500601603022E66Ad1
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,1
4. c2t500601603022E66Ad2
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,2
5. c2t500601603022E66Ad3
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,3
6. c2t500601603022E66Ad4
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,4
7. c2t500601603022E66Ad5
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,5
8. c2t500601603022E66Ad6
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,6
9. c2t500601603022E66Ad7
/pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w500601603022e66a,7

由於安裝過程中，在建立ocr和vot共享磁碟時，需要兩個伺服器具有相同的名稱，因此建立如下的連結。在racnode1上：

# mkdir /dev/rac
# ln -s -f /dev/rdsk/c2t0d2s1 /dev/rac/ocr
# ln -s -f /dev/rdsk/c2t0d2s3 /dev/rac/vot
# chown oracle:oinstall /dev/rdsk/c2t0d2s1
# chown oracle:oinstall /dev/rdsk/c2t0d2s3

在racnode2上：

# mkdir /dev/rac
# ln -s -f /dev/rdsk/c2t500601603022E66Ad2s1 /dev/rac/ocr
# ln -s -f /dev/rdsk/c2t500601603022E66Ad2s3 /dev/rac/vot
# chown oracle:oinstall /dev/rdsk/c2t500601603022E66Ad2s1
# chown oracle:oinstall /dev/rdsk/c2t500601603022E66Ad2s3

注意不要使用s0作為共享裸裝置，否則在安裝完成後執行root.sh檔案時會出現Failed to upgrade Oracle Cluster Registry configuration的錯誤資訊。

下面可以開始安裝了，啟動Xmanager，登陸racnode1執行：

# xhost +
access control disabled, clients can connect from any host
su - oracle
Sun Microsystems Inc. SunOS 5.8 Generic Patch October 2001
$ cd /data/cluster_disk
$ ./runInstaller

啟動圖形介面後，點選next。這時Oracle會提示輸入inventory路徑和作業系統組資訊：預設的就是剛才建立的/data/oracle/oraInventory目錄和oinstall，點選next。

下面Oracle會提示OraCrs10g_home1目錄的路徑，這裡預設是ORACLE_HOME的路徑：/data/oracle/product/10.2/database將其修改為/data/oracle/product/10.2/crs，並且選擇簡體中文語句，點選next。

下面Oracle會自動檢測系統是否滿足安裝需要，如果根據上面一篇文章中的內容進行了設定，這裡的檢查成功，然後進入下一步。

進入Cluster的配置，預設的Cluster Name是crs，可以修改也可以採用預設設定。Oracle會自動將安裝節點的網路配置列出來，這裡需要手工將racnode2的節點資訊：racnode2、racnode2-priv和racnode2-vip新增進去。點選next。

下面會列出可用的網路卡資訊，檢查配置的PUBLIC、PRIVATE配置是否和hosts檔案中的一致。由於當前系統配置172.25開頭的ip，Oracle的bug會認為這個ip是Private IP，因此，會將兩個網路卡的屬性都設定為PRIVATE，這裡需要手工的將子網為172.25.0.0的網路卡設定為PUBLIC。修改之後，點選next。

進入ocr的配置階段。由於ocr使用的共享磁碟來自儲存，本身已經採用了raid0的配置，所以這裡選擇External Redundancy，並在OCR Location的位置輸入剛才設定好的/dev/rac/ocr，然後點選next。

進入Voting Disk配置，出於同樣的原因選擇External Redundancy，並在Voting Disk Location的位置輸入配置好的/dev/rac/vot，點選next。

出現彙總也，點選install開始安裝。

安裝完畢需要分別在兩個節點用root先後執行兩個指令碼。前後在racnode1和racnode2上執行下面的指令碼，輸入結果是一樣的。

# . /data/oracle/oraInventory/orainstRoot.sh
Changing permissions of /data/oracle/oraInventory to 770.
Changing groupname of /data/oracle/oraInventory to oinstall.
The execution of the script is complete

對於第二個指令碼，racnode1和racnode2的結果就不同了：

# . /data/oracle/product/10.2/crs/root.sh
WARNING: directory '/data/oracle/product/10.2' is not owned by root
WARNING: directory '/data/oracle/product' is not owned by root
WARNING: directory '/data/oracle' is not owned by root
WARNING: directory '/data' is not owned by root
ln: cannot create /data/oracle/product/10.2/crs/lib/libskgxn2.so: File exists
ln: cannot create /data/oracle/product/10.2/crs/lib32/libskgxn2.so: File exists
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/data/oracle/product/10.2' is not owned by root
WARNING: directory '/data/oracle/product' is not owned by root
WARNING: directory '/data/oracle' is not owned by root
WARNING: directory '/data' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: racnode1 racnode1-priv racnode1
node 2: racnode2 racnode2-priv racnode2
Creating OCR keys for user 'root', privgrp 'other'..
Operation successful.
Now formatting voting device: /dev/rac/vot
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
racnode1
CSS is inactive on these nodes.
racnode2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.

上面是racnode1上的執行結果，而racnode2上執行：

# . /data/oracle/product/10.2/crs/root.sh
WARNING: directory '/data/oracle/product/10.2' is not owned by root
WARNING: directory '/data/oracle/product' is not owned by root
WARNING: directory '/data/oracle' is not owned by root
WARNING: directory '/data' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/data/oracle/product/10.2' is not owned by root
WARNING: directory '/data/oracle/product' is not owned by root
WARNING: directory '/data/oracle' is not owned by root
WARNING: directory '/data' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: racnode1 racnode1-priv racnode1
node 2: racnode2 racnode2-priv racnode2
clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
racnode1
racnode2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "ce0" is not public. Public interfaces should be used to configure virtual IPs.

指令碼顯示出現了錯誤。這個錯誤就是前面提到了多次的Oracle的bug，將PUBLIC的interface認為是private的，導致無法配置vip。這個問題的詳細描述會在最後一篇文章問題彙總中詳細描述，這裡只給出解決辦法。

最簡單的方法是啟動vipca圖形介面手頭配置：

# cd /data/oracle/product/10.2/crs/bin/
# ./vipca

在Xmanager中啟動一個終端，輸入上述命令，啟動vipca圖形介面。點選next，出現所有可用的網路介面，由於ce0配置的是PUBLIC INTERFACT，這裡選擇ce0，點選next，在出現的配置中IP Alias Name分別填入：racnode1-vip和racnode2-vip，IP address處填入：172.25.198.224和172.25.198.225。這裡如果你的配置是正確的，那麼你填完一個IP，Oracle會自動將剩下三個配置補齊。點選next，出現彙總頁面，檢查無誤後，點選Finish。

Oracle會執行6個步驟，Create VIP application resource、Create GSD application resource、Create ONS application resource、Start VIP application resource、Start GSD application resource、Start ONS application resource。

全部成功後點選OK，結束VIPCA的配置。

這個時候可以返回到剛才的Clusterware安裝介面，點選OK。

這個時候Oracle會嘗試啟動兩個工具並最終執行一下驗證程式。全部檢查完成，跳到安裝結束畫面，點選Exit結束Clusterware的安裝。