Oracle RAC 10g for Solaris環境解除安裝（二）

介紹RAC環境的安裝的文章很多，但是介紹RAC環境解除安裝的並不多見，如果RAC環境解除安裝的不徹底，很可能會影響RAC環境的再次搭建。

這一篇介紹CLUSTER軟體的解除安裝。

Oracle RAC 10g for Solaris環境解除安裝（一）：http://yangtingkun.itpub.net/post/468/502731

 

 

上一篇中解除安裝了資料庫，並刪除了資料庫軟體。

在文章的開頭已經提過，RAC環境的解除安裝不當會導致問題。正常情況下，應該首先解除安裝CLUSTER所做的修改，然後再解除安裝CLUSTERWARE。

正常情況下應該在ORA_CLS_HOME目錄下執行：/install/rootdelete.sh remote nosharedhome –force，在RAC的每個節點上都應該執行這個操作，然後在其中一個節點上執行/install/rootdeinstall.sh。刪除ASM例項對裸裝置頭的修改。

但是如果解除安裝的時候沒有注意，很容易沒有進行CLUSTER的清理就直接解除安裝了CLUSTER環境。

比如下面不進行清理而直接解除安裝CLUSTERWARE：

$ cd $ORACLE_HOME/../crs/oui/bin
$ ./runInstaller
Starting Oracle Universal Installer...

Checking installer requirements...

Checking operating system version: must be 5.8, 5.9 or 5.10.    Actual 5.8
                                      Passed

Checking Temp space: must be greater than 150 MB.   Actual 11267 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed

All installer requirements met.

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-04-22_05-45-29PM. Please wait ...$ Oracle Universal Installer, Version 10.2.0.1.0 Production
Copyright (C) 1999, 2005, Oracle. All rights reserved.

仍然點選Deinstall Products。選擇CLUSTERWARE的安裝目錄，這裡是OraCrs10g_home1，然後點選Remove進行刪除：

Oracle會彈出解除安裝的元件詳細明細，確認後，開始解除安裝。

Oracle會彈出一個提示，說明ClusterWare的安裝目錄/data/oracle/product/10.2/crs將會在解除安裝後被刪除，如果確認刪除按yes，不刪除這個目錄按no，cancel則取消操作。這裡選擇yes繼續。

在刪除CLUSTER的過程中也出現了告警，說明在racnode2和racnode3節點上，有些目錄沒有許可權刪除，這種警告是正常的，可以忽略。

解除安裝結束後，檢查日誌檔案，檔案的位置仍然是在ORACLE_BASE/oraInventory/log目錄下：

INFO: Saving serialized component list for /data/oracle/product/10.2/crs
INFO: Updating XML inventory.
INFO: Saving comps.xml for /data/oracle/product/10.2/crs
INFO: Current Inventory:
        Oracle Home: OraCrs10g_home1

INFO: Updating files in Oracle home '/data/oracle/product/10.2/crs' to remote nodes 'racnode2,racnode3'.
INFO: List of files to be excluded from:null
WARNING: Could not rename file '/tmp/OraInstall2010-04-22_06-34-16PM/installCopyFile.lst' to '/tmp/OraInstall2010-04-22_06-34-16PM/i
nstallCopyFile.lst.bak'.
WARNING: Could not rename file '/tmp/OraInstall2010-04-22_06-34-16PM/installRemoveFile.lst' to '/tmp/OraInstall2010-04-22_06-34-16PM
/installRemoveFile.lst.bak'.
SEVERE: java.io.FileNotFoundException: /tmp/OraInstall2010-04-22_06-34-16PM/installRemoveFile.lst (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.(FileInputStream.java:106)
        at java.io.FileInputStream.(FileInputStream.java:66)
        at java.io.FileReader.(FileReader.java:41)
        at oracle.sysman.oii.oiix.OiixFileOps.appendFilePw(OiixFileOps.java:2349)
        at oracle.sysman.oii.oiix.OiixFileOps.appendFile(OiixFileOps.java:2373)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.appendFilesToOH(OiipgClusterLogger.java:673)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.appendFilesToOH(OiipgClusterLogger.java:626)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.updateListFiles(OiipgClusterLogger.java:319)
        at oracle.sysman.oii.oiif.oiifw.OiifwClusterCopyWCCE.doOperation(OiifwClusterCopyWCCE.java:435)
        at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
        at oracle.sysman.oii.oiif.oiifw.OiifwActionsPhaseWCDE.doOperation(OiifwActionsPhaseWCDE.java:606)
        at oracle.sysman.oii.oiic.OiicDeinstallSession$OiicActionsThread.run(OiicDeinstallSession.java:2431)

SEVERE: java.io.FileNotFoundException: /tmp/OraInstall2010-04-22_06-34-16PM/installDirFile.lst (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.(FileInputStream.java:106)
        at java.io.FileInputStream.(FileInputStream.java:66)
        at java.io.FileReader.(FileReader.java:41)
        at oracle.sysman.oii.oiix.OiixFileOps.appendFilePw(OiixFileOps.java:2349)
        at oracle.sysman.oii.oiix.OiixFileOps.appendFile(OiixFileOps.java:2373)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.appendFilesToOH(OiipgClusterLogger.java:673)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.appendFilesToOH(OiipgClusterLogger.java:628)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.updateListFiles(OiipgClusterLogger.java:319)
        at oracle.sysman.oii.oiif.oiifw.OiifwClusterCopyWCCE.doOperation(OiifwClusterCopyWCCE.java:435)
        at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
        at oracle.sysman.oii.oiif.oiifw.OiifwActionsPhaseWCDE.doOperation(OiifwActionsPhaseWCDE.java:606)
        at oracle.sysman.oii.oiic.OiicDeinstallSession$OiicActionsThread.run(OiicDeinstallSession.java:2431)

SEVERE: java.io.FileNotFoundException: /tmp/OraInstall2010-04-22_06-34-16PM/installCopyFile.lst (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.(FileInputStream.java:106)
        at java.io.FileInputStream.(FileInputStream.java:66)
        at java.io.FileReader.(FileReader.java:41)
        at oracle.sysman.oii.oiix.OiixFileOps.appendFilePw(OiixFileOps.java:2349)
        at oracle.sysman.oii.oiix.OiixFileOps.appendFile(OiixFileOps.java:2373)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.appendFilesToOH(OiipgClusterLogger.java:673)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.appendFilesToOH(OiipgClusterLogger.java:629)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterLogger.updateListFiles(OiipgClusterLogger.java:319)
        at oracle.sysman.oii.oiif.oiifw.OiifwClusterCopyWCCE.doOperation(OiifwClusterCopyWCCE.java:435)
        at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
        at oracle.sysman.oii.oiif.oiifw.OiifwActionsPhaseWCDE.doOperation(OiifwActionsPhaseWCDE.java:606)
        at oracle.sysman.oii.oiic.OiicDeinstallSession$OiicActionsThread.run(OiicDeinstallSession.java:2431)

INFO: Error in copying cluster list file to central inventory; OUI-10185:Unable to copy '/tmp/OraInstall2010-04-22_06-34-16PM/instal
lCopyFile.lst' to '/data/oracle/oraInventory/logs/installCopyFile.lst.2010-04-22_06-34-16PM'. OUI-10186:The source file '/tmp/OraIns
tall2010-04-22_06-34-16PM/installCopyFile.lst' does not exist.
SEVERE: oracle.sysman.oii.oiip.oiipg.OiipgRemoteFileOperationException: Could not remove listed directories based on '/tmp/OraInstal
l2010-04-22_06-34-16PM/installRemoveDirFile.lst' from nodes 'racnode2,racnode3'. [PRKC-1002 : All the submitted commands did not exe
cute successfully]
----------------------------------------------------------------------------------
racnode2:
    rm: Unable to remove directory /data/oracle/product/10.2/crs/OPatch: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/bin: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cdata: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cfgtoollogs/vipca: File exists
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cfgtoollogs: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/clone: Permission denied
.
.
.
    rm: Unable to remove directory /data/oracle/product/10.2/crs/tg4sybs: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/tg4tera: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs: File exists
----------------------------------------------------------------------------------
racnode3:
    rm: Unable to remove directory /data/oracle/product/10.2/crs/OPatch: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/bin: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cfgtoollogs: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/clone: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/crs/admin: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/crs/demo: Permission denied
.
.
.
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cdata/crs: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cdata: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs: File exists
----------------------------------------------------------------------------------

        at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.throwOiipgRemoteFileOperationException(OiipgClusterOps.java:2699)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.removeListedDirsFromNodes(OiipgClusterOps.java:2120)
        at oracle.sysman.oii.oiif.oiifw.OiifwClusterCopyWCCE.doOperation(OiifwClusterCopyWCCE.java:700)
        at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
        at oracle.sysman.oii.oiif.oiifw.OiifwActionsPhaseWCDE.doOperation(OiifwActionsPhaseWCDE.java:606)
        at oracle.sysman.oii.oiic.OiicDeinstallSession$OiicActionsThread.run(OiicDeinstallSession.java:2431)
Caused by: oracle.ops.mgmt.cluster.RemoteFileOperationException: PRKC-1002 : All the submitted commands did not execute successfully
        at oracle.ops.mgmt.cluster.ClusterCmd.removeListedDirsFromNodes(ClusterCmd.java:4274)
        at oracle.ops.mgmt.cluster.ClusterCmd.removeListedDirsFromNodes(ClusterCmd.java:4083)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.removeListedDirsFromNodes(OiipgClusterOps.java:2104)
        ... 4 more

WARNING: Could not remove listed directories based on '/tmp/OraInstall2010-04-22_06-34-16PM/installRemoveDirFile.lst' from nodes 'ra
cnode2,racnode3'. [PRKC-1002 : All the submitted commands did not execute successfully]
----------------------------------------------------------------------------------
racnode2:
    rm: Unable to remove directory /data/oracle/product/10.2/crs/OPatch: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/bin: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cdata: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cfgtoollogs/vipca: File exists
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cfgtoollogs: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/clone: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/crs/admin: Permission denied
.
.
.
    rm: Unable to remove directory /data/oracle/product/10.2/crs/tg4tera: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs: File exists
----------------------------------------------------------------------------------
racnode3:
    rm: Unable to remove directory /data/oracle/product/10.2/crs/OPatch: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/bin: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cfgtoollogs: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/clone: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs/crs/admin: Permission denied
.
.
.
    rm: Unable to remove directory /data/oracle/product/10.2/crs/cdata: Permission denied
    rm: Unable to remove directory /data/oracle/product/10.2/crs: File exists
----------------------------------------------------------------------------------

Refer to '/data/oracle/oraInventory/logs/installActions2010-04-22_06-34-16PM.log' for details. You may fix the errors on the require
d remote nodes. Refer to the install guide for error recovery. Click 'Yes' if you want to proceed. Click 'No' to exit the install. D
o you want to continue?
INFO: User Selected: Yes/OK

INFO: Setting variable 'CLUSTER_NODES' to 'racnode1,racnode2,racnode3,'. Received the value from a code block.
INFO: Local node 'racnode1' is specified by the user.
INFO: Setting variable 'REMOTE_NODES' to 'racnode2,racnode3,'. Received the value from a code block.
INFO: Removing Cluster Oracle homes
INFO: Running command '/tmp/OraInstall2010-04-22_06-34-16PM/oui/bin/runInstaller -jreLoc ../../jre -paramFile /tmp/OraInstall2010-04
-22_06-34-16PM/oui/clusterparam.ini  -silent -ignoreSysPrereqs -removeHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/c
rs LOCAL_NODE=racnode2 -remoteInvocation -invokingNodeName racnode1 -logFilePath "/data/oracle/oraInventory/logs" -timestamp 2010-04
-22_06-34-16PM' on the nodes 'racnode2'.
INFO: Invoking OUI on cluster nodes racnode2
INFO: /tmp/OraInstall2010-04-22_06-34-16PM/oui/bin/runInstaller -jreLoc ../../jre -paramFile /tmp/OraInstall2010-04-22_06-34-16PM/ou
i/clusterparam.ini  -silent -ignoreSysPrereqs -removeHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs LOCAL_NODE=rac
node2 -remoteInvocation -invokingNodeName racnode1 -logFilePath "/data/oracle/oraInventory/logs" -timestamp 2010-04-22_06-34-16PM
INFO: Running command '/tmp/OraInstall2010-04-22_06-34-16PM/oui/bin/runInstaller -jreLoc ../../jre -paramFile /tmp/OraInstall2010-04
-22_06-34-16PM/oui/clusterparam.ini  -silent -ignoreSysPrereqs -removeHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/c
rs LOCAL_NODE=racnode3 -remoteInvocation -invokingNodeName racnode1 -logFilePath "/data/oracle/oraInventory/logs" -timestamp 2010-04
-22_06-34-16PM' on the nodes 'racnode3'.
INFO: Invoking OUI on cluster nodes racnode3
INFO: /tmp/OraInstall2010-04-22_06-34-16PM/oui/bin/runInstaller -jreLoc ../../jre -paramFile /tmp/OraInstall2010-04-22_06-34-16PM/ou
i/clusterparam.ini  -silent -ignoreSysPrereqs -removeHome -noClusterEnabled ORACLE_HOME=/data/oracle/product/10.2/crs LOCAL_NODE=rac
node3 -remoteInvocation -invokingNodeName racnode1 -logFilePath "/data/oracle/oraInventory/logs" -timestamp 2010-04-22_06-34-16PM
INFO: The flags set for this home (/data/oracle/product/10.2/database) are:clean
INFO: Home:/data/oracle/product/10.2/database oraclehomeproperties.xml is not saved because the flags indicate no changes to this fi
le.
INFO: The flags set for this home (/data/oracle/product/10.2/crs) are:localnode|nodelist|crs|removed
INFO: Home:/data/oracle/product/10.2/crs oraclehomeproperties.xml is not saved because the flags indicate no changes to this file.
INFO: Saving serialized component list for /data/oracle/product/10.2/crs
INFO: Updating XML inventory.
INFO: Current Inventory:

INFO: Deleting the directory: /data/oracle/product/10.2/crs
INFO: This deinstallation was successful

在上面的日誌檔案中可以看到錯誤主要是由於沒有許可權刪除root使用者建立的目錄，可以利用root來刪除這個目錄，CLUSTERWARE的解除安裝完成。

但是由於沒有進行清理工作，因此節點上還有一些程式沒有被停止：

bash-2.03$ ps -ef|grep ora
  oracle  2534     1  0   Feb 21 ?        0:00 /data/oracle/product/10.2/crs/opmn/bin/ons -d
  oracle 15523 15403  0 09:40:28 pts/1    0:00 bash
  oracle  1309  1308  0   Feb 21 ?        0:00 /bin/sh -c ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/racnode1/c
  oracle  1932     1  0   Feb 21 ?        1:02 /data/oracle/product/10.2/database/bin/tnslsnr LISTENER1_RACNODE1 -inherit
  oracle  1308  1150  0   Feb 21 ?        0:00 sh -c /bin/sh -c 'ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/rac
    root   377     1  0   Feb 21 ?       152:16 /data/oracle/product/10.2/crs/bin/crsd.bin reboot
  oracle   375     1  0   Feb 21 ?        0:00 sh -c sh -c 'ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/racnode1
    root  1216  1147  0   Feb 21 ?        0:00 /data/oracle/product/10.2/crs/bin/oprocd run -t 1000 -m 500 -f
  oracle  1562  1151  0   Feb 21 ?        1:01 /data/oracle/product/10.2/crs/bin/evmlogger.bin -o /data/oracle/product/10.2/cr
  oracle  1151   375  0   Feb 21 ?       16:10 /data/oracle/product/10.2/crs/bin/evmd.bin
  oracle  1310  1309  0   Feb 21 ?       85:41 /data/oracle/product/10.2/crs/bin/ocssd.bin
  oracle  2536  2534  0   Feb 21 ?        0:03 /data/oracle/product/10.2/crs/opmn/bin/ons -d
  oracle 17996 15523  0 09:42:04 pts/1    0:00 grep ora
  oracle 15403 15401  0 09:40:23 pts/1    0:00 -sh
bash-2.03$ su -
Password:
Sun Microsystems Inc.   SunOS 5.8       Generic Patch   October 2001
You have new mail.
# /etc/init.d/init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Shutdown has begun. The daemons should exit soon.
# ps -ef|grep ora
  oracle  2534     1  0   Feb 21 ?        0:00 /data/oracle/product/10.2/crs/opmn/bin/ons -d
  oracle 15523 15403  0 09:40:28 pts/1    0:00 bash
  oracle  1309  1308  0   Feb 21 ?        0:00 /bin/sh -c ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/racnode1/c
  oracle  1932     1  0   Feb 21 ?        1:02 /data/oracle/product/10.2/database/bin/tnslsnr LISTENER1_RACNODE1 -inherit
  oracle  1308  1150  0   Feb 21 ?        0:00 sh -c /bin/sh -c 'ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/rac
    root   377     1  0   Feb 21 ?       152:16 /data/oracle/product/10.2/crs/bin/crsd.bin reboot
  oracle   375     1  0   Feb 21 ?        0:00 sh -c sh -c 'ulimit -c unlimited; cd /data/oracle/product/10.2/crs/log/racnode1
    root  1216  1147  0   Feb 21 ?        0:00 /data/oracle/product/10.2/crs/bin/oprocd run -t 1000 -m 500 -f
  oracle  1562  1151  0   Feb 21 ?        1:01 /data/oracle/product/10.2/crs/bin/evmlogger.bin -o /data/oracle/product/10.2/cr
  oracle  1151   375  0   Feb 21 ?       16:10 /data/oracle/product/10.2/crs/bin/evmd.bin
  oracle  1310  1309  0   Feb 21 ?       85:41 /data/oracle/product/10.2/crs/bin/ocssd.bin
    root 18632 18287  0 09:42:34 pts/1    0:00 grep ora
  oracle  2536  2534  0   Feb 21 ?        0:03 /data/oracle/product/10.2/crs/opmn/bin/ons -d
  oracle 15403 15401  0 09:40:23 pts/1    0:00 -sh

可以看到，透過/etc/init.d/init.crs已經無法關閉已啟動的cluster程式了。其他兩個節點的情況類似。

如果說這些程式重啟後就會消失，那麼ASM例項對裸裝置頭的修改沒有清除，這樣以後重建RAC的時候會發現以前的磁碟組檔案仍然存在，且沒有辦法重新使用。

首先找到ocr對應的裸裝置：

bash-2.03$ more /var/opt/oracle/ocr.loc
ocrconfig_loc=/dev/rac/ocr
local_only=FALSE
bash-2.03$ ls -l /dev/rac/ocr
lrwxrwxrwx   1 root     other         18 Nov 26 16:52 /dev/rac/ocr -> /dev/rdsk/c2t1d2s1

然後利用dd手工清理ocr裝置：

# dd if=/dev/zero f=/dev/rdsk/c2t1d2s1 skip=25 bs=4k count=2560
2560+0 records in
2560+0 records out

清除/var/opt/oracle目錄下所有的檔案，否則會導致以後安裝CLUSTERWARE的時候報錯，Oracle會認為系統中已經配置過CLUSTERWARE。

至於留下的cluster相關的程式，無論是KILL的方式，還是REBOOT的方式，總之重啟系統是不可避免的了。