環境說明：

OS:Redhat 7.5
DB:11.2.0.4.0 RAC
ASM:NFS

磁碟初始化

檢查磁碟大小

root@chenj-ch-05:/oradata3#df -h /oradata*
Filesystem                       Size  Used Avail Use% Mounted on
10.1.10.3:/test_ora19c_nfsB1     100G  710M  100G   1% /oradata1
10.1.10.3:/test_ora19c_01_nfsB1   30G   40M   30G   1% /oradata2
10.1.10.3:/test_ora19c_02_nfsB1   30G   38M   30G   1% /oradata3

檢查磁碟掛載情況

root@chenj-ch-05:/oradata3# mount 
10.1.10.3:/test_ora19c_nfsB1 on /oradata1 type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.10.3,mountvers=3,mountport=635,mountproto=tcp,local_lock=none,addr=10.1.10.3)
10.1.10.3:/test_ora19c_01_nfsB1 on /oradata2 type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.10.3,mountvers=3,mountport=635,mountproto=tcp,local_lock=none,addr=10.1.10.3)
10.1.10.3:/test_ora19c_02_nfsB1 on /oradata3 type nfs (rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.10.3,mountvers=3,mountport=635,mountproto=tcp,local_lock=none,addr=10.1.10.3)

建立共享磁碟檔案

dd if=/dev/zero of=/oradata1/test_asm/ocr1  bs=1024k count=2048
dd if=/dev/zero of=/oradata1/test_asm/ocr2  bs=1024k count=2048
dd if=/dev/zero of=/oradata1/test_asm/ocr3  bs=1024k count=2048
dd if=/dev/zero of=/oradata1/test_asm/data1  bs=1024k count=10240
dd if=/dev/zero of=/oradata1/test_asm/data2  bs=1024k count=10240
dd if=/dev/zero of=/oradata1/test_asm/data3  bs=1024k count=10240
dd if=/dev/zero of=/oradata1/test_asm/data4  bs=1024k count=10240
dd if=/dev/zero of=/oradata1/test_asm/data5  bs=1024k count=10240

修改許可權

chown grid.oinstall /oradata1/test_asm -R
chmod 660 /oradata1/test_asm/*

靜默安裝grid

grid@chenj-ch-05:/home/grid$nohup sh /oracle/tmp/grid/runInstaller -silent -noconfig -waitForCompletion -ignorePrereq -ignoreInternalDriverError -showProgress -responseFile /home/grid/grid_install.rsp > /home/grid/install_0304.log &

靜默安裝grid日誌

Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 3949 MB Passed
Checking swap space: must be greater than 150 MB. Actual 16338 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2022-03-04_02-34-22PM. Please wait ...[WARNING] [INS-30011] The SYS password entered does not conform to the Oracle recommended standards.
CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
ACTION: Provide a password that conforms to the Oracle recommended standards.
[WARNING] [INS-30011] The ASMSNMP password entered does not conform to the Oracle recommended standards.
CAUSE: Oracle recommends that the password entered should be at least 8 characters in length, contain at least 1 uppercase character, 1 lower case character and 1 digit [0-9].
ACTION: Provide a password that conforms to the Oracle recommended standards.
[WARNING] [INS-32016] The selected Oracle home contains directories or files.
CAUSE: The selected Oracle home contained directories or files.
ACTION: To start with an empty Oracle home, either remove its contents or choose another location.
You can find the log of this install session at:
/oracle/oraINventory/logs/installActions2022-03-04_02-34-22PM.log
Prepare in progress.
.................................................. 9% Done.
Prepare successful.
Copy files in progress.
.................................................. 15% Done.
.................................................. 20% Done.
.................................................. 25% Done.
.................................................. 30% Done.
.................................................. 35% Done.
.................................................. 40% Done.
.................................................. 45% Done.
........................................
Copy files successful.
Link binaries in progress.
Link binaries successful.
.................................................. 62% Done.
Setup files in progress.
Setup files successful.
.................................................. 76% Done.
Perform remote operations in progress.
.................................................. 89% Done.
Perform remote operations successful.
The installation of Oracle Grid Infrastructure 11g was successful.
Please check '/oracle/oraINventory/logs/silentInstall2022-03-04_02-34-22PM.log' for more details.
.................................................. 94% Done.
Execute Root Scripts in progress.
As a root user, execute the following script(s):
1. /oracle/oraINventory/orainstRoot.sh
2. /oracle/grid/product/11.2/root.sh
Execute /oracle/oraINventory/orainstRoot.sh on the following nodes:
[chenj-ch-05, chenj-ch-06]
Execute /oracle/grid/product/11.2/root.sh on the following nodes:
[chenj-ch-05, chenj-ch-06]
.................................................. 100% Done.
Execute Root Scripts successful.
As install user, execute the following script to complete the configuration.
1. /oracle/grid/product/11.2/cfgtoollogs/configToolAllCommands RESPONSE_FILE=<response_file>
Note:
1. This script must be run on the same host from where installer was run.
2. This script needs a small password properties file for configuration assistants that require passwords (refer to install guide documentation).
Successfully Setup Software.

節點1 root.sh執行日誌

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /oracle/grid/product/11.2
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/grid/product/11.2/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'chenj-ch-05'
CRS-2676: Start of 'ora.mdnsd' on 'chenj-ch-05' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'chenj-ch-05'
CRS-2676: Start of 'ora.gpnpd' on 'chenj-ch-05' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'chenj-ch-05'
CRS-2672: Attempting to start 'ora.gipcd' on 'chenj-ch-05'
CRS-2676: Start of 'ora.cssdmonitor' on 'chenj-ch-05' succeeded
CRS-2676: Start of 'ora.gipcd' on 'chenj-ch-05' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'chenj-ch-05'
CRS-2672: Attempting to start 'ora.diskmon' on 'chenj-ch-05'
CRS-2676: Start of 'ora.diskmon' on 'chenj-ch-05' succeeded
CRS-2676: Start of 'ora.cssd' on 'chenj-ch-05' succeeded
ASM created and started successfully.
Disk Group GD_TEST_OCR created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 2c78cf2ab1ac4f99bfbeafadec4caa01.
Successful addition of voting disk 003f25c63a7e4f15bfe6ce115377e75d.
Successful addition of voting disk 083b82cd51cc4fb0bf44ba96af95c99c.
Successfully replaced voting disk group with +GD_TEST_OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c78cf2ab1ac4f99bfbeafadec4caa01 (/oradata1/test_asm/ocr1) [GD_TEST_OCR]
 2. ONLINE   003f25c63a7e4f15bfe6ce115377e75d (/oradata1/test_asm/ocr2) [GD_TEST_OCR]
 3. ONLINE   083b82cd51cc4fb0bf44ba96af95c99c (/oradata1/test_asm/ocr3) [GD_TEST_OCR]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'chenj-ch-05'
CRS-2676: Start of 'ora.asm' on 'chenj-ch-05' succeeded
CRS-2672: Attempting to start 'ora.GD_TEST_OCR.dg' on 'chenj-ch-05'
CRS-2676: Start of 'ora.GD_TEST_OCR.dg' on 'chenj-ch-05' succeeded
FirstNode configuration failed at /oracle/grid/product/11.2/crs/install/crsconfig_lib.pm line 9379.
/oracle/grid/product/11.2/perl/bin/perl -I/oracle/grid/product/11.2/perl/lib -I/oracle/grid/product/11.2/crs/install /oracle/grid/product/11.2/crs/install/rootcrs.pl execution failed

執行節點2 root.sh

完整日誌：

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/grid/product/11.2/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to inittab
ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow: 
2022-03-04 15:24:05.980: 
[client(55011)]CRS-2101:The OLR was formatted using version 3.
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node chenj-ch-05, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

檢視crs

grid@chenj-ch-06:/home/grid$crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GD_TEST_OCR.dg
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
ora.asm
               ONLINE  ONLINE       chenj-ch-05              Started             
               ONLINE  ONLINE       chenj-ch-06              Started             
ora.gsd
               OFFLINE OFFLINE      chenj-ch-05                                  
               OFFLINE OFFLINE      chenj-ch-06                                  
ora.net1.network
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
ora.ons
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       chenj-ch-05                                  
ora.cvu
      1        OFFLINE OFFLINE                                                   
ora.chenj-ch-05.vip
      1        ONLINE  ONLINE       chenj-ch-05                                  
ora.chenj-ch-06.vip
      1        ONLINE  ONLINE       chenj-ch-06                                  
ora.oc4j
      1        ONLINE  ONLINE       chenj-ch-06                                  
ora.scan1.vip
      1        ONLINE  ONLINE       chenj-ch-05

節點1：執行configToolAllCommands

生成rsp檔案

su - grid
vi /home/grid/cfgrsp.properties
oracle.assistants.asm|S_ASMPASSWORD=oracle
oracle.assistants.asm|S_ASMMONITORPASSWORD=oracle
oracle.crs|S_BMCPASSWORD=oracle

開始執行

/oracle/ggrid@chenj-ch-05:/home/grid$/oracle/grid/product/11.2/cfgtoollogs/configToolAllCommands RESPONSE_FILE=/home/grid/cfgrsp.properties
Setting the invPtrLoc to /oracle/grid/product/11.2/oraInst.loc
perform - mode is starting for action: configure
Mar 4, 2022 3:40:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn invoke
INFO: Executing ASMCA
Mar 4, 2022 3:40:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn invoke
INFO: Command /oracle/grid/product/11.2/bin/asmca -silent -postConfigureASM -oui_internal  
Mar 4, 2022 3:40:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn.handleProcess() entered.
Mar 4, 2022 3:40:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
Mar 4, 2022 3:40:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: checking secretArguments.
Mar 4, 2022 3:40:46 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: ... GenericInternalPlugIn: starting read loop.
Mar 4, 2022 3:40:49 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: SYS_PASSWORD_PROMPT
Mar 4, 2022 3:40:49 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Processing: SYS_PASSWORD_PROMPT for argument tag -sysAsmPassword
Mar 4, 2022 3:40:49 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: ASMSNMP_PASSWORD_PROMPT
Mar 4, 2022 3:40:49 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Processing: ASMSNMP_PASSWORD_PROMPT for argument tag -asmMonitorPassword
Mar 4, 2022 3:40:49 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: End of argument passing to stdin
Mar 4, 2022 3:40:50 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 
Mar 4, 2022 3:40:50 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: PostConfiguration completed successfully
Mar 4, 2022 3:40:50 PM oracle.install.driver.oui.config.GenericInternalPlugIn handleProcess
INFO: Read: 
perform - mode finished for action: configure
You can see the log file: /oracle/grid/product/11.2/cfgtoollogs/oui/configActions2022-03-04_03-39-48-PM.logrid/product/11.2/cfgtoollogs/configToolAllCommands RESPONSE_FILE=/home/grid/cfgrsp.properties

檢視執行日誌

實際呼叫如下命令：

/oracle/grid/product/11.2/oui/bin/runInstaller -nowait -noconsole -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true "CLUSTER_NODES={chenj-ch-05,chenj-ch-06}" ORACLE_HOME=/oracle/grid/product/11.2

日誌：

vi /oracle/grid/product/11.2/cfgtoollogs/oui/configActions2022-03-04_03-39-48-PM.logrid/product/11.2/cfgtoollogs/configToolAllCommands RESPONSE_FILE=/home/grid/cfgrsp.properties
###################################################
The action configuration is performing
------------------------------------------------------
The plug-in Update Inventory is running
/oracle/grid/product/11.2/oui/bin/runInstaller -nowait -noconsole -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true "CLUSTER_NODES={chenj-ch-05,chenj-ch-06}" ORACLE_HOME=/oracle/grid/product/11.2
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 16338 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /oracle/oraINventory
The plug-in Update Inventory has successfully been performed
------------------------------------------------------
------------------------------------------------------
The plug-in Oracle Net Configuration Assistant is running
Parsing command line arguments:
    Parameter "orahome" = /oracle/grid/product/11.2
    Parameter "orahnam" = Ora11g_gridinfrahome1
    Parameter "instype" = typical
    Parameter "inscomp" = client,oraclenet,javavm,server,ano
    Parameter "insprtcl" = tcp
    Parameter "cfg" = local
    Parameter "authadp" = NO_VALUE
    Parameter "responsefile" = /oracle/grid/product/11.2/network/install/netca_typ.rsp
    Parameter "silent" = true
    Parameter "silent" = true
Done parsing command line arguments.
Oracle Net Services Configuration:
Profile configuration complete.
Profile configuration complete.
chenj-ch-05...
chenj-ch-06...
Oracle Net Listener Startup:
    Listener started successfully.
Listener configuration complete
......
Post-check for cluster services setup was unsuccessful on all the nodes.
The plug-in Oracle Cluster Verification Utility has failed its perform method
------------------------------------------------------
The action configuration has failed its perform method
###################################################

Oracle軟體安裝

root@chenj-ch-05:/oracle#cp -r oraINventory oraINventory_bak

靜默安裝

nohup sh /oracle/tmp/database/runInstaller -silent -noconfig -waitForCompletion -ignorePrereq -ignoreInternalDriverError -showProgress -responseFile /home/oracle/rsp/db_install.rsp > /home/oracle/install_0304.log &

安裝日誌

Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB.   Actual 3949 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 16338 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2022-03-04_03-47-30PM. Please wait ...
You can find the log of this install session at:
 /oracle/oraINventory/logs/installActions2022-03-04_03-47-30PM.log
Prepare in progress.
..................................................   9% Done.
Prepare successful.
Copy files in progress.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   36% Done.
..................................................   41% Done.
..................................................   46% Done.
..................................................   51% Done.
..................................................   56% Done.
Copy files successful.
..........
Link binaries in progress.
Link binaries successful.
..................................................   77% Done.
Setup files in progress.
..................................................   94% Done.
Setup files successful.
The installation of Oracle Database 11g was successful.
Please check '/oracle/oraINventory/logs/silentInstall2022-03-04_03-47-30PM.log' for more details.
Execute Root Scripts in progress.
As a root user, execute the following script(s):
        1. /oracle/db/product/11.2/root.sh
Execute /oracle/db/product/11.2/root.sh on the following nodes: 
[chenj-ch-05, chenj-ch-06]
..................................................   100% Done.
Execute Root Scripts successful.
Successfully Setup Software.

執行指令碼

root@chenj-ch-05:/root#/oracle/db/product/11.2/root.sh
Check /oracle/db/product/11.2/install/root_chenj-ch-05_2022-03-04_15-56-21.log for the output of root script
root@chenj-ch-05:/root#vi /oracle/db/product/11.2/install/root_chenj-ch-05_2022-03-04_15-56-21.log
root@chenj-ch-06:/oracle#/oracle/db/product/11.2/root.sh
Check /oracle/db/product/11.2/install/root_chenj-ch-06_2022-03-04_15-57-10.log for the output of root script
root@chenj-ch-06:/oracle#vi /oracle/db/product/11.2/install/root_chenj-ch-06_2022-03-04_15-57-10.log

日誌如下：

Performing root user operation for Oracle 11g
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/db/product/11.2
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
Finished product-specific root actions.

檢查安裝

oracle@chenj-ch-05:/home/oracle$sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Fri Mar 4 15:55:38 2022
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL>

建立磁碟組

SQL> select name,total_mb,free_mb from v$asm_diskgroup;
NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
GD_TEST_OCR                          6144       5218
SQL>
CREATE DISKGROUP GD_TEST_DATA
    EXTERNAL REDUNDANCY
    DISK '/oradata1/test_asm/data1','/oradata1/test_asm/data2','/oradata1/test_asm/data3';
SQL> select name,state,type from v$asm_diskgroup;
NAME                           STATE       TYPE
------------------------------ ----------- ------
GD_TEST_OCR                    MOUNTED     NORMAL
GD_TEST_DATA                   MOUNTED     EXTERN

節點2：

SQL> select name,state,type from v$asm_diskgroup;
NAME                           STATE       TYPE
------------------------------ ----------- ------
GD_TEST_OCR                    MOUNTED     NORMAL
GD_TEST_DATA                   DISMOUNTED
SQL> alter diskgroup GD_TEST_DATA mount;
Diskgroup altered.
SQL> select name,state,type from v$asm_diskgroup;
NAME                           STATE       TYPE
------------------------------ ----------- ------
GD_TEST_OCR                    MOUNTED     NORMAL
GD_TEST_DATA                   MOUNTED     EXTERN

DBCA建庫

oracle@chenj-ch-05:/home/oracle/rsp$dbca -silent -createdatabase -responseFile  /home/oracle/rsp/dbca.rsp

日誌如下：

Cleaning up failed steps
4% complete
Copying database files
5% complete
6% complete
7% complete
13% complete
19% complete
24% complete
30% complete
33% complete
Creating and starting Oracle instance
35% complete
39% complete
43% complete
46% complete
47% complete
48% complete
50% complete
52% complete
Creating cluster database views
54% complete
71% complete
Completing Database Creation
74% complete
77% complete
85% complete
94% complete
100% complete
Look at the log file "/oracle/db/cfgtoollogs/dbca/test/test5.log" for further details.

RAC檢查

oracle@chenj-ch-05:/home/oracle/rsp$srvctl config database
test
oracle@chenj-ch-05:/home/oracle/rsp$srvctl status database -d test
Instance test1 is running on node chenj-ch-05
Instance test2 is running on node chenj-ch-06
grid@chenj-ch-05:/home/grid$crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GD_TEST_DATA.dg
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
ora.GD_TEST_OCR.dg
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
ora.LISTENER.lsnr
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
ora.asm
               ONLINE  ONLINE       chenj-ch-05              Started             
               ONLINE  ONLINE       chenj-ch-06              Started             
ora.gsd
               OFFLINE OFFLINE      chenj-ch-05                                  
               OFFLINE OFFLINE      chenj-ch-06                                  
ora.net1.network
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
ora.ons
               ONLINE  ONLINE       chenj-ch-05                                  
               ONLINE  ONLINE       chenj-ch-06                                  
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       chenj-ch-05                                  
ora.cvu
      1        OFFLINE OFFLINE                                                   
ora.chenj-ch-05.vip
      1        ONLINE  ONLINE       chenj-ch-05                                  
ora.chenj-ch-06.vip
      1        ONLINE  ONLINE       chenj-ch-06                                  
ora.oc4j
      1        ONLINE  ONLINE       chenj-ch-06                                  
ora.scan1.vip
      1        ONLINE  ONLINE       chenj-ch-05                                  
ora.test.db
      1        ONLINE  ONLINE       chenj-ch-05              Open                
      2        ONLINE  ONLINE       chenj-ch-06              Open

資料測試

節點1：

SQL> create user cjc identified by a;
User created.
SQL> grant dba to cjc;
Grant succeeded.
SQL> conn cjc/a
Connected.
SQL> 
SQL> 
SQL> create table t1 as select level as id from dual connect by level <=10;
Table created.
SQL> select * from t1;
        ID
----------
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
10 rows selected.

節點2：

oracle@chenj-ch-06:/home/oracle$sqlplus cjc/a
SQL*Plus: Release 11.2.0.4.0 Production on Fri Mar 4 16:17:40 2022
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select * from t1;
        ID
----------
         1
         2
         3
         4
         5
         6
         7
         8
         9
        10
10 rows selected.

重啟：

oracle@chenj-ch-05:/home/oracle$srvctl stop database -d test
oracle@chenj-ch-05:/home/oracle$srvctl start database -d test

重啟crs

停crs

root@chenj-ch-06:/oracle/grid/product/11.2/bin#./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.crsd' on 'chenj-ch-06'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.GD_TEST_OCR.dg' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.test.db' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.oc4j' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.chenj-ch-06.vip' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.test.db' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.GD_TEST_DATA.dg' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.chenj-ch-06.vip' on 'chenj-ch-06' succeeded
CRS-2672: Attempting to start 'ora.chenj-ch-06.vip' on 'chenj-ch-05'
CRS-2677: Stop of 'ora.GD_TEST_DATA.dg' on 'chenj-ch-06' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'chenj-ch-06' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'chenj-ch-05'
CRS-2677: Stop of 'ora.oc4j' on 'chenj-ch-06' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'chenj-ch-05'
CRS-2676: Start of 'ora.chenj-ch-06.vip' on 'chenj-ch-05' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'chenj-ch-05' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'chenj-ch-05'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'chenj-ch-05' succeeded
CRS-2677: Stop of 'ora.GD_TEST_OCR.dg' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.asm' on 'chenj-ch-06' succeeded
CRS-2674: Start of 'ora.oc4j' on 'chenj-ch-05' failed
CRS-2679: Attempting to clean 'ora.oc4j' on 'chenj-ch-05'
CRS-2681: Clean of 'ora.oc4j' on 'chenj-ch-05' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.ons' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.net1.network' on 'chenj-ch-06' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'chenj-ch-06' has completed
CRS-2677: Stop of 'ora.crsd' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.crf' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.ctssd' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.evmd' on 'chenj-ch-06'
CRS-2673: Attempting to stop 'ora.asm' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.mdnsd' on 'chenj-ch-06' succeeded
CRS-2677: Stop of 'ora.evmd' on 'chenj-ch-06' succeeded
CRS-2677: Stop of 'ora.crf' on 'chenj-ch-06' succeeded
CRS-2677: Stop of 'ora.asm' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'chenj-ch-06' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.cssd' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.gipcd' on 'chenj-ch-06' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'chenj-ch-06'
CRS-2677: Stop of 'ora.gpnpd' on 'chenj-ch-06' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'chenj-ch-06' has completed
CRS-4133: Oracle High Availability Services has been stopped.

啟動crs過程

root@chenj-ch-06:/oracle/grid/product/11.2/bin#./crsctl start crs
2022-03-04 16:50:50.088: 
[ohasd(69993)]CRS-2112:The OLR service started on node chenj-ch-06.
2022-03-04 16:50:50.093: 
[ohasd(69993)]CRS-1301:Oracle High Availability Service started on node chenj-ch-06.
2022-03-04 16:50:50.093: 
[ohasd(69993)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2022-03-04 16:50:53.370: 
[/oracle/grid/product/11.2/bin/orarootagent.bin(70033)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). 
2022-03-04 16:50:57.737: 
[gpnpd(70128)]CRS-2328:GPNPD started on node chenj-ch-06. 
2022-03-04 16:51:00.056: 
[cssd(70196)]CRS-1713:CSSD daemon is started in clustered mode
2022-03-04 16:51:01.902: 
[ohasd(69993)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2022-03-04 16:51:01.902: 
[ohasd(69993)]CRS-2769:Unable to failover resource 'ora.diskmon'.
2022-03-04 16:51:20.542: 
[cssd(70196)]CRS-1707:Lease acquisition for node chenj-ch-06 number 2 completed
2022-03-04 16:51:21.814: 
[cssd(70196)]CRS-1605:CSSD voting file is online: /oradata1/test_asm/ocr1; details in /oracle/grid/product/11.2/log/chenj-ch-06/cssd/ocssd.log.
2022-03-04 16:51:21.858: 
[cssd(70196)]CRS-1605:CSSD voting file is online: /oradata1/test_asm/ocr2; details in /oracle/grid/product/11.2/log/chenj-ch-06/cssd/ocssd.log.
2022-03-04 16:51:21.898: 
[cssd(70196)]CRS-1605:CSSD voting file is online: /oradata1/test_asm/ocr3; details in /oracle/grid/product/11.2/log/chenj-ch-06/cssd/ocssd.log.
2022-03-04 16:51:25.529: 
[cssd(70196)]CRS-1601:CSSD Reconfiguration complete. Active nodes are chenj-ch-05 chenj-ch-06 .
2022-03-04 16:51:27.063: 
[ctssd(70307)]CRS-2403:The Cluster Time Synchronization Service on host chenj-ch-06 is in observer mode.
2022-03-04 16:51:27.309: 
[ctssd(70307)]CRS-2407:The new Cluster Time Synchronization Service reference node is host chenj-ch-05.
2022-03-04 16:51:27.310: 
[ctssd(70307)]CRS-2401:The Cluster Time Synchronization Service started on host chenj-ch-06.
2022-03-04 16:51:29.059: 
[ohasd(69993)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2022-03-04 16:51:29.059: 
[ohasd(69993)]CRS-2769:Unable to failover resource 'ora.diskmon'.
2022-03-04 16:51:47.564: 
[crsd(70414)]CRS-1012:The OCR service started on node chenj-ch-06.
2022-03-04 16:51:47.572: 
[evmd(70329)]CRS-1401:EVMD started on node chenj-ch-06.
2022-03-04 16:51:49.614: 
[crsd(70414)]CRS-1201:CRSD started on node chenj-ch-06.
2022-03-04 16:51:57.716: 
[client(70701)]CRS-4743:File /oracle/grid/product/11.2/oc4j/j2ee/home/OC4J_DBWLM_config/system-jazn-data.xml was updated from OCR(Size: 13365(New), 13378(Old) bytes)

檢視會話

SQL> select inst_id,count(*) from gv$session group by inst_id;
   INST_ID   COUNT(*)
---------- ----------
         1         41
         2         39

排錯過程

問題一： crsconfig_lib.pm line 9379

報錯：

FirstNode configuration failed at /oracle/grid/product/11.2/crs/install/crsconfig_lib.pm line 9379.
/oracle/grid/product/11.2/perl/bin/perl -I/oracle/grid/product/11.2/perl/lib -I/oracle/grid/product/11.2/crs/install /oracle/grid/product/11.2/crs/install/rootcrs.pl execution failed

手動執行：

/oracle/grid/product/11.2/perl/bin/perl -I/oracle/grid/product/11.2/perl/lib -I/oracle/grid/product/11.2/crs/install /oracle/grid/product/11.2/crs/install/rootcrs.pl
...
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr
CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'chenj-ch-05'
FirstNode configuration failed at /oracle/grid/product/11.2/crs/install/crsconfig_lib.pm line 9379.

檢查crsd.log

grid@chenj-ch-05:/oracle/grid/product/11.2/log/chenj-ch-05/crsd$vi crsd.log 
2022-03-04 15:06:08.395: [UiServer][1295947520]{1:56663:491} Container [ Name: ORDER
        MESSAGE:
        TextMessage[CRS-2674: Start of 'ora.oc4j' on 'chenj-ch-05' failed]
        MSGTYPE:
        TextMessage[1]
        OBJID:
        TextMessage[ora.oc4j]
        WAIT:
        TextMessage[0]
]
...
2022-03-04 15:11:15.902: [   CRSPE][1298048768]{1:56663:598} Op 0x7fddfc1266d0 has 3 WOs
2022-03-04 15:11:15.903: [   CRSPE][1298048768]{1:56663:598} ICE has queued an operation. Details: Operation [START of [ora.oc4j 1 1] on [chenj-ch-05] : local=0, unplanned=00x7fddfc1266d0] cannot run cause it needs W lock for: WO for Placement Path RI:[ora.oc4j 1 1] server [chenj-ch-05] target states [ONLINE INTERMEDIATE ], locked by op [START of [ora.oc4j 1 1] on [chenj-ch-05] : local=0, unplanned=00x7fddfc123e60]. Owner: CRS-2682: It is locked by 'grid' for command 'Start Resource' issued from 'chenj-ch-05

手動啟動

grid@chenj-ch-05:/oracle/grid/product/11.2/log/chenj-ch-05/crsd$srvctl start oc4j -v
OC4J could not be started
PRCR-1079 : Failed to start resource ora.oc4j
CRS-2674: Start of 'ora.oc4j' on 'chenj-ch-05' failed
CRS-2632: There are no more servers to try to place resource 'ora.oc4j' on that would satisfy its placement policy

先跳過，不處理，ora.oc4不影響安裝

問題二: gpnptool rc=256

節點1執行root.sh遇到的另一個錯誤：

錯誤：

Failed to sign a peer profile for Oracle Cluster GPnP. gpnptool rc=256
Creation of Oracle GPnP peer profile failed for chenj-ch-05
Failed to create GPnP peer profile for chenj-ch-05 at /oracle/grid/product/11.2/crs/install/crsconfig_lib.pm line 5013.
/oracle/grid/product/11.2/perl/bin/perl -I/oracle/grid/product/11.2/perl/lib -I/oracle/grid/product/11.2/crs/install /oracle/grid/product/11.2/crs/install/rootcrs.pl execution failed

問題原因：

在安裝grid階段，執行root.sh前，更新了31537677 Opatch，導致root.sh執行失敗。

回退OPatch

opatch rollback -id 31537677

問題三： CRS-2101

執行節點2 root.sh

日誌：

報錯，hang住

ohasd failed to start
Failed to start the Clusterware. Last 20 lines of the alert log follow: 
2022-03-04 15:24:05.980: 
[client(55011)]CRS-2101:The OLR was formatted using version 3.

解決方案：

root@chenj-ch-06:/root#cd /var/tmp/.oracle/
root@chenj-ch-06:/var/tmp/.oracle#ls
npohasd
root@chenj-ch-06:/var/tmp/.oracle#ll -rth
total 0
prw-r--r-- 1 root root 0 Mar  3 21:41 npohasd
root@chenj-ch-06:/var/tmp/.oracle#rm npohasd 
rm: remove fifo ‘npohasd’? y
root@chenj-ch-06:/var/tmp/.oracle#touch npohasd
root@chenj-ch-06:/var/tmp/.oracle#chmod 755 npohasd 
root@chenj-ch-06:/var/tmp/.oracle#ll -rth
total 0
-rwxr-xr-x 1 root root 0 Mar  3 21:48 npohasd

問題四: Oracle Grid Infrastructure is not installed

###執行configToolAllCommands

必須執行configToolAllCommands，如果不執行，後面安裝oracle時，會提示下面錯誤：

  CAUSE: Before you can install Oracle RAC, you must install Oracle Grid Infrastructure on all servers (Oracle Clusterware and Oracle ASM) to create a cluster.
  ACTION: Oracle Grid Infrastructure is not installed. Install it either from the separate installation media included in your media pack, or install it by downloading it from Electronic Product Delivery (EPD) or the Oracle Technology Network (OTN). Oracle Grid Infrastructure normally is installed by a different operating system user than the one used for Oracle Database. It may need to be installed by your system administrator. See the installation guide for more details.

問題五: ORA-12545

安裝oracle軟體時遇到如下錯誤：

執行root.sh時，提示rootadd_rdbms.sh、rootadd_filemap.sh 檔案找不到，sqlplus登入時提示ORA-12545

Performing root user operation for Oracle 11g
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /oracle/db/product/11.2
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
/oracle/db/product/11.2/root.sh: line 14: /oracle/db/product/11.2/rdbms/install/rootadd_rdbms.sh: No such file or directory
/oracle/db/product/11.2/root.sh: line 16: /oracle/db/product/11.2/rdbms/install/rootadd_filemap.sh: No such file or directory

sqlplus登入報錯ORA-12545

oracle@chenj-ch-05:/home/oracle$source .bash_profile 
oracle@chenj-ch-05:/home/oracle$sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Fri Mar 4 10:04:57 2022
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
ERROR:
ORA-12545: Connect failed because target host or object does not exist

透過history發現，在安裝oracle時，解決軟體包只解壓了1個，第2個包居然忘解壓了，靜默安裝時沒有報錯，安裝後發現少了很多檔案。

解除安裝Oracle軟體，重新安裝，具體需要刪除$ORACLE_BASE，$ORACLE_HOME，/etc/ora*，/var/tmp/.oracle/*等檔案。

#####chenjuchao 20220304 20:50#####

Oracle 11g RAC Silent Install For NFS

相關文章