root script (root.sh or rootupgrade.sh) Failed to start resource ora.ons

11GR2 Grid安裝或升級時，在執行/root.sh或rootupgrade.sh 時由於ora.ons 不能開啟而失敗，失敗開啟有多種情況；
 現象：

• $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_$HOSTNAME.log

2010-10-11 19:10:51: J2EE (OC4J) Container Resource Add ... passed ...
2010-10-11 19:10:51: starting nodeapps...
2010-10-11 19:10:51: DHCP_flag=0
2010-10-11 19:10:51: nodes_to_start=racnode1
2010-10-11 19:11:13: exit value of start nodeapps/vip is 2
2010-10-11 19:11:13: output for start nodeapps is  PRCR-1013 : Failed to start resource ora.onsPRCR-1064:Failed to start resource ora.onson node racnode1 CRS-5016: Process "/u01/app/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log" onsctl start: ons failed to start CRS-2674: Start of 'ora.ons' on 'racnode1' failed
2010-10-11 19:11:13: output of startnodeapp after removing already started mesgs is PRCR-1013 : Failed to start resource ora.ons PRCR-1064 : Failed to start resource ora.ons on node racnode1 CRS-5016: Process "/u01/app/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log" onsctl start: ons failed to start CRS-2674: Start of 'ora.ons' on 'racnode1' failed
2010-10-11 19:11:13: /u01/app/11.2.0/grid/bin/srvctl start nodeapps -n racnode1 ... failed

• $GRID_HOME/log/$HOSTNAME/agent/crsd/oraagent_$GRID_USER/oraagent_$GRID_USER.log

2010-10-11 19:10:58.841: [ora.ons][1800] [start] Utils:execCmd Running the binary from /u01/app/11.2.0/grid/opmn/bin/onsctli
2010-10-11 19:11:06.155: [ora.ons][1800] [start]
(:CLSN00010:)onsctl start: ons failed to start

2010-10-11 19:11:06.156: [ora.ons][1800] [start] (:CLSN00010:)Utils:execCmd scls_process_join() uret 2

2010-10-11 19:11:06.156: [ora.ons][1800] [start] Exception type=2 string=CRS-5016: Process "/u01/app/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/racnode1/agent/crsd/oraagent_grid/oraagent_grid.log"

2010-10-11 19:11:06.159: [ora.ons][1800] [start] execCmd ret = 2
2010-10-11 19:11:06.159: [ora.ons][1800] [start]
(:CLSN00010:)onsctl start: ons failed to start

2010-10-11 19:11:06.159: [ora.ons][1800] [start] Exception type=2 string=onsctl start: ons failed to start

2010-10-11 19:11:06.160: [ora.ons][1800] [start] OnsAgent::start }
2010-10-11 19:11:06.160: [ora.ons][1800] [start] clsn_agent::start }
2010-10-11 19:11:06.160: [    A**][1800] Command: start for resource: ora.ons racnode1 1 completed with status: SUCCESS
2010-10-11 19:11:06.161: [    A**][1800] Executing command: check for resource: ora.ons racnode1 1
2010-10-11 19:11:06.162: [ora.ons][1800] [check] OnsAgent::check {
2010-10-11 19:11:06.162: [ora.ons][1800] [check] onsctl ping
2010-10-11 19:11:06.162: [ora.ons][1800] [check] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
2010-10-11 19:11:06.162: [ora.ons][1800] [check] Utils:execCmd action = 3 flags = 6 ohome = /u01/app/11.2.0/grid/opmn/ cmdname = onsctli.
2010-10-11 19:11:06.163: [ora.ons][1800] [check] Utils:execCmd Running the binary from /u01/app/11.2.0/grid/opmn/bin/onsctli
2010-10-11 19:11:06.282: [ora.ons][1800] [check]
(:CLSN00010:)ons is not running ...

2010-10-11 19:11:06.282: [ora.ons][1800] [check] execCmd ret = 2
2010-10-11 19:11:06.283: [ora.ons][1800] [check]
(:CLSN00010:)ons is not running ...

2010-10-11 19:11:06.283: [ora.ons][1800] [check] return val = 1
2010-10-11 19:11:06.283: [ora.ons][1800] [check] OnsAgent::check }
2010-10-11 19:11:06.283: [    A**][1800] check for resource: ora.ons racnode1 1 completed with status: OFFLINE
解決方案：

情況 1:

If $GRID_HOME/opmn/logs/ons.log.* has the following error:

10/10/11 19:10:59 [internal] getaddrinfo(::0, 6200, 1) failed (Hostname and service name not provided or found): Connection timed out

The cause is wrong IP address is defined for 'localhost', 'localhost' must have IP 127.0.0.1.

The solution is to ensure both DNS and local /etc/hosts file has correct setting for 'localhost' - DNS or local hosts file will be used depending on name resolution setting in (/etc/nsswitch.conf, or /etc/netsvc.conf depend on platform), once the setting is fixed, refer to note 942166.1 or note 969254.1 to proceed.

 情況 2:

If $GRID_HOME/opmn/logs/ons.log.* has the following error:

[2012-02-06T18:34:19-08:00] [ons] [ERROR:1] [17] [ons-listener] any: BIND (Address already in use)

The cause is some other process is using ons port:

grep port $GRID_HOME/opmn/conf/ons.config

localport=6100          # line added by Agent
remoteport=6200         # line added by Agent

lsof | grep 6200 | grep LISTEN
opmn      18026    oracle    9u     IPv4          191328268                  TCP racnode1:6200 (LISTEN)

From above we can see that process 'opmn' with pid 18026 already using the same port

It is also possible old release ons process was not terminated properly.

The solution is to ensure ons port is not being used by any other processes.
If old release ons is still running, please stop it before running rootupgrade.sh.

情況 3:

If $GRID_HOME/opmn/logs/ons.log.* has the following error:

[2012-02-14T13:53:26-08:00] [ons] [NOTIFICATION:1] [104] [ons-internal] ONS server initiated
[2012-02-14T13:53:26-08:00] [ons] [ERROR:1] [17] [ons-listener] 0000:0000:0000:0000:0000:0000:0000:0001,6100: BIND (Cannot assign requested address)

The likely cause is IPv6 is partially configured. 11gR2 Grid Infrastructure does not support IPv6. The workaround is to set the following parameter in $GRID_HOME/opmn/conf/ons.config and ons.config.:

interface=ipv4