ORA-29701 from 10.2 DB after ASM upgraded to 11.2 in HP-UX Itanium_1323826.1

In this Document

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.
***Checked for relevance on 05-Sep-2013***

Symptoms

non-RAC - 112010(asm) - 112010(db) - 10205(db) - HP-UX Itanium (11.31)

After migrating 10.2 ASM to 11.2, now only the 10.2 database instances error out with ORA-29701: unable to connect to Cluster Manager


10.2 database alert log:
~~~~~~~~~~~~~~~~~~~~~~~~
ALTER DATABASE MOUNT
Fri Mar 11 14:26:17 2011
ORA-00202: control file:
'+NSHDEVHORA01_DATA/devphrm4/controlfile/current.394.683466541'
ORA-17503: ksfdopn:2 Failed to open file
+NSHDEVHORA01_DATA/devphrm4/controlfile/current.394.683466541
ORA-15001: diskgroup "NSHDEVHORA01_DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Manager

10.2 database sqlplus session:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SQL> create tablespace test_asm datafile '+NSHDEVHORA01_DATA' size 10M;
create tablespace test_asm datafile '+NSHDEVHORA01_DATA' size 10M
*
ERROR at line 1:
ORA-01119: error in creating database file '+NSHDEVHORA01_DATA'
ORA-17502: ksfdcre:4 Failed to create file +NSHDEVHORA01_DATA
ORA-15001: diskgroup "NSHDEVHORA01_DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Manager


Additional information
~~~~~~~~~~~~~~~~~~~~~~~
* This is not affecting any of the 11.2 database instances or any non-ASM 10.2 database instances.
* Same OS software owner for all homes (11.2 asm_home, 11.2 or 10.2 db_home)
.

Cause

This issue looks similar to Bug 10054912 (DB IS UNABLE TO COMMUNICATE WITH ASM INSTANCE) and to do that in pre-11.2 and for HP only, we cannot use hostnames > 8 characters.

* Checked an 'uname' showing a 8 character value for the hostname. Also 'netconf' shows 8 characters for hostname.
* Checked the /tmp/.oracle socket files and the configuration suses a 12 character value for hostname.

In 11.1, there is a restriction for hostname, which is documented in 11.1 database release note on HP-UX IA:

http://download.oracle.com/docs/cd/B28359_01/relnotes.111/b32071/toc.htm#CHDEHEIB
6.5 Hostname Length Restriction for Oracle Clusterware on HP-UX Itanium
On HP-UX Itanium, the hostname for cluster member nodes must be no greater than 8 characters.
.

Solution

To fix this:

1) As root:

- set ORACLE HOME to GRID HOME:
- cd $ORACLE_HOME/crs/install
- perl roothas.pl -delete -force

2) As root:

Edit file: /etc/rc.config.d/netconf
- set the hostname and nodename to the first 8 characters ONLY
- reboot server

3) As root:

- set ORACLE HOME to GRID HOME:
- cd $ORACLE_HOME
- ./root.sh # this rebuilds the 'cluster' with the 8 character name.

4) As the Oracle Restart (gi_home) Software Owner :

Add/start all the components into the Oracle Restart configuration:

$GRID_HOME/bin/srvctl add listener -l LISTENER
$GRID_HOME/bin/srvctl start listener -l LISTENER

$GRID_HOME/bin/srvctl add asm -l LISTENER -d +ASM
$GRID_HOME/bin/srvctl start asm

$GRID_HOME/bin/srvctl add database -d -o -a
$GRID_HOME/bin/srvctl start database -d

5) Change to the 10g home and start up the 10g databases that use ASM, and it works.