Unable to startup CRS as ASM failed to startup with "ORA-01017: invalid username/password; logon denied" (Doc ID 1681849.1)

In this Document

Cause

Solution

References

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

One of the node CRS was not coming up properly.

ASM did not come up automatically - it failed to start with "ORA-01017: invalid username/password; logon denied" error in the /log//agent/ohasd/oraagent_grid/oraagent_grid.log

alert.log
~~~~~~~~~~~~~~~~~~~~
[client(3990)]CRS-10001:10-Jun-14 14:35 ACFS-9156: Detecting control device '/dev/ofsctl'.
[client(4002)]CRS-10001:10-Jun-14 14:35 ACFS-9322: completed
2014-06-10 14:36:18.128:
[ohasd(48895)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2014-06-10 14:36:18.128:
[ohasd(48895)]CRS-2807:Resource 'ora.crsd' failed to start automatically.

oraagent_grid.log
~~~~~~~~~~~~~~~~~~
2014-06-10 14:35:15.585: [ora.asm][3219122496]{0:0:2} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/11.2.0.4/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/11.2.0.4/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2014-06-10 14:35:15.587: [ora.asm][3219122496]{0:0:2} [start] Container:start oracle home /u01/app/11.2.0.4/grid
2014-06-10 14:35:15.587: [ora.asm][3219122496]{0:0:2} [start] InstConnection::connectInt: server not attached
2014-06-10 14:35:15.621: [ora.asm][3219122496]{0:0:2} [start] ORA-01017: invalid username/password; logon denied <<<<<<<<<

2014-06-10 14:36:12.960: [ A**][2622531904]{0:0:2} A** Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.asm 1 1] ID 4100:456
2014-06-10 14:36:12.960: [ CRSPE][2543831360]{0:0:2} Received reply to action [Clean] message ID: 456
2014-06-10 14:36:12.960: [ CRSPE][2543831360]{0:0:2} Got agent-specific msg: ORA-01017: invalid username/password; logon denied

ASM instance can be started manually using sqlplus "/as sysasm" but the CRSD fails to start with same error.

crsd.log
~~~~~~~~~
2014-06-05 11:05:28.617: [ CRSMAIN][4003182320] Initializing OCR
[ CLWAL][4003182320]clsw_Initialize: OLR initlevel [70000]
2014-06-05 11:05:29.938: [ OCRASM][4003182320]proprasmo: Error in open/create file in dg [DATA]
[ OCRASM][4003182320]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge

2014-06-05 11:05:29.938: [ OCRASM][4003182320]ASM Error Stack : ORA-01017: invalid username/password; logon denied <<<<<<<<

2014-06-05 11:05:30.985: [ OCRASM][4003182320]proprasmo: kgfoCheckMount returned [7]
2014-06-05 11:05:30.985: [ OCRASM][4003182320]proprasmo: The ASM instance is down
2014-06-05 11:05:30.985: [ OCRRAW][4003182320]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2014-06-05 11:05:30.985: [ OCRRAW][4003182320]proprioo: No OCR/OLR devices are usable