How to Restore ASM Password File if Lost ( ORA-01017 ORA-15077 ) (Doc ID 1644005.1)

In this Document

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 and later
Information in this document applies to any platform.

Goal

If the ASM password file is lost due to recreation of the DG which is hosting the password file or due to some other reason. Then you need to restore the same with the backup of the ASM Password file

Solution

If the ASM password file is not available then the clusterware stack will not come up and you will see errors in clusterware alert log and crsd log. Other than holding the password authentication the password file also holds the credentials for CRS to connect to ASM instance.

In the current scenario, the diskgroup hosting the ASM passwordfile was recreated, resulting in the loss of the original ASM Passwordfile. In such a situation you will notice errors as below

From crsd.log
===========

2014-02-02 14:05:43.885: [ OCRRAW][3619968576]proprinit: Could not open raw device
2014-02-02 14:05:43.887: [ OCRAPI][3619968576]a_init:16!: Backend init unsuccessful : [26]
2014-02-02 14:05:43.888: [ CRSOCR][3619968576] OCR context init failure. Error: PROC-26: Error while accessing the physical storage
ORA-01017: invalid username/password; logon denied
ORA-15077: could not locate ASM instance serving a required diskgroup

2014-02-02 14:05:43.888: [ CRSMAIN][3619968576] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage
ORA-01017: invalid username/password; logon denied
ORA-15077: could not locate ASM instance serving a required diskgroup

2014-02-02 14:05:43.888: [ CRSD][3619968576][PANIC] CRSD exiting: Could not init OCR, code: 26
2014-02-02 14:05:43.889: [ CRSD][3619968576] Done.

From alertvmrac1.log
================

2014-02-02 14:05:12.136:
[ctssd(3423)]CRS-2409:The clock on host vmrac1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2014-02-02 14:05:23.300:
[/oracle/12.1.0/grid/bin/orarootagent.bin(3175)]CRS-5818:Aborted command 'start' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:11:9} in /oracle/12.1.0/grid/log/vmrac1/agent/ohasd/orarootagent_root/orarootagent_root.log.
2014-02-02 14:05:26.028:
[/oracle/12.1.0/grid/bin/orarootagent.bin(3175)]CRS-5017:The resource action "ora.storage start" encountered the following error:
Storage agent start action aborted. For details refer to "(:CLSN00107:)" in "/oracle/12.1.0/grid/log/vmrac1/agent/ohasd/orarootagent_root/orarootagent_root.log".
2014-02-02 14:05:29.941:
[ohasd(2818)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.storage'. Details at (:CRSPE00163:) {0:11:9} in /oracle/12.1.0/grid/log/vmrac1/ohasd/ohasd.log.
2014-02-02 14:05:43.880:
[crsd(9115)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /oracle/12.1.0/grid/log/vmrac1/crsd/crsd.log.
2014-02-02 14:05:43.888:
[crsd(9115)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
ORA-01017: invalid username/password; logon denied
ORA-15077: could not locate ASM instance serving a required diskgroup
]. Details at (:CRSD00111:) in /oracle/12.1.0/grid/log/vmrac1/crsd/crsd.log.

In such a scenario the following things can be done

Either

Restore the ASM Passwordfile from backup. The backup of ASM Passwordfile should be performed manually after every change, as the automatic backup of ASM passwordfile is currently not being performed. An Enhancement Request to include ASM passwordfile into autobackup is already in place.

How to take a backup of ASM password file and restore the same

----Taking backup----

ASMCMD> pwget --asm
+DGP_01/ASM/PASSWORD/pwdasm.256.844043619

ASMCMD> ls +DGP_01/ASM/PASSWORD/pwdasm.256.844043619
pwdasm.256.844043619

ASMCMD> pwcopy --asm +DGP_01/ASM/PASSWORD/pwdasm.256.844043619
+DGP_02/pwdasm_bk
copying +DGP_01/ASM/PASSWORD/pwdasm.256.844043619 -> +DGP_02/pwdasm_bk

ASMCMD> pwget --asm
+DGP_02/pwdasm_bk

ASMCMD> pwset --asm +DGP_01/ASM/PASSWORD/pwdasm.256.844043619

ASMCMD> pwget --asm
+DGP_01/ASM/PASSWORD/pwdasm.256.844043619

-----Restoring the password file-----

ASMCMD> pwcopy --asm +DGP_02/pwdasm_bk +DGP_03/pwdasm03

ASMCMD> pwget --asm
+DGP_03/pwdasm03

ASMCMD> ls +DGP_03/pwdasm03
pwdasm03

Or

If manual backup of the ASM passwordfile is not available then, please deconfigure and re-configure the clusterware by following the document "Doc ID 1377349.1"

References

NOTE:1377349.1 - How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure

BUG:19675509 - ORA-1017 ORA-15077 COULD NOT START CRSD AFTER CHANGING ASM PASSWD