Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.1 and later
Information in this document applies to any platform.
Checked for relevance on 10-May-2013

Symptoms

Note About Platform Specific: Although '/etc/sysconfig/oracleasm' is only applicable to Linux environments, similar errors could occur and affect non-Linux environments that employ multipath devices.

Mount ASM disk group fails with: ORA-15186, ORA-15025, ORA-15063

alert_+ASM.log:
~~~~~~~~~~~~~~~~~~
...
NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:DISCO01BKP"
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:DISCO02BKP"
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:DISCO03BKP"
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:DISCO04BKP"
ORA-15186: ASMLIB error function = [asm_open(global)], error = [1], mesg = [Operation not permitted]
ORA-15025: could not open disk "ORCL:DISCO05BKP"
ERROR: no PST quorum in group: required 2, found 0
...
ERROR: diskgroup DATA2 was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA2" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA2"

Cause

All the asmlib disks appear in ASM's v$asm_disk view, however their HEADER_STATUS=UNKNOWN

However, ASMLIB sees the disks:

$ /etc/init.d/oracleasm listdisks
DISCO01BKP
DISCO02BKP
DISCO03BKP
DISCO04BKP
DISCO05BKP

A list of the asmlib disks, show their major and minor numbers (first set of numbers from the left):

$ ls -l /dev/oracleasm/disks
brw-rw---- 1 oracle oinstall 8, 1 Dec 8 10:31 DISCO01BKP
brw-rw---- 1 oracle oinstall 8, 17 Dec 8 10:31 DISCO02BKP
brw-rw---- 1 oracle oinstall 8, 33 Dec 8 10:31 DISCO03BKP
brw-rw---- 1 oracle oinstall 8, 49 Dec 8 10:31 DISCO04BKP
brw-rw---- 1 oracle oinstall 8, 65 Dec 8 10:31 DISCO05BKP

A 'cat /proc/partitions', show the asmlib devices are linked to the 'sd' devices instead the 'multipath (dm)' devices. These asmlib devices used to be linked to the multipath devices before the problem:

major minor #blocks name
8 1 209712478 sda1
8 17 209712478 sdb1
8 33 209712478 sdc1
8 49 209712478 sdd1
8 65 209712478 sde1
...
253 0 209715200 dm-0
253 1 209715200 dm-1
253 2 209715200 dm-2
253 3 209715200 dm-3
253 4 209715200 dm-4

Solution

The asmlib file (/etc/sysconfig/oracleasm) to do with mapping asmlib devices to multipath devices, seems is not configure for multipathing:

ORACLEASM_ENABLED=true
ORACLEASM_UID=oracle
ORACLEASM_GID=oinstall
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""

After ORACLEASM_SCANORDER and ORACLEASM_EXCLUDE were updated in /etc/sysconfig/oracleasm, ASM was able to display the asmlib disks with v$asm_disk.header_status=MEMBER. Thereafter, ASM was able to mount the diskgroup.

ORACLEASM_SCANORDER="mpath dm"
ORACLEASM_SCANEXCLUDE="sd"