Error 'Multipathd: Asm!.Asm_ctl_spec: Failed To Store Path Info' found In /var/log/messages (Doc ID 1268895.1)

In this Document

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.1.0.1 [Release 11.2 to 12.1]
Linux x86-64
This note will provide the way how to get around the issue when multipathd gives the following error message on crs startup.

Symptoms

The following message will be seen in /var/log/messages when CRS stack is started.

/var/log/messages

When stopping the clusterware :

Nov 15 15:35:25 idsdr31 multipathd: ofsctl: remove path (uevent)
Nov 15 15:35:25 idsdr31 multipathd: ofsctl: spurious uevent, path not in pathvec
Nov 15 15:35:25 idsdr31 multipathd: uevent trigger error
Nov 15 15:35:26 idsdr31 kernel: ACFSK-0039: Module unloaded.
Nov 15 15:35:26 idsdr31 multipathd: asm!.asm_ctl_vbg8: remove path (uevent)
Nov 15 15:35:26 idsdr31 multipathd: asm!.asm_ctl_vbg8: spurious uevent, path not in pathvec
Nov 15 15:35:26 idsdr31 multipathd: uevent trigger error

When starting the clusterware:

Nov 15 15:39:01 idsdr31 kernel: [Oracle OKS] Module load succeeded. Build information: (LOW DEBUG)- (USM_11.2.0.2.0_LINUX.X64_100804.1) , Built on 2010/08/17 00:52:36
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_spec: add path (uevent)
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_spec: failed to store path info
Nov 15 15:39:01 idsdr31 multipathd: uevent trigger error
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vmb: add path (uevent)
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vmb: failed to store path info
Nov 15 15:39:01 idsdr31 multipathd: uevent trigger error
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vdbg: add path (uevent)
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vdbg: failed to store path info
Nov 15 15:39:01 idsdr31 multipathd: uevent trigger error
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vbg0: add path (uevent)
Nov 15 15:39:01 idsdr31 kernel: [Oracle ADVM] Module load succeeded. Build information: (LOW DEBUG) - USM_11.2.0.2.0_LINUX.X64_100804.1 Built on 2010/08/17 00:55:46
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vbg0: failed to store path info
Nov 15 15:39:01 idsdr31 multipathd: uevent trigger error
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vbg1: add path (uevent)
Nov 15 15:39:01 idsdr31 multipathd: asm!.asm_ctl_vbg1: failed to store path info
Nov 15 15:39:01 idsdr31 multipathd: uevent trigger error

Changes

Recently 11.2.0.2 has been installed or upgraded from previous version.

Cause

The following devices have been created under /dev/asm as part of configuring ADVM/ACFS.

[root@node bin]# ls -la /dev/asm

total 0
drwxrwx--- 2 root asmadmin 280 Nov 4 10:49 .
drwxr-xr-x 17 root root 4960 Nov 4 07:59 ..
brwxrwx--- 1 root asmadmin 252, 0 Nov 4 07:59 .asm_ctl_spec
brwxrwx--- 1 root asmadmin 252, 10 Nov 4 07:59 .asm_ctl_vbg0
brwxrwx--- 1 root asmadmin 252, 11 Nov 4 07:59 .asm_ctl_vbg1
brwxrwx--- 1 root asmadmin 252, 12 Nov 4 07:59 .asm_ctl_vbg2
brwxrwx--- 1 root asmadmin 252, 13 Nov 4 07:59 .asm_ctl_vbg3
brwxrwx--- 1 root asmadmin 252, 14 Nov 4 07:59 .asm_ctl_vbg4
brwxrwx--- 1 root asmadmin 252, 15 Nov 4 07:59 .asm_ctl_vbg5
brwxrwx--- 1 root asmadmin 252, 16 Nov 4 07:59 .asm_ctl_vbg6
brwxrwx--- 1 root asmadmin 252, 17 Nov 4 07:59 .asm_ctl_vbg7
brwxrwx--- 1 root asmadmin 252, 18 Nov 4 07:59 .asm_ctl_vbg8
brwxrwx--- 1 root asmadmin 252, 1 Nov 4 07:59 .asm_ctl_vdbg
brwxrwx--- 1 root asmadmin 252, 2 Nov 4 07:59 .asm_ctl_vmb

Solution

1] In some platforms, opening /dev/ofsctl by a 3rd party application could prevent CRS stack from being stopped properly. Refer to note 1417294.1

2] This message can be suppressed by adding the following 2 lines in blacklist element in /etc/multipath.conf file.

/etc/multipath.conf

blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd|vd)[a-z]*"
wwid "360060e80058a350000008a35000000a2"
# wwid "*"
devnode "^asm/*" # Here
devnode "ofsctl" # Here

}

NOTE>
1] Reboot or mkinitrd command may be required in order to reflect the changes in /etc/multipath.conf. Contact your Linux Vendor to get the exact method for this.
2] Recently,"oracleafd" is reported with 12.1.0.x version. Basically, any device name in message file not used for multipath can be listed as blacklist.