Force ASM to Scan the Multipathed Device First using ASMLIB/oracleasm on RHEL 5/OL 5 and RHEL 6/OL 6 (Doc ID 1679409.1)

In this Document

Goal

Additional Resources

References

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.4 to 11.2.0.4 [Release 10.2 to 11.2]
Linux x86-64
Linux x86

Goal

ASMLib/oracleasm can only be implemented on Linux OS. When using multipathing, you want to FORCE oracleasm to scan the multipathed devices first.

Solution

On all nodes:

1) Shutdown all (DB and ASM):
You should stop all before restart ASMLib (eg: DB, ASM and HAS (crsctl stop crs for RAC) if you are in 11.2 - crsctl stop has)

If the diskgroups are mounted, can fail with:

/etc/init.d/oracleasm restart
Unmounting ASMlib driver filesystem: umount: /dev/oracleasm: device is busy
umount: /dev/oracleasm: device is busy
Unable to unmount ASMlib driver filesystem
[FAILED]
[root@asmstorage ~]#

ASMLib will not allow that

Then, on each node, one at a time do the following steps that are dependent on the type of multipathing software:

2) For backup, make a copy of your current /etc/sysconfig/oracleasm and /etc/sysconfig/oracleasm_dev_oracleasm files

a) If using DM-Multipathing or Linux OS Native Multipathing:

1. Then, you want to change the following parameters in the /etc/sysconfig/oracleasm_dev_oracleasm file to which the /etc/sysconfig/oracleasm file is really linked to:

For example:

lrwxrwxrwx 1 root root 24 Jun 13 09:58 /etc/sysconfig/oracleasm -> oracleasm-_dev_oracleasm

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="mpath dm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

b) If using EMC PowerPath Multipathing:

2. Then, you want to change the following parameters in the /etc/sysconfig/oracleasm_dev_oracleasm file to which the /etc/sysconfig/oracleasm file is really linked to:

For example:

lrwxrwxrwx 1 root root 24 Jun 13 09:58 /etc/sysconfig/oracleasm -> oracleasm-_dev_oracleasm

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="emcpower"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sd"

c) If using Hitachi Dynamic Link Manager:

3. Then, you want to change the following parameters in the /etc/sysconfig/oracleasm_dev_oracleasm file to which the /etc/sysconfig/oracleasm file is really linked to:

For example:

lrwxrwxrwx 1 root root 24 Jun 13 09:58 /etc/sysconfig/oracleasm -> oracleasm-_dev_oracleasm

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="sddlm"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=""

3) As root user, save the file (oracleasm-_dev_oracleasm)

4) Restart oracleasm (as root):

# service oracleasm restart

or

# /etc/init.d/oracleasm restart

5) check /dev/oracleasm/disks/*:

ls -ltr /dev/oracleasm/disks/*

a) If using DM-Multipathing or Linux OS Native Multipathing, all of the Major Numbers should equal 253.

For example:

# ls -l /dev/oracleasm/disks
brw-rw---- 1 grid asmadmin 253, 9 Jun 4 10:35 ASM01
brw-rw---- 1 grid asmadmin 253, 4 Jun 4 10:35 ASM02
brw-rw---- 1 grid asmadmin 253, 7 Jun 4 10:35 ASM03
brw-rw---- 1 grid asmadmin 253, 11 Jun 4 10:35 ASM04
brw-rw---- 1 grid asmadmin 253, 20 Jun 4 10:35 ASM05

2. The above Major Numbers should match the multipathed device (dm*) in the file, /proc/partitions.

For example:

# cat /proc/partitions
major minor #blocks name
...
253 8 524281275 dm-8
253 9 524281220 dm-9
253 3 524281275 dm-3
253 4 524281220 dm-4
253 6 524281275 dm-6
253 7 524281220 dm-7
253 10 524281275 dm-10
253 11 524281220 dm-11
253 19 524281275 dm-19
253 20 524281220 dm-20
...

b) If using EMC PowerPath Multipathing, all of the Major Numbers should equal 120.

For example:

# ls -l /dev/oracleasm/disks
brw-rw---- 1 grid asmadmin 120, 11 Jun 4 10:35 ASM01
brw-rw---- 1 grid asmadmin 120, 5 Jun 4 10:35 ASM02
brw-rw---- 1 grid asmadmin 120, 3 Jun 4 10:35 ASM03
brw-rw---- 1 grid asmadmin 120, 7 Jun 4 10:35 ASM04
brw-rw---- 1 grid asmadmin 120, 9 Jun 4 10:35 ASM05

2. The above Major Numbers should match the multipathed device (emcpower*) in the file, /proc/partitions.

# cat /proc/partitions
major minor #blocks name
...
120 2 524281275 emcpowerb
120 3 524281220 emcpowerb1
120 4 524281275 emcpowerc
120 5 524281220 emcpowerc1
120 6 524281275 emcpowerd
120 7 524281220 emcpowerd1
120 8 524281275 emcpowere
120 9 524281220 emcpowere1
120 10 524281275 emcpowerf
120 11 524281220 emcpowerf1
...

c) If using Hitachi Dynamic Link Manager:

For example:

# ls -l /dev/oracleasm/disks
brw-rw---- 1 grid asmadmin 252, 224 Jun 4 10:35 ASM01
brw-rw---- 1 grid asmadmin 252, 240 Jun 4 10:35 ASM02
brw-rw---- 1 grid asmadmin 251, 0 Jun 4 10:35 ASM03
brw-rw---- 1 grid asmadmin 251, 16 Jun 4 10:35 ASM04
brw-rw---- 1 grid asmadmin 251, 32 Jun 4 10:35 ASM05

2. The above should match the multipathed device (sddlm*) in the file, /proc/partitions.

PLEASE NOTE: For Hitachi Dynamic Link Manager, the Major Numbers can be different, but must match to a sddlm* device

When the /proc/partitions file has listed sddlm* devices, then that is normally an indication of Hitachi Dynamic Link Manager is being used.

# cat /proc/partitions
major minor #blocks name
...

252 224 204802560 sddlmao

252 240 204802560 sddlmap

251 0 204802560 sddlmba

251 16 204802560 sddlmbb

251 32 215040000 sddlmbc

...

d) If using AIX (Please note that ASMLib/oracleasm can only be implemented on Linux OS.):

Please note that AIX can use Linux OS with different types of multipathing software, but the multipathing setup/SCAN depends on the Storage Vendor.

However, most AIX Customers' set up multipathing with mknod and there is a built in MPIO (MultiPath I/O) software in AIX 7.0.

If Customer uses HDS or EMC storage than the drivers come from the vendor.

[root@ocotillop1] / # lsdev -Cc disk

hdisk38 Available 06-09-02 MPIO IBM 2107 FC Disk

hdisk39 Available 06-09-02 MPIO IBM 2107 FC Disk

hdisk40 Available 06-09-02 MPIO IBM 2107 FC Disk

hdisk41 Available 06-09-02 MPIO IBM 2107 FC Disk

hdisk42 Available 06-09-02 MPIO IBM 2107 FC Disk

hdisk43 Available 06-09-02 MPIO IBM 2107 FC Disk

hdisk44 Available 06-09-02 MPIO IBM 2107 FC Disk

Beginning with AIX? 5.2, the multiple path I/O function uses a unique device identifier (UDID) to identify each MPIO-capable device, regardless of the path on which it was discovered.

The UDID is saved in the device configuration database. When a device is discovered, the UDIDs in the database are checked to determine whether the device is new or whether the discovery is another path to an existing device. When multiple paths to a device are detected, the device driver or the Path Control Manager kernel extension decides which path to use for a particular request.

e) If using Solaris OS, please see the following Documents (ASMLIb/oracleasm can NOT be used):

The following documents are only for Solaris MPIO references:

How to Verify the Health of Disk Logical Unit Managed Under Solaris Multipathing Software (MPxIO) (Doc ID 1002465.1)

Multipathing Troubleshooting on Solaris [TM] 10 Operating System (OS) (Doc ID 1003914.1)

Additional Resources

Community Discussions: Storage Management MOS Community
Still have questions? Use the above community to search for similar discussions or start a new discussion on this subject.

References

NOTE:555603.1 - Configuration and Use of Device Mapper Multipathing on Oracle Linux (OL)
NOTE:580153.1 - How To Setup ASM on Linux Using ASMLIB Disks, Raw Devices, Block Devices or UDEV Devices?
NOTE:602952.1 - How To Setup ASM & ASMLIB On Native Linux Multipath Mapper disks?
NOTE:1458768.1 - How to Convert ASMLib Disks from SD Devices to Multipath Devices
NOTE:1365511.1 - How to Configure LUNs for ASM Disks using WWID, DM-Multipathing, and ASMLIB on RHEL 5/OL 5 and RHEL 6/OL 6