ASMLib Devices Not Discovered with Diskstring as 'ORCL:*'_1444115.1

rongshiyuan發表於2014-03-11

ASMdev

ASMLib Devices Not Discovered with Diskstring as 'ORCL:*' (Doc ID 1444115.1)

In this Document

Symptoms

Cause

Solution

References

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.1 to 11.2.0.3 [Release 10.2 to 11.2]
Linux x86-64
Linux x86
***Checked for relevance on 06-Jan-2014***

Symptoms

Symptom #1]

1. This issue only happens with ASMLib installation on Linux server.

2. ASMLib is installed properly.

# rpm -qa |grep oracleasm

oracleasmlib-2.0.4-1.el5
oracleasm-support-2.1.7-1.el5
oracleasm-2.6.18-274.17.1.el5-2.0.5-1.el5
oracleasm-2.6.18-274.17.1.el5-debuginfo-2.0.5-1.el5

3. ASM instance itself can be started but no diskgroup can be mounted with asm_diskstring='ORCL:*'

4. After asm_diskstring is changed to '/dev/oracleasm/disks/*', all ASM devices are visible from v$asm_disk and diskgroups can be mounted.

5. ASMLib disks can be discoverable with the following command as root but not discoverable with grid or oracle user.

# /usr/sbin/oracleasm-discover 'ORCL:*'

Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:ARCH [318761667 blocks (163205973504 bytes), maxio 512]
Discovered disk: ORCL:DATA [3145719717 blocks (1610608495104 bytes), maxio 512]
Discovered disk: ORCL:FLASH [104856192 blocks (53686370304 bytes), maxio 512]

[grid]$ /usr/sbin/oracleasm-discover 'ORCL:*'

Unable to open ASMLib

6. The permisssion of ASMLib shows "executable" for others.

$ ls -l -rwxr-xr-x 1 root root 18176 Mar 19 2009 /opt/oracle/extapi/*/asm/orcl/1/libasm.so

7. ASM RBAL trace shows the following similar error when ASM instance is stared.

   WARNING::lib=/opt/oracle/extapi/64/asm err:8 rc:opendir location:skgdllOpenDi
   errbug=13
   msgbuf=Permission denied other=Permission denied

Symptom #2 ]

1. Recently, the following parameters from /etc/sysconfig/oracleasm have been changed and "/etc/init.d/oracleasm scandisks" has been run.

/etc/sysconfig/oracleasm

From

ORACLEASM_SCANORDER=
ORACLEASM_SCANEXCLUDE=

ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

If EMC powerpath is used, the value above should be ORACLEASM_SCANORDER="emcpower"

2. The following output shows that single path devices have been excluded from discovery path.

$ /usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=oracle
ORACLEASM_GID=oinstall
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

If EMC powerpath is used, the value above should be ORACLEASM_SCANORDER="emcpower"

3. The following output shows that ASMLib devices has been instantiated with single path device. Please note major# as "8".

$ ls -l /dev/oracleasm/disks/*
brw-rw---- 1 oracle oinstall 8, 17 Apr 23 22:53 /dev/oracleasm/disks/CRS1
brw-rw---- 1 oracle oinstall 8, 18 Apr 23 22:53 /dev/oracleasm/disks/DATA1

Please also note the creation time for ASMLib devices is before the change of ORACLEASM_SCANORDER="dm" and "/etc/init.d/oracleasm scandisks" command is run .

4. v$asm_disk view returns all ASM device information but v$asm_disk.header_status colume shows the status as "UNKNOWN".

SQL> select group_number gn,disk_number dn, mount_status, header_status,mode_status,state, total_mb, free_mb, label, path from v$asm_disk order by group_number, disk_number;

GN   DN MOUNT_STATUS HEADER_STATU MODE_STATUS STATE         TOTAL_MB    FREE_MB        LABEL          PATH
---- ---- ----------------------------------------------------------------------------------------------------------------------------------------
   0    0 CLOSED      UNKNOWN    ONLINE      NORMAL                  0             0       CRS1             ORCL:CRS1
   0    1 CLOSED      UNKNOWN        ONLINE      NORMAL                  0              0       DATA1          ORCL:DATA1

5. From ASM alert.log, ASM diskgroups are not mounted with ORA-15186 error.

6. On GRID installation, OUI does not show ASMLib devices and ASMLib devices are discoverable only with discovery string "/dev/oracleasm/disks/*". In this case, it's impossible to check v$asm_disk.header_status as Oracle ASM is not installed yet.

Cause

Case #1 ]

This issue could happen with the wrong permission of the following path for asmlib. But the cause for the wrong permission is not known.

$ stat /opt/oracle/extapi/*/asm/orcl/1/libasm.so

stat: cannot stat `/opt/oracle/extapi/64/asm/orcl/1/libasm.so': Permission denied

$ ls -la /opt/oracle
total 20
drwxr-x--- 4 root root 4096 Jun 16 2009 . << drwxr-x--- 2 root oinstall 4096 Aug 18 2010 bin
drwxr-xr-x 3 root root 4096 Jun 14 2009 extapi

Or this could be from wrong ASMLib installation ( 32bit Vs. 64bit ). Note that ASMLib shows it as 32b it and uname command shows it as 64 bit.

$ /usr/sbin/oracleasm-discover
Using ASMLib from /opt/oracle/extapi/32/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]

# uname -a
Linux hostname 2.6.32-100.34.1.el6uek.x86_64 #1 SMP Wed May 25 17:46:45 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Case #2 ]

"/etc/init.d/oracleasm scandisks" command will not create new ASMLib devices if devices with the same ASMLib label alredy exist in /dev/oracleasm/disks/* directory.

Solution

Solution #1 ]

ASMlib library is loaded by RBAL process when ASM instance is started. If RBAL is not able to load ASMLib library on startup for any reason ( usually premission issue if ASMLib rpm is installed properly ), ASMLib devices can not be discovered with asm_diskstring='ORCL:*' .

Change the permission for /opt/oracle directory so that "others" on the server can have "excutable" permission.

# chmod o+x /opt/oracle

And then check whether the following ASMlib can be accessible from grid or oracle user.

$ stat /opt/oracle/extapi/*/asm/orcl/1/libasm.so

Restart ASM instance and check whether ASMLib is loaded properly in the following ways.

1. The following command should return one row if RBAL process has loaded libasm.so library on startup properly.

$ ps -ef | grep rbal_+ASM | grep -v grep | awk -F ' ' '{print $2}' | xargs lsof -p | grep libasm.so

2. From ASM alert.log, find the following similar message right after "ALTER DISKGROUP ALL MOUNT" message and check if ASMLib shows the correct bit ( 32 Vs. 64bit ) information

SQL> ALTER DISKGROUP ALL MOUNT
...

NOTE: Loaded library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so

3. If ASMLib shows 32bit information either from "/usr/sbin/oracleasm-discover" and "uname -a" command shows as 64bit server, 64bit ASMLib should be reinstalled.

Solution #2 ]

Running "/etc/init.d/oracleasm scandisks" command is not good enough after changing ORACLEASM_SCANORDER="dm" from /etc/sysconfig/oracleasm file. ASMlib should be restarted using the following way.

1. Shutdown all databases.
2. Shutdown ASM instance
3. Make sure that there is no process loading libasm.so and no user is in /dev/oracleasm directory before restarting ASMLib.

# lsof | grep libasm.so | grep -v grep | awk -F ' ' '{print $2}' | xargs ps
# lsof | grep /dev/oracleasm | grep -v grep | awk -F ' ' '{print $2}' | xargs ps

Otherwise, ASMLib restart will fail with the following message.

     # /sbin/service oracleasm restart
     Dropping Oracle ASMLib disks:                              [ OK ]
     Shutting down the Oracle ASMLib driver:                [FAILED]

4. Restart ASMLib
As root,

# /sbin/service oracleasm restart

5. Check
5-1. v$asm_disk.header_status column shows proper status - PROVISIONED/MEMBER/FORMER etc. except "UNKNOWN".

5-2. All ASM devices under /dev/oracleasm/disks/* shows major# as "253" or the one for the multipath devices seen from the output of "cat /proc/partitions".

$ ls -l /dev/oracleasm/disks/*
brw-rw---- 1 oracle oinstall 253, 1 Apr 23 22:53 /dev/oracleasm/disks/CRS1
brw-rw---- 1 oracle oinstall 253, 2 Apr 23 22:53 /dev/oracleasm/disks/DATA1

5-3. Check whether GRID OUI shows all ASMLib devices with "ORCL:*' discovery string.

References

NOTE:402741.1 - Adding Disk To Diskgroup Fails With ASMLIB error function = [asm_io]
NOTE:1297099.1 - ORA-15081: failed to submit an I/O operation to a disk
NOTE:1436169.1 - KFOD (On ASMLIB) Is Reporting " " State For The "User" & "Group" Columns.
NOTE:359266.1 - FAQ ASMLIB CONFIGURE,VERIFY, TROUBLESHOOT
NOTE:738480.1 - Oracle Universal installer (OUI) or DBCA does not show any asmlib disks
NOTE:452770.1 - TROUBLESHOOTING - ASM disk not found/visible/discovered issues
NOTE:1410243.1 - V$ASM_DISK View Shows Both "ORCL:*" & "/dev/oracleasm/disks/*" Paths On 11.2 Release.
NOTE:1059847.1 - 11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC USING ASMLIB
NOTE:1314022.1 - ORA-15180: Could Not Open Dynamic Library ASM Library (libasm.so Open err: open)
NOTE:1395851.1 - ASMCA Can Not Discover All The ASMLIB Disks On The Second RAC Node (ASM 11.2).