TROUBLESHOOTING - ASM disk not found/visible/discovered issues

Purpose

This note will assist in troubleshooting disk not found / visible / discovered issues with ASM

Typically this means that the disk in question is not found in the v$asm_disk view

Common errors indicating that a disk is missing/not found are :

ORA-15063: ASM discovered an insufficient number of disks for diskgroup s%
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "%" is missing

Troubleshooting Steps

If an existing diskgroup is suddenly missing a disk ...

Determine what disk is missing

If on non-RAC ... or ALL RAC nodes exhibit these errors

    a) Locate and open the ASM alert log (this may require multiple logs for RAC)
    b) Locate the last successfull mount of the diskgroup (this will show a list of disks)
    c) Locate each successful ALTER DISKGROUP ... ADD DISK since that time
    d) Combine the lists of disks found in b) and c) above
    e) Compare this list against those shown in V$ASM_DISK and determine the missing disk(s)

If on RAC and at least one node can mount the diskgroup

    Compare the V$ASM_DISK entries on the node that can mount to those that cannot
            this will show which disk(s) is missing


The following is a set of steps that will assist in resolving disk not found issues

1) ASM_DISKSTRING is not set correctly

    Examine the ASM_DISKSTRING setting in the parameter file or via SHOW PARAMETER

    If ASM_DISKSTRING is NOT SET ... then the following default is used

    Default ASM_DISKSTRING per OS

     Operating System Default            Search String
     =======================================
    Solaris (32/64 bit)                        /dev/rdsk/*
    Windows NT/XP                          * 
    Linux (32/64 bit)                          /dev/raw/* 

    LINUX (ASMLIB)                         ORCL:*
    LINUX (ASMLIB)                         /dev/oracleasm/disks/* ( as a workaround )

    HPUX                                       /dev/rdsk/* 
    HP-UX(Tru 64)                            /dev/rdisk/*
    AIX                                            /dev/rhdisk*

   IF ASM_DISKSTRING is SET ... then verify that the setting includes the disks that are needed to be seen by ASM


2) Operating system drive ownership

     Make sure that the disk is owned by the OS user who installed the ASM Oracle Home ... and that the disk is mounted correctly (with the correct owner)

3) Operating system drive permissions

   Make sure that the permissions are set correctly at the disk level ... 660 is normal ... but if there are problems use 777 as a test

4) RAC is being used

     If RAC is in use ... then ALL disks need to be visible on all nodes where ASM is / will be running ... before an attempt is made to add the disks to a diskgroup

5) Use OS utilities to determine which disk cannot be found

TRUSSing or STRACEing the RBAL process while selecting * from v$asm_disk can often show errors in the path of the command

EXAMPLE:

SESSION #1

strace -f -o /tmp/rbal.trc -p
 
truss -ef -o /tmp/rbal.out -p

SESSION #2

select * from v$asm_disk

SESSION #3



Examine the rbal.out for errors:

1147090: 1871929: chdir("dev/") = 0
1147090: 1871929: statx("rhdisk8, ", 0x0FFFFFFFFFFFAA80, 176, 010) Err#2 ENOENT

<< This says that rhdisk8 cannot be found >>

========================================================================

NOTE ... If a crash occured during an Add or Drop of a disk ... then the disk(s) in question may still be part of the diskgroup ... so all of the steps above need to take into consideration this (these) disks

PORT SPECIFIC ISSUES

1) HEWLETT PACKARD (HP)

If this is problem is occuring on HP (RISC or Itanium) ... and all of the above are not helping ... and the ct is using HP Logical Volume Manager (LVM) then

  Note.433770.1 Cannot Discover Disks in ASM After Upgrade on 10.2.0.3 on HP-UX  Itanium

  Note.434500.1  When Starting A Database With The 10.2.0.3 Executables, Error "ORA-15059 invalid device type for ASM disk"

2) IBM AIX

Note.353761.1 Assigning a Physical Volume ID (PVID) To An Existing ASM Disk Corrupts the ASM Disk Header