ORA-15041 IN A DISKGROUP ALTHOUGH FREE_MB REPORTS SUFFICIENT SPACE

In this Document
  Purpose
  Scope and Application
  ORA-15041 IN A DISKGROUP ALTHOUGH FREE_MB REPORTS SUFFICIENT SPACE
     Capacity of disks within ASM diskgroup are different
     ASM instance is shutdowned normal/immediate before a rebalance is completed
     Disk is DROPPING / HUNG state
     After an add disk command the rebalance is still in place

Applies to:

Purpose

ORA-15041 is the most common space error reported when disk free space is not sufficient 
to complete file allocation request. Due to imbalanced space distribution, ORA-15041 can still
be encountered although ASM views reports sufficient free space. This article is intended to 
help the reader to understand the most common reasons for ORA-15041 error although sufficient 
free space is reported in ASM views.

Error:  ORA-15041  (ORA-15041) 
Text:   diskgroup space exhausted  
--------------------------------------------------------------------------- 
Cause:  The diskgroup ran out of space.  
Action: Add more disks to the diskgroup, or delete some existing files.  

Scope and Application

This article is intended to help the reader to understand the most common reasons 
for ORA-15041 error in ASM diskgroups.

ORA-15041 IN A DISKGROUP ALTHOUGH FREE_MB REPORTS SUFFICIENT SPACE

ASM spreads file extents evenly accross all the disks disks on a diskgroup accordingly with 
capacity, free space within diskgroup and its redundancy type. Total free Mega-bytes in an 
ASM diskgroup is reported in FREE_MB column but the maximum space that can be allocated 
actually changes with the type of redundancy as summarized below:  

Redundancy type       Max.Space  
----------------      ------------------------- 
External              FREE_MB of diskgroup 
Normal                1/2 FREE_MB of diskgroup 
High                  1/3 FREE_MB of diskgroup 

v$ASM_DISK.FREE_MB column is simply a sum of free megabyte space reported in each v$asm_disk. 
On the other hand USABLE_FILE_MB in V$ASM_DISKGROUP also indicates the amount of free space, 
adjusted for mirroring, that is available for new allocations. 

Although FREE_MB and USABLE FILE_MB columns reports sufficient free space, an ORA-15041 error 
can still be encountered due to imbalanced free space between disks. The reason for this is 
that one disk lacking sufficient free space makes it impossible to do any allocation in a 
disk group because every file must be evenly allocated across all disks per ASM stripping 
policy. 

Unbalanced disk configuration and certain operations on ASM disks can create this type of 
problem. The problem frequently should be resolved after a succesfull rebalance as far 
as all disks have the same storage capacity and there is no underlying hardware problems.  

The most common reasons can be classified as follows: 

1- Capacity of disks within ASM diskgroup are different 
2- ASM instance is shutdowned normal/immediate before a rebalance is ompleted. 
3- Disk is DROPPING / HUNG state 
4- After an add disk command the rebalance is still in place 
    
    
    
    

Capacity of disks within ASM diskgroup are different

When an asm diskgroup is having the disks in different capacity, one disk lacking free 
space makes it impossible to do any allocation in the other disks as well. 
This is expected behaviour because every file must be evenly allocated across 
all disks. Rebalancing and allocation attempts to make the percentage of allocated 

    
    
space about the same on every disk. File allocation may fail with an ORA-15041 
in case of imbalanced space distribution. 

Extents are allocated evenly accross the disks accordingly with the capacity 
(TOTAL_MB) of disks. If all ASM disks are of the same size (e.g. 10 disks, 50GB 
each), ASM allocator places extents on each disk in a sequence. The first disk 
allocation is chosen randomly, but all subsequent disks for extent allocation 
are chosen to evenly spread each file across all disks and to evenly fill all
disks.
    
On the other hand, if ASM disks are not of the same size (e.g. disk 1 is 10GB, 

    
    
disk 2 is 50GB and disks 3-10 are 10GB), ASM allocator will place one extent on 
disk 1, five extents on disk 2, one extent on disk 3 and so on. This is to 
ensure balanced disk utilization.

Extent allocation also differs with the type of redundancy. If redundancy is     
NORMAL/HIGH, no allocation is possible when free space in any of the disks 

    
    
is not sufficient for the requested allocation size. On the other hand, in 
an external redundancy diskgroup, ASM distributes the extents evenly across 
the disks accordingly with the capacity (TOTAL_MB) of disks and the allocation 
continue till there exists at least two disks having enough space to complete 
the allocation.
            
The following test demonstrates the space allocation behaviour according
to redundancy type when there exists disks with different capacity in the 
diskgroups.

  
  
  
  

    
DG_EXT: External redundancy diskgroup with 3 disks: 447M, 447M, 70M bytes.
DG_NOR: Normal redundancy diskgroup with 3 disks: 447M, 447M, 70M bytes.

ASM/Database instance version is 10.2.0.3.

    
In order to test the file allocation on different redundancy types 
(external/normal), files with different sizes are created on diskgroups. 
In order to ensure the same amount of free space left after each file creation, 
size of the files created on external redundancy diskgroup is double 
the amount of the file sizes created on normal redundancy diskgroup. The 
following table demonstrates the free space after each file creation and the 
point where ORA-15041 error is encountered on each type of diskgroup.

FREE_MB(0): Initial free space at each disk. 
FREE_MB(1): After 200M/100M files are created external/normal redundancy 
diskgroup.
FREE_MB(2): After 100/50M files are created external/normal redundancy 
diskgroup.
FREE_MB(3): After 500M/- file is created on external redundancy diskgroup.

    


    
NAME              TOTAL_MB FREE_MB(0)  FREE_MB(1)  FREE_MB(2) FREE_MB(3)
                                       +200M/+100M +100M/+50M  +500M/-
------------    ---------- ---------- ------------ ---------- ----------
DG_EXT_0001            447        423          330        283         50 **
DG_EXT_0000            447        421          326        279         48 **
DG_EXT_0002             70         66           51         43          8 **
DG_NOR_0001            447        395          301    ORA-15041 *      X
DG_NOR_0000            447        395          301    ORA-15041 *      X
DG_NOR_0002             70         18            1    ORA-15041 *      X
    

* File allocation (50MB) on  normal redundancy diskgroup fails with ORA-15041 
when there is no more than 50M at a single disk. This is mainly because normal 
redundancy diskgroup can’t allocate primary/secondary extents of file on 
separate disks due to insufficient space. 
    
** On the other hand, file allocation (100M + 500M) succeeds for external 
redundancy diskgroup as this type of redundancy only stripes data over 
available free space in all disks. Further tests show that file creation in 
external redundancy is possible as long as there is some space in at least 
two disks in the diskgroup. 

  

ASM instance is shutdowned normal/immediate before a rebalance is completed
  
A rebalance can be stopped if ASM instance is shutdowned and it is expected 
that rebalance should resume after the instance is restarted. However, due to 
a known issue (Unpublished Bug 5089819) if ASM instance is shutdowned with 
normal/immediate option, rebalance doesn't kick off again upon a new startup.
  
ASM instance requires to either do a shutdown abort or restart rebalance manually.
  

    
NAME                           GROUP_NUMBER   TOTAL_MB    FREE_MB
------------------------------ ------------ ---------- ----------
DG_NORMAL                                 2        894        386

    

NAME                           GROUP_NUMBER   TOTAL_MB    FREE_MB
------------------------------ ------------ ---------- ----------
DG_NORMAL_0000                            2        447        193
DG_NORMAL_0001                            2        447        193

      
A new disk is being added:

alter diskgroup dg_normal add disk '/dev/hdb10';
    

select * from v$asm_operation;
    
GROUP_NUMBER OPERA STAT      POWER      SOFAR   EST_WORK   EST_RATE
------------ ----- ---- ---------- ---------- ---------- ----------
 2           REBAL RUN           1          75       194        181

ASM instance is shutdowned:

SQL> shutdown immediate
ASM diskgroups dismounted
ASM instance shutdown
SQL>


Upon startup, there is no relabalance operation going on and free
space in asm disks is not balanced.

NAME                           GROUP_NUMBER   TOTAL_MB    FREE_MB
------------------------------ ------------ ---------- ----------
DG_NORMAL                                 2       1341        781

NAME                           GROUP_NUMBER   TOTAL_MB    FREE_MB
------------------------------ ------------ ---------- ----------
DG_NORMAL_0000                            2        447        240
DG_NORMAL_0001                            2        447        239
DG_NORMAL_0002                            2        447        302


    
Normally, we should be able create a file with approx. 350-400M as 
asm diskgroup reports sufficient space.
    

SQL> alter tablespace test1 add datafile '+DG_NORMAL' size 370m;
    
*
ERROR at line 1:
ORA-01119: error in creating database file '+DG_NORMAL'
ORA-17502: ksfdcre:4 Failed to create file +DG_NORMAL
ORA-15041: diskgroup space exhausted
    

SQL> alter diskgroup dg_normal rebalance power 11;
SQL> alter tablespace test1 add datafile '+DG_NORMAL' size 370m;

Tablespace altered

A new rebalance remedies this situation. Diskgroup has the following free
space figures after 370MB file is created.

    


    
NAME                           GROUP_NUMBER   TOTAL_MB    FREE_MB
------------------------------ ------------ ---------- ----------
DG_NORMAL                                 2       1341         70
    

NAME                           GROUP_NUMBER   TOTAL_MB    FREE_MB
------------------------------ ------------ ---------- ----------
DG_NORMAL_0000                            2        447         24
DG_NORMAL_0001                            2        447         22
DG_NORMAL_0002                            2        447         24
  


  

Disk is DROPPING / HUNG state


  
Free space in an asm diskgroup can be imbalanced if a drop disk fails for any 
reason (lack of space, disk crash, etc.). Disks may be  stuck in the DROPPING 
state in this case. 

The most common reasons for DROPPING state are that a careless drop disk command 
is submitted on a diskgroup runing with full capacity or dropping the disk reduces 
the amount of available disk space to less than that required for all the existing 
extents. After a drop disk command, a rebalance is triggered and completed however 
there exits disks at DROPPING state in this case. 

It is not possible to allocate space from diskgroup any more as no free space is 
also reported in v$asm_diskgroup. To resolve the problem, you can either add more 
disks to provide extra space or undrop the disk to roll back the drop. 

- Add more disks
Adding more disks provides starts a rebalance implicity and provides extra space for the rebalance 
to complete. Once the data is copied out of the dropping disks, they will be expelled out of the diskgroup. 


    
alter diskgroup  add disk 'path';
  

- Undrop the disk
when an undrop command is issued, it simply rolls back the drop. If the disks 
dropping has not gone too far, ASM will be able to re-integrate the disks back into 
the diskgroup. UNDROP DISKS triggers a rebalance implicitly which rolls back the drop 
and make the space again available to diskgroup. Space should be balanced between disks 
once the command is completed.

    
alter diskgroup  undrop disks;
  
    
    
    
    
    
    
While disks are runing near to capacity, imagine a drop disk brings the 
disk state to HUNG. Drop disk can't be completed as due to lack of space as 
current extents can't be fit into the remaining disks.

NAME                             TOTAL_MB    FREE_MB STATE
------------------------------ ---------- ---------- --------
DG2_0001                              447         97 NORMAL
DG2_0000                              447         90 NORMAL
DG2_0002                              447         97 NORMAL


      
      
NAME                             TOTAL_MB    FREE_MB TYPE
------------------------------ ---------- ---------- ------
DG2                                  1341        284 EXTERN
alter diskgroup DG2 drop disk DG2_0002;
      
      

While rebalance is runing, disk state stays at DROPPING but it changes to
HUNG after rebalance is completed.
NAME                             TOTAL_MB    FREE_MB STATE
------------------------------ ---------- ---------- --------
DG2_0001                              447          7 NORMAL
DG2_0000                              447          6 NORMAL
DG2_0002                              447        271 DROPPING
      
      
      
      
After rebalance is complete, disk state is HUNG as disk can't be expelled out 
from the diskgroup.      
      
      
NAME                             TOTAL_MB    FREE_MB STATE
------------------------------ ---------- ---------- --------
DG2_0001                              447          0 NORMAL
DG2_0000                              447          0 NORMAL
DG2_0002                              447        284 HUNG


alter diskgroup DG2 undrop disks;

      
      
Undrop disk triggers a new rebalance implicitly and resolves the problem.
This state  can also be resolved with ADD DISK by providing extra space for 
the rebalance to complete. Once the data is copied out of the dropping disks 
they will be expelled out of the diskgroup.  
    
    



After an add disk command the rebalance is still in place

    
When a disk is added to a disk group its space is not immediately available 
for allocation. Since every file must be evenly allocated, extents must be 
rebalanced off other disks to the new disk to make space evenly available. 
    

Free space will be available in the course of time while rebalance is
progressing. Since rebalance takes a while, users may not be able to allocate 
files and could get out of space errors (ORA-15041). 

  
  
  
As a workaround, WAIT option with ADD disk command can be used. If the WAIT 
option given with add disk, the command doesn't return until rebalance is 
complete.  This may provide more intuitive to users who run disks with near 
to full capacity.     

   
  

    
The following test shows how free space at each disk change while rebalance is 
going.

NAME                           GROUP_NUMBER   TOTAL_MB    FREE_MB
------------------------------ ------------ ---------- ----------
TEST_DG                                   2        894         48
    

PATH                 GROUP_NUMBER   TOTAL_MB    FREE_MB
-------------------- ------------ ---------- ----------
/dev/hdb8                       2        447         24
/dev/hdb9                       2        447         24
      
      
Two more 457MB disks are added to diskgroup.
      
      
alter diskgroup test_dg add disk '/dev/hdb10','/dev/hdb11';
      
      
      
Free space is reported immediately in v$asm_diskgroup however it is imbalanced 
as rebalance is not completed yet.
      
      
NAME          TOTAL_MB    FREE_MB(1) FREE_MB(2) ... FREE_MB(3)
------------- ---------- ---------- ----------      ----------
TEST_DG         1788            888        887             885

      
      
      

PATH             TOTAL_MB FREE_MB(1) FREE_MB(2) ... FREE_MB(3) 
-------------- ---------- ---------- ----------     ----------
/dev/hdb8             447         36        144            221
/dev/hdb9             447         36        144            220 
/dev/hdb10            447        409        300            223
/dev/hdb11            447        407        299            221

Free space is getting balanced while the rebalance is progressing. Till the
rebalance is completed large file allocations may fail with ORA-15041 errors 
although free space is reported in v$asm_diskgroup. 

As a workaround, WAIT option can be used with add disk command.  When the WAIT 
option is used, add disk command doesn't return until rebalance is complete. 
This may provide more intuitive results when running disks with near to full 
capacity.