ASM device error ORA-27041 ORA-15025 ORA-15081 (Doc ID 1487475.1)

In this Document

Applies to:

Symptoms

 1. All ASM diskgroups are currently mounted

2. Majority of the business applications are running without any issue.

3. There is no issue when connecting to RDBMS using RDBMS OS user ( example "oracle" user )

4. Role separation  between ASM devices and RDBMS has been implemented.
    For example,  grid for ASM instance & oracle for RDBMS instance

5. The following error is reported in RDBMS alert.log.

Cause

This issue can happen with wrong permission of oracle binary from RDBMS home or wrong permission of ASM devices.  The group owner of oracle binary from RDBMS home should be the same as the group owner of all ASM devices with setuid and setgid bit configured, so that any user connecting RDBMS can have the group priviledge configured for oracle binary.

For example, if all ASM devices are owned by "asmadmin" group, the group of oracle binary should be set to "asmadmin" with setgid priviledge.

And the group owner of ASM devices should be the same as the one specified as sysasm group on GRID installation and can be checked with the following command.

$ cat $GRID_HOME/rdbms/lib/config.[sc] | grep -i SS_ASM_GRP


Examples of wrong configuration for role separation between GRID and RDBMS.

Solution

1. Implement the correct permission for oracle binary from GRID and RDBMS Home and ASM devices.

2. How to add setuid and setgid bit to oracle binary

    Shutdown RDBMS ( or ASM ) instance

    $ chmod 6751 oracle

    And start up instance

3. How to change the group permission of oracle binary from RDBMS home.

    - See Oracle Note 1084186.1

   << Confirm sysasm group name using "$ cat $GRID_HOME/rdbms/lib/config.[sc]  | grep -i SS_ASM_GRP "

4. How to change the group permission of ASMLib devices.

4-1. Stop CRS ( or ASM instance )  from all nodes but this can be done in a rolling fashion in a situation where CRS stack can not be stopped at the same time.

 << This will shutdown RDBMS/ASM and CRS

4-2. Stop oracleasm 

 << If ASMlib is not stopped properly, see Solution #2 ] in Oracle  Note 1444115.1 .

4-3. Reconfigure  ASMlib with the correct group ( ex. asmadmin )

4-4. Startup ASMlib 

 
4-5. Check whether group infromation from all ASM disks in /dev/oracleasm/disks/* has been changed to correct group.

4-6. Start CRS
  

 

Simple Workaround

1. Give read permission to the world  for all ASM devices to check whether the issue is the same described in the Cause section because this error only happens on read operation from ASM devices.
    But this temporary workaround won't work on next reboot or on ASMlib restart as ASMLib restart will restore the original permission.
  

 
2. If the group owner of oracle binary from RDBMS home has changed to conform to the group owner of ASM devices other than sysasm ( SS_ASM_GRP) group, chances are that the next RDBMS startup with srvctl will change the group owner to the original group, which is an expected behaviour. But the issue doesn't happen with sqlplus.




Appendix I

If  the impact of this issue  and which application or user causing this issue  have not been determined, run OSWatcher till the same error is reported and find process id in RDBMS alert.log to check which application or OS user is responsible for this error from oswps data.

- OS Watcher Black Box User Guide (Doc ID 301137.1)  

 

References