今天做資料庫巡檢，發現一套Datagard資料庫standby備庫日誌沒有應用，查alert.log裡有報錯，ORA-07445: exception encountered: core dump [kggibr()+52]

系統版本：AIX 6.1 (64-bit)

[root@M-DB02 /]#uname -a

AIX M-DB02 1 6 00F86DD34C00

[root@M-DB02 /]#bootinfo -K

64

資料庫版本：11.2.0.1

SQL> select * from v$version;

BANNER

--------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

PL/SQL Release 11.2.0.1.0 - Production

CORE 11.2.0.1.0 Production

TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production

NLSRTL Version 11.2.0.1.0 - Production

alert.log相關資訊：

Exception [type: SIGSEGV, Invalid permissions for mapped object] [ADDR:0x0] [PC:0x100137314, kggibr()+52] [flags: 0x0, count: 1]

Errors in file /oracle/diag/orcl/diag/rdbms/orcl2/orcl1/trace/orcl1_pr04_8781910.trc (incident=24251):

ORA-07445: exception encountered: core dump [kggibr()+52] [SIGSEGV] [ADDR:0x0] [PC:0x100137314] [Invalid permissions for mapped object] []

Incident details in: /oracle/diag/orcl/diag/rdbms/orcl2/orcl1/incident/incdir_24251/orcl1_pr04_8781910_i24251.trc

Tue Feb 03 18:14:51 2015

Trace dumping is performing id=[cdmp_20150203181451]

Tue Feb 03 18:14:52 2015

Sweep [inc][24251]: completed

Sweep [inc2][24251]: completed

Tue Feb 03 18:15:51 2015

Recovery Slave PR04 died

Tue Feb 03 18:15:51 2015

Errors with log /oracle/archivelog/1_9738_834401436.arc

MRP0: Background Media Recovery terminated with error 448

Errors in file /oracle/diag/orcl/diag/rdbms/orcl2/orcl1/trace/orcl1_pr00_24051932.trc:

ORA-00448: normal completion of background process

Recovery interrupted!

Recovered data files to a consistent state at change 1501068094

Errors in file /oracle/diag/orcl/diag/rdbms/orcl2/orcl1/trace/orcl1_pr00_24051932.trc:

ORA-00448: normal completion of background process

Errors in file /oracle/diag/orcl/diag/rdbms/orcl2/orcl1/trace/orcl1_mrp0_21954746.trc:

ORA-10878: parallel recovery slave died unexpectedly

MRP0: Background Media Recovery process shutdown (orcl1)

trace檔案資訊：

/oracle/diag/orcl/diag/rdbms/orcl2/orcl1/trace/orcl1_pr04_8781910.trc

Exception [type: SIGSEGV, Invalid permissions for mapped object] [ADDR:0x0] [PC:0x100137314, kggibr()+52] [flags: 0x0, count: 1]

Incident 24251 created, dump file: /oracle/diag/orcl/diag/rdbms/orcl2/orcl1/incident/incdir_24251/orcl1_pr04_8781910_i24251.trc

ORA-07445: exception encountered: core dump [kggibr()+52] [SIGSEGV] [ADDR:0x0] [PC:0x100137314] [Invalid permissions for mapped object] []

ssexhd: crashing the process...

Shadow_Core_Dump = partial

/oracle/diag/orcl/diag/rdbms/orcl2/orcl1/incident/incdir_24251/orcl1_pr04_8781910_i24251.trc：

dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0)

----- SQL Statement (None) -----

Current SQL information unavailable - no cursor.

----- Call Stack Trace -----

calling call entry argument values in hex

location type point (? means dubious value)

-------------------- -------- -------------------- ----------------------------

skdstdst()+40 bl 107bf3838 10932CE80 ? 000002004 ?

000000001 ? 000000003 ?

000000001 ? 000000001 ?

ksedst1()+104 call skdstdst() 110B59E50 ? 000002004 ?

1109CCFA0 ? 10932D66C ?

1109CCFA0 ? 000000000 ?

110B59F80 ? 700000007 ?

ksedst()+40 call ksedst1() 3030100137314 ? 002050033 ?

10932D660 ? 700000000024D ?

000000000 ? 000000000 ?

10932CEB0 ? 000000000 ?

……

分析：

ORA-7445和ORA-600錯誤都是很常見的錯誤，一般是由內部錯誤或者是BUG引起的，
當oracle伺服器程式從作業系統收到一個致命的錯誤資訊時會丟擲ora-07445錯誤，我們需要關注第一個引數的型別。

這裡我懷疑是BUG引起的

去MOS上，果然搜到了一個Bug 9728806

Bug 9728806 ORA-7445 [kggibr()+52] during recovery on IBM AIX POWER Systems

This note gives a brief overview of bug 9728806.
The content was last updated on: 28-JUN-2013

Affects:

*Product (Component)*	Oracle Server (Rdbms)
Range of versionsbelieved to be affected	Versions >= 11.1 but BELOW 12.1
*Versions confirmedas being affected*	11.2.0.1 11.1.0.7
Platforms affected	IBM AIX Power Based Systems (64bit)

Fixed:

This issue is fixed in

Description

ORA-7445 [kggibr()+52] during RECOVERY when platform is

IBM AIX on POWER Systems (64-bit). This bug is more commonly

seen in RDBMS version 11.2.0.1.

This is a fix to prevent a parallel recovery slave process

from crashing with an ORA-07445 during recovery.

Rediscovery Notes:

Look for all the following:

 - a parallel recovery slave crashes with a ORA-07445 with

   krp_next_cv_for_apply() on the stack

    - for example in 11.2.0.1 on AIX the top of the crashed

      process's stack looks like this:

       -> krp_next_cv_for_apply() -> kggibr()+52 -> ORA-07445

 - the recovery coordinator process (eg MRP on a standby

   database) terminates recovery and writes the following

   message to its tracefile:

    ORA-10878: parallel recovery slave died unexpectedly

   This error may also be accompanied by

    ORA-16766 Redo Apply is stopped

 - the memory model of the underlying platform is non-TSO

   (eg AIX.PPC64)

Workaround

 Restart recovery or use serial recovery

 eg: ALTER DATABASE RECOVER MANAGED STANDBY

      DATABASE USING CURRENT LOGFILE DISCONNECT noparallel

從BUG描述來看，這個BUG出現在AIX 64位系統上，資料庫版本11.2.0.1.

the recovery coordinator process (eg MRP on a standby database) terminates recovery and writes the following message to its tracefile:

這一段描述也和我的環境相符合，基本和我遇到的這個情況是一致的.

下面Workaround也給出瞭解決辦法
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT noparallel
或才升級資料庫到11.2.0.2以上版本

至此問題解決

另：Oracle也提供了一個工具ORA-600/ORA-7445/ORA-700 Error Look-up Tool用來診斷ORA-600 和ORA-7445/ ORA-700錯誤,
ORA-600/ORA-7445/ORA-700 Error Look-up Tool工具的使用方法見我的另一篇部落格http://blog.itpub.net/29702473/viewspace-1426297/

ORA-07445: [kggibr()+52]錯誤解決方法

今天做資料庫巡檢，發現一套Datagard資料庫standby備庫日誌沒有應用，查alert.log裡有報錯，ORA-07445: exception encountered: core dump [kggibr()+52]

相關文章