ORA-27300 ORA-27301 ORA-27302問題處理

上週客戶AIX環境Oracle 10.2.0.4 RAC環境中遇到ORA-2730*一系列問題後節點1重啟。我們具體看一下這個問題

環境介紹：
AIX 6100-06
Oracle 10.2.0.4 RAC 使用裸裝置管理

1.系統出現268DA6A3錯誤
$ errpt -aj 268DA6A3
---------------------------------------------------------------------------
LABEL:          DR_DMA_MAPPER_FAIL
IDENTIFIER:     268DA6A3

Date/Time:       Thu Sep 18 18:29:00 GMT+08:00 2014
Sequence Number: 12463954
Machine Id:      00F7025F4C00
Class:           S
Type:            TEMP
WPAR:            Global
Resource Name:   DR_KER_MEM     

Description
Memory related DR operation failed

Probable Causes
DMA Mapper DR handler failure

Failure Causes
DMA specific memory mapper failed

        Recommended Actions
        Try DR operation on other memory resources

Detail Data
Return Code
           1          -1
Memory Address
0000 0007 E84C 0000
Handler Address
0000 0000 0413 0840
Module Name
/usr/lib/drivers/pci/pci_busdd
---------------------------------------------------------------------------
LABEL:          DR_DMA_MAPPER_FAIL
IDENTIFIER:     268DA6A3

Date/Time:       Thu Sep 18 18:28:00 GMT+08:00 2014
Sequence Number: 12463947
Machine Id:      00F7025F4C00
Class:           S
Type:            TEMP
WPAR:            Global
Resource Name:   DR_KER_MEM     

Description
Memory related DR operation failed

Probable Causes
DMA Mapper DR handler failure

Failure Causes
DMA specific memory mapper failed

        Recommended Actions
        Try DR operation on other memory resources

Detail Data
Return Code
           1          22
Memory Address
0000 0006 72B5 0000
Handler Address
0000 0000 0443 3680
Module Name
/usr/lib/drivers/headd

2.叢集CRSD程式出現報錯並最終導致程式中止
2014-09-18 18:28:05.716: [  CRSEVT][16919]32CAAMonitorHandler :: 0:Could not execute /oracle/product/10.2/crs/bin/racgwrap(check) for ora.vip
category: 1234, operation: scls_process_spawn, loc: read_pipe, OS error: 12, other: EOF on read pipe
2014-09-18 18:28:05.750: [  CRSAPP][16919]32CheckResource error for ora.vip error code = -1
……
[  OCRAPI][3368]procr_ctx_set_invalid_no_abort: ctx set to invalid
2014-09-18 18:36:51.127: [ CSSCLNT][11080]clsssRecvMsg: comm error received, comrc 11, con (114b682f0), msg (114b63150), msgl 144

2014-09-18 18:36:51.136: [ CSSCLNT][7994]clsssRecvMsg: comm error received, comrc 11, con (1134b5890), msg (1135b99f0), msgl 144

2014-09-18 18:36:51.178: [ CSSCLNT][7994]clssgsGGetStatus:  communications failed (0/3/324770392)

2014-09-18 18:36:51.178: [ CSSCLNT][7994]clssgsGGetStatus: returning 8

2014-09-18 18:36:51.156: [ CSSCLNT][11080]clssgsGGetStatus:  communications failed (0/3/0)

2014-09-18 18:36:51.178: [ CSSCLNT][11080]clssgsGGetStatus: returning 8

2014-09-18 18:36:51.218: [  CRSEVT][11080]32Error in clssgsgrpstat rc =8
2014-09-18 18:36:51.232: [    CRSD][7994][PANIC]32 termination by CSS, ret=
2014-09-18 18:36:51.241: [    CRSD][7994]32Done.

3.資料庫alert日誌中出現status 12錯誤資訊
Thu Sep 18 18:30:35 2014
Process startup failed, error stack:
Thu Sep 18 18:30:35 2014
Errors in file /oracle/admin/orcl/bdump/orcl1_psp0_11469288.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3

4.之後節點2將節點1重啟，並接管了節點1的vip
5.節點1重啟之後,資料庫alert日誌中報
AUTO SGA: Disabling background sga auto-tuning.
Thu Sep 18 18:47:46 2014
Error 0 in kwqmnpartition(), aborting txn
Thu Sep 18 18:47:49 2014
ORA-376 encountered when generating server alert SMG-4120
Thu Sep 18 18:47:50 2014
Errors in file /oracle/admin/orcl/bdump/orcl1_smon_10551752.trc:
ORA-01595: error freeing extent (12) of rollback segment (133))
ORA-00376: file 2 cannot be read at this time
ORA-01110: data file 2: '/dev/vx/rdsk/vgorc/lvorcl_undotbs1_1'

使用recover datafile之後資料庫恢復。

問題分析：
關於ORA-2730* status 12故障，主要可能有兩種情況導致
1.伺服器資源耗盡,比如記憶體或交換空間，或者是一些其他資源，在有些系統中可能是nproc或maxnproc引數設定太小需要進行調整（參考 文件 579365_1）
2.AIX系統需要安裝IV37048補丁（IV37048 CIFS_FS LEAVES BEHIND DEFUNCT KERNEL PROCESSES）（參考MOS 文件 ID 1541121.1）
  如果是這種情況，伺服器可能出現下面現象：
  -AIX系統不可用
  -系統命令，比如ps命令返回fork或malloc錯誤
  -無法連線到資料庫
  -命令列HANG住
  -殭屍程式

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/23850820/viewspace-1277602/，如需轉載，請註明出處，否則將追究法律責任。