ORA-27300: OS system dependent operation:fork failed with status: 12

Oracle 版本：11.2.0.4.161018 ，作業系統：AIX 7100-04-02-1614 ，系統架構：單例項。

 

資料庫報錯無法正常 fork 程式，後臺報錯。

 

1 、alert 日誌報錯，無法fork 程式，直接報錯。

根據ORA-27300: OS system dependent operation:fork failed with status: 12 ，匹配到bug ，記憶體不足，導致程式無法fork 。

 

 

*** 2021-01-01 01:56:33.588

*** SESSION ID:(28.1) 2021-01-01 01:56:33.588

*** CLIENT ID:() 2021-01-01 01:56:33.588

*** SERVICE NAME:(SYS$BACKGROUND) 2021-01-01 01:56:33.588

*** MODULE NAME:() 2021-01-01 01:56:33.588

*** ACTION NAME:() 2021-01-01 01:56:33.588

 

Process startup failed, error stack:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

 

*** 2021-01-01 01:56:41.760

Process startup failed, error stack:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

 

*** 2021-01-01 02:00:31.593

Process startup failed, error stack:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

 

*** 2021-01-01 02:00:37.594

Process startup failed, error stack:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

 

2 、OS 同樣有交換空間溢位的報錯，具體如下：

 

SYSTEM RUNNING OUT OF PAGING SPACE

Detail Data

PROGRAM

oracle

USER'S PROCESS ID:

              54593176

PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS

         196

---------------------------------------------------------------------------

LABEL:          PGSP_KILL

IDENTIFIER:     C5C09FFA

 

Date/Time:       Fri Jan  1 02:02:50 CST 2021

Sequence Number: 5100

Machine Id:      00CE70D74C00

Node Id:         GXDB01

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSVMM         

 

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

 

Probable Causes

SYSTEM RUNNING OUT OF PAGING SPACE

 

Failure Causes

INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM

PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

 

        Recommended Actions

        DEFINE ADDITIONAL PAGING SPACE

        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

3 、檢查OS 記憶體的剩餘情況，發現從01.35 分，記憶體剩餘維持在比較低的值，虛擬記憶體明顯降低，虛擬記憶體已經不夠用了。

 

4 、檢查業務負載00:50 磁碟讀寫壓力明顯增加，00:50 網路IO 的讀寫開始明顯增加，00:50 檔案系統Paging 明顯增加，但是到01:25 才出現明顯的記憶體Paging.

 

明顯看出從00:00 之後RUNQUEUE 明顯增加, 系統負載增加，導致記憶體使用增多，記憶體不夠用觸發Oracle bug, 導致ORA-27300 錯誤，無法正常fork 出程式。

 

1 、排查當天的業務增長情況，看是否有明顯的業務增長。

2 、記憶體擴容，預留足夠的記憶體給OS, 記憶體分配（PGA+SGA  60%~70% 左右）

 

參考檔案：

診斷 ORA-27300 ORA-27301 ORA-27302 錯誤 (Doc ID 2179478.1)

參考 Note 465002.1  Database Crash With Error ORA-00490 

Note 580552.1 Ora-04030 Ora-07445 Ora-27300 Ora-27301 Ora-27302 Crashed Database

 Note 560309.1> /> Database Cannot Start Due to Lack of Memory

 

記憶體資訊收集命令參考：

# amepat

 

# svmon -U -t 10 -O summary=basic -i 60