ORA-27300，ORA-27301，ORA-27302，ORA-04030導致crashed database

今天檢查資料庫的時候發現的alert日誌報錯，OS:HP UNIX 11.31 DB:Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 – 64bit Prodction RAC

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn5

ORA-27303: additional information: skgpspawn5

Mon Mar 14 15:23:00 EAT 2011

Process J001 died, see its trace file

Mon Mar 14 15:23:00 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:23:00 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn5

ORA-27303: additional information: skgpspawn5

Mon Mar 14 15:46:09 EAT 2011

Process J000 died, see its trace file

Mon Mar 14 15:46:09 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:46:09 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Mon Mar 14 15:47:09 EAT 2011

Process startup failed, error stack:

Mon Mar 14 15:47:09 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

Mon Mar 14 15:47:10 EAT 2011

Process J000 died, see its trace file

Mon Mar 14 15:47:10 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:47:10 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Process startup failed, error stack:

Mon Mar 14 15:53:07 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

Mon Mar 14 15:53:08 EAT 2011

Process J001 died, see its trace file

Mon Mar 14 15:53:08 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:53:08 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Process startup failed, error stack:

Mon Mar 14 15:53:23 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

Mon Mar 14 15:53:24 EAT 2011

Process m000 died, see its trace file

Mon Mar 14 15:53:24 EAT 2011

ksvcreate: Process(m000) creation failed

 

Errors in file /oracle/admin/scrk/bdump/scrk2_dbw0_3639.trc:

ORA-04030: out of process memory when trying to allocate 1049136 bytes (KSFD vec heap,KS

FD contig I/O buf)

Mon Mar 14 15:53:55 EAT 2011

System state dump is made for local instance

Mon Mar 14 15:53:55 EAT 2011

DBW0: terminating instance due to error 4030

Mon Mar 14 15:53:55 EAT 2011

System State dumped to trace file /oracle/admin/scrk/bdump/scrk2_diag_3620.trc

Mon Mar 14 15:53:58 EAT 2011

Shutting down instance (abort)

License high water mark = 174

Mon Mar 14 15:54:00 EAT 2011

Instance terminated by DBW0, pid = 3639

Mon Mar 14 15:54:03 EAT 2011

Instance terminated by USER, pid = 9322

Mon Mar 14 16:25:27 EAT 2011

Starting ORACLE instance (normal)

透過查詢metalink [ID 579365.1] ：

ORA-27300: OS system dependent operation:fork failed with status: 11錯誤是由於：Maximum number of PROCESSES allowed per user may be too low (for example: maxuproc on AIX, maxuprc on HP and Solaris.  On Solaris 10, check project.max-lwps)。不過alert日誌的ORA-27303: additional information: 是skgpspawn5不是文件中的skgpspawn3。由於對OS不熟，此問題暫時擱置，以備以後查詢。

ORA-27300: OS system dependent operation:fork failed with status: 12 錯誤是由於：STATUS 12 - ENOMEM Not enough core / memory ;During an exec or a break, the program asked for more memory than the one available by the system. This error also occurs when there are too many segmentation registers which are required for the arrangement of text data or stack segments.

ORA-04030: out of process memory when trying to allocate 1049136 bytes (KSFD vec heap,KS

FD contig I/O buf)是由於dbw0沒有請求到足夠的記憶體導致了實力的重啟。根據metalink  [ID 580552.1]在system log應當還有WARNING: Sorry, no swap space to grow stack for pid ***** (oracle)的警告資訊，可是在此時間段內沒有發現，此問題暫時擱置以待複查。