ORA-27300: OS system dependent operation:fork failed with status: 12
Oracle 版本:11.2.0.4.161018 ,作業系統:AIX 7100-04-02-1614 ,系統架構:單例項。
資料庫報錯無法正常 fork 程式,後臺報錯。
1 、alert 日誌報錯,無法fork 程式,直接報錯。
根據ORA-27300: OS system dependent operation:fork failed with status: 12 ,匹配到bug ,記憶體不足,導致程式無法fork 。
*** 2021-01-01 01:56:33.588
*** SESSION ID:(28.1) 2021-01-01 01:56:33.588
*** CLIENT ID:() 2021-01-01 01:56:33.588
*** SERVICE NAME:(SYS$BACKGROUND) 2021-01-01 01:56:33.588
*** MODULE NAME:() 2021-01-01 01:56:33.588
*** ACTION NAME:() 2021-01-01 01:56:33.588
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
*** 2021-01-01 01:56:41.760
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
*** 2021-01-01 02:00:31.593
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
*** 2021-01-01 02:00:37.594
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
2 、OS 同樣有交換空間溢位的報錯,具體如下:
SYSTEM RUNNING OUT OF PAGING SPACE
Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
54593176
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
196
---------------------------------------------------------------------------
LABEL: PGSP_KILL
IDENTIFIER: C5C09FFA
Date/Time: Fri Jan 1 02:02:50 CST 2021
Sequence Number: 5100
Machine Id: 00CE70D74C00
Node Id: GXDB01
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSVMM
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE
Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE
Recommended Actions
DEFINE ADDITIONAL PAGING SPACE
REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)
3 、檢查OS 記憶體的剩餘情況,發現從01.35 分,記憶體剩餘維持在比較低的值,虛擬記憶體明顯降低,虛擬記憶體已經不夠用了。
4 、檢查業務負載00:50 磁碟讀寫壓力明顯增加,00:50 網路IO 的讀寫開始明顯增加,00:50 檔案系統Paging 明顯增加,但是到01:25 才出現明顯的記憶體Paging.
明顯看出從00:00 之後RUNQUEUE 明顯增加, 系統負載增加,導致記憶體使用增多,記憶體不夠用觸發Oracle bug, 導致ORA-27300 錯誤,無法正常fork 出程式。
1 、排查當天的業務增長情況,看是否有明顯的業務增長。
2 、記憶體擴容,預留足夠的記憶體給OS, 記憶體分配(PGA+SGA 60%~70% 左右)
參考文件:
診斷 ORA-27300 ORA-27301 ORA-27302 錯誤 (Doc ID 2179478.1)
參考 Note 465002.1 Database Crash With Error ORA-00490
Note 580552.1 Ora-04030 Ora-07445 Ora-27300 Ora-27301 Ora-27302 Crashed Database
Note 560309.1> /> Database Cannot Start Due to Lack of Memory
記憶體資訊收集命令參考:
# amepat
# svmon -U -t 10 -O summary=basic -i 60
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/10201716/viewspace-2939063/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- ORA-27300: OS system dependent operation:fork failed with statusAI
- 3.4.3 Restoring the System to Normal OperationRESTORM
- fork failed - Cannot allocate memoryAI
- Failed to get D-Bus connection: Operation not permittedAIMIT
- Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERRORAIDNNError
- rabbitMq的status報錯Error: unable to perform an operation on node ‘rabbit……MQErrorORM
- docker安裝matlab mount failed: Operation not permitted.DockerMatlabAIMIT
- SAP CRM One order裡user status和system status的mapping邏輯APP
- python 中 os.fork() 與 os.waitpid(pid,*args, **args) 說明PythonAI
- remount of /system failed: Read-only file system原因及解決REMAI
- Error - RtlWerpReportException failed with status code :-1073741823. Will try to launch the processErrorExceptionAI
- 解決STS或者Eclipse卡死在remote system explorer operation問題EclipseREM
- 執行systemctl status ssh返回“Failed to get properties: Connection timed out”AI
- RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED解決辦法ErrorDNNAI
- gitlab-runner之build failed with exit status 1問題分析GitlabUIAI
- PHP報錯getimagesize(): SSL operation failed with code 1問題解決方案PHPAI
- HTTP Status 500 - Request processing failed; nested exception is org.springframework.jdbc.BadSqlGramHTTPAIExceptionSpringFrameworkJDBCSQL
- 記錄Windows failed fast startup with error status 0xC00000D4.WindowsAIASTError
- dbca建立資料庫時報錯ORA-13516: AWR Operation failed: CATPROC not valid資料庫AI
- command 'gcc' failed with exit status 1錯誤問題的解決辦法GCAI
- MIT6.S081 - Lecture3: OS Organization and System CallsMIT
- MySQL建立使用者報錯 ERROR 1396 (HY000): Operation CREATE USER failed for 'afei'@'%'MySqlErrorAI
- ORA-27300 ORA-27301
- ORACLE 12C opatch fuser與ChecksystemCommandAvailable failedOracleAI
- Realcase: Failed to upgrade SQL Server 2016 SP2 CU11. (Installation success or error status: 1648)AISQLServerError
- [Javascript] ++operationJavaScript
- 部署在IIS中服務出現異常 Failed to load resource: the server responded with a status of 503 (Service Unavailable)AIServer
- Sector size導致ORA-27061: waiting for async I/Os failed with root.shAI
- Deep Embedding Learning for Text-Dependent Speaker Verification
- tidb Error: Operation aborted by user answer '' (cliutil.operation_aborted)TiDBError
- [已解決] [HiveCatalog]Kerberos GSS initiate failed, No valid credentials provided, Cannot read from System.inHiveROSAIIDE
- Fork/Join框架框架
- Fork/Join 框架框架
- Fork倉庫
- [Javascript] Encapsulate chunk operationJavaScript
- macOS: sudo : Operation not permittedMacMIT
- TIC Read Status
- git status 命令Git