ORACLE OMS啟動失敗之BEA-300040 I/O error while writing node manager status

        今天處理一個oracle最佳化管理工具oms啟動異常的問題。oms問題處理之前，oms所在伺服器上的一套asm和db所用

的審計目錄檔案系統inode耗盡導致asm和db異常當機，經清理審計目錄的日誌後asm和db正常啟動，但是oms啟動異常。

一、OMS啟動異常分析

1、OMS啟動異常資訊

[oracle@orcl bin]$ ./emctl start oms

Oracle Enterprise Manager Cloud Control 12c Release 4  

Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

Starting Oracle Management Server...

Starting WebTier...

WebTier Successfully Started

Oracle Management Server Could Not Be Started

Check EM Server log file for details: /u01/app/oracle/oms/domains/GCDomain/servers/ORCL_OMS1/logs/ORCL_OMS1.out

Oracle Management Server is Down

[oracle@ orcl bin]$ 

2、檢視 ORCL_OMS1.out日誌

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.audit.AuditServiceImpl registerInternal

WARNING: Cannot register to audit service for component "JPS".

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.policystore.ldap.LdapPolicyStore initial

INFO: Container is not ready at the point null

<Mar 19, 2020 5:55:53 PM CST> <Error> <NodeManager> <BEA-300040> <I/O error while writing node manager status change message.

java.io.IOException: No space left on device

        at java.io.UnixFileSystem.createFileExclusively(Native Method)

        at java.io.File.checkAndCreate(File.java:1705)

        at java.io.File.createTempFile0(File.java:1726)

        at java.io.File.createTempFile(File.java:1803)

        at weblogic.nodemanager.util.ConcurrentUnixFile.write(ConcurrentUnixFile.java:38)

        Truncated. see log file for complete stacktrace

> 

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <Waiting for the process to die: 6906>

<Mar 19, 2020 5:55:54 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

3、根據.out日誌提示，跟之前asm和db啟動異常報錯一致，也是之前oracle資料庫審計日誌過多導致檔案系統inode耗盡

其他程式無法寫入導致。

二、問題處理

1、對於oms來說，單純的重啟oms無法解決問題，因為nodemanager程式未正常停止

2、使用ps -ef|grep java，把oms相關的java程式殺掉kill -9 java_process_number

3、再次嘗試啟動oms程式，成功啟動