資料庫伺服器記憶體不夠，DBW0 hang死導致sqlplus連不上

後臺"sqlplus / as sysdba"去連線，不能正常連線

alter日誌報錯：

Errors in file /opt/app/oracle/admin/gc/bdump/gc_pmon_14178.trc:

ORA-00471: Message 471 not found; No message file for product=RDBMS, facility=ORA

Tue Jul 11 15:19:30 2023

WARNING: inbound connection timed out (ORA-3136)

Tue Jul 11 15:19:32 2023

WARNING: inbound connection timed out (ORA-3136)

Tue Jul 11 15:19:33 2023

WARNING: inbound connection timed out (ORA-3136)

Tue Jul 11 15:19:34 2023

WARNING: inbound connection timed out (ORA-3136)

Tue Jul 11 15:19:37 2023

根據alert日誌檢查gc_pmon_14178.trc日誌：

** 2023-07-11 15:18:27.293

kssxdl: error deleting SO: 0x2d30ebc48, type: 53, owner: 0x2e5fcf180, flag: I/-/-/0x00:

ORA-00471: Message 471 not found; No message file for product=RDBMS, facility=ORA

*** 2023-07-11 15:18:37.318

Background process DBW0 found dead

Oracle pid = 5

OS pid (from detached process) = 14184

OS pid (from process state) = 14184

dtp = 0x60012cf0, proc = 0x2e59c30f0

Dump of memory from 0x0000000060012CF0 to 0x0000000060012D38

發現Background process DBW0 found dead，檢查作業系統message日誌:

Jul 11 15:17:04 ytg-db kernel: [ 5308]  1001  5308  2556379     1543   0       0             0 oracle

Jul 11 15:17:04 ytg-db kernel: [ 5310]  1001  5310  2556413     1537   3       0             0 oracle

Jul 11 15:17:04 ytg-db kernel: [ 5312]  1001  5312  2556413     1589   2       0             0 oracle

Jul 11 15:17:04 ytg-db kernel: Out of memory: Kill process 4956 (oracle) score 128 or sacrifice child

Jul 11 15:17:04 ytg-db kernel: Killed process 4956, UID 1001, (oracle) total-vm:10227268kB, anon-rss:2248kB, file-rss:2613572kB

Jul 11 15:17:12 ytg-db kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Jul 11 15:17:12 ytg-db kernel: java cpuset=/ mems_allowed=0

Jul 11 15:17:12 ytg-db kernel: Pid: 1191, comm: java Tainted: G           --------------- H  2.6.32-431.23.3.el6.x86_64 #1

Jul 11 15:17:12 ytg-db kernel: Call Trace:

Jul 11 15:17:12 ytg-db kernel: [<ffffffff810d0431>] ? cpuset_print_task_mems_allowed+0x91/0xb0

Jul 11 15:17:12 ytg-db kernel: [<ffffffff81122810>] ? dump_header+0x90/0x1b0

Jul 11 15:17:12 ytg-db kernel: [<ffffffff8122833c>] ? security_real_capable_noaudit+0x3c/0x70

Jul 11 15:17:12 ytg-db kernel: [<ffffffff81122c92>] ? oom_kill_process+0x82/0x2a0

Jul 11 15:17:12 ytg-db kernel: [<ffffffff81122bd1>] ? select_bad_process+0xe1/0x120

Jul 11 15:17:12 ytg-db kernel: [<ffffffff811230d0>] ? out_of_memory+0x220/0x3c0

Jul 11 15:17:12 ytg-db kernel: [<ffffffff8112f9ef>] ? __alloc_pages_nodemask+0x89f/0x8d0

Jul 11 15:17:12 ytg-db kernel: [<ffffffff811678ea>] ? alloc_pages_current+0xaa/0x110

發現“Jul 11 15:17:04 ytg-db kernel: Out of memory: Kill process 4956 (oracle) score 128 or sacrifice child”，這是由於記憶體不足的情況，Linux中觸發保護機制，保全系統不發生嚴重問題，優先殺掉體量較大的程式，來解決記憶體不足問題。

所以發現這種情況，首先考慮的是增加記憶體。