Linux AS3.0 u6 報錯, 系統Hang住 ：kernel: (6747) ERROR: status = -999

Linux AS3.0 u6 版本， 2.4.21-37.ELsmp
Oracle9.2.0.4 RAC OCFS
4/22 下午 17：50左右發生問題，節點2系統hang住，只有按電源才能重新啟動，後來重啟後OK . [@more@]

-------------------------------------------------------------------------
節點2上的 Linux Log :


Apr 22 00:43:57 DELL-RAC02 telnetd[6710]: ttloop: peer died: EOF
Apr 22 00:48:51 DELL-RAC02 telnetd[6813]: ttloop: peer died: EOF
Apr 22 00:53:50 DELL-RAC02 telnetd[6953]: ttloop: peer died: EOF
Apr 22 00:57:43 DELL-RAC02 kernel: (6747) ERROR: status = -999, Common/ocfsgendlm.c, 1146
Apr 22 00:57:43 DELL-RAC02 kernel: (6747) ERROR: status = -999, Common/ocfsgencreate.c, 482
Apr 22 00:57:43 DELL-RAC02 kernel: (6747) ERROR: status = -999, Linux/ocfsmain.c, 2099
Apr 22 00:57:43 DELL-RAC02 kernel: (6747) ERROR: status = -999, Linux/ocfsmain.c, 2427
Apr 22 00:58:50 DELL-RAC02 telnetd[7103]: ttloop: peer died: EOF
Apr 22 01:03:50 DELL-RAC02 telnetd[7259]: ttloop: peer died: EOF



Apr 22 04:44:01 DELL-RAC02 telnetd[18930]: ttloop: peer died: EOF
Apr 22 04:48:51 DELL-RAC02 telnetd[19204]: ttloop: peer died: EOF
Apr 22 04:53:50 DELL-RAC02 telnetd[19334]: ttloop: peer died: EOF
Apr 22 04:55:52 DELL-RAC02 kernel: (18959) ERROR: status = -999, Common/ocfsgendlm.c, 1146
Apr 22 04:55:52 DELL-RAC02 kernel: (18959) ERROR: status = -999, Common/ocfsgencreate.c, 482
Apr 22 04:55:52 DELL-RAC02 kernel: (18959) ERROR: status = -999, Linux/ocfsmain.c, 2099
Apr 22 04:55:52 DELL-RAC02 kernel: (18959) ERROR: status = -999, Linux/ocfsmain.c, 2427
Apr 22 04:58:50 DELL-RAC02 telnetd[19420]: ttloop: peer died: EOF
Apr 22 05:03:50 DELL-RAC02 telnetd[19506]: ttloop: peer died: EOF


Apr 22 08:56:20 DELL-RAC02 telnetd[31190]: ttloop: peer died: EOF
Apr 22 09:01:20 DELL-RAC02 telnetd[31358]: ttloop: peer died: EOF
Apr 22 09:03:38 DELL-RAC02 kernel: (30831) ERROR: status = -999, Common/ocfsgendlm.c, 1146
Apr 22 09:03:38 DELL-RAC02 kernel: (30831) ERROR: status = -999, Common/ocfsgencreate.c, 482
Apr 22 09:03:38 DELL-RAC02 kernel: (30831) ERROR: status = -999, Linux/ocfsmain.c, 2099
Apr 22 09:03:38 DELL-RAC02 kernel: (30831) ERROR: status = -999, Linux/ocfsmain.c, 2427
Apr 22 09:06:20 DELL-RAC02 telnetd[31492]: ttloop: peer died: EOF
Apr 22 09:11:20 DELL-RAC02 telnetd[31628]: ttloop: peer died: EOF



Apr 22 10:41:20 DELL-RAC02 telnetd[3097]: ttloop: peer died: EOF
Apr 22 10:42:33 DELL-RAC02 kernel: (3065) ERROR: status = -999, Common/ocfsgendlm.c, 1146
Apr 22 10:42:33 DELL-RAC02 kernel: (3065) ERROR: status = -999, Common/ocfsgencreate.c, 482
Apr 22 10:42:33 DELL-RAC02 kernel: (3065) ERROR: status = -999, Linux/ocfsmain.c, 2099
Apr 22 10:42:33 DELL-RAC02 kernel: (3065) ERROR: status = -999, Linux/ocfsmain.c, 2427
Apr 22 10:46:20 DELL-RAC02 telnetd[3282]: ttloop: peer died: EOF


Apr 22 12:41:20 DELL-RAC02 telnetd[6788]: ttloop: peer died: EOF
Apr 22 12:46:20 DELL-RAC02 telnetd[6979]: ttloop: peer died: EOF
Apr 22 12:47:13 DELL-RAC02 kernel: (6950) ERROR: status = -999, Common/ocfsgendlm.c, 1146
Apr 22 12:47:13 DELL-RAC02 kernel: (6950) ERROR: status = -999, Common/ocfsgencreate.c, 482
Apr 22 12:47:13 DELL-RAC02 kernel: (6950) ERROR: status = -999, Linux/ocfsmain.c, 2099
Apr 22 12:47:13 DELL-RAC02 kernel: (6950) ERROR: status = -999, Linux/ocfsmain.c, 2427
Apr 22 12:51:20 DELL-RAC02 telnetd[7136]: ttloop: peer died: EOF
Apr 22 12:56:19 DELL-RAC02 telnetd[7364]: ttloop: peer died: EOF
Apr 22 13:01:20 DELL-RAC02 telnetd[7520]: ttloop: peer died: EOF



Apr 22 16:41:22 DELL-RAC02 telnetd[16110]: ttloop: peer died: EOF
Apr 22 16:46:24 DELL-RAC02 telnetd[16227]: ttloop: peer died: EOF
Apr 22 16:47:05 DELL-RAC02 kernel: (16194) ERROR: status = -999, Common/ocfsgendlm.c, 1146
Apr 22 16:47:05 DELL-RAC02 kernel: (16194) ERROR: status = -999, Common/ocfsgencreate.c, 482
Apr 22 16:47:05 DELL-RAC02 kernel: (16194) ERROR: status = -999, Linux/ocfsmain.c, 2099
Apr 22 16:47:05 DELL-RAC02 kernel: (16194) ERROR: status = -999, Linux/ocfsmain.c, 2427
Apr 22 16:51:24 DELL-RAC02 telnetd[16363]: ttloop: peer died: EOF
Apr 22 16:56:24 DELL-RAC02 telnetd[16509]: ttloop: peer died: EOF


以下是重新啟動Server之後報的錯 。 17:55左右重新啟動 。


Apr 22 17:55:18 DELL-RAC02 kernel: Attached scsi disk sdm at scsi2, channel 0, id 1, lun 2
Apr 22 17:55:18 DELL-RAC02 kernel: SCSI device sdh: 555745280 512-byte hdwr sectors (284542 MB)
Apr 22 17:55:18 DELL-RAC02 rpc.statd[11267]: Version 1.0.6 Starting
Apr 22 17:55:18 DELL-RAC02 kernel: sdh: sdh1
Apr 22 17:55:18 DELL-RAC02 kernel: SCSI device sdi: 545259520 512-byte hdwr sectors (279173 MB)
Apr 22 17:55:18 DELL-RAC02 kernel: sdi:<6>Device 08:80 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:80, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:80 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:80, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: unable to read partition table
Apr 22 17:55:18 DELL-RAC02 kernel: SCSI device sdj: 524288000 512-byte hdwr sectors (268435 MB)
Apr 22 17:55:18 DELL-RAC02 kernel: sdj: sdj1
Apr 22 17:55:18 DELL-RAC02 kernel: SCSI device sdk: 555745280 512-byte hdwr sectors (284542 MB)
Apr 22 17:55:18 DELL-RAC02 kernel: sdk:<6>Device 08:a0 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:a0, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:a0 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:a0, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: unable to read partition table
Apr 22 17:55:18 DELL-RAC02 kernel: SCSI device sdl: 545259520 512-byte hdwr sectors (279173 MB)
Apr 22 17:55:18 DELL-RAC02 kernel: sdl: sdl1 sdl2 sdl3 sdl4
Apr 22 17:55:18 DELL-RAC02 kernel: SCSI device sdm: 524288000 512-byte hdwr sectors (268435 MB)
Apr 22 17:55:18 DELL-RAC02 kernel: sdm:<6>Device 08:c0 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:c0, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:c0 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:c0, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: unable to read partition table
Apr 22 17:55:18 DELL-RAC02 kernel: Journalled Block Device driver loaded
Apr 22 17:55:18 DELL-RAC02 kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
Apr 22 17:55:18 DELL-RAC02 kernel: EXT3-fs: write access will be enabled during recovery.
Apr 22 17:55:18 DELL-RAC02 kernel: kjournald starting. Commit interval 5 seconds




Apr 22 17:55:18 DELL-RAC02 kernel: input1: USB HID v1.10 Mouse [Tripplite B015-000 R0.74 USB to PS2 adapter.] on usb2:2.1
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:20 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:20, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:20 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:20, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:20 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:20, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:20 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:20, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:40 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:40, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:40 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:40, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:40 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:40, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:40 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:40, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:60 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:60, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:60 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:60, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:60 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:60, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:60 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:60, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:80 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:80, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:80 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:80, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:80 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:80, sector 0
Apr 22 17:55:18 DELL-RAC02 kernel: Device 08:80 not ready.
Apr 22 17:55:18 DELL-RAC02 kernel: I/O error: dev 08:80, sector 0



Apr 22 23:11:42 DELL-RAC02 telnetd[29182]: ttloop: peer died: EOF
Apr 22 23:16:41 DELL-RAC02 telnetd[29266]: ttloop: peer died: EOF
Apr 22 23:21:41 DELL-RAC02 telnetd[29392]: ttloop: peer died: EOF
Apr 22 23:26:35 DELL-RAC02 kernel: scsi(2): LOOP DOWN detected.
Apr 22 23:26:41 DELL-RAC02 telnetd[29468]: ttloop: peer died: EOF
Apr 22 23:27:21 DELL-RAC02 kernel: scsi(2): LOOP DEAD detected.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Path Bus 2 Tgt 1 Lun 0 to CK200040700048 is dead.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Killing bus 2 to CLARiiON CK200040700048 port B1.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Path Bus 2 Tgt 1 Lun 1 to CK200040700048 is dead.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Path Bus 2 Tgt 1 Lun 2 to CK200040700048 is dead.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Path Bus 2 Tgt 0 Lun 1 to CK200040700048 is dead.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Killing bus 2 to CLARiiON CK200040700048 port A1.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Path Bus 2 Tgt 0 Lun 2 to CK200040700048 is dead.
Apr 22 23:27:21 DELL-RAC02 kernel: Error:emcp:Mpc:Path Bus 2 Tgt 0 Lun 0 to CK200040700048 is dead.
Apr 22 23:31:41 DELL-RAC02 telnetd[29628]: ttloop: peer died: EOF
Apr 22 23:36:42 DELL-RAC02 telnetd[29792]: ttloop: peer died: EOF
Apr 22 23:41:42 DELL-RAC02 telnetd[29844]: ttloop: peer died: EOF
Apr 22 23:46:42 DELL-RAC02 telnetd[29944]: ttloop: peer died: EOF
Apr 22 23:51:42 DELL-RAC02 telnetd[30024]: ttloop: peer died: EOF








節點1上的報錯資訊：

Apr 22 23:11:48 DELL-RAC01 telnetd[17014]: ttloop: peer died: EOF
Apr 22 23:16:48 DELL-RAC01 telnetd[17060]: ttloop: peer died: EOF
Apr 22 23:21:55 DELL-RAC01 telnetd[17106]: ttloop: peer died: EOF
Apr 22 23:26:48 DELL-RAC01 telnetd[17148]: ttloop: peer died: EOF
Apr 22 23:27:15 DELL-RAC01 kernel: ocfs: Removing DELL-PRI02 (node 0) from clustered device (232,83)
Apr 22 23:27:16 DELL-RAC01 kernel: ocfs: Removing DELL-PRI02 (node 1) from clustered device (232,97)
Apr 22 23:27:16 DELL-RAC01 kernel: ocfs: Removing DELL-PRI02 (node 1) from clustered device (232,65)
Apr 22 23:27:16 DELL-RAC01 kernel: ocfs: Removing DELL-PRI02 (node 0) from clustered device (232,82)
Apr 22 23:27:21 DELL-RAC01 kernel: ocfs: Adding DELL-PRI02 (node 1) to clustered device (232,65)
Apr 22 23:27:21 DELL-RAC01 kernel: ocfs: Adding DELL-PRI02 (node 0) to clustered device (232,82)
Apr 22 23:27:21 DELL-RAC01 kernel: ocfs: Adding DELL-PRI02 (node 0) to clustered device (232,83)
Apr 22 23:27:22 DELL-RAC01 kernel: ocfs: Adding DELL-PRI02 (node 1) to clustered device (232,97)
Apr 22 23:31:48 DELL-RAC01 telnetd[17198]: ttloop: peer died: EOF
Apr 22 23:36:48 DELL-RAC01 telnetd[17244]: ttloop: peer died: EOF
Apr 22 23:41:48 DELL-RAC01 telnetd[17284]: ttloop: peer died: EOF
Apr 22 23:46:48 DELL-RAC01 telnetd[17326]: ttloop: peer died: EOF
Apr 22 23:51:55 DELL-RAC01 telnetd[17382]: ttloop: peer died: EOF