斷電導致RAC -- kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).

對Oracle 10g RAC資料庫進行例行檢查時發現兩個節點在同一時間段有相同的報錯資訊，應該是和磁碟方面有關係，由於採用的是共享磁碟櫃，懷疑是連線磁碟櫃的通訊出現問題。 [@more@]

查詢Storage的相關資訊，發現果然是連線Server與Storage的一臺交換機斷電過，冗餘的兩臺交換機，一臺接UPS, 一臺接市電，防止UPS損壞導致系統異常。剛好前兩天由於特殊情況停過電 。

Jun 8 04:04:08 gnd-rac01 syslogd 1.4.1: restart.
Jun 8 04:04:08 gnd-rac01 snmpd[1271]: NET-SNMP version 5.1.2
Jun 8 05:32:53 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 8 05:32:53 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 8 05:32:53 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 8 05:32:53 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 8 12:58:48 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 12:59:33 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 12:59:33 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 12:59:33 gnd-rac01 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:00:09 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:09 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP occured (f8f7).
Jun 8 13:00:09 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f7f7).
Jun 8 13:00:19 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:20 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP UP detected (4 Gbps).
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is alive.
Jun 8 13:27:15 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 0> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac01 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:36 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:28:45 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:28:46 gnd-rac01 kernel: qla2xxx 0000:01:00.0: LOOP UP detected (4 Gbps).
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is alive.
Jun 8 13:33:11 gnd-rac01 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is alive.
Jun 9 05:28:11 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 05:28:11 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 05:28:11 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 05:28:12 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 08:51:25 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 08:51:25 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 08:51:25 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 08:51:25 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 11:02:16 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 11:02:16 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 11:02:16 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 11:02:16 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 14:17:29 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 14:17:29 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 14:17:29 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 14:17:29 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 15:00:39 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 15:00:39 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 15:00:39 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 15:00:39 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 9 22:08:21 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 9 22:08:21 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 9 22:08:21 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 9 22:08:21 gnd-rac01 kernel: hda: ATAPI reset complete
Jun 10 08:24:17 gnd-rac01 kernel: ide-cd: cmd 0x1e timed out
Jun 10 08:24:17 gnd-rac01 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 10 08:24:17 gnd-rac01 kernel: hda: irq timeout: error=0x00
Jun 10 08:24:17 gnd-rac01 kernel: hda: ATAPI reset complete

Jun 8 04:04:13 gnd-rac02 syslogd 1.4.1: restart.
Jun 8 04:04:13 gnd-rac02 snmpd[13034]: NET-SNMP version 5.1.2
Jun 8 08:43:24 gnd-rac02 kernel: ide-cd: cmd 0x1e timed out
Jun 8 08:43:24 gnd-rac02 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 8 08:43:24 gnd-rac02 kernel: hda: irq timeout: error=0x00
Jun 8 08:43:24 gnd-rac02 kernel: hda: ATAPI reset complete
Jun 8 11:34:34 gnd-rac02 kernel: ide-cd: cmd 0x1e timed out
Jun 8 11:34:34 gnd-rac02 kernel: hda: irq timeout: status=0xd0 { Busy }
Jun 8 11:34:34 gnd-rac02 kernel: hda: irq timeout: error=0x00
Jun 8 11:34:34 gnd-rac02 kernel: hda: ATAPI reset complete
Jun 8 12:58:48 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 12:59:33 gnd-rac02 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 12:59:33 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 12:59:33 gnd-rac02 last message repeated 2 times
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 12:59:33 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:00:09 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:09 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP occured (f8f7).
Jun 8 13:00:09 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f7f7).
Jun 8 13:00:19 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).
Jun 8 13:00:20 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LOOP UP detected (4 Gbps).
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is alive.
Jun 8 13:04:44 gnd-rac02 kernel: Info:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is alive.
Jun 8 13:27:15 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LOOP DOWN detected (2).
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 0 1> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 0> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: SCSI error : <3 0 1 2> return code = 0x10000
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP A5.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 0 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 1 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Killing bus 3 to Clariion CK200074100595 port SP B5.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 2 to CK200074100595 is dead.
Jun 8 13:28:00 gnd-rac02 kernel: Error:Mpx:Path Bus 3 Tgt 1 Lun 0 to CK200074100595 is dead.
Jun 8 13:28:34 gnd-rac02 kernel: qla2xxx 0000:01:00.0: LIP reset occured (f8f7).