記一次Linux故障

 

1. Linux message日誌

Nov  8 09:39:24 QH-WAP-BILL-2 kernel: SCSI error : <2 0 0 3> return code = 0x10000
Nov  8 09:39:24 QH-WAP-BILL-2 last message repeated 8 times
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 0 to CK200083800438 is dead.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Killing bus 1 to Clariion   CK200083800438 port SP B4.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 6 to CK200083800438 is dead.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 3 to CK200083800438 is dead.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 1 Tgt 0 Lun 2 to CK200083800438 is dead.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 2 Tgt 0 Lun 3 to CK200083800438 is dead.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Killing bus 2 to Clariion   CK200083800438 port SP A4.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 2 Tgt 0 Lun 0 to CK200083800438 is dead.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 2 Tgt 0 Lun 6 to CK200083800438 is dead.
Nov  8 09:39:24 QH-WAP-BILL-2 kernel: Error:Mpx:Path Bus 2 Tgt 0 Lun 2 to CK200083800438 is dead.
Nov  8 09:39:44 QH-WAP-BILL-2 su(pam_unix)[7793]: session opened for user oracle by (uid=0)
Nov  8 09:39:44 QH-WAP-BILL-2 su(pam_unix)[7793]: session closed for user oracle
Nov  8 09:39:44 QH-WAP-BILL-2 su(pam_unix)[7796]: session opened for user oracle by (uid=0)
Nov  8 09:39:44 QH-WAP-BILL-2 su(pam_unix)[7796]: session closed for user oracle
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:All paths to 60060160D1202200CADA7F7BEE5BE111 are dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:60060160D1202200CADA7F7BEE5BE111 is dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:All paths to 60060160D12022009A587FC0B05CE111 are dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 135092518
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 135092519
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 411853570
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 411853571
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 411853572
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 411853573
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 439351437
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 58313812
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 58490880
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerx1, logical block 58511725
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerx1
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Aborting journal on device emcpowerx1.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: ext3_abort called.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: EXT3-fs error (device emcpowerx1): ext3_journal_start_sb: Detected aborted journal
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Remounting filesystem read-only
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:60060160D12022009A587FC0B05CE111 is dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:All paths to 60060160D1202200966C9BFEEC5BE111 are dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:60060160D1202200966C9BFEEC5BE111 is dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:All paths to 60060160E72022002E8EAECDD23EE111 are dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: Error:Mpx:60060160E72022002E8EAECDD23EE111 is dead.
Nov  8 09:40:24 QH-WAP-BILL-2 kernel: EXT3-fs error (device emcpowerx1) in start_transaction: Journal has aborted
Nov  8 09:40:40 QH-WAP-BILL-2 Had[31267]: VCS ERROR V-16-1-13067 (QH-WAP-BILL-2) Agent is calling clean for resource(mamoradb) because the resource became OFFLINE unexpectedly, on its own.
Nov  8 09:40:40 QH-WAP-BILL-2 su(pam_unix)[8037]: session opened for user oracle by (uid=0)
Nov  8 09:40:40 QH-WAP-BILL-2 su(pam_unix)[8037]: session closed for user oracle
Nov  8 09:40:40 QH-WAP-BILL-2 su(pam_unix)[8047]: session opened for user oracle by (uid=0)
Nov  8 09:40:40 QH-WAP-BILL-2 su(pam_unix)[8047]: session closed for user oracle
Nov  8 09:40:42 QH-WAP-BILL-2 su(pam_unix)[8082]: session opened for user oracle by (uid=0)
Nov  8 09:40:42 QH-WAP-BILL-2 su(pam_unix)[8082]: session closed for user oracle
Nov  8 09:40:42 QH-WAP-BILL-2 su(pam_unix)[8085]: session opened for user oracle by (uid=0)
Nov  8 09:40:44 QH-WAP-BILL-2 su(pam_unix)[8085]: session closed for user oracle
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: printk: 8 messages suppressed.
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerc1, logical block 1602
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerc1
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerc1, logical block 1602
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerc1
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerc1, logical block 1602
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerc1
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: Buffer I/O error on device emcpowerc1, logical block 0
Nov  8 09:40:45 QH-WAP-BILL-2 kernel: lost page write due to I/O error on emcpowerc1
Nov  8 09:40:46 QH-WAP-BILL-2 kernel: __journal_remove_journal_head: freeing b_frozen_data
Nov  8 09:40:46 QH-WAP-BILL-2 kernel: __journal_remove_journal_head: freeing b_frozen_data
Nov  8 09:40:48 QH-WAP-BILL-2 Had[31267]: VCS ERROR V-16-1-10205 Group oracle_sg is faulted on system QH-WAP-BILL-2
Nov  8 09:42:20 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 09:43:01 QH-WAP-BILL-2 kernel: qla2400 0000:08:00.0: LIP reset occured (f7f7).
Nov  8 09:43:02 QH-WAP-BILL-2 kernel: qla2400 0000:08:00.0: LIP occured (f7f7).
Nov  8 09:43:02 QH-WAP-BILL-2 kernel: qla2400 0000:08:00.0: LOOP UP detected (2 Gbps).
Nov  8 09:43:27 QH-WAP-BILL-2 kernel: qla2400 0000:08:00.1: LIP reset occured (f7f7).
Nov  8 09:43:27 QH-WAP-BILL-2 kernel: qla2400 0000:08:00.1: LIP occured (f7f7).
Nov  8 09:43:27 QH-WAP-BILL-2 kernel: qla2400 0000:08:00.1: LOOP UP detected (4 Gbps).
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 1 Tgt 0 Lun 3 to CK200083800438 is alive.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Not all paths to 60060160D1202200CADA7F7BEE5BE111 are dead.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:60060160D1202200966C9BFEEC5BE111 is alive.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 1 Tgt 0 Lun 2 to CK200083800438 is alive.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Not all paths to 60060160D1202200966C9BFEEC5BE111 are dead.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:60060160D12022009A587FC0B05CE111 is alive.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 1 Tgt 0 Lun 0 to CK200083800438 is alive.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Not all paths to 60060160D12022009A587FC0B05CE111 are dead.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 1 Tgt 0 Lun 6 to CK200083800438 is alive.
Nov  8 09:44:24 QH-WAP-BILL-2 kernel: Info:Mpx:Not all paths to 60060160E72022002E8EAECDD23EE111 are dead.
Nov  8 09:44:34 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 2 Tgt 0 Lun 0 to CK200083800438 is alive.
Nov  8 09:44:34 QH-WAP-BILL-2 kernel: Info:Mpx:60060160E72022002E8EAECDD23EE111 is alive.
Nov  8 09:44:34 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 2 Tgt 0 Lun 6 to CK200083800438 is alive.
Nov  8 09:44:34 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 2 Tgt 0 Lun 2 to CK200083800438 is alive.
Nov  8 09:44:34 QH-WAP-BILL-2 kernel: Info:Mpx:60060160D1202200CADA7F7BEE5BE111 is alive.
Nov  8 09:44:34 QH-WAP-BILL-2 kernel: Info:Mpx:Path Bus 2 Tgt 0 Lun 3 to CK200083800438 is alive.
Nov  8 09:47:48 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 09:52:52 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 09:58:06 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 10:03:10 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 10:07:29 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 10:07:39 QH-WAP-BILL-2 sshd(pam_unix)[10671]: session opened for user root by (uid=0)
Nov  8 10:08:54 QH-WAP-BILL-2 sshd(pam_unix)[10671]: session closed for user root
Nov  8 10:09:14 QH-WAP-BILL-2 sshd(pam_unix)[10714]: session opened for user root by (uid=0)
Nov  8 10:11:04 QH-WAP-BILL-2 su(pam_unix)[11165]: session opened for user oracle by root(uid=0)
Nov  8 10:12:38 QH-WAP-BILL-2 sshd(pam_unix)[11253]: session opened for user root by (uid=0)
Nov  8 10:12:42 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 10:14:07 QH-WAP-BILL-2 sshd(pam_unix)[11414]: session opened for user root by (uid=0)
Nov  8 10:15:49 QH-WAP-BILL-2 su(pam_unix)[11752]: session opened for user oracle by root(uid=0)
Nov  8 10:17:44 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 10:19:34 QH-WAP-BILL-2 sshd(pam_unix)[11950]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=qh-wap-snmp  user=root
Nov  8 10:19:40 QH-WAP-BILL-2 sshd(pam_unix)[11950]: session opened for user root by (uid=0)
Nov  8 10:22:22 QH-WAP-BILL-2 snmpd[11064]: send response: 
Nov  8 10:23:35 QH-WAP-BILL-2 sshd(pam_unix)[11253]: session closed for user root
Nov  8 10:23:35 QH-WAP-BILL-2 sshd(pam_unix)[11414]: session closed for user root
Nov  8 10:23:35 QH-WAP-BILL-2 su(pam_unix)[11752]: session closed for user oracle
Nov  8 10:23:39 QH-WAP-BILL-2 sshd(pam_unix)[10714]: session closed for user root
Nov  8 10:23:39 QH-WAP-BILL-2 su(pam_unix)[11165]: session closed for user oracle
Nov  8 10:26:40 QH-WAP-BILL-2 kernel: kjournald starting.  Commit interval 5 seconds
Nov  8 10:26:40 QH-WAP-BILL-2 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
Nov  8 10:26:40 QH-WAP-BILL-2 kernel: EXT3 FS on emcpowerx1, internal journal
Nov  8 10:26:40 QH-WAP-BILL-2 kernel: EXT3-fs: recovery complete.
Nov  8 10:26:40 QH-WAP-BILL-2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov  8 10:26:51 QH-WAP-BILL-2 kernel: kjournald starting.  Commit interval 5 seconds
Nov  8 10:26:51 QH-WAP-BILL-2 kernel: EXT3-fs warning: checktime reached, running e2fsck is recommended
Nov  8 10:26:51 QH-WAP-BILL-2 kernel: EXT3 FS on emcpowerc1, internal journal
Nov  8 10:26:51 QH-WAP-BILL-2 kernel: EXT3-fs: recovery complete.
Nov  8 10:26:51 QH-WAP-BILL-2 kernel: EXT3-fs: mounted filesystem with ordered data mode.
Nov  8 10:28:04 QH-WAP-BILL-2 sshd(pam_unix)[12880]: session opened for user root by (uid=0)
......

 

2. oracle alert日誌

Thu Nov  8 09:40:24 2012
Errors in file /bases/oracle/admin/CDMAWAP/bdump/cdmawap_lgwr_24753.trc:
ORA-00345: redo log write error block 832401 count 2
ORA-00312: online log 5 thread 1: '/bases/oracle/oradata/CDMAWAP/redo05.log'
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 832400
Thu Nov  8 09:40:24 2012
KCF: write/open error block=0x1c9e nline=1
     file=39 /bases/oracle/oradata/CDMAWAP/undotbs02.dbf
Thu Nov  8 09:40:24 2012
Errors in file /bases/oracle/admin/CDMAWAP/bdump/cdmawap_ckpt_24755.trc:
ORA-00206: error in writing (block 192, # blocks 1) of controlfile
ORA-00202: controlfile: '/bases/oracle/oradata/CDMAWAP/control03.ctl'
ORA-27072: File I/O error
Linux Error: 30: Read-only file system
Additional information: 191
ORA-00206: error in writing (block 192, # blocks 1) of controlfile
ORA-00202: controlfile: '/bases/oracle/oradata/CDMAWAP/control02.ctl'
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 191
ORA-00206: error in writing (block 192, # blocks 1) of controlfile
ORA-00202: controlfile: '/bases/oracle/oradata/CDMAWAP/control01.ctl'
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 191
     error=27072 txt: 'Linux Error: 5: Input/output error
Additional information: 7325'
Thu Nov  8 09:40:24 2012
Errors in file /bases/oracle/admin/CDMAWAP/bdump/cdmawap_ckpt_24755.trc:
ORA-00221: error on write to controlfile
ORA-00206: error in writing (block 192, # blocks 1) of controlfile
ORA-00202: controlfile: '/bases/oracle/oradata/CDMAWAP/control03.ctl'
ORA-27072: File I/O error
Linux Error: 30: Read-only file system
Additional information: 191
ORA-00206: error in writing (block 192, # blocks 1) of controlfile
ORA-00202: controlfile: '/bases/oracle/oradata/CDMAWAP/control02.ctl'
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 191
ORA-00206: error in writing (block 192, # blocks 1) of controlfile
ORA-00202: controlfile: '/bases/oracle/oradata/CDMAWAP/control01.ctl'
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 191
Thu Nov  8 09:40:24 2012
CKPT: terminating instance due to error 221
Instance terminated by CKPT, pid = 24755
Thu Nov  8 10:11:39 2012