hp rx6600兩臺oracle雙機互備伺服器其中一臺經常自動關機的故障診斷
hp rx6600兩臺oracle資料庫雙機互備伺服器其中一臺經常自動關機，剛好在做巡檢時遇到了就順便檢查一下原因.檢查經常出故障的一臺小機日誌資訊如下:
rx6600-1:[/]#cat /var/adm/syslog/syslog.log Nov 6 10:40:35 rx6600-1 syslogd: restart Nov 6 10:40:35 rx6600-1 vmunix: Found adjacent data tr. Growing size. 0x32a6000 -> 0x72a6000. Nov 6 10:40:35 rx6600-1 vmunix: Pinned PDK malloc pool: base: 0xe000000100d5a000 size=117400K Nov 6 10:40:35 rx6600-1 vmunix: Loaded ACPI revision 2.0 tables. Nov 6 10:40:35 rx6600-1 vmunix: MMIO on this platform supports Write Coalescing. Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 vmunix: MFS is defined: base= 0xe000000100d5a000 size= 5084 KB Nov 6 10:40:35 rx6600-1 vmunix: Unpinned PDK malloc pool: base: 0xe000000108000000 size=393216K Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: cachefs_link(): File system was registered at index 5. Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: GPX emcpgpx_install() success. Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 above message repeats 2 times Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: DM emcpgpx_dm_install() success. Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: VLUMD emcpgpx_vlumd_install() success. Nov 6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: XCRYPT emcpgpx_xcrypt_install() success. Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: nfs3_link(): File system was registered at index 8. Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: mod_fs_reg: Cannot retrieve configured loading phase from KRS for module: cifs. Setting to load at INIT Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 vmunix: 0 sba Nov 6 10:40:35 rx6600-1 vmunix: 0/0 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/0/1/0 rmp3f01 Nov 6 10:40:35 rx6600-1 vmunix: 0/0/1/1 rmp3f01 Nov 6 10:40:35 rx6600-1 vmunix: 0/0/1/2 asio0 Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/0 UsbOhci Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Unknown/0_1 Nov 6 10:40:35 rx6600-1 vmunix: <2.1.3.10.1008.4390.1> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0 UsbMiniBus Nov 6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/hp/Unknown/0_1 Nov 6 10:40:35 rx6600-1 vmunix: <2.305.3.100.1008.4390.1> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0.0 UsbBootKeyboard Nov 6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Standard/hp/Unknown/0_1 Nov 6 10:40:35 rx6600-1 vmunix: <2.307.3.10.1008.4390.1> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1 UsbOhci Nov 6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Multibay/0_a1 Nov 6 10:40:35 rx6600-1 vmunix: <2.1.3.10.1008.294.161> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0 UsbMiniBus Nov 6 10:40:35 rx6600-1 vmunix: Devices/MassStorage-SCSI/USB/BulkOnly/hp/Multibay/0_a1 Nov 6 10:40:35 rx6600-1 vmunix: <2.310.3.150.1008.294.161> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.0 UsbBulkOnlyMS Nov 6 10:40:35 rx6600-1 vmunix: Devices/ScsiControllerAdaptor/USB/BulkOnly/hp/Multibay Nov 6 10:40:35 rx6600-1 vmunix: <2.1000.3.150.1008.294> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16 UsbScsiAdaptor Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:36 rx6600-1 above message repeats 5 times Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/Avocent/KVMAdaptor/1_0 Nov 6 10:40:35 rx6600-1 vmunix: <2.1.3.10.1572.833.256> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1 UsbMiniBus Nov 6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/Avocent/KVMAdaptor/1_0 Nov 6 10:40:35 rx6600-1 vmunix: <2.305.3.100.1572.833.256> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.0 UsbBootKeyboard Nov 6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Boot/Avocent/KVMAdaptor/1_0 Nov 6 10:40:35 rx6600-1 vmunix: <2.307.3.100.1572.833.256> Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.1 UsbBootMouse Nov 6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached. Identification String: Nov 6 10:40:36 rx6600-1 above message repeats 2 times Nov 6 10:40:35 rx6600-1 vmunix: 0/0/2/2 UsbEhci Nov 6 10:40:35 rx6600-1 vmunix: 0/0/4/0 gvid_core Nov 6 10:40:35 rx6600-1 vmunix: 0/1 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/2 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/2/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/2/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/2/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/2/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/3 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/3/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/3/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/3/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/3/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/4 lba Nov 6 10:40:35 rx6600-1 vmunix: sasd: Claimed HP PCI/PCI-X SAS MPT adapter at hardware path 0/4/1/0 Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0 sasd Nov 6 10:40:35 rx6600-1 vmunix: 0/4/2/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/4/2/1 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/5 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/5/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/6 lba Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0 PCItoPCI Nov 6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/6/1/0/4/0 (FC Port 1 on HBA) Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0 fcd Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/6/0 iether Nov 6 10:40:35 rx6600-1 vmunix: 0/7 lba Nov 6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/0. Controller firmware version is 01.03.35.70 Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/0 mpt Nov 6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/1. Controller firmware version is 01.03.35.70 Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/1 mpt Nov 6 10:40:35 rx6600-1 vmunix: 120 processor Nov 6 10:40:35 rx6600-1 vmunix: 121 processor Nov 6 10:40:35 rx6600-1 vmunix: 122 processor Nov 6 10:40:35 rx6600-1 vmunix: 123 processor Nov 6 10:40:35 rx6600-1 vmunix: 124 processor Nov 6 10:40:35 rx6600-1 vmunix: 125 processor Nov 6 10:40:35 rx6600-1 vmunix: 126 processor Nov 6 10:40:35 rx6600-1 vmunix: 127 processor Nov 6 10:40:35 rx6600-1 vmunix: 250 pdh Nov 6 10:40:35 rx6600-1 vmunix: 250/0 ipmi Nov 6 10:40:35 rx6600-1 vmunix: 250/1 asio0 Nov 6 10:40:35 rx6600-1 vmunix: 250/2 acpi_node Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1 fcd_fcp Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.1 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.2 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.3 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.4 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1 fcd_fcp Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0 fcd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.1 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.2 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.3 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.1 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.4 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.2 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.3 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.4 sdisk Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7.0 sctl Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0 sasd_vbus Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0 tgt Nov 6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0.0 sdisk Nov 6 10:40:35 rx6600-1 vmunix: Boot device's HP-UX HW path is: 0/4/1/0.0.0.0.0 Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:35 rx6600-1 vmunix: System Console is on the Built-In Serial Interface Nov 6 10:40:35 rx6600-1 vmunix: iether0: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/2/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether1: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/3/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether2: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/0 Nov 6 10:40:35 rx6600-1 vmunix: iether4: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/5/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether5: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/6/1/0/6/0 Nov 6 10:40:35 rx6600-1 vmunix: iether3: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/1 Nov 6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x3 configured as ROOT Nov 6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as SWAP Nov 6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as DUMP Nov 6 10:40:35 rx6600-1 vmunix: Swap device table: (start & size given in 512-byte blocks) Nov 6 10:40:35 rx6600-1 vmunix: entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216 Nov 6 10:40:35 rx6600-1 vmunix: Dump device table: (start & size given in 1-Kbyte blocks) Nov 6 10:40:35 rx6600-1 vmunix: entry 0000000000000000 - major is 31, minor is 0x30000; start = 2349940, size = 8388604 Nov 6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 1 Nov 6 10:40:35 rx6600-1 vmunix: Create STCP device files Nov 6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 2 Nov 6 10:40:35 rx6600-1 vmunix: $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $ Nov 6 10:40:35 rx6600-1 vmunix: Memory Information: Nov 6 10:40:35 rx6600-1 vmunix: physical page size = 4096 bytes, logical page size = 4096 bytes Nov 6 10:40:35 rx6600-1 vmunix: Physical: 25133536 Kbytes, lockable: 18994328 Kbytes, available: 22051156 Kbytes Nov 6 10:40:35 rx6600-1 vmunix: Nov 6 10:40:36 rx6600-1 nettl[832]: nettl starting up. Nov 6 10:40:48 rx6600-1 sshd[986]: Server listening on :: port 22. Nov 6 10:40:48 rx6600-1 sshd[986]: Server listening on 0.0.0.0 port 22. Nov 6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found CLTS loopback transport Nov 6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS loopback transport Nov 6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS ORD loopback transport Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for udp Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for tcp Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticlts Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticotsord Nov 6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticots Nov 6 10:40:50 rx6600-1 inetd[1100]: Reading configuration Nov 6 10:40:50 rx6600-1 inetd[1100]: ftp/tcp: Added service, server /usr/lbin/ftpd Nov 6 10:40:50 rx6600-1 inetd[1100]: telnet/tcp: Added service, server /usr/lbin/telnetd Nov 6 10:40:50 rx6600-1 inetd[1100]: tftp/udp: Added service, server /usr/lbin/tftpd Nov 6 10:40:50 rx6600-1 inetd[1100]: login/tcp: Added service, server /usr/lbin/rlogind Nov 6 10:40:50 rx6600-1 inetd[1100]: shell/tcp: Added service, server /usr/lbin/remshd Nov 6 10:40:50 rx6600-1 inetd[1100]: exec/tcp: Added service, server /usr/lbin/rexecd Nov 6 10:40:50 rx6600-1 inetd[1100]: ntalk/udp: Added service, server /usr/lbin/ntalkd Nov 6 10:40:50 rx6600-1 inetd[1100]: auth/tcp: Added service, server /usr/lbin/identd Nov 6 10:40:50 rx6600-1 inetd[1100]: printer/tcp: Added service, server /usr/sbin/rlpdaemon Nov 6 10:40:51 rx6600-1 inetd[1100]: daytime/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: daytime/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: time/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: echo/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: echo/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: discard/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: discard/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: chargen/tcp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: chargen/udp: Added service, server internal Nov 6 10:40:51 rx6600-1 inetd[1100]: kshell/tcp: Added service, server /usr/lbin/remshd Nov 6 10:40:51 rx6600-1 inetd[1100]: klogin/tcp: Added service, server /usr/lbin/rlogind Nov 6 10:40:51 rx6600-1 inetd[1100]: dtspc/tcp: Added service, server /usr/dt/bin/dtspcd Nov 6 10:40:51 rx6600-1 inetd[1100]: recserv/tcp: Added service, server /usr/lbin/recserv Nov 6 10:40:51 rx6600-1 inetd[1100]: swat/tcp: Added service, server /opt/samba/bin/swat Nov 6 10:40:51 rx6600-1 inetd[1100]: registrar/tcp: Added service, server /etc/opt/resmon/lbin/registrar Nov 6 10:40:51 rx6600-1 inetd[1100]: hacl-probe/tcp: Added service, server /opt/cmom/lbin/cmomd Nov 6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/udp: Added service, server /usr/lbin/cmclconfd Nov 6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/tcp: Added service, server /usr/lbin/cmclconfd Nov 6 10:40:51 rx6600-1 inetd[1100]: instl_boots/udp: Added service, server /opt/ignite/lbin/instl_bootd Nov 6 10:40:51 rx6600-1 inetd[1100]: omni/tcp: Added service, server /opt/omni/lbin/inet Nov 6 10:40:51 rx6600-1 inetd[1100]: rpc.cmsd/udp: Added service, server /usr/dt/bin/rpc.cmsd Nov 6 10:40:51 rx6600-1 inetd[1100]: rpc.ttdbserver/tcp: Added service, server /usr/dt/bin/rpc.ttdbserver Nov 6 10:40:51 rx6600-1 inetd[1100]: Configuration complete Nov 6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: restore (user ID real=0 effective=0) Nov 6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: config (user ID real=0 effective=0) Nov 6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: save (user ID real=0 effective=0) Nov 6 10:40:54 rx6600-1 su: + tty?? root-sfmdb Nov 6 10:41:06 rx6600-1 cimserver[1706]: starting Nov 6 10:41:29 rx6600-1 cimserver[1707]: PGS10026: THE CIM SERVER IS LISTENING ON HTTPS PORT 5,989. Nov 6 10:41:29 rx6600-1 cimserver[1707]: PGS10028: THE CIM SERVER IS LISTENING ON THE LOCAL CONNECTION SOCKET. Nov 6 10:41:29 rx6600-1 cimserver[1707]: PGS10030: STARTED HP-UX WBEM Services VERSION A.02.07. Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/100dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/75dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_korean/75dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Cannot initialize font path element: "/usr/lib/X11/fonts/hp_chinese_t/75dpi/" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ttfjpn.st" Nov 6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ifojpn.st" Nov 6 10:41:34 rx6600-1 pwgrd: Started at Thu Nov 6 10:41:34 2014, pid = 1798 Nov 6 10:41:34 rx6600-1 diagmond[1833]: started Nov 6 10:41:34 rx6600-1 /usr/sbin/envd[1837]: VXPBFt6/, 2"6A3vEdVCND< ~ Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2180]: Setting STREAMS-HEAD high water value to 131072. Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one mpctl succeeded: ncpus = 8. Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 2 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 3 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd do_one bind 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd do_one bind 1 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd do_one bind 2 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd do_one bind 3 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd do_one bind 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd do_one bind 5 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one bind 7 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd 5 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2197]: nfsd 5 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd 3 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd 2 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2200]: nfsd 2 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd 1 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2201]: nfsd 1 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2199]: nfsd 3 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd 4 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2202]: nfsd 4 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd 7 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2219]: nfsd 7 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd do_one bind 6 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd 0 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2220]: nfsd 0 0 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd 6 1 sock 4 Nov 6 10:41:50 rx6600-1 /usr/sbin/nfsd[2221]: nfsd 6 0 sock 4 Nov 6 10:41:53 rx6600-1 krsd[2300]: Delay time is 300 seconds Nov 6 10:41:53 rx6600-1 sfd[2301]: daemon already running. Nov 6 10:41:54 rx6600-1 sfd[2314]: starting the daemon. Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [0] (2,4,60) Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [1] (20,40,300) Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SetLogMask:: EventLogMask set to 0x66 Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Using hostname localhost community public debug 0 Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Daemon created successfully. Starting it now Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled. Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled. Nov 6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f ***** Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!# Nov 6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a Nov 6 10:49:14 rx6600-1 EMS [2928]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/system/events/ipmi_fpl/ipmi_fpl" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 191889410 -r /system/events/ipmi_fpl/ipmi_fpl -n 191889409 -a Nov 6 18:48:12 rx6600-1 EMS [2970]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641922 -a Nov 6 19:00:00 rx6600-1 su: + tty?? root-oracle Nov 7 08:00:00 rx6600-1 su: + tty?? root-root
從如下資訊看到伺服器已經出問題了，且資訊已經指出可以執行
/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令來檢視詳細資訊
Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled. Nov 6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled. Nov 6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f ***** Nov 6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!# Nov 6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a
執行/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令來檢視詳細資訊
rx6600-1:[/]#/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a
ARCHIVED MONITOR DATA:
Event Time..........: Thu Nov 6 10:45:42 2014
Severity............: MAJORWARNING
Monitor.............: ia64_corehw
Event #.............: 101011
System..............: rx6600-1
Summary:
System temperature is out of normal range.
Description of Error:
The system temperature is not within normal operating range. It is higher
than required operating range.
這個錯誤描述是說系統的溫度超出了正常範圍，下面資訊說明了可能的原因
Probable Cause / Recommended Action:
Something may be blocking the cooling intakes of the fans. Check for
obstruction.
One or more fans may be operating at lower speed than normal. Check the
fan performance.
Check for problems with the room air conditioning.
If the problem is not fixed, the operating temperature may become
non-recoverable, in which case there are chances that the hardware may be
damaged. At that temperature level, on Integrity servers, the firmware
will shutdown the system automatically. However on HP 9000 servers, the
action specified in the envd config file will be taken - which may be to
shutdown the system automatically.
For information on the sensor that generated this event, refer to FRU ID
in Event Details section.
上面的資訊是說，可能需要清理一下風機，或者風機效能出現問題，或者檢查空調情況，如果不是這些原因造成那麼可能是硬體出現問題了。下面的論斷事件的資料:
Additional Event Data:
System IP Address...: 10.138.129.5
Event Id............: 0x545ae0d600000000
Monitor Version.....: B.01.00
Event Class.........: System
Client Configuration File...........:
/var/stm/config/tools/monitor/default_ia64_corehw.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: ia64 hp server rx6600
EMS Version.....................: A.04.20
STM Version.....................: C.58.00
System Serial Number............: SGH48045VY
Latest information on this event:
v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v
Event Details :
Event Date .............: Thu Nov 6 10:44:08 2014
Sensor Number ..........: 0xdb
Sensor Type ............: Temperature
Sensor Class ...........: Threshold based
Sensor Reading/Offset...: 0x07 (Offset)
Event Type.............: Assertion
Entity ID ..............: 3
Generic Message.........:
Temperature : Upper non-critical - going high
Entity FRU Id Info......:
processor (Sensor ID: Processor 2)
從上面的Event Details資訊可以看到，感測器型別是溫度方面的問題，感測器類別是基於閾值，事件型別是斷言，是說2號cpu的溫度已經超過了閾值.經過檢查不是機房空調，通風口堵塞問題，需要聯絡小機廠商來進行一步檢查是什麼原因造成cpu溫度超過閾值，平時cpu使用率只有10%。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26015009/viewspace-1323699/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- 兩臺HP RX6600共享儲存實現HA，同步兩臺機器vg map的命令
- MySQL雙機互備熱備自動切換KVMySql
- hp-ux 雙機互備安裝oracle遇到的幾點問題UXOracle
- 兩臺Linux完美實現雙機熱備Linux
- HP平臺，9i RAC instance 2被驅逐故障診斷
- 雙機熱備、雙機互備與雙機雙工的區別
- 雙機熱備、雙機互備與 雙機雙工的區別
- hp-ux 雙機互備安裝oracle遇到的幾點問題(二)UXOracle
- AIX平臺HA雙機互備環境下升級兩個oracle 11g資料庫AIOracle資料庫
- Oracle故障診斷Oracle
- HP UNIX雙機常見操作步驟和相關命令
- mysql雙機互備方式配置MySql
- 印表機的常見故障解決方法 HP 5000印表機為例
- Oracle在HP RX6600小機上實現HA後的測試方法Oracle
- 在一臺機器配置兩個listener(Oracle)（轉）Oracle
- 9 Oracle Data Guard 故障診斷Oracle
- ASM磁碟故障診斷（一）ASM
- 風機故障診斷學習資源（更新中）
- 讓雙CPU的linux機器自動關機(轉)Linux
- Linux平臺Oracle開機自啟動設定LinuxOracle
- oracle 10046事件故障診斷一例Oracle事件
- 部落格連結—Oracle故障診斷Oracle
- oracle雙機熱備份Oracle
- AIX平臺HA雙機互備環境定製exp邏輯匯出指令碼AI指令碼
- 設定兩臺ＲＡＣ 主機的信任關係
- 聽音識故障，人工智慧“診斷”機器新形式人工智慧
- SANGFOR NGAF雙機主備專線故障的排查
- 在同一臺機器下安裝兩個Oracle Software 版本Oracle
- HP UNIX開機自動掛載與開機自動執行命令
- oracle雙機熱備份方法Oracle
- Oracle___診斷案例__資料庫的exp故障Oracle資料庫
- 一臺機器上安裝兩個MysqlMySql
- HP-UX ServiceGuard雙機命令UX
- 光纖故障診斷和故障排查
- Oracle Windows平臺自動排程備份指令碼OracleWindows指令碼
- MySQL資料庫診斷：InnoDB關機問題MySql資料庫
- [經驗]HP小機一次無故當機的經歷總結
- 如何診斷伺服器關閉的原因伺服器