hp rx6600兩臺oracle雙機互備伺服器其中一臺經常自動關機的故障診斷

eric0435發表於2014-11-07

hp rx6600兩臺oracle資料庫雙機互備伺服器其中一臺經常自動關機,剛好在做巡檢時遇到了就順便檢查一下原因.檢查經常出故障的一臺小機日誌資訊如下:

rx6600-1:[/]#cat /var/adm/syslog/syslog.log
Nov  6 10:40:35 rx6600-1 syslogd: restart
Nov  6 10:40:35 rx6600-1 vmunix: Found adjacent data tr.  Growing size.  0x32a6000 -> 0x72a6000.
Nov  6 10:40:35 rx6600-1 vmunix: Pinned PDK malloc pool: base: 0xe000000100d5a000  size=117400K
Nov  6 10:40:35 rx6600-1 vmunix: Loaded ACPI revision 2.0 tables.
Nov  6 10:40:35 rx6600-1 vmunix: MMIO on this platform supports Write Coalescing.
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1 vmunix: MFS is defined: base= 0xe000000100d5a000  size= 5084 KB
Nov  6 10:40:35 rx6600-1 vmunix: Unpinned PDK malloc pool: base: 0xe000000108000000  size=393216K
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: cachefs_link(): File system was registered at index 5.
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: GPX emcpgpx_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1  above message repeats 2 times
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: DM emcpgpx_dm_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: VLUMD emcpgpx_vlumd_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: XCRYPT emcpgpx_xcrypt_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: nfs3_link(): File system was registered at index 8.
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: mod_fs_reg: Cannot retrieve configured loading phase from KRS for module: cifs. Setting to load at INIT
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1 vmunix: 0 sba
Nov  6 10:40:35 rx6600-1 vmunix: 0/0 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/1/0 rmp3f01
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/1/1 rmp3f01
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/1/2 asio0
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/0 UsbOhci
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Unknown/0_1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1.3.10.1008.4390.1>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0 UsbMiniBus
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/hp/Unknown/0_1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.305.3.100.1008.4390.1>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0.0 UsbBootKeyboard
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Standard/hp/Unknown/0_1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.307.3.10.1008.4390.1>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1 UsbOhci
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Multibay/0_a1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1.3.10.1008.294.161>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0 UsbMiniBus
Nov  6 10:40:35 rx6600-1 vmunix: Devices/MassStorage-SCSI/USB/BulkOnly/hp/Multibay/0_a1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.310.3.150.1008.294.161>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.0 UsbBulkOnlyMS
Nov  6 10:40:35 rx6600-1 vmunix: Devices/ScsiControllerAdaptor/USB/BulkOnly/hp/Multibay
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1000.3.150.1008.294>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16 UsbScsiAdaptor
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:36 rx6600-1  above message repeats 5 times
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/Avocent/KVMAdaptor/1_0
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1.3.10.1572.833.256>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1 UsbMiniBus
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/Avocent/KVMAdaptor/1_0
Nov  6 10:40:35 rx6600-1 vmunix:  <2.305.3.100.1572.833.256>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.0 UsbBootKeyboard
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Boot/Avocent/KVMAdaptor/1_0
Nov  6 10:40:35 rx6600-1 vmunix:  <2.307.3.100.1572.833.256>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.1 UsbBootMouse
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:36 rx6600-1  above message repeats 2 times
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/2 UsbEhci
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/4/0 gvid_core
Nov  6 10:40:35 rx6600-1 vmunix: 0/1 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/2 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/2/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/2/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/2/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/2/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/3 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/3/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/3/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/3/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/3/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/4 lba
Nov  6 10:40:35 rx6600-1 vmunix: sasd: Claimed HP PCI/PCI-X SAS MPT adapter at hardware path 0/4/1/0 
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0 sasd
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/2/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/2/1 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/5 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/5/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/6 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/6/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/7 lba
Nov  6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/0. Controller firmware version is 01.03.35.70
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/0 mpt
Nov  6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/1. Controller firmware version is 01.03.35.70
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/1 mpt
Nov  6 10:40:35 rx6600-1 vmunix: 120 processor
Nov  6 10:40:35 rx6600-1 vmunix: 121 processor
Nov  6 10:40:35 rx6600-1 vmunix: 122 processor
Nov  6 10:40:35 rx6600-1 vmunix: 123 processor
Nov  6 10:40:35 rx6600-1 vmunix: 124 processor
Nov  6 10:40:35 rx6600-1 vmunix: 125 processor
Nov  6 10:40:35 rx6600-1 vmunix: 126 processor
Nov  6 10:40:35 rx6600-1 vmunix: 127 processor
Nov  6 10:40:35 rx6600-1 vmunix: 250 pdh
Nov  6 10:40:35 rx6600-1 vmunix: 250/0 ipmi
Nov  6 10:40:35 rx6600-1 vmunix: 250/1 asio0
Nov  6 10:40:35 rx6600-1 vmunix: 250/2 acpi_node
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1 fcd_fcp
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.1 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.2 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.3 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.4 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1 fcd_fcp
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.1 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.2 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.3 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.1 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.4 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.2 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.3 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.4 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0 sasd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: Boot device's HP-UX HW path is: 0/4/1/0.0.0.0.0
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1 vmunix:     System Console is on the Built-In Serial Interface
Nov  6 10:40:35 rx6600-1 vmunix: iether0: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/2/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether1: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/3/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether2: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/0
Nov  6 10:40:35 rx6600-1 vmunix: iether4: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/5/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether5: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/6/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether3: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/1
Nov  6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x3 configured as ROOT
Nov  6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as SWAP
Nov  6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as DUMP
Nov  6 10:40:35 rx6600-1 vmunix:     Swap device table:  (start & size given in 512-byte blocks)
Nov  6 10:40:35 rx6600-1 vmunix:         entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216
Nov  6 10:40:35 rx6600-1 vmunix:     Dump device table:  (start & size given in 1-Kbyte blocks)
Nov  6 10:40:35 rx6600-1 vmunix:         entry 0000000000000000 - major is 31, minor is 0x30000; start = 2349940, size = 8388604
Nov  6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 1
Nov  6 10:40:35 rx6600-1 vmunix: Create STCP device files
Nov  6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 2
Nov  6 10:40:35 rx6600-1 vmunix:      $Revision: vmunix:    B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
Nov  6 10:40:35 rx6600-1 vmunix: Memory Information:
Nov  6 10:40:35 rx6600-1 vmunix:     physical page size = 4096 bytes, logical page size = 4096 bytes
Nov  6 10:40:35 rx6600-1 vmunix:     Physical: 25133536 Kbytes, lockable: 18994328 Kbytes, available: 22051156 Kbytes
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:36 rx6600-1 nettl[832]: nettl starting up.
Nov  6 10:40:48 rx6600-1 sshd[986]: Server listening on :: port 22.
Nov  6 10:40:48 rx6600-1 sshd[986]: Server listening on 0.0.0.0 port 22.
Nov  6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found CLTS loopback transport
Nov  6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS loopback transport
Nov  6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS ORD loopback transport
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for udp
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for tcp
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticlts
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticotsord
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticots
Nov  6 10:40:50 rx6600-1 inetd[1100]: Reading configuration
Nov  6 10:40:50 rx6600-1 inetd[1100]: ftp/tcp: Added service, server /usr/lbin/ftpd
Nov  6 10:40:50 rx6600-1 inetd[1100]: telnet/tcp: Added service, server /usr/lbin/telnetd
Nov  6 10:40:50 rx6600-1 inetd[1100]: tftp/udp: Added service, server /usr/lbin/tftpd
Nov  6 10:40:50 rx6600-1 inetd[1100]: login/tcp: Added service, server /usr/lbin/rlogind
Nov  6 10:40:50 rx6600-1 inetd[1100]: shell/tcp: Added service, server /usr/lbin/remshd
Nov  6 10:40:50 rx6600-1 inetd[1100]: exec/tcp: Added service, server /usr/lbin/rexecd
Nov  6 10:40:50 rx6600-1 inetd[1100]: ntalk/udp: Added service, server /usr/lbin/ntalkd
Nov  6 10:40:50 rx6600-1 inetd[1100]: auth/tcp: Added service, server /usr/lbin/identd
Nov  6 10:40:50 rx6600-1 inetd[1100]: printer/tcp: Added service, server /usr/sbin/rlpdaemon
Nov  6 10:40:51 rx6600-1 inetd[1100]: daytime/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: daytime/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: time/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: echo/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: echo/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: discard/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: discard/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: chargen/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: chargen/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: kshell/tcp: Added service, server /usr/lbin/remshd
Nov  6 10:40:51 rx6600-1 inetd[1100]: klogin/tcp: Added service, server /usr/lbin/rlogind
Nov  6 10:40:51 rx6600-1 inetd[1100]: dtspc/tcp: Added service, server /usr/dt/bin/dtspcd
Nov  6 10:40:51 rx6600-1 inetd[1100]: recserv/tcp: Added service, server /usr/lbin/recserv
Nov  6 10:40:51 rx6600-1 inetd[1100]: swat/tcp: Added service, server /opt/samba/bin/swat
Nov  6 10:40:51 rx6600-1 inetd[1100]: registrar/tcp: Added service, server /etc/opt/resmon/lbin/registrar
Nov  6 10:40:51 rx6600-1 inetd[1100]: hacl-probe/tcp: Added service, server /opt/cmom/lbin/cmomd
Nov  6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/udp: Added service, server /usr/lbin/cmclconfd
Nov  6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/tcp: Added service, server /usr/lbin/cmclconfd
Nov  6 10:40:51 rx6600-1 inetd[1100]: instl_boots/udp: Added service, server /opt/ignite/lbin/instl_bootd
Nov  6 10:40:51 rx6600-1 inetd[1100]: omni/tcp: Added service, server /opt/omni/lbin/inet
Nov  6 10:40:51 rx6600-1 inetd[1100]: rpc.cmsd/udp: Added service, server /usr/dt/bin/rpc.cmsd
Nov  6 10:40:51 rx6600-1 inetd[1100]: rpc.ttdbserver/tcp: Added service, server /usr/dt/bin/rpc.ttdbserver
Nov  6 10:40:51 rx6600-1 inetd[1100]: Configuration complete
Nov  6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: restore  (user ID real=0 effective=0)
Nov  6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: config  (user ID real=0 effective=0)
Nov  6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: save  (user ID real=0 effective=0)
Nov  6 10:40:54 rx6600-1 su: + tty?? root-sfmdb
Nov  6 10:41:06 rx6600-1 cimserver[1706]: starting
Nov  6 10:41:29 rx6600-1 cimserver[1707]: PGS10026:  THE CIM SERVER IS LISTENING ON HTTPS PORT 5,989.
Nov  6 10:41:29 rx6600-1 cimserver[1707]: PGS10028: THE CIM SERVER IS LISTENING ON THE LOCAL CONNECTION SOCKET.
Nov  6 10:41:29 rx6600-1 cimserver[1707]: PGS10030:  STARTED HP-UX WBEM Services VERSION A.02.07.
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/100dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/75dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_korean/75dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Cannot initialize font path element: "/usr/lib/X11/fonts/hp_chinese_t/75dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ttfjpn.st"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ifojpn.st"
Nov  6 10:41:34 rx6600-1 pwgrd: Started at Thu Nov  6 10:41:34 2014, pid = 1798
Nov  6 10:41:34 rx6600-1 diagmond[1833]: started
Nov  6 10:41:34 rx6600-1 /usr/sbin/envd[1837]: VXPBFt6/, 2"6A3vEdVCND< ~ Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2180]: Setting STREAMS-HEAD high water value to 131072. Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one mpctl succeeded: ncpus = 8. Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 2 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 3 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd do_one bind 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd do_one bind 1 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd do_one bind 2 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd do_one bind 3 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd do_one bind 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd do_one bind 5 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one bind 7 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd 5 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2197]: nfsd 5 0  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd 3 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd 2 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2200]: nfsd 2 0  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd 1 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2201]: nfsd 1 0  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2199]: nfsd 3 0  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd 4 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2202]: nfsd 4 0  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd 7 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2219]: nfsd 7 0  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd do_one bind 6 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: Return from t_optmgmt(XTI_DISTRIBUTE) 0 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd 0 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2220]: nfsd 0 0  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd 6 1  sock 4 Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2221]: nfsd 6 0  sock 4 Nov  6 10:41:53 rx6600-1 krsd[2300]: Delay time is 300 seconds Nov  6 10:41:53 rx6600-1 sfd[2301]: daemon already running. Nov  6 10:41:54 rx6600-1 sfd[2314]: starting the daemon. Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [0] (2,4,60) Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [1] (20,40,300) Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SetLogMask:: EventLogMask set to 0x66  Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Using hostname localhost community public debug 0 Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Daemon created successfully.  Starting it now Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled. Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled. Nov  6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f *****
Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!# Nov  6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------   Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw"     (Threshold:  >= " 3")   
 Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 
Nov  6 10:49:14 rx6600-1 EMS [2928]: ------ EMS Event Notification ------   Value: "CRITICAL (5)" for Resource: "/system/events/ipmi_fpl/ipmi_fpl"     (Threshold:  >= " 3")    
Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 191889410 -r /system/events/ipmi_fpl/ipmi_fpl -n 191889409 -a 
Nov  6 18:48:12 rx6600-1 EMS [2970]: ------ EMS Event Notification ------   Value: "CRITICAL (5)" for Resource: "/system/events/ia64_corehw/core_hw"     (Threshold:  >= " 3")    
Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641922 -a 
Nov  6 19:00:00 rx6600-1 su: + tty?? root-oracle
Nov  7 08:00:00 rx6600-1 su: + tty?? root-root

從如下資訊看到伺服器已經出問題了,且資訊已經指出可以執行
/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令來檢視詳細資訊

Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled.
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled.
Nov  6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array
Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f *****
Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!# Nov  6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------   Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw"     (Threshold:  >= " 3")   
Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 

執行/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令來檢視詳細資訊

rx6600-1:[/]#/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 

ARCHIVED MONITOR DATA:

Event Time..........: Thu Nov  6 10:45:42 2014
Severity............: MAJORWARNING
Monitor.............: ia64_corehw
Event #.............: 101011              
System..............: rx6600-1

Summary:
     System temperature is out of normal range. 


Description of Error:

     The system temperature is not within normal operating range. It is higher
     than required operating range.

這個錯誤描述是說系統的溫度超出了正常範圍,下面資訊說明了可能的原因

Probable Cause / Recommended Action:

     Something may be blocking the cooling intakes of the fans. Check for
     obstruction.
     One or more fans may be operating at lower speed than normal. Check the
     fan performance.

     Check for problems with the room air conditioning.

     If the problem is not fixed, the operating temperature may become
     non-recoverable, in which case there are chances that the hardware may be
     damaged.  At that temperature level, on Integrity servers, the firmware
     will shutdown the system automatically. However on HP 9000 servers, the
     action specified in the envd config file will be taken - which may be to
     shutdown the system automatically.

     For information on the sensor that generated this event, refer to FRU ID
     in Event Details section.

上面的資訊是說,可能需要清理一下風機,或者風機效能出現問題,或者檢查空調情況,如果不是這些原因造成那麼可能是硬體出現問題了。下面的論斷事件的資料:

Additional Event Data: 
     System IP Address...: 10.138.129.5
     Event Id............: 0x545ae0d600000000
     Monitor Version.....: B.01.00
     Event Class.........: System
     Client Configuration File...........:
     /var/stm/config/tools/monitor/default_ia64_corehw.clcfg 
     Client Configuration File Version...: A.01.00 
          Qualification criteria met.
               Number of events..: 1 
     Associated OS error log entry id(s): 
          None
     Additional System Data:
          System Model Number.............: ia64 hp server rx6600 
          EMS Version.....................: A.04.20 
          STM Version.....................: C.58.00 
          System Serial Number............: SGH48045VY 
     Latest information on this event:
          

v-v-v-v-v-v-v-v-v-v-v-v-v    D  E  T  A  I  L  S    v-v-v-v-v-v-v-v-v-v-v-v-v


Event Details :

     Event Date .............: Thu Nov  6 10:44:08 2014
     Sensor Number ..........: 0xdb
     Sensor Type ............: Temperature
     Sensor Class ...........: Threshold based
     Sensor Reading/Offset...: 0x07 (Offset)
     Event  Type.............: Assertion
     Entity ID ..............: 3
     Generic Message.........: 
       Temperature :  Upper non-critical - going high 
     Entity FRU Id Info......: 
       processor (Sensor ID: Processor 2)

從上面的Event Details資訊可以看到,感測器型別是溫度方面的問題,感測器類別是基於閾值,事件型別是斷言,是說2號cpu的溫度已經超過了閾值.經過檢查不是機房空調,通風口堵塞問題,需要聯絡小機廠商來進行一步檢查是什麼原因造成cpu溫度超過閾值,平時cpu使用率只有10%。

來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/26015009/viewspace-1323699/,如需轉載,請註明出處,否則將追究法律責任。

相關文章