ORACLE RAC 11.2.0.4 FOR RHEL6叢集無法啟動的處理
ORACLE RAC 11.2.0.4 FOR RHEL6叢集無法啟動與RHEL的NetworkManager服務有關係，本文介紹一例相關
故障的處理。
1、問題描述：
oracle rac 11.2.0.4 for rhel6主機重啟，重啟之後發現oracle rac叢集無法啟動。
rac2主機啟動後，發現rac叢集啟動不了，檢查叢集程式狀態，發現cssd一直處於starting狀態
[root@rac2 bin]# pwd
/u01/oracle/app/grid/home/bin
[root@rac2 bin]# ./crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE Instance Shutdown
ora.cluster_interconnect.haip
1 ONLINE OFFLINE
ora.crf
1 ONLINE ONLINE rac2
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE OFFLINE STARTING
ora.cssdmonitor
1 ONLINE ONLINE rac2
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE OFFLINE
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE ONLINE rac2
ora.gpnpd
1 ONLINE ONLINE rac2
ora.mdnsd
1 ONLINE ONLINE rac2
[root@rac2 bin]#
2、問題分析
--由於叢集啟動到cssd無法繼續正常啟動，檢視cssd的日誌ocssd.log
發現有報錯：has a disk HB, but no network HB
[root@rac2 bin]# tail -f /u01/oracle/app/grid/home/log/rac2/cssd/ocssd.log
2018-12-06 10:16:35.506: [ CSSD][2489263872]clssgmJoinGrock: global grock CRF- new client 0x7f268c118700 with con 0x7f2600004253, requested num -1, flags 0x4000e00
2018-12-06 10:16:35.506: [ CSSD][2489263872]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2018-12-06 10:16:35.506: [ CSSD][2489263872]clssgmDiscEndpcl: gipcDestroy 0x4253
2018-12-06 10:16:35.506: [ CSSD][2489263872]clssgmDeadProc: proc 0x7f268c116f40
2018-12-06 10:16:35.506: [ CSSD][2489263872]clssgmDestroyProc: cleaning up proc(0x7f268c116f40) con(0x4224) skgpid ospid 2599 with 0 clients, refcount 0
2018-12-06 10:16:35.506: [ CSSD][2489263872]clssgmDiscEndpcl: gipcDestroy 0x4224
2018-12-06 10:16:35.754: [ CSSD][2275362560]clssnmvDHBValidateNcopy: node 1, rac1, has a disk HB, but no network HB , DHB has rcfg 439510562, wrtcnt, 336931, LATS 4294670080, lastSeqNo 336928, uniqueness 1544015571, timestamp 1544025004/8530004
2018-12-06 10:16:35.992: [ CSSD][2261169920]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2018-12-06 10:16:36.288: [ CSSD][2265900800]clssnmvDHBValidateNcopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 439510562, wrtcnt, 336933, LATS 4294670610, lastSeqNo 336930, uniqueness 1544015571, timestamp 1544025004/8530494
2018-12-06 10:16:36.775: [ CSSD][2275362560]clssnmvDHBValidateNcopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 439510562, wrtcnt, 336934, LATS 4294671100, lastSeqNo 336931, uniqueness 1544015571, timestamp 1544025005/8531004
2018-12-06 10:16:36.993: [ CSSD][2261169920]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2018-12-06 10:16:36.993: [GIPCHALO][2280093440] gipchaLowerProcessNode: no valid interfaces found to node for 4294671320 ms, node 0x7f267c0bd080 { host 'rac1', haName 'CSS_raccls', srcLuid 43347245-1561779d, dstLuid 00000000-00000000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [572 : 572], createTime 4294099030, sentRegister 1, localMonitor 1, flags 0x4 }
2018-12-06 10:16:37.007: [GIPCHDEM][2485143296] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0x115c700 [0000000000000010] { gipchaContext : host 'rac2', name 'CSS_raccls', luid '43347245-00000000', numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2018-12-06 10:16:37.289: [ CSSD][2265900800]clssnmvDHBValidateNcopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 439510562, wrtcnt, 336936, LATS 4294671610, lastSeqNo 336933, uniqueness 1544015571, timestamp 1544025005/8531494
2018-12-06 10:16:37.784: [ CSSD][2275362560]clssnmvDHBValidateNcopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 439510562, wrtcnt, 336937, LATS 4294672110, lastSeqNo 336934, uniqueness 1544015571, timestamp 1544025006/8532004
2018-12-06 10:16:37.907: [ CSSD][2489263872]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0x7f268c0593b0)
2018-12-06 10:16:37.907: [ CSSD][2489263872]clssgmShutDown: Received abortive shutdown request from client.
2018-12-06 10:16:37.907: [ CSSD][2489263872]###################################
2018-12-06 10:16:37.907: [ CSSD][2489263872]clssscExit: CSSD aborting from thread GMClientListener
2018-12-06 10:16:37.907: [ CSSD][2489263872]###################################
2018-12-06 10:16:37.907: [ CSSD][2489263872](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
[root@rac2 bin]#
--根據cssd的日誌ocssd.log檢查節點間通訊發現rac1到rac2的public網路卡地址無法互相ping通
[grid@rac1 ~]$ ping rac2
PING rac2 (20.20.20.37) 56(84) bytes of data.
From rac1 (20.20.20.34) icmp_seq=1 Destination Host Unreachable
From rac1 (20.20.20.34) icmp_seq=2 Destination Host Unreachable
From rac1 (20.20.20.34) icmp_seq=3 Destination Host Unreachable
^C
--- rac2 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4998ms
pipe 3
[grid@rac1 ~]$
[root@rac2 bin]# ping rac1
PING rac1 (20.20.20.34) 56(84) bytes of data.
From rac2 (20.20.20.37) icmp_seq=2 Destination Host Unreachable
From rac2 (20.20.20.37) icmp_seq=3 Destination Host Unreachable
From rac2 (20.20.20.37) icmp_seq=4 Destination Host Unreachable
From rac2 (20.20.20.37) icmp_seq=5 Destination Host Unreachable
From rac2 (20.20.20.37) icmp_seq=6 Destination Host Unreachable
From rac2 (20.20.20.37) icmp_seq=7 Destination Host Unreachable
^C
--- rac1 ping statistics ---
20 packets transmitted, 0 received, +15 errors, 100% packet loss, time 19395ms
pipe 4
[root@rac2 bin]#
--但是rac1和rac2的priv私有網路卡是通的
[root@rac2 bin]# ping rac1priv
PING rac1priv (172.25.25.1) 56(84) bytes of data.
64 bytes from rac1priv (172.25.25.1): icmp_seq=1 ttl=64 time=0.636 ms
^C
--- rac1priv ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 727ms
rtt min/avg/max/mdev = 0.636/0.636/0.636/0.000 ms
[root@rac2 bin]# ping rac2priv
PING rac2priv (172.25.25.2) 56(84) bytes of data.
64 bytes from rac2priv (172.25.25.2): icmp_seq=1 ttl=64 time=0.018 ms
64 bytes from rac2priv (172.25.25.2): icmp_seq=2 ttl=64 time=0.030 ms
^C
--- rac2priv ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1300ms
rtt min/avg/max/mdev = 0.018/0.024/0.030/0.006 ms
[root@rac2 bin]#
[grid@rac1 ~]$ ping rac1priv
PING rac1priv (172.25.25.1) 56(84) bytes of data.
64 bytes from rac1priv (172.25.25.1): icmp_seq=1 ttl=64 time=0.016 ms
^C
--- rac1priv ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 897ms
rtt min/avg/max/mdev = 0.016/0.016/0.016/0.000 ms
[grid@rac1 ~]$ ping rac2priv
PING rac2priv (172.25.25.2) 56(84) bytes of data.
64 bytes from rac2priv (172.25.25.2): icmp_seq=1 ttl=64 time=0.162 ms
^C
--- rac2priv ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 819ms
rtt min/avg/max/mdev = 0.162/0.162/0.162/0.000 ms
[grid@rac1 ~]$
--檢視rac2的網路介面和網路卡IP地址資訊，發現四張網路卡的IP地址均為叢集私有網路卡地址bond1的IP，
雖然bond0有IP地址，但是ping不通
[root@rac2 ~]# ifconfig -a
bond0 Link encap:Ethernet HWaddr 08:00:27:11:69:E3
inet addr:20.20.20.37 Bcast:20.20.20.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe11:69e3/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:355 errors:0 dropped:0 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:220099 (214.9 KiB) TX bytes:4611 (4.5 KiB)
bond1 Link encap:Ethernet HWaddr 08:00:27:72:BE:6E
inet addr:172.25.25.2 Bcast:172.25.25.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe72:be6e/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:296 errors:0 dropped:0 overruns:0 frame:0
TX packets:188 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:185467 (181.1 KiB) TX bytes:27072 (26.4 KiB)
eth0 Link encap:Ethernet HWaddr 08:00:27:11:69:E3
inet addr:172.25.25.2 Bcast:172.25.25.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:110 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:88432 (86.3 KiB) TX bytes:0 (0.0 b)
eth1 Link encap:Ethernet HWaddr 08:00:27:9D:E6:00
inet addr:172.25.25.2 Bcast:172.25.25.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:245 errors:0 dropped:0 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:131667 (128.5 KiB) TX bytes:4611 (4.5 KiB)
eth2 Link encap:Ethernet HWaddr 08:00:27:72:BE:6E
inet addr:172.25.25.2 Bcast:172.25.25.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:230 errors:0 dropped:0 overruns:0 frame:0
TX packets:188 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:127461 (124.4 KiB) TX bytes:27072 (26.4 KiB)
eth3 Link encap:Ethernet HWaddr 08:00:27:CA:0D:9C
inet addr:172.25.25.2 Bcast:172.25.25.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:66 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:58006 (56.6 KiB) TX bytes:0 (0.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:20 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1141 (1.1 KiB) TX bytes:1141 (1.1 KiB)
[root@rac2 ~]#
--如果使用root執行service network restart之後，bond網路卡均失去IP地址，並且外部機器均無法與rac2網路互通
--這種bond異常的狀況，與rhel的NetworkManager服務管理網路服務有關
3、問題處理
在rhel主機使用網路卡bond的狀況下，rhel的NetworkManager管理網路卡服務，會導致bond網路卡繫結ip地址異常，需要
關閉NetworkManager並設定其不隨機啟動。
--關閉NetworkManager並設定其不隨機啟動
--重啟網路服務，檢查bond網路卡正常
[root@rac2 ~]# ifconfig -a
bond0 Link encap:Ethernet HWaddr 08:00:27:11:69:E3
inet addr:20.20.20.37 Bcast:20.20.20.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe11:69e3/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:5988 errors:0 dropped:0 overruns:0 frame:0
TX packets:173 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5637836 (5.3 MiB) TX bytes:21701 (21.1 KiB)
bond1 Link encap:Ethernet HWaddr 08:00:27:72:BE:6E
inet addr:172.25.25.2 Bcast:172.25.25.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe72:be6e/64 Scope:Link
UP BROADCAST MASTER MULTICAST MTU:1500 Metric:1
RX packets:7780 errors:0 dropped:0 overruns:0 frame:0
TX packets:2532 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5549280 (5.2 MiB) TX bytes:368936 (360.2 KiB)
eth0 Link encap:Ethernet HWaddr 08:00:27:11:69:E3
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:2171 errors:0 dropped:0 overruns:0 frame:0
TX packets:73 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2021551 (1.9 MiB) TX bytes:7343 (7.1 KiB)
eth1 Link encap:Ethernet HWaddr 08:00:27:9D:E6:00
BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3817 errors:0 dropped:0 overruns:0 frame:0
TX packets:100 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3616285 (3.4 MiB) TX bytes:14358 (14.0 KiB)
eth2 Link encap:Ethernet HWaddr 08:00:27:72:BE:6E
BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:5916 errors:0 dropped:0 overruns:0 frame:0
TX packets:2532 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3764442 (3.5 MiB) TX bytes:368936 (360.2 KiB)
eth3 Link encap:Ethernet HWaddr 08:00:27:CA:0D:9C
BROADCAST SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1864 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1784838 (1.7 MiB) TX bytes:0 (0.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:811 errors:0 dropped:0 overruns:0 frame:0
TX packets:811 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4242930 (4.0 MiB) TX bytes:4242930 (4.0 MiB)
[root@rac2 ~]#
--檢查rac2節點叢集，發現單節點能夠啟動
[grid@rac2 ~]$ crsctl status res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE rac2 Started
ora.cluster_interconnect.haip
1 ONLINE ONLINE rac2
ora.crf
1 ONLINE ONLINE rac2
ora.crsd
1 ONLINE ONLINE rac2
ora.cssd
1 ONLINE ONLINE rac2
ora.cssdmonitor
1 ONLINE ONLINE rac2
ora.ctssd
1 ONLINE ONLINE rac2 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE rac2
ora.evmd
1 ONLINE ONLINE rac2
ora.gipcd
1 ONLINE ONLINE rac2
ora.gpnpd
1 ONLINE ONLINE rac2
ora.mdnsd
1 ONLINE ONLINE rac2
[grid@rac2 ~]$
小插曲，相比節點rac2，rac1的NetworkManager也在執行，節點rac2的eth網路卡卻均為public網路卡地址，
雖然rac1的eth網路卡均為public網路卡地址，但是rac1節點的叢集例項能正常啟動。
[root@rac1 ~]# service NetworkManager status
NetworkManager (pid 1848) 正在執行...
[root@rac1 ~]#
--節點rac2的eth網路卡卻均為public網路卡地址
[grid@rac1 ~]$ ifconfig -a
bond0 Link encap:Ethernet HWaddr 08:00:27:F8:54:9E
inet addr:20.20.20.34 Bcast:20.20.20.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fef8:549e/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:42512 errors:0 dropped:0 overruns:0 frame:0
TX packets:1542 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:39197489 (37.3 MiB) TX bytes:170518 (166.5 KiB)
bond0:1 Link encap:Ethernet HWaddr 08:00:27:F8:54:9E
inet addr:20.20.20.26 Bcast:20.20.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond0:2 Link encap:Ethernet HWaddr 08:00:27:F8:54:9E
inet addr:20.20.20.28 Bcast:20.20.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond0:3 Link encap:Ethernet HWaddr 08:00:27:F8:54:9E
inet addr:20.20.20.25 Bcast:20.20.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
bond1 Link encap:Ethernet HWaddr 08:00:27:6F:A8:F7
inet addr:172.25.25.1 Bcast:172.25.25.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe6f:a8f7/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:27140 errors:0 dropped:0 overruns:0 frame:0
TX packets:43504 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14877318 (14.1 MiB) TX bytes:39723988 (37.8 MiB)
bond1:1 Link encap:Ethernet HWaddr 08:00:27:6F:A8:F7
inet addr:169.254.220.217 Bcast:169.254.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
eth0 Link encap:Ethernet HWaddr 08:00:27:F8:54:9E
inet addr:20.20.20.34 Bcast:20.20.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:22881 errors:0 dropped:0 overruns:0 frame:0
TX packets:1391 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:20771703 (19.8 MiB) TX bytes:158110 (154.4 KiB)
eth1 Link encap:Ethernet HWaddr 08:00:27:76:6A:BA
inet addr:20.20.20.34 Bcast:20.20.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:19631 errors:0 dropped:0 overruns:0 frame:0
TX packets:151 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:18425786 (17.5 MiB) TX bytes:12408 (12.1 KiB)
eth2 Link encap:Ethernet HWaddr 08:00:27:6F:A8:F7
inet addr:20.20.20.34 Bcast:20.20.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:13048 errors:0 dropped:0 overruns:0 frame:0
TX packets:43504 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1914331 (1.8 MiB) TX bytes:39723988 (37.8 MiB)
eth3 Link encap:Ethernet HWaddr 08:00:27:CA:32:4E
inet addr:20.20.20.34 Bcast:20.20.20.255 Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:14092 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:12962987 (12.3 MiB) TX bytes:0 (0.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:79993 errors:0 dropped:0 overruns:0 frame:0
TX packets:79993 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:55828936 (53.2 MiB) TX bytes:55828936 (53.2 MiB)
[grid@rac1 ~]$
--雖然rac1的eth網路卡均為public網路卡地址，但是rac1節點的叢集例項能正常啟動
[grid@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.ARCHDG.dg ora....up.type ONLINE ONLINE rac1
ora.CRSDG.dg ora....up.type ONLINE ONLINE rac1
ora.DATADG.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac1
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.orcl.db ora....se.type ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora.rac2.vip ora....t1.type ONLINE ONLINE rac1
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type ONLINE ONLINE rac1
[grid@rac1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE rac1 Started
ora.cluster_interconnect.haip
1 ONLINE ONLINE rac1
ora.crf
1 ONLINE ONLINE rac1
ora.crsd
1 ONLINE ONLINE rac1
ora.cssd
1 ONLINE ONLINE rac1
ora.cssdmonitor
1 ONLINE ONLINE rac1
ora.ctssd
1 ONLINE ONLINE rac1 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE rac1
ora.evmd
1 ONLINE ONLINE rac1
ora.gipcd
1 ONLINE ONLINE rac1
ora.gpnpd
1 ONLINE ONLINE rac1
ora.mdnsd
1 ONLINE ONLINE rac1
[grid@rac1 ~]$
繼續處理叢集啟動異常的問題。
--關閉節點rac1的NetworkManager服務並並設定其不開機啟動，然後重啟主機
[root@rac1 ~]# service NetworkManager status
NetworkManager (pid 1848) 正在執行...
[root@rac1 ~]# service NetworkManager stop
停止 NetworkManager 守護程式： [確定]
[root@rac1 ~]#
[root@rac1 ~]# chkconfig NetworkManager off
[root@rac1 ~]#
[root@rac1 ~]# reboot
Broadcast message from root@rac1
(/dev/pts/0) at 0:16 ...
The system is going down for reboot NOW!
[root@rac1 ~]#
4、問題處理結果，調整完2個節點的NetworkManager後，同時重啟2臺伺服器後，oracle rac叢集恢復正常
--調整完2個節點的NetworkManager後，檢查叢集狀態，oracle rac叢集恢復正常
[grid@rac2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.CRSDG.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.DATADG.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2
ora.cvu
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.orcl.db
1 ONLINE ONLINE rac2 Open
2 ONLINE ONLINE rac1 Open
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac2
[grid@rac2 ~]$
[grid@rac1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE rac1 Started
ora.cluster_interconnect.haip
1 ONLINE ONLINE rac1
ora.crf
1 ONLINE ONLINE rac1
ora.crsd
1 ONLINE ONLINE rac1
ora.cssd
1 ONLINE ONLINE rac1
ora.cssdmonitor
1 ONLINE ONLINE rac1
ora.ctssd
1 ONLINE ONLINE rac1 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE rac1
ora.evmd
1 ONLINE ONLINE rac1
ora.gipcd
1 ONLINE ONLINE rac1
ora.gpnpd
1 ONLINE ONLINE rac1
ora.mdnsd
1 ONLINE ONLINE rac1
[grid@rac1 ~]$
[grid@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.ARCHDG.dg ora....up.type ONLINE ONLINE rac1
ora.CRSDG.dg ora....up.type ONLINE ONLINE rac1
ora.DATADG.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac2
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac1
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.orcl.db ora....se.type ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type ONLINE ONLINE rac2
[grid@rac1 ~]$
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29357786/viewspace-2284397/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle Linux 6.7中 Oracle 11.2.0.4 RAC叢集CRS異常處理OracleLinux
- oracle 11.2.0.4 rac叢集等待事件enq: TM - contentionOracle事件ENQ
- ORACLE RAC 11.2.0.4 for RHEL6.8無法啟動之ORA000205&ORA17503&ORA01174Oracle
- 11.2.0.4 RAC CSSD服務無法啟動故障 unable to set priority to 4CSS
- RAC節點hang住, oracle bug導致了cpu過高，無法啟動叢集隔離Oracle
- 紅色警報 ORACLE RAC 11.2.0.4 FOR SOLARIS 10 ASM 和DB因叢集心跳丟失重啟OracleASM
- ORACLE RAC 11.2.0.4 ASM加盤導致叢集重啟之ASM sga設定過小OracleASM
- Oracle日常問題處理-資料庫無法啟動Oracle資料庫
- ORACLE rac 11.2.0.4 for rhel7.8 upgrade to 19.11.1 報錯ORA-29516處理Oracle
- ORACLE 11.2.0.4 for solaris更換硬體後主機時間改變導致一節點叢集服務無法啟動Oracle
- Oracle 11.2.0.4 awr過期快照無法自動清理Oracle
- Oracle叢集技術 | 叢集的自啟動系列（一）Oracle
- Oracle 11.2.0.4 Dataguard兩則故障處理Oracle
- Oracle 12c RAC CSSD程式無法啟動real time模式OracleCSS模式
- Oracle 11gr2修改RAC叢集的scan ip,並處理ORA-12514問題Oracle
- 搭建Kubernetes叢集時DNS無法解析問題的處理過程DNS
- 私有IP丟失造成Oracle 12C RAC叢集節點不能啟動Oracle
- Oracle 11gR2 RAC 叢集服務啟動與關閉總結Oracle
- oracle rac 11.2.0.3 升級到11.2.0.4Oracle
- oracle RAC 診斷叢集狀態命令Oracle
- Mac環境下MySQL無法啟動的處理方法MacMySql
- Oracle 12c叢集啟動故障Oracle
- Oracle：Oracle RAC 11.2.0.4 升級為 19cOracle
- Oracle RAC自啟動Oracle
- Solaris下Oracle RAC 11.2.0.4 安裝方法Oracle
- 華納雲：如何解決hadoop叢集無法啟動的問題？Hadoop
- Oracle叢集（RAC）時間同步（ntp和CTSS）Oracle
- 記一次oracle 19c RAC叢集重啟單節點DB啟動異常（二）Oracle
- Oracle 10g RAC故障處理Oracle 10g
- ORACLE無法OPEN，處理三板斧Oracle
- Oracle RAC日常運維-NetworkManager導致叢集故障Oracle運維
- 【ASK_ORACLE】Relink RAC叢集詳細步驟Oracle
- Oracle 叢集的自啟動，OLR與套接字檔案Oracle
- 處理mysql無法啟動且啟動時服務沒報錯的異常情況MySql
- RAC節點啟動失敗--ASM無法連線ASM
- 沃趣微講堂 | Oracle叢集技術（三）：被誤傳的叢集自啟動Oracle
- Oracle RAC的自定義service自啟動Oracle
- 沃趣微講堂 | Oracle叢集技術（二）：GI與Oracle RACOracle