Oracle叢集（RAC）時間同步（ntp和CTSS）
Oracle叢集（RAC）時間同步（ntp和CTSS）
http://blog.itpub.net/26736162/viewspace-2157130/
crsctl stat res -t -init
ps -ef|grep ctss
crsctl check ctss
cluvfy comp clocksync -n all -verbose
crsctl start res ora.ctssd -init
crsctl stop res ora.ctssd -init
Network Time Protocol Setting
l You have two options for time synchronization: an operating system configured network time protocol (NTP), or Oracle Cluster Time Synchronization Service.
l Oracle Cluster Time Synchronization Service is designed for organizations whose cluster servers are unable to access NTP services.
l If you use NTP, then the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons, then ctssd starts up in active mode and synchronizes time among cluster members without contacting an external time server..
可以採用作業系統的NTP服務，也可以使用Oracle自帶的服務ctss，如果ntp沒有啟用，那麼Oracle會自動啟用自己的ctssd程式。
從oracle 11gR2 RAC開始使用Cluster Time Synchronization Service(CTSS)同步各節點的時間,當安裝程式發現NTP協議處於非活動狀態時，安裝叢集時間同步服務將以活動模式（active）自動進行安裝並同步所有節點的時間。如果發現配置了 NTP，則以觀察者模式（observer mode）啟動叢集時間同步服務，Oracle Clusterware不會在叢集中進行活動的時間同步。
在RAC中，叢集的時間應該是保持同步的，否則可能導致很多問題，例如：依賴於時間的應用會造成資料的錯誤，各種日誌列印的順序紊亂，這將會影響問題的診斷，嚴重的可能會導致叢集當機或者重新啟動叢集時節點無法加入叢集。
在Oracle 11gR2前，叢集的時間是由NTP同步的，而在11gR2後，Oracle引入了CTSS元件，如果系統沒有配置NTP，則由CTSS來同步叢集時間。
NTP和CTSS是可以共存的，且NTP的優先順序要高於CTSS，也就是說，如果系統中同時有NTP和CTSS，那麼叢集的時間是由NTP同步的，CTSS會處於觀望（Observer）模式，只有當叢集關閉所有的NTP服務，CTSS才會處於啟用（Active）模式。在一個叢集中，只要有一個節點的ntp處於活動狀態，那麼叢集的所有節點的CTSS都會處於觀望（Observer）模式。
需要注意的是，要讓CTSS處於啟用（Active）模式，則不僅要關閉ntp服務（/sbin/service ntpd stop），還要刪除/etc/ntp.conf檔案（mv /etc/ntp.conf /etc/ntp.conf.bak），否則不能啟用CTSS。
1.1.1 CTSS同步模式
關閉NTP：
/sbin/service ntpd stop
mv /etc/ntp.conf /etc/ntp.conf.bak
service ntpd status
chkconfig ntpd off
[root@raclhr-11gR2-N2 ~]# ps -ef|grep ctss
root 19678 1 0 19:22 ? 00:00:02 /u01/app/11.2.0/grid/bin/octssd.bin reboot
root 20970 20623 0 19:35 pts/4 00:00:00 grep ctss
[root@raclhr-11gR2-N2 ~]#
[root@raclhr-11gR2-N2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE raclhr-11gr2-n2 Started
ora.cluster_interconnect.haip
1 ONLINE ONLINE raclhr-11gr2-n2
ora.crf
1 ONLINE ONLINE raclhr-11gr2-n2
ora.crsd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.cssd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.cssdmonitor
1 ONLINE ONLINE raclhr-11gr2-n2
ora.ctssd
1 ONLINE ONLINE raclhr-11gr2-n2 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.gipcd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.gpnpd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.mdnsd
1 ONLINE ONLINE raclhr-11gr2-n2
[root@raclhr-11gR2-N2 ~]#
節點1的ctss狀態：
[root@raclhr-11gR2-N1 ~]# crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0
[root@raclhr-11gR2-N1 ~]#
節點1的octssd的日誌：
/u01/app/11.2.0/grid/log/raclhr-11gr2-n1/ctssd/octssd.log
2018-06-30 19:25:56.369: [ CTSS][899475200]sclsctss_gvss2: NTP default pid file not found
2018-06-30 19:25:56.369: [ CTSS][899475200]sclsctss_gvss8: Return [0] and NTP status [1].
2018-06-30 19:25:56.369: [ CTSS][899475200]ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2018-06-30 19:25:57.002: [ CTSS][916338432]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xcc], offset[0 ms]}, length=[8].
2018-06-30 19:26:01.263: [ CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].
2018-06-30 19:26:01.264: [ CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg
2018-06-30 19:26:01.264: [ CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc4] nodenum [2] hostname [raclhr-11gr2-n2] )
2018-06-30 19:26:09.267: [ CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].
節點1的octssd.log中記錄沒有發現ntp服務，ctss服務為啟用模式。
節點2的ctss狀態：
[root@raclhr-11gR2-N2 ~]# crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0
[root@raclhr-11gR2-N2 ~]#
節點2的octssd的日誌：
/u01/app/11.2.0/grid/log/raclhr-11gr2-n2/ctssd/octssd.log
2018-06-30 19:28:49.539: [ CTSS][839321344]sclsctss_gvss2: NTP default pid file not found
2018-06-30 19:28:49.539: [ CTSS][839321344]sclsctss_gvss8: Return [0] and NTP status [1].
2018-06-30 19:28:49.539: [ CTSS][839321344]ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].
2018-06-30 19:29:05.544: [ CTSS][839321344]ctsselect_msm: CTSS mode is [0xc4]
2018-06-30 19:29:05.544: [ CTSS][839321344]ctssslave_swm1_2: Ready to initiate new time sync process.
2018-06-30 19:29:05.545: [ CTSS][839321344]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].
2018-06-30 19:29:05.546: [ CTSS][845625088]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].
2018-06-30 19:29:05.546: [ CTSS][845625088]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].
2018-06-30 19:29:05.547: [ CTSS][839321344]ctssslave_swm2_3: Received time sync message from master.
2018-06-30 19:29:05.547: [ CTSS][839321344]ctssslave_swm: The system time difference is too small [243] usec. Not adjusting time.
2018-06-30 19:29:05.547: [ CTSS][839321344]ctssslave_swm17: LT [1530358145sec 546888usec], MT [1530358145sec 140655884523349usec], Delta [2314usec]
2018-06-30 19:29:05.547: [ CTSS][839321344]ctssslave_swm19: The offset is [243 usec] and sync interval set to [1]
2018-06-30 19:29:05.547: [ CTSS][839321344]ctssslave_swm: Received from master (mode [0xcc] nodenum [1] hostname [raclhr-11gr2-n1] )
2018-06-30 19:29:05.547: [ CTSS][839321344]ctsselect_msm: Sync interval returned in [1]
2018-06-30 19:29:05.547: [ CTSS][845625088]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler
2018-06-30 19:29:07.910: [ CTSS][860387072]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xc4], offset[0 ms]}, length=[8].
節點2的octssd.log中記錄沒有發現ntp服務，ctss服務為啟用模式，同步時間的主節點是節點1，並且會告訴叢集的時間有差異，但是因為差異過小，無需調整。
校驗叢集的時間：
cluvfy comp clocksync -n all -verbose
雖然叢集時間不一致，但是這種情況下校驗結果是通過的，而且略微的差異範圍內叢集也會自動同步回來。
[grid@raclhr-11gR2-N1 ~]$ cluvfy comp clocksync -n all -verbose
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
raclhr-11gr2-n2 passed
raclhr-11gr2-n1 passed
Result: CTSS resource check passed
Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
raclhr-11gr2-n2 Active
raclhr-11gr2-n1 Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
Node Name Time Offset Status
------------ ------------------------ ------------------------
raclhr-11gr2-n2 0.0 passed
raclhr-11gr2-n1 0.0 passed
Time offset is within the specified limits on the following set of nodes:
"[raclhr-11gr2-n2, raclhr-11gr2-n1]"
Result: Check of clock time offsets passed
Oracle Cluster Time Synchronization Services check passed
Verification of Clock Synchronization across the cluster nodes was successful.
1.1.2 NTP同步模式
開啟NTP:
mv /etc/ntp.conf.bak /etc/ntp.conf
service ntpd status
/sbin/service ntpd start
# chkconfig ntpd off
ps -ef|grep ntp
節點1 ：
[root@raclhr-11gR2-N1 ~]# crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.
[root@raclhr-11gR2-N1 ~]# crsctl stat res -t -init
ora.ctssd
1 ONLINE ONLINE raclhr-11gr2-n1 OBSERVER
節點1的ctss日誌：
/u01/app/11.2.0/grid/log/raclhr-11gr2-n1/ctssd/octssd.log
2018-06-30 20:51:29.388: [ CTSS][899475200]sclsctss_gvss1: NTP default config file found
2018-06-30 20:51:29.389: [ CTSS][899475200]sclsctss_gvss8: Return [0] and NTP status [2].
2018-06-30 20:51:29.389: [ CTSS][899475200]ctss_check_vendor_sw: Vendor time sync software is detected. status [2].
2018-06-30 20:51:29.389: [ CTSS][899475200]ctss_check_vendor_sw: Ctssd is switching to observer role
2018-06-30 20:51:29.389: [ CTSS][899475200]clsctsselect_update_mbrdata: Updating pridata: { version[1] node[1] swversion[186647296] mode[0xee] }.
2018-06-30 20:51:29.639: [ CRSCCL][671086336]clsCclGetPriMemberData: Detected pridata change for node[1]. Retrieving it to the cache.
2018-06-30 20:51:31.434: [ CTSS][916338432]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xee], offset[0 ms]}, length=[8].
2018-06-30 20:51:35.258: [ CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].
2018-06-30 20:51:35.258: [ CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg
2018-06-30 20:51:35.259: [ CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc4] nodenum [2] hostname [raclhr-11gr2-n2] )
2018-06-30 20:51:35.656: [ CRSCCL][671086336]clsCclGetPriMemberData: Detected pridata change for node[2]. Retrieving it to the cache.
2018-06-30 20:51:43.240: [ CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].
2018-06-30 20:51:43.240: [ CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg
2018-06-30 20:51:43.240: [ CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc6] nodenum [2] hostname [raclhr-11gr2-n2] )
2018-06-30 20:51:51.217: [ CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].
2018-06-30 20:51:51.217: [ CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg
2018-06-30 20:51:51.218: [ CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc6] nodenum [2] hostname [raclhr-11gr2-n2] )
2018-06-30 20:51:59.194: [ CTSS][901576448]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [1].
2018-06-30 20:51:59.194: [ CTSS][901576448]ctsscomm_msg_hndlr: Received sync msg
2018-06-30 20:51:59.195: [ CTSS][901576448]ctsscomm_msg_hndlr: Received from slave ( mode [0xc6] nodenum [2] hostname [raclhr-11gr2-n2] )
節點1的octssd.log中記錄發現ntp服務，ctss服務會自動切換到觀望模式。
2018-06-30 20:57:27.608: [ CTSS][839321344]ctsselect_msm: CTSS mode is [0xc6]
2018-06-30 20:57:27.608: [ CTSS][839321344]ctssslave_swm1_2: Ready to initiate new time sync process.
2018-06-30 20:57:27.609: [ CTSS][839321344]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].
2018-06-30 20:57:27.612: [ CTSS][845625088]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].
2018-06-30 20:57:27.613: [ CTSS][845625088]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].
2018-06-30 20:57:27.613: [ CTSS][839321344]ctssslave_swm2_3: Received time sync message from master.
2018-06-30 20:57:27.613: [ CTSS][839321344]ctssslave_swm17: LT [1530363447sec 613028usec], MT [1530363447sec 140655884569984usec], Delta [4410usec]
2018-06-30 20:57:27.613: [ CTSS][839321344]ctssslave_swm19: The offset is [19748 usec] and sync interval set to [1]
2018-06-30 20:57:27.613: [ CTSS][839321344]ctssslave_swm: Received from master (mode [0xee] nodenum [1] hostname [raclhr-11gr2-n1] )
2018-06-30 20:57:27.613: [ CTSS][839321344]ctsselect_msm: Sync interval returned in [1]
2018-06-30 20:57:27.613: [ CTSS][845625088]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler
節點2的octssd.log中也會記錄發現ntp服務，ctss服務為觀望模式，並且同步時間的主節點是節點1。
1.1.3 模擬叢集時間不一致
如果在我們生產系統中碰到叢集時間不一致會導致什麼結果，我們的排查思路是怎麼樣的，以下是模擬叢集時間不一致的場景。
更改節點2的時間，向後推移2天：
將系統時間設定成2018年07月02日的命令如下：
#date -s 07/02/2018
將系統時間設定成下午23點23分06秒的命令如下。
#date -s 23:23:06
[root@raclhr-11gR2-N2 ctssd]# crsctl stat res -t -init
ora.ctssd
1 ONLINE ONLINE raclhr-11gr2-n2 ACTIVE:172768000
[root@raclhr-11gR2-N2 ctssd]# crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 172768000
172768000毫秒大約為2天:
SYS@lhrrac11> select 172768000/1000/24/60/60 from dual;
172768000/1000/24/60/60
-----------------------
1.99962963
更改節點2的時間後，在ASM和DB的alert日誌中產生了以下的告警資訊：
Time drift detected. Please check VKTM trace file for more details.
drift表示漂移。
[grid@raclhr-11gR2-N2 trace]$ pwd
/u01/app/grid/diag/asm/+asm/+ASM2/trace
[grid@raclhr-11gR2-N2 trace]$ ll -lrt *vktm*
-rw-r----- 1 grid oinstall 136 May 17 14:09 +ASM2_vktm_29999.trm
-rw-r----- 1 grid oinstall 1847 May 17 14:09 +ASM2_vktm_29999.trc
-rw-r----- 1 grid oinstall 529 Jun 4 14:52 +ASM2_vktm_32504.trm
-rw-r----- 1 grid oinstall 7238 Jun 4 14:52 +ASM2_vktm_32504.trc
-rw-r----- 1 grid oinstall 78 Jun 4 14:59 +ASM2_vktm_14800.trm
-rw-r----- 1 grid oinstall 1079 Jun 4 14:59 +ASM2_vktm_14800.trc
-rw-r----- 1 grid oinstall 90 Jun 4 17:26 +ASM2_vktm_14991.trm
-rw-r----- 1 grid oinstall 1200 Jun 4 17:26 +ASM2_vktm_14991.trc
-rw-r----- 1 grid oinstall 89 Jun 29 10:05 +ASM2_vktm_17961.trm
-rw-r----- 1 grid oinstall 1200 Jun 29 10:05 +ASM2_vktm_17961.trc
-rw-r----- 1 grid oinstall 191 Jul 2 21:35 +ASM2_vktm_19774.trm
-rw-r----- 1 grid oinstall 3171 Jul 2 21:35 +ASM2_vktm_19774.trc
[grid@raclhr-11gR2-N2 trace]$ cat +ASM2_vktm_19774.trc
*** 2018-06-30 19:22:12.650
VKTM running at (1)millisec precision with DBRM quantum (100)ms
[Start] HighResTick = 1530357732650537
kstmrmtickcnt = 0 : ksudbrmseccnt[0] = 1530357732
kstmchkdrift (kstmhighrestimecntkeeper:highres): Time stalled at 1530363888044519
*** 2018-06-10 20:04:00.000
kstmchkdrift (kstmhighrestimecntkeeper:highres): Time jumped forward by
(172844812599)usec at (1528632240000738) whereas (1000000) is allowed
usec代表微秒，msec表示毫秒，1s=1000ms=1000000us
VKTM程式發現系統時間變了，alert日誌會產生相應的告警資訊，從產生的trace檔案中可知，系統向前推進了172844812599微秒，也即為2天，也就是我們模擬更改的時間，而允許的差異範圍為1秒。
SYS@lhrrac11> select 172844812599/1000/1000/24/60/60 from dual;
172844812599/1000/1000/24/60/60
-------------------------------
2.00051866
節點2的octssd.log中和ctss狀態都記錄了偏移的時間：
2018-07-02 21:54:39.330: [ CTSS][1400497920]ctsselect_msm: CTSS mode is [0x84]
2018-07-02 21:54:39.330: [ CTSS][1400497920]ctssslave_swm1_2: Ready to initiate new time sync process.
2018-07-02 21:54:39.330: [ CTSS][1400497920]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].
2018-07-02 21:54:39.331: [ CTSS][1404700416]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].
2018-07-02 21:54:39.331: [ CTSS][1404700416]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctssslave_swm2_3: Received time sync message from master.
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctssslave_swm: The magnitude [172757997797] of the offset [172757997797 usec] is larger than [86400000000 usec] sec which is the CTSS limit.
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctssslave_swm: The magnitude of the systime diff is larger than max adjtime limit. Offset [172757997797] usec will be changed to max adjtime limit [+/- 131071].
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctssslave_swm15: The CTSS master is behind this node. The local time offset [-131071 usec] is being adjusted. Sync method [2]
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctssslave_swm17: LT [1530539679sec 331583usec], MT [1530366921sec 139882790197210usec], Delta [1267usec]
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctssslave_swm19: The offset is [131071 usec] and sync interval set to [4]
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctssslave_swm: Received from master (mode [0x8c] nodenum [1] hostname [raclhr-11gr2-n1] )
2018-07-02 21:54:39.331: [ CTSS][1400497920]ctsselect_msm: Sync interval returned in [4]
2018-07-02 21:54:39.331: [ CTSS][1404700416]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler
叢集的時間同步校驗也是失敗的，校驗結果是需要同步節點2的時間，此時因為叢集時間差異較大，同步服務往往是無法做到的，只有手工同步才能修復。
校驗叢集的時間同步：
[grid@raclhr-11gR2-N2 ~]$ cluvfy comp clocksync -n all -verbose
Verifying Clock Synchronization across the cluster nodes
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
raclhr-11gr2-n2 passed
raclhr-11gr2-n1 passed
Result: CTSS resource check passed
Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
raclhr-11gr2-n2 Active
raclhr-11gr2-n1 Active
CTSS is in Active state. Proceeding with check of clock time offsets on all nodes...
Reference Time Offset Limit: 1000.0 msecs
Check: Reference Time Offset
Node Name Time Offset Status
------------ ------------------------ ------------------------
raclhr-11gr2-n2 1.727568E8 failed
raclhr-11gr2-n1 0.0 passed
Result: PRVF-9661 : Time offset is greater than acceptable limit on node "raclhr-11gr2-n2" [actual = "1.727568E8", acceptable = "1000.0" ]
PRVF-9652 : Cluster Time Synchronization Services check failed
Verification of Clock Synchronization across the cluster nodes was unsuccessful.
Checks did not pass for the following node(s):
raclhr-11gr2-n2
1.727568E8表示科學計數法，為1.7*10的8次方，即172756800ms，即2天。
在沒有同步時間之前，重啟節點2是無法正常啟動的，從以下命令可知是在ctss這一步有問題，通過重新更改正確時間後，叢集才能正常啟動。
[root@raclhr-11gR2-N2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE Instance Shutdown
ora.cluster_interconnect.haip
1 ONLINE ONLINE raclhr-11gr2-n2
ora.crf
1 ONLINE ONLINE raclhr-11gr2-n2
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.cssdmonitor
1 ONLINE ONLINE raclhr-11gr2-n2
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.gpnpd
1 ONLINE ONLINE raclhr-11gr2-n2
ora.mdnsd
1 ONLINE ONLINE raclhr-11gr2-n2
檢視叢集的告警日誌：
/u01/app/11.2.0/grid/log/raclhr-11gr2-n2/alertraclhr-11gr2-n2.log
2018-07-02 22:05:36.344
[ctssd(30350)]CRS-2405:The Cluster Time Synchronization Service on host raclhr-11gr2-n2 is shutdown by user
2018-07-02 22:05:40.689
[ctssd(30358)]CRS-2407:The new Cluster Time Synchronization Service reference node is host raclhr-11gr2-n1.
2018-07-02 22:05:40.689
[ctssd(30358)]CRS-2401:The Cluster Time Synchronization Service started on host raclhr-11gr2-n2.
2018-07-02 22:05:42.704
[ctssd(30358)]CRS-2404:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid/log/raclhr-11gr2-n2/ctssd/octssd.log.
2018-07-02 22:05:43.395
[ctssd(30358)]CRS-2402:The Cluster Time Synchronization Service aborted on host raclhr-11gr2-n2. Details at in /u01/app/11.2.0/grid/log/raclhr-11gr2-n2/ctssd/octssd.log.
2018-07-02 22:05:44.404
[ohasd(29989)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2018-07-02 22:05:44.405
[ohasd(29989)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
2018-07-02 22:05:44.405
[ohasd(29989)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.
2018-07-02 22:05:44.405
[ohasd(29989)]CRS-2807:Resource 'ora.evmd' failed to start automatically.
檢視octssd.log
2018-07-02 22:05:42.702: [ CTSS][1805252352]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [3].
2018-07-02 22:05:42.702: [ CTSS][1805252352]ctsscomm_recv_cb4_2: Receive active version change msg. Old active version [186647296] New active version [186647296].
2018-07-02 22:05:42.702: [ CTSS][1805252352]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].
2018-07-02 22:05:42.702: [ CTSS][1805252352]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctssslave_swm2_3: Received time sync message from master.
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctssslave_swm: sendtime{sec[1530540340], usec[690191]}, receivetime{sec[1530540342], usec[702977]}.
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctssslave_swm: The RTT of sync msg [2012786] is too large for time sync to be accurate. Recommends retry. Returns [17].
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctssslave_swm: Received from master (mode [0x8c] nodenum [1] hostname [raclhr-11gr2-n1] )
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctsselect_monitor_steysync_mode: Failed in clsctssslave_sync_with_master [17]. Retries [0/3].
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctssslave_swm1_1: Waiting for last time sync process to finish. sync_state[6].
2018-07-02 22:05:42.703: [ CTSS][1805252352]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctssslave_swm1_2: Ready to initiate new time sync process.
2018-07-02 22:05:42.703: [ CTSS][1798948608]ctssslave_swm2_1: Waiting for time sync message from master. sync_state[2].
2018-07-02 22:05:42.704: [ CTSS][1805252352]ctsscomm_recv_cb2: Receive incoming message event. Msgtype [2].
2018-07-02 22:05:42.704: [ CTSS][1805252352]ctssslave_msg_handler4_1: Waiting for slave_sync_with_master to finish sync process. sync_state[3].
2018-07-02 22:05:42.704: [ CTSS][1798948608]ctssslave_swm2_3: Received time sync message from master.
2018-07-02 22:05:42.704: [ CTSS][1798948608]ctssslave_swm: The magnitude [172752141259 usec] of the offset [172752141259 usec] is larger than [86400000000 usec] sec which is the CTSS limit.
2018-07-02 22:05:42.704: [ CTSS][1798948608]ctsselect_monitor_steysync_mode: Failed in clsctssslave_sync_with_master [12]: Time offset is too much to be corrected
2018-07-02 22:05:42.704: [ CTSS][1805252352]ctssslave_msg_handler4_3: slave_sync_with_master finished sync process. Exiting clsctssslave_msg_handler
2018-07-02 22:05:43.395: [ CTSS][2023593728]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xd0], offset[172752141 ms]}, length=[8].
2018-07-02 22:05:43.395: [ CTSS][1798948608]ctsselect_monitor_steysync_mode: CTSS daemon exiting [12].
2018-07-02 22:05:43.395: [ CTSS][1798948608]CTSS daemon aborting
2018-07-02 22:05:44.398: [ CTSS][2023593728]ctss_checkcb: clsdm requested check alive. checkcb_data{mode[0xd0], offset[172752141 ms]}, length=[8].
下面開始修復系統：
將系統時間設定成2018年06月30日的命令如下：
#date -s 06/30/2018
將系統時間設定成下午23點23分06秒的命令如下。
#date -s 22:14:06
然後重啟CRS服務：
crsctl stop crs -f
crsctl start crs
然後ctss自動同步時間：
[root@raclhr-11gR2-N2 ctssd]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ctssd
1 ONLINE ONLINE raclhr-11gr2-n2 ACTIVE:100
[root@raclhr-11gR2-N2 ctssd]# crsctl stat res -t -init
ora.ctssd
1 ONLINE ONLINE raclhr-11gr2-n2 ACTIVE:0
小麥苗課程
小麥苗課堂開課啦，如下是現有的課程，歡迎諮詢小麥苗：
|
課程名稱 |
課時 |
上課時間 |
價格 |
|
OCP（從入門到專家） |
每年1期，35課時左右/期 |
20：00-22：00 |
1588(可優惠) |
|
OCM認證 |
每年N期，9課時/期 |
20：00-22：00 |
22888 |
|
高可用課程（rac+dg+ogg） |
每年1期，20課時左右/期 |
20：00-22：00 |
1888(可優惠) |
|
Oracle初級入門 |
每年1期，15課時左右/期 |
20：00-22：00 |
800 |
|
Oracle健康檢查指令碼 |
可微信或微店購買。 |
88 |
|
|
Oracle資料庫技能直通車 |
包含如下3個課程： ①《11g OCP網路課程培訓》（面向零基礎） 價值1600元 ②《11g OCM網路班課程培訓》（Oracle技能合集）價值10000+元 ③《RAC + DG + OGG 高可用網路班課程》 價值2000元 以上3個課程全部打包只要5888，只要5888所有課程帶回家，終身指導！所有課程都是線上講課，不是播放視訊，課件全部贈送！ 注意：以上OCP和OCM課程只包括培訓課程，不包括考試費用。OCM提供培訓+視訊，但是不提供練習環境和資料。報名一次，OCP和高可用的課程可以免費終身迴圈聽課。 |
5888 |
|
|
OCP+高可用（rac+dg+ogg） |
報名OCP+高可用課程，可以優惠300元，優惠後的價格為3188. |
3188(可優惠) |
|
注意：
1、每次上課前30分鐘答疑。
2、OCM實時答疑，提供和考試一樣的練習模擬環境，只要按照老師講的方式來練習，可以保證100%通過。
3、授課方式：YY語音網路直播講課（非視訊） + QQ互動答疑 + 視訊複習。
4、OCP課時可以根據大家學習情況進行增加或縮減。
5、以上所有課程均可迴圈聽課。
6、12c OCM課程私聊。
7、Oracle初級入門課程，只教大家最實用+最常用的Oracle操作維護知識。
8、以上所有課程，可以加小麥苗微信（lhrbestxh）或QQ（646634621）詳聊，優惠多多。
|
培訓專案 |
連線地址 |
|
DB筆試面試歷史連線 |
|
|
OCP培訓說明連線 |
|
|
OCM培訓說明連線 |
|
|
高可用（RAC+DG+OGG）培訓說明連線 |
|
|
OCP最新題庫解析歷史連線（052） |
|
|
微店地址 |
https://weidian.com/s/793741433?wfr=c&ifr=shopdetail |
About Me
.............................................................................................................................................
● 本文作者：小麥苗，部分內容整理自網路，若有侵權請聯絡小麥苗刪除
● 本文在itpub（http://blog.itpub.net/26736162/abstract/1/）、部落格園（http://www.cnblogs.com/lhrbest）和個人微信公眾號（xiaomaimiaolhr）上有同步更新
● 本文itpub地址：http://blog.itpub.net/26736162/abstract/1/
● 本文部落格園地址：http://www.cnblogs.com/lhrbest
● 本文pdf版、個人簡介及小麥苗雲盤地址：http://blog.itpub.net/26736162/viewspace-1624453/
● 資料庫筆試面試題庫及解答：http://blog.itpub.net/26736162/viewspace-2134706/
● DBA寶典今日頭條號地址：http://www.toutiao.com/c/user/6401772890/#mid=1564638659405826
.............................................................................................................................................
● QQ群號：230161599（滿）、618766405
● 微信群：可加我微信，我拉大家進群，非誠勿擾
● 聯絡我請加QQ好友（646634621），註明新增緣由
● 於 2018-06-01 06:00 ~ 2018-06-31 24:00 在魔都完成
● 最新修改時間：2018-06-01 06:00 ~ 2018-06-31 24:00
● 文章內容來源於小麥苗的學習筆記，部分整理自網路，若有侵權或不當之處還請諒解
● 版權所有，歡迎分享本文，轉載請保留出處
.............................................................................................................................................
● 小麥苗的微店：https://weidian.com/s/793741433?wfr=c&ifr=shopdetail
● 小麥苗出版的資料庫類叢書：http://blog.itpub.net/26736162/viewspace-2142121/
● 小麥苗OCP、OCM、高可用網路班：http://blog.itpub.net/26736162/viewspace-2148098/
.............................................................................................................................................
使用微信客戶端掃描下面的二維碼來關注小麥苗的微信公眾號（xiaomaimiaolhr）及QQ群（DBA寶典）、新增小麥苗微信，學習最實用的資料庫技術。
小麥苗的微信公眾號 小麥苗的DBA寶典QQ群2 小麥苗的微信二維碼 小麥苗的微店
.............................................................................................................................................
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26736162/viewspace-2157130/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle叢集時間同步Oracle
- Linux叢集環境下NTP伺服器時間同步Linux伺服器
- Oracle rac時間不同步Oracle
- NTP時間同步
- Hadoop叢集是配置時間同步！Hadoop
- Linux使用ntp時間伺服器同步時間Linux伺服器
- 使用NTP原理進行時間同步
- Oracle RAC啟動因CTSS導致的異常Oracle
- oracle RAC 診斷叢集狀態命令Oracle
- 時間同步協議NTP - 原理&實踐協議
- LINUX 解決時間同步問題（NTP）Linux
- 分散式叢集伺服器時間同步方案分散式伺服器
- Ubuntu 時間不準，怎麼設定NTP時間同步？Ubuntu
- NTP時間同步伺服器（時鐘同步）工作原理介紹伺服器
- oracle 11.2.0.4 rac叢集等待事件enq: TM - contentionOracle事件ENQ
- Windows ntp時間同步設定（bat指令碼）WindowsBAT指令碼
- Linux系統時間同步方法小結(NTP)Linux
- Oracle RAC日常運維-NetworkManager導致叢集故障Oracle運維
- 【ASK_ORACLE】Relink RAC叢集詳細步驟Oracle
- NTP時間同步伺服器（區域網時鐘同步）解決方案伺服器
- Oracle Linux 6.7中 Oracle 11.2.0.4 RAC叢集CRS異常處理OracleLinux
- 論企業生產網NTP時鐘同步（網路時間同步服務）
- Oracle 例項和RAC叢集下資料庫日誌目錄合集Oracle資料庫
- Centos下部署NTP時間伺服器同步環境CentOS伺服器
- CentOS7使用NTP搭建時間同步伺服器CentOS伺服器
- 沃趣微講堂 | Oracle叢集技術（二）：GI與Oracle RACOracle
- Oracle歸檔日誌所在目錄時間不對&&Oracle叢集日誌時間顯示錯誤Oracle
- ADMIN05 - 管理使用者和組、tar備份和恢復、NTP時間同步
- MongoDB叢集同步MongoDB
- oracle 12C RAC 12.1.0.2 叢集日誌（cluster log)目錄Oracle
- 大資料叢集服務啟停指令碼/常用埠/時間同步大資料指令碼
- 時間伺服器-NTP伺服器
- Oracle叢集軟體管理-新增和刪除叢集節點Oracle
- rac叢集日常維護命令
- 紅色警報 ORACLE RAC 11.2.0.4 FOR SOLARIS 10 ASM 和DB因叢集心跳丟失重啟OracleASM
- ORACLE RAC 11.2.0.4 FOR RHEL6叢集無法啟動的處理Oracle
- Elasticsearch跨叢集同步Elasticsearch
- NTP校時（時間同步伺服器）IPC網路攝像機時鐘問題排查伺服器