學習oracle 10.2.0.1 rac叢集程式cssd的原理及機制之系列一
結論
1,cssd程式訪問表決磁碟2,如果RAC節點間私網通訊出現故障,不是馬上會導致RAC節點驅逐,有個時間間隔,大約為1分鐘
由CSSD的日誌看出
[ CSSD]2015-11-19 03:13:19.281 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(2) checkin(s)
[ CSSD]2015-11-19 03:13:20.282 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(3) checkin(s)
[ CSSD]2015-11-19 03:13:21.283 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(4) checkin(s)
中間略
[ CSSD]2015-11-19 03:14:13.340 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(56) checkin(s)
[ CSSD]2015-11-19 03:14:14.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(57) checkin(s)
[ CSSD]2015-11-19 03:14:15.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(58) checkin(s)
[ CSSD]2015-11-19 03:14:16.342 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(59) checkin(s)
[ CSSD]2015-11-19 03:14:14.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(57) checkin(s)
[ CSSD]2015-11-19 03:14:15.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(58) checkin(s)
[ CSSD]2015-11-19 03:14:16.342 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(59) checkin(s)
發生驅逐節點2
[ CSSD]2015-11-19 03:14:17.344 [1205823808] >TRACE: clssnmPollingThread: Eviction started for node jingfa2 (2), flags 0x000d, state 3, wt4c 0
3,RAC節點間私網出現通訊故障時,先是CSSD日誌有資訊,爾後是CRSD日誌,最後或同時是ALERT日誌(DB ALERT)
所以分析cssd日誌分析最為重要
4,一定要重點針對性分析cssd日誌中的函式比如:clssnmPollingThread
5,RAC私網通訊故障,最終會在ALERT中出現LMON及LMD的TRC檔案,可見LMON是監控RAC節點的健康執行的程式
引發問題
1, cssd日誌中的函式比如:clssnmPollingThread含義是什麼,從哪兒獲知這些函式的含義2, 分析RAC節點重啟或驅逐這些函式的區別又是什麼
3,故障出現時應重點關注哪些函式,並從這些函式獲取到故障的原因呢
測試
---禁用節點2的私網網路卡
[root@jingfa2 ~]# ifconfig eth1
eth1 Link encap:Ethernet HWaddr 08:00:27:5C:4F:73
inet addr:10.10.10.9 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe5c:4f73/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:120143 errors:0 dropped:0 overruns:0 frame:0
TX packets:127532 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:61973575 (59.1 MiB) TX bytes:67650524 (64.5 MiB)
Base address:0xd040 Memory:f0820000-f0840000
[root@jingfa2 ~]# ifconfig eth1 down
---觀察節點1的alert,crsd,cssd log
---先是cssd log出現如下資訊
[ CSSD]2015-11-19 03:11:44.329 [1163864384] >TRACE: clssgmClientConnectMsg: Connect from con(0xf9ecd60) proc(0xfa31db0) pid() proto(10:2:1:1)
[ CSSD]2015-11-19 03:12:44.621 [1163864384] >TRACE: clssgmClientConnectMsg: Connect from con(0xf9ecfa0) proc(0xfa31db0) pid() proto(10:2:1:1)
用函式clssnmpollingthread檢測到節點2通訊不正常,可見如果RAC 2個節點如果1分鐘不能通訊,即發生節點驅逐,且重啟有問題的節點(主機重啟喲)
[ CSSD]2015-11-19 03:13:19.281 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(2) checkin(s)
[ CSSD]2015-11-19 03:13:20.282 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(3) checkin(s)
[ CSSD]2015-11-19 03:13:21.283 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(4) checkin(s)
[ CSSD]2015-11-19 03:13:22.285 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(5) checkin(s)
[ CSSD]2015-11-19 03:13:23.287 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(6) checkin(s)
[ CSSD]2015-11-19 03:13:24.287 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(7) checkin(s)
[ CSSD]2015-11-19 03:13:25.288 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(8) checkin(s)
[ CSSD]2015-11-19 03:13:26.289 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(9) checkin(s)
[ CSSD]2015-11-19 03:13:27.290 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(10) checkin(s)
[ CSSD]2015-11-19 03:13:28.291 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(11) checkin(s)
[ CSSD]2015-11-19 03:13:29.293 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(12) checkin(s)
[ CSSD]2015-11-19 03:13:30.293 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(13) checkin(s)
[ CSSD]2015-11-19 03:13:31.295 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(14) checkin(s)
[ CSSD]2015-11-19 03:13:32.296 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(15) checkin(s)
[ CSSD]2015-11-19 03:13:33.297 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(16) checkin(s)
[ CSSD]2015-11-19 03:13:34.297 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(17) checkin(s)
中間略
[ CSSD]2015-11-19 03:14:13.340 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(56) checkin(s)
[ CSSD]2015-11-19 03:14:14.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(57) checkin(s)
[ CSSD]2015-11-19 03:14:15.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(58) checkin(s)
[ CSSD]2015-11-19 03:14:16.342 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(59) checkin(s)
[ CSSD]2015-11-19 03:14:14.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(57) checkin(s)
[ CSSD]2015-11-19 03:14:15.341 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(58) checkin(s)
[ CSSD]2015-11-19 03:14:16.342 [1205823808] >TRACE: clssnmPollingThread: node jingfa2 (2) missed(59) checkin(s)
發生驅逐節點2
[ CSSD]2015-11-19 03:14:17.344 [1205823808] >TRACE: clssnmPollingThread: Eviction started for node jingfa2 (2), flags 0x000d, state 3, wt4c 0
[ CSSD]2015-11-19 03:14:17.344 [1226803520] >TRACE: clssnmDoSyncUpdate: Initiating sync 3
[ CSSD]2015-11-19 03:14:17.344 [1226803520] >TRACE: clssnmSetupAckWait: Ack message type (11)
[ CSSD]2015-11-19 03:14:17.344 [1226803520] >TRACE: clssnmSetupAckWait: node(1) is ALIVE
[ CSSD]2015-11-19 03:14:17.344 [1226803520] >TRACE: clssnmSetupAckWait: node(2) is ALIVE
[ CSSD]2015-11-19 03:14:17.344 [1226803520] >TRACE: clssnmSendSync: syncSeqNo(3)
[ CSSD]2015-11-19 03:14:17.344 [1226803520] >TRACE: clssnmWaitForAcks: Ack message type(11), ackCount(2)
[ CSSD]2015-11-19 03:14:17.344 [1153374528] >TRACE: clssnmHandleSync: Acknowledging sync: src[1] srcName[jingfa1] seq[9] sync[3]
[ CSSD]2015-11-19 03:14:17.381 [577131712] >USER: NMEVENT_SUSPEND [00][00][00][06]
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmWaitForAcks: node(2) is expiring, msg type(11)
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmWaitForAcks: done, msg type(11)
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmDoSyncUpdate: node(0) missCount(20369) state(0)
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmDoSyncUpdate: node(2) missCount(60) state(3)
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmSetupAckWait: Ack message type (13)
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmSetupAckWait: node(1) is ACTIVE
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmSendVote: syncSeqNo(3)
[ CSSD]2015-11-19 03:14:18.346 [1226803520] >TRACE: clssnmWaitForAcks: Ack message type(13), ackCount(1)
[ CSSD]2015-11-19 03:14:18.346 [1153374528] >TRACE: clssnmSendVoteInfo: node(1) syncSeqNo(3) --可見cssd與表決磁碟有關
[ CSSD]2015-11-19 03:14:19.347 [1226803520] >TRACE: clssnmWaitForAcks: done, msg type(13)
[ CSSD]2015-11-19 03:14:19.347 [1226803520] >TRACE: clssnmCheckDskInfo: Checking disk info...
節點2 DOWN,一直持續讀取節點2的磁碟心跳
[ CSSD]2015-11-19 03:14:20.364 [1132394816] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(3) wrtcnt(19832) LATS(19649184) Disk lastSeqNo(19832)
[ CSSD]2015-11-19 03:14:21.378 [1132394816] >TRACE: clssnmReadDskHeartbeat: node(2) is down. rcfg(3) wrtcnt(19833) LATS(19650204) Disk lastSeqNo(19833)
然後crsd log出現資訊
可見crsd程式和cssd程式的作用不同,所以RAC叢集出問題先要檢視crsd的log,也就是說crsd的日誌只是現象
2015-11-19 03:15:16.749: [ CRSEVT][1607604544]0Processing member leave for jingfa2, incarnation: 3 --節點2離開即驅逐
2015-11-19 03:15:16.750: [ CRSEVT][1607604544]0Do failover for: jingfa2
2015-11-19 03:15:17.156: [ OCRSRV][1555155264]s_update_remote_cache_int: FAILED TO RCV ACK FROM node 2 retcode 7 --接受不到節點2的反饋
2015-11-19 03:15:19.805: [ CRSRES][1586624832]0startRunnable: setting CLI values
2015-11-19 03:15:19.966: [ CRSRES][1586624832]0Attempting to start `ora.jingfa2.vip` on member `jingfa1` --在節點1 啟動節點2原來的VIP
2015-11-19 03:15:24.288: [ CRSAPP][1586624832]0StartResource error for ora.jingfa2.vip error code = 1
2015-11-19 03:15:26.545: [ CRSRES][1586624832]0Start of `ora.jingfa2.vip` on member `jingfa1` failed.
2015-11-19 03:15:26.716: [ CRSRES][1586624832]0startRunnable: setting CLI values
2015-11-19 03:15:26.769: [ CRSRES][1628584256]0startRunnable: setting CLI values
2015-11-19 03:15:26.783: [ CRSRES][1628584256]0Attempting to start `ora.jingfa.db` on member `jingfa1` --在節點1啟動原節點2的DB資源
2015-11-19 03:15:26.838: [ CRSRES][1586624832]0Attempting to start `ora.jingfa.zxy_jingfa.jingfa1.srv` on member `jingfa1`
2015-11-19 03:15:26.999: [ CRSRES][1712503104]0jingfa1 : CRS-1019: Resource ora.jingfa2.ASM2.asm (application) cannot run on jingfa1
2015-11-19 03:15:27.016: [ CRSRES][1691523392]0startRunnable: setting CLI values
2015-11-19 03:15:27.018: [ CRSRES][1691523392]0Attempting to start `ora.jingfa.jingfa1.inst` on member `jingfa1` --原理同上
2015-11-19 03:15:27.919: [ CRSAPP][1586624832]0StartResource error for ora.jingfa.zxy_jingfa.jingfa1.srv error code = 1
2015-11-19 03:15:28.139: [ CRSRES][1586624832]0Start of `ora.jingfa.zxy_jingfa.jingfa1.srv` on member `jingfa1` failed.
2015-11-19 03:15:29.485: [ OCRSRV][1240459584]th_select_handler: Failed to retrieve procctx from ht. constr = [216713920] retval lht [-27] Signal CV.
2015-11-19 03:15:39.202: [ CRSAPP][1691523392]0StartResource error for ora.jingfa.jingfa1.inst error code = 1
2015-11-19 03:15:41.328: [ CRSRES][1691523392]0Start of `ora.jingfa.jingfa1.inst` on member `jingfa1` failed.
2015-11-19 03:15:41.380: [ CRSRES][1691523392]0CRS-1006: No more members to consider
2015-11-19 03:15:41.421: [ CRSAPP][1628584256]0StartResource error for ora.jingfa.db error code = 1
2015-11-19 03:15:41.578: [ CRSRES][1628584256]0Start of `ora.jingfa.db` on member `jingfa1` failed.
2015-11-19 03:15:41.593: [ CRSEVT][1607604544]0Post recovery done evmd event for: jingfa2
2015-11-19 03:15:41.595: [ CRSEVT][1607604544]0Processing RecoveryDone
同時alert出現資訊
Thu Nov 19 03:15:27 2015
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 92.168.1.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 1
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/10.2.0/db_1/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
processes = 150
event = 10708 trace name context forever,level 10
__shared_pool_size = 339738624
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 4194304
spfile = +DATA/jingfa/spfilejingfa.ora
_ksmg_lock_check_interval= 2
_lm_dd_interval = 120
sga_target = 599785472
control_files = +DATA/jingfa/control01.ctl, +DATA/jingfa/control02.ctl, +DATA/jingfa/control03.ctl
db_block_size = 8192
__db_cache_size = 243269632
compatible = 10.2.0.1.0
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
_kgl_time_to_wait_for_locks= 25
remote_login_passwordfile= EXCLUSIVE
db_domain =
local_listener = (address=(protocol=tcp)(host=92.168.1.21)(port=1521))
remote_listener = LISTENERS_JINGFA
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/admin/jingfa/bdump
user_dump_dest = /u01/app/oracle/admin/jingfa/udump
core_dump_dest = /u01/app/oracle/admin/jingfa/cdump
audit_file_dest = /u01/app/oracle/admin/jingfa/adump
db_name = jingfa
open_cursors = 300
pga_aggregate_target = 199229440
Cluster communication is configured to use the following interface(s) for this instance
10.10.10.8
Thu Nov 19 03:15:28 2015
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=2618
DIAG started with pid=3, OS id=2625
PSP0 started with pid=4, OS id=2630
LMON started with pid=5, OS id=2632
LMD0 started with pid=6, OS id=2659
LMS0 started with pid=7, OS id=2661
MMAN started with pid=8, OS id=2665
DBW0 started with pid=9, OS id=2667
LGWR started with pid=10, OS id=2669
CKPT started with pid=11, OS id=2671
SMON started with pid=12, OS id=2673
RECO started with pid=13, OS id=2675
CJQ0 started with pid=14, OS id=2677
MMON started with pid=15, OS id=2679
MMNL started with pid=16, OS id=2681
Thu Nov 19 03:15:28 2015
lmon registered with NM - instance id 1 (internal mem no 0)
Thu Nov 19 03:15:28 2015
Reconfiguration started (old inc 0, new inc 2)
pseudo shared rm latch used
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Thu Nov 19 03:15:29 2015
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Thu Nov 19 03:15:29 2015
LMS 0: 0 GCS shadows traversed, 0 replayed
Thu Nov 19 03:15:29 2015
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=17, OS id=2683
Thu Nov 19 03:15:29 2015
ALTER DATABASE MOUNT
Thu Nov 19 03:15:29 2015
This instance was first to mount
Thu Nov 19 03:15:29 2015
Starting background process ASMB
ASMB started with pid=19, OS id=2713
Starting background process RBAL
RBAL started with pid=20, OS id=2717
Thu Nov 19 03:15:32 2015
SUCCESS: diskgroup DATA was mounted
Thu Nov 19 03:15:37 2015
Errors in file /u01/app/oracle/admin/jingfa/udump/jingfa1_ora_2710.trc:
ORA-00600: internal error code, arguments: [kccsbck_first], [2], [623757018], [], [], [], [], []
SUCCESS: diskgroup DATA was dismounted
Thu Nov 19 03:15:38 2015
ORA-600 signalled during: ALTER DATABASE MOUNT...
Thu Nov 19 03:15:38 2015
Trace dumping is performing id=[cdmp_20151119031538]
Thu Nov 19 03:15:40 2015
Shutting down instance (abort)
License high water mark = 1
Instance terminated by USER, pid = 2991
同時檢視節點2 的alert,crsd,cssd log
先是cssd log有資訊:
然後是alert
Thu Nov 19 04:06:51 2015
ospid 4607: network interface with IP address 10.10.10.9 shows failure
requested interface 10.10.10.9 is not UP. Check output from ifconfig command
過一會兒alert
Thu Nov 19 04:07:54 2015
Error: KGXGN aborts the instance (6)
Thu Nov 19 04:07:55 2015
Errors in file /u01/app/oracle/admin/jingfa/bdump/jingfa2_lmon_4605.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Thu Nov 19 04:07:55 2015
Errors in file /u01/app/oracle/admin/jingfa/bdump/jingfa2_lmd0_4607.trc:
ORA-29702: error occurred in Cluster Group Service operation
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/9240380/viewspace-1841309/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- 基於oracle 10.2.0.1 rac學習lmon程式系列六之增補一Oracle
- 【RAC】Oracle叢集重要程式evmd、cssd和crsd發源地及跟蹤探索OracleCSS
- 基於oracle 10.2.0.1 rac學習lmon程式系列六Oracle
- 基於oracle 10.2.0.1 rac學習lms程式系列四Oracle
- 轉:Oracle叢集重要程式evmd、cssd和crsd發源地及跟蹤探索OracleCSS
- Oracle系列:Oracle RAC叢集體系結構Oracle
- Oracle叢集技術 | 叢集的自啟動系列(一)Oracle
- oracle 10.2.0.1 rac的lmd程式的含義之一Oracle
- Oracle RAC叢集、程式、日誌簡介Oracle
- oracle rac_cssd程式故障重啟相關OracleCSS
- 用strace跟蹤分析oracle 10.2.0.1 rac lmd程式系列二Oracle
- Kafka學習之(五)搭建kafka叢集之Zookeeper叢集搭建Kafka
- rac維護工具集系列(三)--叢集層crsctl系列
- rac維護工具集系列(四)--叢集層OCR系列
- 如何檢視Oracle RAC叢集的叢集名稱(CLUSTER NAME)Oracle
- MongoDB學習4:MongoDB複製集機制和原理,搭建複製集MongoDB
- Spring5.0原始碼學習系列之淺談懶載入機制原理Spring原始碼
- 學習叢集。
- 【RAC】使用crsctl工具控制Oracle RAC叢集的啟停Oracle
- Kafka學習之(六)搭建kafka叢集Kafka
- Java學習之反射機制及應用場景Java反射
- 小丸子學Hadoop系列之——部署Hbase叢集Hadoop
- MySQL學習系列之InnoDB下事務隔離機制MySql
- redis原理及叢集主從配置Redis
- Docker 容器搭建及 Redis 叢集原理DockerRedis
- Elasticsearch(二)--叢集原理及優化Elasticsearch優化
- Quartz叢集原理及配置應用quartz
- 從v$sysstat的指標ges messages sent理解oracle 10.2.0.1 rac lmd程式系列三指標Oracle
- Oracle 12c RAC CSSD程式無法啟動real time模式OracleCSS模式
- Redis學習之管道機制Redis
- ZooKeeper學習筆記一:叢集搭建筆記
- oracle RAC 診斷叢集狀態命令Oracle
- oracle 11g rac 叢集操作命令Oracle
- 3 安裝配置oracle叢集和RACOracle
- 小丸子學Hadoop系列之——部署Hadoop叢集Hadoop
- 每天學習一點JVM之:垃圾回收機制JVM
- 基於oracle 10.2.0.1 rac死鎖deadlock檢測時間相關隱含引數及機制之一_lm_dd_intervalOracle
- 沃趣微講堂 | Oracle叢集技術(五):叢集中的三種心跳機制Oracle