ORACLE11GR2 RAC檔案系統變更成ASM EXTEND RAC及高可用測試
本來一直都有玩下ASM EXTEND RAC這樣的想法，苦於沒有資源測試，等。。。。。
老天不負有心人啊~哈哈！終於有資源玩了。
2套儲存：EMS跟HDS，分別放在不同的機房。
由於原測試系統用的是檔案系統，故要將其先改為ASM，再建立ASM EXTEND RAC。
此次修改成ASM EXTEND RAC遇到一系列問題，雖然解決這些問題有過苦惱，但EXTEND RAC成功完成之後，有種莫名的成就感，
各位看官大問題解決之後有木有同感~ 呵呵
1．系統環境
1.1 OS及DB版本
主機OS版本：AIX 7.1 ("7100-02-03-1334")
ORACLE版本：oracle 11.2.0.3 PSU10
是否RAC：是
節點個數：4個
儲存：HDS 100G,EMS 50G
ASM或檔案系統：賽門鐵克VERITAS卷管理工具搭建叢集檔案系統
1.2 硬體
RAM : 128
SWAP: 13G
1.3 AIX /TMP檔案系統
8GB
1.4 AIX JDK & JRE
IBM JDK 1.6.0.00 (64 BIT)
1.5 目錄詳細
/oracle 50GB
/oraclelog 30GB
/ocrvote 2G
/archivelog 400G
/oradata 850
1.6 主機IP配置資訊
100.15.64.180 testdb1
100.15.64.181 testdb2
100.15.64.182 testdb3
100.15.64.183 testdb4
100.15.64.184 testdb1-vip
100.15.64.185 testdb2-vip
100.15.64.186 testdb3-vip
100.15.64.187 testdb4-vip
100.15.64.188 testdb-scan
7.154.64.1 testdb1-priv
7.154.64.2 testdb2-priv
7.154.64.3 testdb3-priv
7.154.64.4 testdb4-priv
2．檔案系統更換成ASM
2.1磁碟許可權及屬性修改
chown grid:asmadmin /dev/vx/rdmp/remc0_04a1
chown grid:asmadmin /dev/vx/rdmp/rhitachi_v0_11cd
chmod 660 /dev/vx/rdmp/remc0_04a1
chmod 660 /dev/vx/rdmp/rhitachi_v0_11cd
(注：由於測試庫使用的是賽門鐵克的儲存多路徑軟體，故無需修改磁碟屬性)
2.2建立ASM例項
su – grid
export DISPLAY=100.15.70.169:0.0
asmca
(注：建立OCTVOTE磁碟組選NORMAL冗餘，建立2個故障組，最少3塊磁碟，建議選用3塊磁碟，當asm的故障組如果有多餘3塊盤，votedisk遷移到這個磁碟組也只用其中的3塊盤。使用crsctl query css votedisk只看到votedisk放在3塊盤上。磁碟組的可用空間以其故障組總大小最小的為準)
2.3建立ASM磁碟組SYSDG,DATADG並修改磁碟組引數
su – grid
export DISPLAY=100.15.70.169:0.0
asmca
注：同一邊的儲存放在一個故障組中。
oracle 11G之後的ASM需要將rdbms的compatible引數修改為11.2.0.0，這個引數預設的是10.2.0.0，如果這個引數不修改，後面如果使用兩個故障組，其中一個故障組故障修復後，將故障組線上online的時候會報如下錯:
ORA-15283: ASM operation requires compatible.rdbms of 11.1.0.0.0 or higher
修改命令：
alter diskgroup SYSDG set attribute 'compatible.rdbms'='11.2.0.0';
select name,COMPATIBILITY,DATABASE_COMPATIBILITY from v$asm_diskgroup;
----compatibility對應asm的版本，
DATABASE_COMPATIBILITY --- 相容資料庫版本
2.4將檔案系統資料檔案遷移至ASM中
因為本次測試沒建庫，所以不涉及資料檔案遷移，如需遷移，使用RMAN實現。
2.5將OCR,VOTEDISK遷移至磁碟組OCRVOTE中
1)檢視ocr跟votedisk
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3296
Available space (kbytes) : 258824
ID : 1187520997
Device/File Name : /ocrvote/ocr1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a948649dc0e14f65bf171ba2ca496962 (/ocrvote/votedisk1) []
2. ONLINE a5f290d560684f47bf82eb3d34db5fc7 (/ocrvote/votedisk2) []
3. ONLINE 49617fb984fc4fcdbf5b7566a9e1778f (/ocrvote/votedisk3) []
Located 3 voting disk(s).
2)檢視資源狀態
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb1
ora.cvu
1 ONLINE ONLINE testdb1
ora.oc4j
1 ONLINE ONLINE testdb1
ora.scan1.vip
1 ONLINE ONLINE testdb1
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
3)備份OCR
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -manualbackup
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -showbackup
4)將OCR增加到磁碟組中並刪除原有檔案系統中的OCR
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -add +OCRVOTE
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3336
Available space (kbytes) : 258784
ID : 1187520997
Device/File Name : /ocrvote/ocr1
Device/File integrity check succeeded
Device/File Name : +OCRVOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -delete /ocrvote/ocr1
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3336
Available space (kbytes) : 258784
ID : 1187520997
Device/File Name : +OCRVOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
5)將votedisk遷移至檔案系統中
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a948649dc0e14f65bf171ba2ca496962 (/ocrvote/votedisk1) []
2. ONLINE a5f290d560684f47bf82eb3d34db5fc7 (/ocrvote/votedisk2) []
3. ONLINE 49617fb984fc4fcdbf5b7566a9e1778f (/ocrvote/votedisk3) []
Located 3 voting disk(s).
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl replace votedisk +OCRVOTE
CRS-4256: Updating the profile
Successful addition of voting disk 3a5e5e8622024f17bf0c1a4594e303f5.
Successful addition of voting disk 92ff4555f7064f70bf3c022bd687dbc5.
Successful addition of voting disk 19a1fed74b7f4fb6bf780d43b5427dc9.
Successful deletion of voting disk a948649dc0e14f65bf171ba2ca496962.
Successful deletion of voting disk a5f290d560684f47bf82eb3d34db5fc7.
Successful deletion of voting disk 49617fb984fc4fcdbf5b7566a9e1778f.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 3a5e5e8622024f17bf0c1a4594e303f5 (/dev/vx/rdmp/emc0_04a1) [OCRVOTE]
2. ONLINE 92ff4555f7064f70bf3c022bd687dbc5 (/dev/vx/rdmp/hitachi_vsp0_11cc) [OCRVOTE]
3. ONLINE 19a1fed74b7f4fb6bf780d43b5427dc9 (/dev/vx/rdmp/emc0_04c1) [OCRVOTE]
Located 3 voting disk(s).
3.將NFS新增至磁碟組OCTVOTE中，作為第三塊仲裁盤
asm extend rac需要在2套儲存之外的地方放置一臺linux的pc server，並需要在這臺server上建立一個檔案系統。 將此檔案系統以NFS形式掛載到asm extend rac的伺服器端，NFS上需要使用dd命令生成盤。
3.1NFS伺服器資訊
系統版本：Linux el5 x86_64
3.2NFS伺服器建立grid使用者
groupadd -g 1000 oinstall
groupadd -g 1100 asmadmin
useradd -u 1100 -g oinstall -G oinstall,asmadmin -d /home/grid -c "GRID Software Owner" grid
注：建議nfs伺服器使用者ID、組ID跟生產庫一致
3.3在NFS伺服器建立目錄並賦權，DD出一個盤
cd /oradata
mkdir votedisk
chown 1100:1100 votedisk
3.4修改NFS伺服器上的/etc/exports檔案，並重啟NFS
vi /etc/exports
新增如下行
/oradata/votedisk *(rw,sync,all_squash,anonuid=1100,anongid=1100)
service nfs stop
service nfs start
3.5檢視nfs是否包含新增的votedisk目錄
[root@ywtcdb ~]# exportfs -v
/oradata 100.15.64.*(rw,wdelay,no_root_squash,no_subtree_check,anonuid=65534,anongid=65534)
/oradata/votedisk
<world>(rw,wdelay,root_squash,all_squash,no_subtree_check,anonuid=1100,anongid=1100)
(注：紅色部分為新增部分)
3.6修改生產主機的/etc/filesystems檔案，將目錄設為自動隨機掛載(每個節點執行)
su - root
mkdir /voting_disk
chown grid:asmadmin /voting_disk
vi /etc/filesystems
新增如下內容：
/voting_disk:
dev = "/oradata/votedisk"
vfs = nfs
nodename = ywtcdb
mount = true
options = rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=600,vers=3,proto=tcp,noac,sec=sys
account = false
(注：嚴格按照/etc/filesystems的已有選項進行配置，包括標點符號，空格等，建議使用smit nfs命令進行nfs配置，並在命令配置完成之後修改/etc/filesystems檔案中對應掛載目錄的options屬性，options屬性必須是rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=600,vers=3,proto=tcp,noac,sec=sys)
使用smit nfs命令設定啟動自動掛載nfs
#smit nfs
[TOP] [Entry Fields]
* Pathname of mount point [/voting_disk]
* Pathname of remote directory [/oradata/votedisk]
* Host where remote directory resides [ywtcdb]
Mount type name []
* Security method [sys]
* Mount now, add entry to /etc/filesystems or both? both
* /etc/filesystems entry will mount the directory yes
3.7手動掛載目錄(每個節點執行)
/usr/sbin/nfso –p -o nfs_use_reserved_ports=1
或nfso -p -o nfs_use_reserved_ports=1
su - root
mount -v nfs -o rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=600,vers=3,proto=tcp,noac,sec=sys 100.15.57.125:/oradata/votedisk /voting_disk
注:命令中的100.15.57.125問NFS伺服器的IP, /oradata/votedisk為NFS伺服器的目錄，/voting_disk為生產主機的目錄。
3.8使用dd命令生成一塊盤(任一生產節點)
dd if=/dev/zero of=/voting_disk/vote_disk_nfs bs=1M count=1000
3.9將新生成盤加到磁碟組OCRVOTE中
su – grid
export DISPLAY=100.15.70.169:0.0
asmca
在asmca中要先改變Disk Discovery Path
修改前：
/dev/vx/rdmp/*
修改後：
/voting_disk/vote_disk_nfs, /dev/vx/rdmp/*
將盤/voting_disk/vote_disk_nfs加到磁碟組OCRVOTE中的一個新的故障組中，新增完成之後我們可以看到磁碟組OCRVOTE有3個故障組。
3.10檢查votedisk是否在新增盤上
$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 89210622f0864ff0bf9517205691e679 (/voting_disk/vote_disk_nfs) [OCRVOTE]
2. ONLINE 55c4ee685a824ff3bf6ce510bf09468e (/dev/vx/rdmp/remc0_04a1) [OCRVOTE]
3. ONLINE 159234e88fe64f55bf0d4571362c3b07 (/dev/vx/rdmp/ rhitachi_v0_11cd) [OCRVOTE]
Located 3 voting disk(s).
3.11開始建庫，建庫完成之後，至此ASM EXTEND RAC建立完成
4.ASM EXTEND RAC高可用測試
4.1 拔掉節點1、節點2的EMC儲存光纖，模擬一邊儲存宕掉
css日誌如下：
節點1:：
2014-05-20 14:46:44.886:
[cssd(4129042)]CRS-1649:An I/O error occured for voting file: /dev/remc0_04a5; details at (:CSSNM00060:) in /oracle/app/11.2.0/grid/log/testdb1/cssd/ocssd.log.
2014-05-20 14:46:44.886:
[cssd(4129042)]CRS-1649:An I/O error occured for voting file: /dev/remc0_04a5; details at (:CSSNM00059:) in /oracle/app/11.2.0/grid/log/testdb1/cssd/ocssd.log.
2014-05-20 14:46:46.051:
[cssd(4129042)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.071:
[cssd(4129042)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
節點2：
2014-05-20 14:46:46.053:
[cssd(4195026)]CRS-1604:CSSD voting file is offline: /dev/remc0_04a5; details at (:CSSNM00069:) in /oracle/app/11.2.0/grid/log/testdb2/cssd/ocssd.log.
2014-05-20 14:46:46.053:
[cssd(4195026)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.071:
[cssd(4195026)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
節點3：
2014-05-20 14:46:46.053:
[cssd(3604942)]CRS-1604:CSSD voting file is offline: /dev/remc0_04a5; details at (:CSSNM00069:) in /oracle/app/11.2.0/grid/log/testdb3/cssd/ocssd.log.
2014-05-20 14:46:46.053:
[cssd(3604942)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.074:
[cssd(3604942)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
節點4：
2014-05-20 14:46:46.053:
[cssd(3015132)]CRS-1604:CSSD voting file is offline: /dev/remc0_04a5; details at (:CSSNM00069:) in /oracle/app/11.2.0/grid/log/testdb4/cssd/ocssd.log.
2014-05-20 14:46:46.053:
[cssd(3015132)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.073:
[cssd(3015132)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
CRS狀態正常:
testdb3:/oracle/app/11.2.0/grid/log/testdb3/cssd(testdb3)$/oracle/app/11.2.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE
testdb4
ora.cvu
1 ONLINE ONLINE testdb3
ora.oc4j
1 ONLINE ONLINE testdb3
ora.scan1.vip
1 ONLINE ONLINE testdb4
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
檢視votedisk如下：
$ /oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 8a31ddf5013d4fb1bfdbb01d6fc6eb7b (/dev/rhitachi_v0_11cc) [OCRVOTE]
2. ONLINE 1ef9486d54b24f8cbf07814d2848a009 (/voting_disk/vote_disk_nfs) [OCRVOTE]
Located 2 voting disk(s).
當把儲存光纖插回去之後手動online磁碟，兩邊儲存會自動同步資料
alter diskgroup SYSDG online disks in failgroup fail_1;
alter diskgroup DATADG online disks in failgroup fail_1;
所有EMC儲存在各節點ASM磁碟組中都自動OFFLINE，保留HDS儲存，各節點例項正常。在測試中我們拔掉hds儲存光纖，現象跟拔掉EMS儲存光纖一致。由此可以得出：當一邊儲存宕掉之後，ASM EXTEND RAC保留好的那邊儲存，各節點例項均正常。當把儲存光纖插回去之後手動online磁碟，兩邊儲存會自動同步資料。
注：存放votedisk的磁碟組在磁碟掛回來之後會自動online磁碟
4.2 reboot節點1、2主機，模擬主機突然宕掉故障
當reboot節點1、2主機，檢視crs資源狀態如下：
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.DATADG.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb3
ora.cvu
1 ONLINE ONLINE testdb3
ora.oc4j
1 ONLINE ONLINE testdb3
ora.scan1.vip
1 ONLINE ONLINE testdb3
ora.testdb.db
1 ONLINE OFFLINE
2 ONLINE OFFLINE
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE INTERMEDIATE testdb4 FAILED OVER
ora.testdb2.vip
1 ONLINE INTERMEDIATE testdb3 FAILED OVER
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
當節點1、2主機起來之後，在檢視CRS狀態如下：
$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb3
ora.cvu
1 ONLINE ONLINE testdb3
ora.oc4j
1 ONLINE ONLINE testdb4
ora.scan1.vip
1 ONLINE ONLINE testdb3
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
當宕掉1個或多個節點時，其VIP會飄至正常節點，所有客戶端重連線到可用節點，當測試主機重啟完成之後，CRS會自動拉起，且VIP會正常回飄。
4.3 模擬public網路中斷
由於主機做了虛擬化，無法拔除網線。使用命令ifconfig en1 down宕掉節點1 public ip所在的網路卡進行測試
1)檢視節點1發現公有IP、VIP及SCAN IP均在網路卡en1上。
root@testdb1:/#netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en1 1500 link#2 0.14.5e.79.5c.ca 5153732 0 4066346 2 0
en1 1500 100.15.64 100.15.64.180 5153732 0 4066346 2 0
en1 1500 100.15.64 100.15.64.184 5153732 0 4066346 2 0
en1 1500 100.15.64 100.15.64.188 5153732 0 4066346 2 0
en2 1500 link#3 0.14.5e.79.5b.e6 40305463 0 44224443 2 0
en2 1500 7.154.64 7.154.64.1 40305463 0 44224443 2 0
en2 1500 169.254 169.254.78.30 40305463 0 44224443 2 0
lo0 16896 link#1 2316784 0 2316787 0 0
lo0 16896 127 127.0.0.1 2316784 0 2316787 0 0
lo0 16896 ::1%1 2316784 0 2316787 0 0
2)使用命令ifconfig en1 down進行測試
root@testdb1:/oracle/app/11.2.0/grid/bin#ifconfig en1 down
3)檢視crs資源狀態發現vip，scan ip均已飄至正常節點
testdb3:/home/oracle(testdb3)$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE OFFLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE OFFLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE OFFLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb2
ora.cvu
1 ONLINE ONLINE testdb2
ora.oc4j
1 ONLINE ONLINE testdb4
ora.scan1.vip
1 ONLINE ONLINE testdb2
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE INTERMEDIATE testdb4 FAILED OVER
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
4)將節點1的en1網路卡啟起來
root@testdb1:/#ifconfig en1 up
5)檢視crs資源狀態發現vip正常回飄
testdb3:/home/oracle(testdb3)$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb2
ora.cvu
1 ONLINE ONLINE testdb2
ora.oc4j
1 ONLINE ONLINE testdb4
ora.scan1.vip
1 ONLINE ONLINE testdb2
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
測試節點(節點1)監聽停止，SCAN LISTENER原來在該節點執行，已漂移到其他可用節點，測試節點 VIP漂移到其他可用節點，當網路卡起來之後(public網路恢復正常)，VIP正常回飄，測試節點監聽自動online，SCAN LISTENER及scan VIP沒回飄。而後我們依次測試宕掉其他節點的public IP所在網路卡，發現SCAN LISTENER漂移至instance_number最小的節點，而vip隨機漂移。
4.4 宕掉監聽測試
透過kill監聽程式實現
原有連線沒有收到影響，新的連線不能連到該節點例項，應用透過TAF或自動重連到另一節點
監聽程式自動重新啟動
4.5 資料庫單個例項crash測試
透過kill pmon程式實現
kill pmon程式後，資料庫例項crash，並且例項自動重啟，重啟完成後會話自動重新連線
4.6 模擬CSSD程式crash
透過kill cssd程式實現
kill cssd程式後，該節點重啟，VIP飄至其他正常節點，主機啟動完成後CRS自動拉起，叢集重新配置。
4.7 模擬CRSD程式crash
透過kill crsd程式實現
kill crsd.bin程式後，一分鐘內該程式自動拉起。原理：crsd程式crash將會被orarootagent檢測到，同時crsd程式會被自動重啟。
4.8 模擬EVMD程式crash
透過kill evmd程式實現
kill evmd.bin程式後，一分鐘內該程式自動拉起。原理：evmd程式crash將被ohasd程式檢測到，evmd、orarootagent和crsd程式將會被重啟
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/28869493/viewspace-2153066/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- oracle rac在加錯asm盤時，變成了檔案系統處理OracleASM
- 12c複製 RAC ASM中的密碼檔案到檔案系統ASM密碼
- 【RAC】RAC本地資料檔案遷移至ASM的方法(3)ASM
- 【RAC】RAC本地資料檔案遷移至ASM的方法(2)ASM
- 【RAC】RAC本地資料檔案遷移至ASM的方法(1)ASM
- Oracle RAC & Data Guard搭建高可用資料庫系統方案Oracle資料庫
- 更改RAC資料庫日誌檔案大小及數量 for ASM資料庫ASM
- Oracle RAC NFS掛載檔案系統OracleNFS
- RAC（ASM）歸檔模式修改ASM模式
- RAC 11G ASM下修改spfile檔案ASM
- oracle10g rac 全域性|例項級引數變更測試Oracle
- Linux RAC OCFS檔案系統與INODESLinux
- lustre檔案系統環境搭建及測試
- oracle之 RAC本地資料檔案遷移至ASMOracleASM
- oracle11gR2 RAC 環境測試修改節點VIP的測試操作記錄Oracle
- oracle11gR2 RAC更換網路卡實施方案Oracle
- ORACLE11GR2 RAC解除安裝ASM例項步驟OracleASM
- oracle dbfs檔案系統介紹及使用測試Oracle
- RAC變更ASMSPFILE儲存位置（轉）ASM
- RMAN Duplicate Database From RAC ASM To RAC ASM [ID 461479.1]DatabaseASM
- 不同cluster 檔案系統之間的遷移及RAC thread問題thread
- ASM與檔案系統之間copy資料檔案--檔案系統到ASMASM
- 20140218_RAC ASM新增磁碟測試ASM
- Linux檔案系統變成只讀Linux
- 【RAC】RAC更換心跳地址和RAC更換儲存主要步驟
- 【RAC】Oracle RAC上線測試場景介紹Oracle
- MySQL——MHA高可用群集部署及故障測試MySql
- MySQL高可用群集MHA部署及故障測試分析MySql
- Oracle RAC序列效能測試Oracle
- RAC下歸檔不放到共享盤的測試
- goldengate + asm + racGoASM
- RAC+ASM+DATAGUARDASM
- 使用OpenFiler來模擬儲存配置RAC中ASM共享盤及多路徑（multipath）的測試ASM
- Oracle10g RAC -- Linux 叢集檔案系統OracleLinux
- linux下Oracle RAC檔案系統相關的文章LinuxOracle
- 如何把rac元件變成單機元件
- 【RAC】Oracle 10g RAC 重建控制檔案Oracle 10g
- asm拷貝檔案到檔案系統ASM