ORACLE11GR2 RAC檔案系統變更成ASM EXTEND RAC及高可用測試
本來一直都有玩下ASM EXTEND RAC這樣的想法，苦於沒有資源測試，等。。。。。
老天不負有心人啊~哈哈！終於有資源玩了。
2套儲存：EMS跟HDS，分別放在不同的機房。
由於原測試系統用的是檔案系統，故要將其先改為ASM，再建立ASM EXTEND RAC。
此次修改成ASM EXTEND RAC遇到一系列問題，雖然解決這些問題有過苦惱，但EXTEND RAC成功完成之後，有種莫名的成就感，
各位看官大問題解決之後有木有同感~ 呵呵
1．系統環境
1.1 OS及DB版本
主機OS版本：AIX 7.1 ("7100-02-03-1334")
ORACLE版本：oracle 11.2.0.3 PSU10
是否RAC：是
節點個數：4個
儲存：HDS 100G,EMS 50G
ASM或檔案系統：賽門鐵克VERITAS卷管理工具搭建叢集檔案系統
1.2 硬體
RAM : 128
SWAP: 13G
1.3 AIX /TMP檔案系統
8GB
1.4 AIX JDK & JRE
IBM JDK 1.6.0.00 (64 BIT)
1.5 目錄詳細
/oracle 50GB
/oraclelog 30GB
/ocrvote 2G
/archivelog 400G
/oradata 850
1.6 主機IP配置資訊
100.15.64.180 testdb1
100.15.64.181 testdb2
100.15.64.182 testdb3
100.15.64.183 testdb4
100.15.64.184 testdb1-vip
100.15.64.185 testdb2-vip
100.15.64.186 testdb3-vip
100.15.64.187 testdb4-vip
100.15.64.188 testdb-scan
7.154.64.1 testdb1-priv
7.154.64.2 testdb2-priv
7.154.64.3 testdb3-priv
7.154.64.4 testdb4-priv
2．檔案系統更換成ASM
2.1磁碟許可權及屬性修改
chown grid:asmadmin /dev/vx/rdmp/remc0_04a1
chown grid:asmadmin /dev/vx/rdmp/rhitachi_v0_11cd
chmod 660 /dev/vx/rdmp/remc0_04a1
chmod 660 /dev/vx/rdmp/rhitachi_v0_11cd
(注：由於測試庫使用的是賽門鐵克的儲存多路徑軟體，故無需修改磁碟屬性)
2.2建立ASM例項
su – grid
export DISPLAY=100.15.70.169:0.0
asmca
(注：建立OCTVOTE磁碟組選NORMAL冗餘，建立2個故障組，最少3塊磁碟，建議選用3塊磁碟，當asm的故障組如果有多餘3塊盤，votedisk遷移到這個磁碟組也只用其中的3塊盤。使用crsctl query css votedisk只看到votedisk放在3塊盤上。磁碟組的可用空間以其故障組總大小最小的為準)
2.3建立ASM磁碟組SYSDG,DATADG並修改磁碟組引數
su – grid
export DISPLAY=100.15.70.169:0.0
asmca
注：同一邊的儲存放在一個故障組中。
oracle 11G之後的ASM需要將rdbms的compatible引數修改為11.2.0.0，這個引數預設的是10.2.0.0，如果這個引數不修改，後面如果使用兩個故障組，其中一個故障組故障修復後，將故障組線上online的時候會報如下錯:
ORA-15283: ASM operation requires compatible.rdbms of 11.1.0.0.0 or higher
修改命令：
alter diskgroup SYSDG set attribute 'compatible.rdbms'='11.2.0.0';
select name,COMPATIBILITY,DATABASE_COMPATIBILITY from v$asm_diskgroup;
----compatibility對應asm的版本，
DATABASE_COMPATIBILITY --- 相容資料庫版本
2.4將檔案系統資料檔案遷移至ASM中
因為本次測試沒建庫，所以不涉及資料檔案遷移，如需遷移，使用RMAN實現。
2.5將OCR,VOTEDISK遷移至磁碟組OCRVOTE中
1)檢視ocr跟votedisk
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3296
Available space (kbytes) : 258824
ID : 1187520997
Device/File Name : /ocrvote/ocr1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a948649dc0e14f65bf171ba2ca496962 (/ocrvote/votedisk1) []
2. ONLINE a5f290d560684f47bf82eb3d34db5fc7 (/ocrvote/votedisk2) []
3. ONLINE 49617fb984fc4fcdbf5b7566a9e1778f (/ocrvote/votedisk3) []
Located 3 voting disk(s).
2)檢視資源狀態
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb1
ora.cvu
1 ONLINE ONLINE testdb1
ora.oc4j
1 ONLINE ONLINE testdb1
ora.scan1.vip
1 ONLINE ONLINE testdb1
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
3)備份OCR
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -manualbackup
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -showbackup
4)將OCR增加到磁碟組中並刪除原有檔案系統中的OCR
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -add +OCRVOTE
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3336
Available space (kbytes) : 258784
ID : 1187520997
Device/File Name : /ocrvote/ocr1
Device/File integrity check succeeded
Device/File Name : +OCRVOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrconfig -delete /ocrvote/ocr1
root@testdb1:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3336
Available space (kbytes) : 258784
ID : 1187520997
Device/File Name : +OCRVOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
5)將votedisk遷移至檔案系統中
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a948649dc0e14f65bf171ba2ca496962 (/ocrvote/votedisk1) []
2. ONLINE a5f290d560684f47bf82eb3d34db5fc7 (/ocrvote/votedisk2) []
3. ONLINE 49617fb984fc4fcdbf5b7566a9e1778f (/ocrvote/votedisk3) []
Located 3 voting disk(s).
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl replace votedisk +OCRVOTE
CRS-4256: Updating the profile
Successful addition of voting disk 3a5e5e8622024f17bf0c1a4594e303f5.
Successful addition of voting disk 92ff4555f7064f70bf3c022bd687dbc5.
Successful addition of voting disk 19a1fed74b7f4fb6bf780d43b5427dc9.
Successful deletion of voting disk a948649dc0e14f65bf171ba2ca496962.
Successful deletion of voting disk a5f290d560684f47bf82eb3d34db5fc7.
Successful deletion of voting disk 49617fb984fc4fcdbf5b7566a9e1778f.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
root@testdb1:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 3a5e5e8622024f17bf0c1a4594e303f5 (/dev/vx/rdmp/emc0_04a1) [OCRVOTE]
2. ONLINE 92ff4555f7064f70bf3c022bd687dbc5 (/dev/vx/rdmp/hitachi_vsp0_11cc) [OCRVOTE]
3. ONLINE 19a1fed74b7f4fb6bf780d43b5427dc9 (/dev/vx/rdmp/emc0_04c1) [OCRVOTE]
Located 3 voting disk(s).
3.將NFS新增至磁碟組OCTVOTE中，作為第三塊仲裁盤
asm extend rac需要在2套儲存之外的地方放置一臺linux的pc server，並需要在這臺server上建立一個檔案系統。 將此檔案系統以NFS形式掛載到asm extend rac的伺服器端，NFS上需要使用dd命令生成盤。
3.1NFS伺服器資訊
系統版本：Linux el5 x86_64
3.2NFS伺服器建立grid使用者
groupadd -g 1000 oinstall
groupadd -g 1100 asmadmin
useradd -u 1100 -g oinstall -G oinstall,asmadmin -d /home/grid -c "GRID Software Owner" grid
注：建議nfs伺服器使用者ID、組ID跟生產庫一致
3.3在NFS伺服器建立目錄並賦權，DD出一個盤
cd /oradata
mkdir votedisk
chown 1100:1100 votedisk
3.4修改NFS伺服器上的/etc/exports檔案，並重啟NFS
vi /etc/exports
新增如下行
/oradata/votedisk *(rw,sync,all_squash,anonuid=1100,anongid=1100)
service nfs stop
service nfs start
3.5檢視nfs是否包含新增的votedisk目錄
[root@ywtcdb ~]# exportfs -v
/oradata 100.15.64.*(rw,wdelay,no_root_squash,no_subtree_check,anonuid=65534,anongid=65534)
/oradata/votedisk
<world>(rw,wdelay,root_squash,all_squash,no_subtree_check,anonuid=1100,anongid=1100)
(注：紅色部分為新增部分)
3.6修改生產主機的/etc/filesystems檔案，將目錄設為自動隨機掛載(每個節點執行)
su - root
mkdir /voting_disk
chown grid:asmadmin /voting_disk
vi /etc/filesystems
新增如下內容：
/voting_disk:
dev = "/oradata/votedisk"
vfs = nfs
nodename = ywtcdb
mount = true
options = rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=600,vers=3,proto=tcp,noac,sec=sys
account = false
(注：嚴格按照/etc/filesystems的已有選項進行配置，包括標點符號，空格等，建議使用smit nfs命令進行nfs配置，並在命令配置完成之後修改/etc/filesystems檔案中對應掛載目錄的options屬性，options屬性必須是rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=600,vers=3,proto=tcp,noac,sec=sys)
使用smit nfs命令設定啟動自動掛載nfs
#smit nfs
[TOP] [Entry Fields]
* Pathname of mount point [/voting_disk]
* Pathname of remote directory [/oradata/votedisk]
* Host where remote directory resides [ywtcdb]
Mount type name []
* Security method [sys]
* Mount now, add entry to /etc/filesystems or both? both
* /etc/filesystems entry will mount the directory yes
3.7手動掛載目錄(每個節點執行)
/usr/sbin/nfso –p -o nfs_use_reserved_ports=1
或nfso -p -o nfs_use_reserved_ports=1
su - root
mount -v nfs -o rw,bg,hard,intr,rsize=32768,wsize=32768,timeo=600,vers=3,proto=tcp,noac,sec=sys 100.15.57.125:/oradata/votedisk /voting_disk
注:命令中的100.15.57.125問NFS伺服器的IP, /oradata/votedisk為NFS伺服器的目錄，/voting_disk為生產主機的目錄。
3.8使用dd命令生成一塊盤(任一生產節點)
dd if=/dev/zero of=/voting_disk/vote_disk_nfs bs=1M count=1000
3.9將新生成盤加到磁碟組OCRVOTE中
su – grid
export DISPLAY=100.15.70.169:0.0
asmca
在asmca中要先改變Disk Discovery Path
修改前：
/dev/vx/rdmp/*
修改後：
/voting_disk/vote_disk_nfs, /dev/vx/rdmp/*
將盤/voting_disk/vote_disk_nfs加到磁碟組OCRVOTE中的一個新的故障組中，新增完成之後我們可以看到磁碟組OCRVOTE有3個故障組。
3.10檢查votedisk是否在新增盤上
$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 89210622f0864ff0bf9517205691e679 (/voting_disk/vote_disk_nfs) [OCRVOTE]
2. ONLINE 55c4ee685a824ff3bf6ce510bf09468e (/dev/vx/rdmp/remc0_04a1) [OCRVOTE]
3. ONLINE 159234e88fe64f55bf0d4571362c3b07 (/dev/vx/rdmp/ rhitachi_v0_11cd) [OCRVOTE]
Located 3 voting disk(s).
3.11開始建庫，建庫完成之後，至此ASM EXTEND RAC建立完成
4.ASM EXTEND RAC高可用測試
4.1 拔掉節點1、節點2的EMC儲存光纖，模擬一邊儲存宕掉
css日誌如下：
節點1:：
2014-05-20 14:46:44.886:
[cssd(4129042)]CRS-1649:An I/O error occured for voting file: /dev/remc0_04a5; details at (:CSSNM00060:) in /oracle/app/11.2.0/grid/log/testdb1/cssd/ocssd.log.
2014-05-20 14:46:44.886:
[cssd(4129042)]CRS-1649:An I/O error occured for voting file: /dev/remc0_04a5; details at (:CSSNM00059:) in /oracle/app/11.2.0/grid/log/testdb1/cssd/ocssd.log.
2014-05-20 14:46:46.051:
[cssd(4129042)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.071:
[cssd(4129042)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
節點2：
2014-05-20 14:46:46.053:
[cssd(4195026)]CRS-1604:CSSD voting file is offline: /dev/remc0_04a5; details at (:CSSNM00069:) in /oracle/app/11.2.0/grid/log/testdb2/cssd/ocssd.log.
2014-05-20 14:46:46.053:
[cssd(4195026)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.071:
[cssd(4195026)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
節點3：
2014-05-20 14:46:46.053:
[cssd(3604942)]CRS-1604:CSSD voting file is offline: /dev/remc0_04a5; details at (:CSSNM00069:) in /oracle/app/11.2.0/grid/log/testdb3/cssd/ocssd.log.
2014-05-20 14:46:46.053:
[cssd(3604942)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.074:
[cssd(3604942)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
節點4：
2014-05-20 14:46:46.053:
[cssd(3015132)]CRS-1604:CSSD voting file is offline: /dev/remc0_04a5; details at (:CSSNM00069:) in /oracle/app/11.2.0/grid/log/testdb4/cssd/ocssd.log.
2014-05-20 14:46:46.053:
[cssd(3015132)]CRS-1626:A Configuration change request completed successfully
2014-05-20 14:46:46.073:
[cssd(3015132)]CRS-1601:CSSD Reconfiguration complete. Active nodes are testdb1 testdb2 testdb3 testdb4 .
CRS狀態正常:
testdb3:/oracle/app/11.2.0/grid/log/testdb3/cssd(testdb3)$/oracle/app/11.2.0/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE
testdb4
ora.cvu
1 ONLINE ONLINE testdb3
ora.oc4j
1 ONLINE ONLINE testdb3
ora.scan1.vip
1 ONLINE ONLINE testdb4
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
檢視votedisk如下：
$ /oracle/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 8a31ddf5013d4fb1bfdbb01d6fc6eb7b (/dev/rhitachi_v0_11cc) [OCRVOTE]
2. ONLINE 1ef9486d54b24f8cbf07814d2848a009 (/voting_disk/vote_disk_nfs) [OCRVOTE]
Located 2 voting disk(s).
當把儲存光纖插回去之後手動online磁碟，兩邊儲存會自動同步資料
alter diskgroup SYSDG online disks in failgroup fail_1;
alter diskgroup DATADG online disks in failgroup fail_1;
所有EMC儲存在各節點ASM磁碟組中都自動OFFLINE，保留HDS儲存，各節點例項正常。在測試中我們拔掉hds儲存光纖，現象跟拔掉EMS儲存光纖一致。由此可以得出：當一邊儲存宕掉之後，ASM EXTEND RAC保留好的那邊儲存，各節點例項均正常。當把儲存光纖插回去之後手動online磁碟，兩邊儲存會自動同步資料。
注：存放votedisk的磁碟組在磁碟掛回來之後會自動online磁碟
4.2 reboot節點1、2主機，模擬主機突然宕掉故障
當reboot節點1、2主機，檢視crs資源狀態如下：
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.DATADG.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb3
ora.cvu
1 ONLINE ONLINE testdb3
ora.oc4j
1 ONLINE ONLINE testdb3
ora.scan1.vip
1 ONLINE ONLINE testdb3
ora.testdb.db
1 ONLINE OFFLINE
2 ONLINE OFFLINE
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE INTERMEDIATE testdb4 FAILED OVER
ora.testdb2.vip
1 ONLINE INTERMEDIATE testdb3 FAILED OVER
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
當節點1、2主機起來之後，在檢視CRS狀態如下：
$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb3
ora.cvu
1 ONLINE ONLINE testdb3
ora.oc4j
1 ONLINE ONLINE testdb4
ora.scan1.vip
1 ONLINE ONLINE testdb3
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
當宕掉1個或多個節點時，其VIP會飄至正常節點，所有客戶端重連線到可用節點，當測試主機重啟完成之後，CRS會自動拉起，且VIP會正常回飄。
4.3 模擬public網路中斷
由於主機做了虛擬化，無法拔除網線。使用命令ifconfig en1 down宕掉節點1 public ip所在的網路卡進行測試
1)檢視節點1發現公有IP、VIP及SCAN IP均在網路卡en1上。
root@testdb1:/#netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en1 1500 link#2 0.14.5e.79.5c.ca 5153732 0 4066346 2 0
en1 1500 100.15.64 100.15.64.180 5153732 0 4066346 2 0
en1 1500 100.15.64 100.15.64.184 5153732 0 4066346 2 0
en1 1500 100.15.64 100.15.64.188 5153732 0 4066346 2 0
en2 1500 link#3 0.14.5e.79.5b.e6 40305463 0 44224443 2 0
en2 1500 7.154.64 7.154.64.1 40305463 0 44224443 2 0
en2 1500 169.254 169.254.78.30 40305463 0 44224443 2 0
lo0 16896 link#1 2316784 0 2316787 0 0
lo0 16896 127 127.0.0.1 2316784 0 2316787 0 0
lo0 16896 ::1%1 2316784 0 2316787 0 0
2)使用命令ifconfig en1 down進行測試
root@testdb1:/oracle/app/11.2.0/grid/bin#ifconfig en1 down
3)檢視crs資源狀態發現vip，scan ip均已飄至正常節點
testdb3:/home/oracle(testdb3)$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE OFFLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE OFFLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE OFFLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb2
ora.cvu
1 ONLINE ONLINE testdb2
ora.oc4j
1 ONLINE ONLINE testdb4
ora.scan1.vip
1 ONLINE ONLINE testdb2
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE INTERMEDIATE testdb4 FAILED OVER
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
4)將節點1的en1網路卡啟起來
root@testdb1:/#ifconfig en1 up
5)檢視crs資源狀態發現vip正常回飄
testdb3:/home/oracle(testdb3)$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.LISTENER.lsnr
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.OCRVOTE.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.SYSDG.dg
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.asm
ONLINE ONLINE testdb1 Started
ONLINE ONLINE testdb2 Started
ONLINE ONLINE testdb3 Started
ONLINE ONLINE testdb4 Started
ora.gsd
OFFLINE OFFLINE testdb1
OFFLINE OFFLINE testdb2
OFFLINE OFFLINE testdb3
OFFLINE OFFLINE testdb4
ora.net1.network
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.ons
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
ora.registry.acfs
ONLINE ONLINE testdb1
ONLINE ONLINE testdb2
ONLINE ONLINE testdb3
ONLINE ONLINE testdb4
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE testdb2
ora.cvu
1 ONLINE ONLINE testdb2
ora.oc4j
1 ONLINE ONLINE testdb4
ora.scan1.vip
1 ONLINE ONLINE testdb2
ora.testdb.db
1 ONLINE ONLINE testdb1 Open
2 ONLINE ONLINE testdb2 Open
3 ONLINE ONLINE testdb3 Open
4 ONLINE ONLINE testdb4 Open
ora.testdb1.vip
1 ONLINE ONLINE testdb1
ora.testdb2.vip
1 ONLINE ONLINE testdb2
ora.testdb3.vip
1 ONLINE ONLINE testdb3
ora.testdb4.vip
1 ONLINE ONLINE testdb4
測試節點(節點1)監聽停止，SCAN LISTENER原來在該節點執行，已漂移到其他可用節點，測試節點 VIP漂移到其他可用節點，當網路卡起來之後(public網路恢復正常)，VIP正常回飄，測試節點監聽自動online，SCAN LISTENER及scan VIP沒回飄。而後我們依次測試宕掉其他節點的public IP所在網路卡，發現SCAN LISTENER漂移至instance_number最小的節點，而vip隨機漂移。
4.4 宕掉監聽測試
透過kill監聽程式實現
原有連線沒有收到影響，新的連線不能連到該節點例項，應用透過TAF或自動重連到另一節點
監聽程式自動重新啟動
4.5 資料庫單個例項crash測試
透過kill pmon程式實現
kill pmon程式後，資料庫例項crash，並且例項自動重啟，重啟完成後會話自動重新連線
4.6 模擬CSSD程式crash
透過kill cssd程式實現
kill cssd程式後，該節點重啟，VIP飄至其他正常節點，主機啟動完成後CRS自動拉起，叢集重新配置。
4.7 模擬CRSD程式crash
透過kill crsd程式實現
kill crsd.bin程式後，一分鐘內該程式自動拉起。原理：crsd程式crash將會被orarootagent檢測到，同時crsd程式會被自動重啟。
4.8 模擬EVMD程式crash
透過kill evmd程式實現
kill evmd.bin程式後，一分鐘內該程式自動拉起。原理：evmd程式crash將被ohasd程式檢測到，evmd、orarootagent和crsd程式將會被重啟
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/28869493/viewspace-2153066/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- 12c複製 RAC ASM中的密碼檔案到檔案系統ASM密碼
- ORACLE11GR2 RAC解除安裝ASM例項步驟OracleASM
- goldengate + asm + racGoASM
- Oracle RAC NFS掛載檔案系統OracleNFS
- oracle11gR2 RAC更換網路卡實施方案Oracle
- RAC+ASM+DATAGUARDASM
- 【RAC】RAC更換心跳地址和RAC更換儲存主要步驟
- 【RAC】asm_diskgroups 引數ASM
- 【RAC】Oracle RAC上線測試場景介紹Oracle
- RAC之grid叢集安裝及ASM磁碟組配置ASM
- ORACLE RAC ASM資料檔案遷移OMF檔案報錯ORA-01276解決OracleASM
- 使用OpenFiler來模擬儲存配置RAC中ASM共享盤及多路徑（multipath）的測試ASM
- 10. Oracle常用高可用方案——10.2. RACOracle
- Oracle RAC序列效能測試Oracle
- RAC安裝【AIX 7 + 11.2.0.4 + ASM】AIASM
- RAC+DG(asm單例項)ASM單例
- 重建共享(db或asm)密碼檔案 in Oracle 19c RAC-20220209ASM密碼Oracle
- 【RAC】Oracle RAC12c+ srvctl操作命令變化(引數更嚴謹了）Oracle
- Oracle RAC引數檔案管理Oracle
- RAC和ASM環境下打patchASM
- RAC的VIP切換測試
- lustre檔案系統環境搭建及測試
- ORACLE11.2.0.4 RAC+ ASM安裝方法 (作業系統CENTOS7.6)OracleASM作業系統CentOS
- RAC之作業系統配置作業系統
- 【RAC】Oracle rac 如何修改公網及vipOracle
- 19c 資料高可用實用配置 RAC + SingleADG
- 19c 探索高可用系列(二) - RAC+DataGuard Broker
- Oracle RAC修改引數檔案位置Oracle
- Oracle RAC日常運維-ASM磁碟擴容Oracle運維ASM
- ASM叢集檔案系統ACFS（ASM Cluster File System）ASM
- Oracle RAC環境下ASM磁碟組擴容OracleASM
- 【RAC】Oracle rac修改IP地址及埠號命令參考Oracle
- 如何修改rac的系統時間
- 檔案系統變成RAW問題解決
- Overview of Oracle Flex ASM In Oracle 19c RAC-20220111ViewOracleFlexASM
- RAC資料庫心跳更換方案資料庫
- 【RAC】作業系統重灌後RAC11g節點重置注意事項作業系統
- MySQL——MHA高可用群集部署及故障測試MySql