Quorum FailGroup
參考：
How to Manually Add NFS voting disk to an Extended Cluster using ASM in 11.2 (Doc ID 1421588.1)
Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)
RAC: Frequently Asked Questions [ID 220970.1]
***********************************************************
什麼是Quorum FailGroup
***********************************************************
Quorum FailGroup中，只儲存Voting Disk，用於RAC on Extended Distance Clusters，做仲裁盤用
要放在其他儲存節點上或者通過NFS共享一個zero-padded檔案作為voting disk
對於使用2個陣列提供保護的環境中，也可以使用Quorum FG
假設使用high冗餘，一共5個投票盤。那麼5個投票盤在2個陣列上的數目肯定不同
假設一個陣列上有3個，另一個陣列上有2個，如果存在3個投票盤的陣列當機，那麼將只有2個投票盤可用
這種情況下，叢集的所有節點將當機
這種情況下，必須在第三個地點放置一個投票盤，在2個陣列上各有2個投票盤。這樣任意一個陣列當機，還可以保證有3個投票盤是可用的
***********************************************************
如何新增Quorum FailGroup
***********************************************************
一般有3個陣列的情況不多，下面測試通過NFS來配置
1.當前情況：
HIGH冗餘，5個投票盤
[oracle@database2 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 7fc99b8d4dc54f27bf967f23524a19e3 (/dev/asm-crs1) [CRSDG]
2. ONLINE 2ffd39d609bc4f71bf2b19de2e71c7a8 (/dev/asm-crs2) [CRSDG]
3. ONLINE 002eb188e9b14ffbbf4d5a607ade51c2 (/dev/asm-crs3) [CRSDG]
4. ONLINE 2319348cf3cc4f6abf116f973d8fd922 (/dev/asm-crs4) [CRSDG]
5. ONLINE 6c3cb875ba7e4fe2bffe97189e2bae25 (/dev/asm-crs5) [CRSDG]
Located 5 voting disk(s).
SQL> select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /dev/asm-crs05 CRSDG_0004 NORMAL
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.配置NFS
2.1 NFS伺服器端
[root@dm01db01 /]# cat /etc/exports
/oracle/votedisk 192.168.123.31(rw,sync,no_root_squash)
/oracle/votedisk 192.168.123.32(rw,sync,no_root_squash)
[root@dm01db01 /]# /etc/rc.d/init.d/portmap start
Starting portmap: [ OK ]
[root@dm01db01 /]# /etc/rc.d/init.d/nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
Stopping RPC idmapd: [ OK ]
Starting RPC idmapd: [ OK ]
2.2 資料庫伺服器端MOUNT檔案系統
# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk
# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048
# chown -R grid.oinstall /u01/app/oracle/votedisk
2.3 新增 quorum fg disk
SQL> alter system set asm_diskstring="/dev/asm*","/u01/app/oracle/votedisk/vote*" sid='*' scope=both;
SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01';
SQL> select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /dev/asm-crs05 CRSDG_0004 NORMAL
1 5 2048 CRSDG_0005 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.4 刪除多餘的磁碟
在quorum fg新增後，可以看到還沒有votedisk切換到其上
這時，我們將刪除有3個投票盤陣列上的一個投票盤
[grid@database1 votedisk]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 83fa917a0e844f23bf27238aff51b57a (/dev/asm-crs05) [CRSDG]
5. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
SQL> alter diskgroup crsdg drop disk CRSDG_0004; <==
[grid@database1 votedisk]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
5. ONLINE 6fc616a6923a4fb2bffca18e44a58533 (/u01/app/oracle/votedisk/votedisk01) [CRSDG] <==
***********************************************************
相關配置
***********************************************************
對於資料庫DiskGroup，建立時應該為不同陣列的盤，指定好FAILGROUP，以確保2份資料（NORMAL冗餘）放在不同的陣列中
修改引數asm_preferred_read_failure_groups，不同的節點，從不同的FG組中讀取資料
***********************************************************
測試
***********************************************************
1.NFS伺服器當機
2014-02-13 16:56:38.476:
[cssd(3577)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 99910 milliseconds
2014-02-13 16:57:28.541:
[cssd(3577)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 49850 milliseconds
2014-02-13 16:57:58.565:
[cssd(3577)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 19830 milliseconds
2014-02-13 16:58:18.573:
[cssd(3577)]CRS-1604:CSSD voting file is offline: /u01/app/oracle/votedisk/votedisk01; details at (:CSSNM00058:) in /u01/app/11.2.0/grid/log/database1/cssd/ocssd.log.
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).
叢集件執行正常此時
恢復
[root@database1 ~]# umount -f /u01/app/oracle/votedisk
[root@database1 ~]# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ ------------------------------ ------------------------------ --------
0 5 2048 /u01/app/oracle/votedisk/voted NORMAL <== 檔案此時並不屬於CRSDG
isk01
0 9 2048 /dev/asm-crs05 NORMAL
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 0 CRSDG_0004 FGQ NORMAL <==
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
[root@database1 votedisk]# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048 <== 重新格式化Voting disk
SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01'; <== 重新新增
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
0 4 2048 /dev/asm-crs05 NORMAL
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
1 5 0 CRSDG_0005 FGQ NORMAL <==
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
SQL> alter diskgroup crsdg drop quorum disk CRSDG_0005 force; <== 將舊的刪除
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
0 4 2048 /dev/asm-crs05 NORMAL <==
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.先於NFS伺服器啟動GRID
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
5. OFFLINE ad0268dcec654f14bfe60497d9490780 () []
Located 5 voting disk(s).
[root@database1 ~]# crsctl stat res -t <== 無法啟動
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[root@database2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE INTERMEDIATE database2 OCR not started <== ASM啟動，但是CRSDG無法MOUNT
ora.cluster_interconnect.haip
1 ONLINE ONLINE database2
ora.crf
1 ONLINE ONLINE database2
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE database2
ora.cssdmonitor
1 ONLINE ONLINE database2
ora.ctssd
1 ONLINE ONLINE database2 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE database2
ora.evmd
1 ONLINE INTERMEDIATE database2
ora.gipcd
1 ONLINE ONLINE database2
ora.gpnpd
1 ONLINE ONLINE database2
ora.mdnsd
1 ONLINE ONLINE database2
SQL> alter diskgroup crsdg mount;
alter diskgroup crsdg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "4" is missing from group number "1"
SQL> alter diskgroup crsdg mount force; <== 強制MOUNT
Diskgroup altered.
[root@database1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.DATADG.dg
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER_1522.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER_1523.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.asm
ONLINE ONLINE database1 Started
ONLINE ONLINE database2 Started
ora.gsd
OFFLINE OFFLINE database1
OFFLINE OFFLINE database2
ora.net1.network
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.ons
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.registry.acfs
ONLINE ONLINE database1
ONLINE ONLINE database2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE database1
ora.cvu
1 OFFLINE OFFLINE
ora.database1.vip
1 ONLINE ONLINE database1
ora.database2.vip
1 ONLINE ONLINE database2
ora.oc4j
1 ONLINE ONLINE database1
ora.orcl.billa3.svc
1 ONLINE ONLINE database1
ora.orcl.db
1 ONLINE ONLINE database1 Open
2 ONLINE ONLINE database2 Open
ora.scan1.vip
1 ONLINE ONLINE database1
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).
恢復和測試1相同
How to Manually Add NFS voting disk to an Extended Cluster using ASM in 11.2 (Doc ID 1421588.1)
Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)
RAC: Frequently Asked Questions [ID 220970.1]
***********************************************************
什麼是Quorum FailGroup
***********************************************************
Quorum FailGroup中，只儲存Voting Disk，用於RAC on Extended Distance Clusters，做仲裁盤用
要放在其他儲存節點上或者通過NFS共享一個zero-padded檔案作為voting disk
對於使用2個陣列提供保護的環境中，也可以使用Quorum FG
假設使用high冗餘，一共5個投票盤。那麼5個投票盤在2個陣列上的數目肯定不同
假設一個陣列上有3個，另一個陣列上有2個，如果存在3個投票盤的陣列當機，那麼將只有2個投票盤可用
這種情況下，叢集的所有節點將當機
這種情況下，必須在第三個地點放置一個投票盤，在2個陣列上各有2個投票盤。這樣任意一個陣列當機，還可以保證有3個投票盤是可用的
***********************************************************
如何新增Quorum FailGroup
***********************************************************
一般有3個陣列的情況不多，下面測試通過NFS來配置
1.當前情況：
HIGH冗餘，5個投票盤
[oracle@database2 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 7fc99b8d4dc54f27bf967f23524a19e3 (/dev/asm-crs1) [CRSDG]
2. ONLINE 2ffd39d609bc4f71bf2b19de2e71c7a8 (/dev/asm-crs2) [CRSDG]
3. ONLINE 002eb188e9b14ffbbf4d5a607ade51c2 (/dev/asm-crs3) [CRSDG]
4. ONLINE 2319348cf3cc4f6abf116f973d8fd922 (/dev/asm-crs4) [CRSDG]
5. ONLINE 6c3cb875ba7e4fe2bffe97189e2bae25 (/dev/asm-crs5) [CRSDG]
Located 5 voting disk(s).
SQL> select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /dev/asm-crs05 CRSDG_0004 NORMAL
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.配置NFS
2.1 NFS伺服器端
[root@dm01db01 /]# cat /etc/exports
/oracle/votedisk 192.168.123.31(rw,sync,no_root_squash)
/oracle/votedisk 192.168.123.32(rw,sync,no_root_squash)
[root@dm01db01 /]# /etc/rc.d/init.d/portmap start
Starting portmap: [ OK ]
[root@dm01db01 /]# /etc/rc.d/init.d/nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
Stopping RPC idmapd: [ OK ]
Starting RPC idmapd: [ OK ]
2.2 資料庫伺服器端MOUNT檔案系統
# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk
# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048
# chown -R grid.oinstall /u01/app/oracle/votedisk
2.3 新增 quorum fg disk
SQL> alter system set asm_diskstring="/dev/asm*","/u01/app/oracle/votedisk/vote*" sid='*' scope=both;
SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01';
SQL> select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /dev/asm-crs05 CRSDG_0004 NORMAL
1 5 2048 CRSDG_0005 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.4 刪除多餘的磁碟
在quorum fg新增後，可以看到還沒有votedisk切換到其上
這時，我們將刪除有3個投票盤陣列上的一個投票盤
[grid@database1 votedisk]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 83fa917a0e844f23bf27238aff51b57a (/dev/asm-crs05) [CRSDG]
5. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
SQL> alter diskgroup crsdg drop disk CRSDG_0004; <==
[grid@database1 votedisk]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
5. ONLINE 6fc616a6923a4fb2bffca18e44a58533 (/u01/app/oracle/votedisk/votedisk01) [CRSDG] <==
***********************************************************
相關配置
***********************************************************
對於資料庫DiskGroup，建立時應該為不同陣列的盤，指定好FAILGROUP，以確保2份資料（NORMAL冗餘）放在不同的陣列中
修改引數asm_preferred_read_failure_groups，不同的節點，從不同的FG組中讀取資料
***********************************************************
測試
***********************************************************
1.NFS伺服器當機
2014-02-13 16:56:38.476:
[cssd(3577)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 99910 milliseconds
2014-02-13 16:57:28.541:
[cssd(3577)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 49850 milliseconds
2014-02-13 16:57:58.565:
[cssd(3577)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 19830 milliseconds
2014-02-13 16:58:18.573:
[cssd(3577)]CRS-1604:CSSD voting file is offline: /u01/app/oracle/votedisk/votedisk01; details at (:CSSNM00058:) in /u01/app/11.2.0/grid/log/database1/cssd/ocssd.log.
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).
叢集件執行正常此時
恢復
[root@database1 ~]# umount -f /u01/app/oracle/votedisk
[root@database1 ~]# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ ------------------------------ ------------------------------ --------
0 5 2048 /u01/app/oracle/votedisk/voted NORMAL <== 檔案此時並不屬於CRSDG
isk01
0 9 2048 /dev/asm-crs05 NORMAL
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 0 CRSDG_0004 FGQ NORMAL <==
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
[root@database1 votedisk]# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048 <== 重新格式化Voting disk
SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01'; <== 重新新增
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
0 4 2048 /dev/asm-crs05 NORMAL
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
1 5 0 CRSDG_0005 FGQ NORMAL <==
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
SQL> alter diskgroup crsdg drop quorum disk CRSDG_0005 force; <== 將舊的刪除
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
0 4 2048 /dev/asm-crs05 NORMAL <==
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.先於NFS伺服器啟動GRID
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
5. OFFLINE ad0268dcec654f14bfe60497d9490780 () []
Located 5 voting disk(s).
[root@database1 ~]# crsctl stat res -t <== 無法啟動
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[root@database2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE INTERMEDIATE database2 OCR not started <== ASM啟動，但是CRSDG無法MOUNT
ora.cluster_interconnect.haip
1 ONLINE ONLINE database2
ora.crf
1 ONLINE ONLINE database2
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE database2
ora.cssdmonitor
1 ONLINE ONLINE database2
ora.ctssd
1 ONLINE ONLINE database2 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE database2
ora.evmd
1 ONLINE INTERMEDIATE database2
ora.gipcd
1 ONLINE ONLINE database2
ora.gpnpd
1 ONLINE ONLINE database2
ora.mdnsd
1 ONLINE ONLINE database2
SQL> alter diskgroup crsdg mount;
alter diskgroup crsdg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "4" is missing from group number "1"
SQL> alter diskgroup crsdg mount force; <== 強制MOUNT
Diskgroup altered.
[root@database1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.DATADG.dg
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER_1522.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER_1523.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.asm
ONLINE ONLINE database1 Started
ONLINE ONLINE database2 Started
ora.gsd
OFFLINE OFFLINE database1
OFFLINE OFFLINE database2
ora.net1.network
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.ons
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.registry.acfs
ONLINE ONLINE database1
ONLINE ONLINE database2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE database1
ora.cvu
1 OFFLINE OFFLINE
ora.database1.vip
1 ONLINE ONLINE database1
ora.database2.vip
1 ONLINE ONLINE database2
ora.oc4j
1 ONLINE ONLINE database1
ora.orcl.billa3.svc
1 ONLINE ONLINE database1
ora.orcl.db
1 ONLINE ONLINE database1 Open
2 ONLINE ONLINE database2 Open
ora.scan1.vip
1 ONLINE ONLINE database1
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).
恢復和測試1相同
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/8242091/viewspace-1081078/，如需轉載，請註明出處，否則將追究法律責任。