最近的一次ASM diskgroup線上遷移記錄

oliseh發表於2016-04-25

目的:一套11.2.0.4 雙節點的RAC當前使用的是陣列裡的FC(Fibre channel)盤,執行一段時間後發現資料庫比較空閒,所以打算將其遷移到同陣列的SATA盤上,以將FC盤預留給對IO要求更高的應用。

遷移方法:由ASM實現儲存管理的資料庫,使用ASM diskgroup線上遷移方式是再好不過的選擇
挑戰:難度雖然不大,但前一次做diskgroup線上遷移還是3年以前的事情,這次特意從頭到尾將整個步驟整理如下,供參考

//////////////////
// 第一部分:新盤準備
//////////////////
1、shajx1主機上當前使用的dg及盤
ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576    743632   350106                0          350106              0             N  AJXDG1/
MOUNTED  EXTERN  N         512   4096  1048576    743632   321451                0          321451              0             N  AJXDG2/
MOUNTED  EXTERN  N         512   4096  1048576      7500     7134                0            7134              0             N  SYSDG1/
MOUNTED  EXTERN  N         512   4096  1048576      7500     7103                0            7103              0             Y  SYSDG2/
MOUNTED  EXTERN  N         512   4096  1048576      7500     7403                0            7403              0             N  SYSDG3/
ASMCMD> lsdsk -G sysdg1
Path
/dev/rhdiskpower44
/dev/rhdiskpower45
ASMCMD> lsdsk -G sysdg2
Path
/dev/rhdiskpower46
/dev/rhdiskpower47
ASMCMD> lsdsk -G sysdg3
Path
/dev/rhdiskpower48
/dev/rhdiskpower49


2、shajx1主機上認到的新盤及大小如下:
/dev/rhdiskpower58     13FD 09E:0 05A:C8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower59     1401 09E:0 06A:C7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower60     1405 09E:0 05A:CA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower61     1409 09E:0 06A:C9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower62     140D 09E:0 05A:CC  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower63     1411 09E:0 06A:CB  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower64     1415 09E:0 06C:D0  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower65     1419 09E:0 05C:D1  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower66     141D 09E:0 06C:D2  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower67     1421 09E:0 05C:D3  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower68     1425 09E:0 06C:D4  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower69     1429 09E:0 06A:CD  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower70     142D 09E:0 05D:D6  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower71     1431 09E:0 06D:D5  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower72     1435 09E:0 05D:D8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower73     1439 09E:0 06D:D7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower74     143D 09E:0 05D:DA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower75     1441 09E:0 06D:D9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower76     1445 09E:0 05D:DC  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower77     1449 09E:0 06D:DB  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower78     144D 09E:0 06B:C0  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower79     1451 09E:0 05B:C1  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower80     1455 09E:0 06B:C2  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower81     1459 09E:0 05B:C3  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower82     145D 09E:0 06B:C4  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower83     1461 09E:0 06A:D6  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower84     1465 09E:0 05A:D5  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower85     1469 09E:0 06A:D8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower86     146D 09E:0 05A:D7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower87     1471 09E:0 06A:DA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower88     1475 09E:0 05A:D9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower89     1479 09E:0 06A:DC  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower90     147D 09E:0 05A:DB  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower91     1481 09E:0 05C:C0  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower92     1485 09E:0 06C:C1  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower93     1489 09E:0 05C:C2  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower94     148D 09E:0 06C:C3  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower95     1491 09E:0 05C:C4  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower96     1495 09E:0 06D:C6  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower97     1499 09E:0 05D:C5  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower98     149D 09E:0 06D:C8  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower99     14A1 09E:0 05D:C7  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower100    14A5 09E:0 06D:CA  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower101    14A9 09E:0 05D:C9  2-Way Mir     N/Grp'd  (M) RW   46478
/dev/rhdiskpower102    40DF 09E:0 10A:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower103    40E0 09E:0 10C:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower104    40E1 09E:0 09B:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower105    40E2 09E:0 09D:D18 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower106    40E3 09E:0 09A:D19 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower107    40E4 09E:0 09C:D19 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower108    40DD 09E:0 09B:C1B 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower109    40DE 09E:0 10D:C1A 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower110    40D5 09E:0 08B:D1D 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower111    40D6 09E:0 08D:D1D 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower112    40D7 09E:0 10A:C1B 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower113    40D8 09E:0 09C:C1A 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower114    40D9 09E:0 10B:C1A 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower115    40DA 09E:0 09D:C19 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower116    40DB 09E:0 09A:C1C 2-Way Mir     N/Grp'd      RW     938
/dev/rhdiskpower117    40DC 09E:0 10C:C1B 2-Way Mir     N/Grp'd      RW     938


其中hdiskpower58~89用於資料檔案,hdiskpower90~hdiskpower101保留用於archivelog檔案系統,hdiskpower102~hdiskpower117用於新建OCR、vote盤所在DG


3、修改hdiskpower58~hdiskpower89、hdiskpower102~hdiskpower117的磁碟屬性及屬主,屬性為crw-rw----,屬主為grid.asmadmin,兩個節點都要修改
shajx1、shajx2都要執行:
i=58
while [ $i -lt 90 ]
do
chown grid.asmadmin /dev/rhdiskpower$i
((i=i+1))
done


i=102
while [ $i -lt 118 ]
do
chown grid.asmadmin /dev/rhdiskpower$i
((i=i+1))
done


4、登陸ASM例項確認新加的盤都能認到,兩個節點都要執行
sqlplus '/as sysasm'
col path format a40
set linesize 160 pagesize 200
select mount_status,header_status,mode_status,os_mb,state,path from v$asm_disk where header_status='CANDIDATE' order by path;
MOUNT_S HEADER_STATU MODE_ST      OS_MB STATE    PATH
------- ------------ ------- ---------- -------- ----------------------------------------
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower0
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower1
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower10
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower102
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower103
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower104
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower105
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower106
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower107
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower108
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower109
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower11
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower110
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower111
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower112
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower113
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower114
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower115
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower116
CLOSED  CANDIDATE    ONLINE         937 NORMAL   /dev/rhdiskpower117
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower2
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower3
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower4
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower5
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower59
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower6
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower60
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower61
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower62
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower63
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower64
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower65
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower66
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower67
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower68
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower69
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower7
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower70
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower71
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower72
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower73
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower74
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower75
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower76
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower77
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower78
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower79
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower8
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower80
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower81
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower82
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower83
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower84
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower85
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower86
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower87
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower88
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower89
CLOSED  CANDIDATE    ONLINE       46477 NORMAL   /dev/rhdiskpower9


5、挑首尾兩塊盤作讀測試及kfed測試,使用oracle或者grid使用者執行:
dd if=/dev/rhdiskpower58 of=/dev/null count=100 bs=8192
100+0 records in.
100+0 records out.


dd if=/dev/rhdiskpower89 of=/dev/null count=100 bs=8192
100+0 records in.
100+0 records out.


kfed read /dev/rhdiskpower58
kfed read /dev/rhdiskpower89


kfed的輸出如下:
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000


MOS裡關於KFBTYP_INVALID的描述:If the disk is not a FORMER disk, then kfed on new disks will return:    kfbh.type: 0 ; 0x002: KFBTYP_INVALID and that is expected


6、修改hdiskpowerX的reserve_policy屬性,兩節點都要執行
i=58
while [ $i -lt 90 ]
do
chdev -l hdiskpower${i} -a reserve_policy=no_reserve
(( i=i+1 ))
done


i=102
while [ $i -lt 118 ]
do
chdev -l hdiskpower${i} -a reserve_policy=no_reserve
(( i=i+1 ))
done


***驗證reserve_policy是否已經成功
i=58
while [ $i -lt 90 ]
do
echo "hdiskpower${i} " `lsattr -El hdiskpower${i} |grep reserve_policy | awk '{print $2}'`
(( i=i+1 ))
done


i=102
while [ $i -lt 118 ]
do
echo "hdiskpower${i} " `lsattr -El hdiskpower${i} |grep reserve_policy | awk '{print $2}'`
(( i=i+1 ))
done


7、選一塊新的盤在上面建立一個Diskgroup
create diskgroup dgnew1 external redundancy disk '/dev/rhdiskpower58';


SQL> SELECT NAME,TOTAL_MB,usable_file_mb,state FROM V$ASM_DISKGROUP where name='DGNEW1';


NAME                             TOTAL_MB USABLE_FILE_MB STATE
------------------------------ ---------- -------------- -----------
DGNEW1                              46477          46427 MOUNTED                <---損耗主要體現在Allocation Table、PST、Disk Directory等各種directory


diskgroup裡所有的metadata見下:


PST
Allocation Table
Free Space Table
Disk Directory  (file number 2)
File Directory   (file number 1)
ACD  (file number 3)
Alias Directory  (file number 6)
Attribute Directory (file number 9)
User and Group Directory
Used Space Directory (file number 8)
How many partners
Continuing Operations Directory  (file number 4)
Template Directory (file number 5)
Volume Directory
staleness_directory_and_registry


8、檢查crs裡新建diskgroup的狀態
grid@qc740701a:/home/grid>crsctl status res ora.DGNEW1.dg -t
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGNEW1.dg
               OFFLINE OFFLINE      qc740701a        <---第二個節點上需要mount
               ONLINE  ONLINE       qc740702a     


因為是RAC環境,在第二個節點上也把這個diskgroup mount起來
su - grid
srvctl start diskgroup -g dgnew1 -n qc740701a


crsctl status res ora.DGNEW1.dg -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DGNEW1.dg
               ONLINE  ONLINE       qc740701a
               ONLINE  ONLINE       qc740702a


9、如果在第8步裡,兩個節點都能mount起DG,那麼將這個diskgroup從N-1個節點都dismount,只留最後一個節點用於drop
qc740702a: dismount
srvctl stop diskgroup -g dgnew1 -n qc740702a


qc740701a: drop
drop diskgroup dgnew1


10、至此測試完畢,新認的盤已可以使用


//////////////////
// 第二部分:遷移ASM spfile, voting disk,ocr所在的DG至SATA盤
//////////////////
1、ASM spfile, voting disk 存放在SYSDG1、SYSDG2中,SYSDG3裡暫時沒有檔案
ASMCMD> spget
+SYSDG1/shajx-cluster/asmparameterfile/registry.253.854489087    <---spfile
ASMCMD> quit


grid@qc740702a:/home/grid>ocrcheck -config   <---ocr file
Oracle Cluster Registry configuration is :
         Device/File Name         :    +sysdg1
         Device/File Name         :    +sysdg2
         
grid@qc740702a:/home/grid>crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   46c6fdafabb74fa1bf2987f4d9a846b5 (/dev/rhdiskpower46) [SYSDG2]   <---voting disk
Located 1 voting disk(s).


2、將hdiskpower102~hdiskpower117新增到SYSDG1、SYSDG2、SYSDG3
SYSDG1:
alter diskgroup sysdg1 add disk '/dev/rhdiskpower102','/dev/rhdiskpower103','/dev/rhdiskpower104','/dev/rhdiskpower105' drop disk SYSDG1_0000,SYSDG1_0001 rebalance power 500;


SYSDG2:
alter diskgroup sysdg2 add disk '/dev/rhdiskpower106','/dev/rhdiskpower107','/dev/rhdiskpower108','/dev/rhdiskpower109' drop disk SYSDG2_0000,SYSDG2_0001 rebalance power 500;


SYSDG3:
alter diskgroup sysdg3 add disk '/dev/rhdiskpower110','/dev/rhdiskpower111','/dev/rhdiskpower112','/dev/rhdiskpower113' drop disk SYSDG3_0000,SYSDG3_0001 rebalance power 500;


3、將hdiskpower58~hdiskpower73新增到AJXDG1
AJXDG1: 透過shell指令碼獲取SQL指令碼
i=58
while [ $i -lt 74 ]
do
echo "alter diskgroup AJXDG1 add disk '/dev/rhdiskpower${i}' rebalance power 0;";
(( i=i+1 ))
done


得到如下指令碼直接執行
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower58' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower59' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower60' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower61' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower62' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower63' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower64' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower65' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower66' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower67' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower68' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower69' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower70' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower71' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower72' rebalance power 0;
alter diskgroup AJXDG1 add disk '/dev/rhdiskpower73' rebalance power 0;


執行如下SQL
select 'alter diskgroup '||d.name||' drop disk '||s.name ||' rebalance power 0;' from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');


得到drop old disk的cmd
alter diskgroup AJXDG1 drop disk AJXDG1_0000 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0001 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0002 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0003 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0004 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0005 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0006 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0007 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0008 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0009 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0010 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0011 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0012 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0013 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0014 rebalance power 0;
alter diskgroup AJXDG1 drop disk AJXDG1_0015 rebalance power 0;


  ---3.1 觀察rebalance前的磁碟狀態
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');


NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG1_0000                    DROPPING      46477      21880 /dev/rhdiskpower12             02-AUG-14
AJXDG1_0001                    DROPPING      46477      21885 /dev/rhdiskpower13             02-AUG-14
AJXDG1_0002                    DROPPING      46477      21883 /dev/rhdiskpower14             02-AUG-14
AJXDG1_0003                    DROPPING      46477      21878 /dev/rhdiskpower15             02-AUG-14
AJXDG1_0004                    DROPPING      46477      21883 /dev/rhdiskpower16             02-AUG-14
AJXDG1_0005                    DROPPING      46477      21874 /dev/rhdiskpower17             02-AUG-14
AJXDG1_0006                    DROPPING      46477      21886 /dev/rhdiskpower18             02-AUG-14
AJXDG1_0007                    DROPPING      46477      21879 /dev/rhdiskpower19             02-AUG-14
AJXDG1_0008                    DROPPING      46477      21879 /dev/rhdiskpower20             02-AUG-14
AJXDG1_0009                    DROPPING      46477      21885 /dev/rhdiskpower21             02-AUG-14
AJXDG1_0010                    DROPPING      46477      21887 /dev/rhdiskpower22             02-AUG-14
AJXDG1_0011                    DROPPING      46477      21884 /dev/rhdiskpower23             02-AUG-14
AJXDG1_0012                    DROPPING      46477      21881 /dev/rhdiskpower24             02-AUG-14
AJXDG1_0013                    DROPPING      46477      21880 /dev/rhdiskpower25             02-AUG-14
AJXDG1_0014                    DROPPING      46477      21885 /dev/rhdiskpower26             02-AUG-14
AJXDG1_0015                    DROPPING      46477      21877 /dev/rhdiskpower27             02-AUG-14
AJXDG1_0017                    NORMAL        46477      46475 /dev/rhdiskpower59             21-APR-16
AJXDG1_0018                    NORMAL        46477      46475 /dev/rhdiskpower60             21-APR-16
AJXDG1_0019                    NORMAL        46477      46475 /dev/rhdiskpower61             21-APR-16
AJXDG1_0020                    NORMAL        46477      46475 /dev/rhdiskpower62             21-APR-16
AJXDG1_0021                    NORMAL        46477      46475 /dev/rhdiskpower63             21-APR-16
AJXDG1_0022                    NORMAL        46477      46475 /dev/rhdiskpower64             21-APR-16
AJXDG1_0023                    NORMAL        46477      46475 /dev/rhdiskpower65             21-APR-16
AJXDG1_0024                    NORMAL        46477      46475 /dev/rhdiskpower66             21-APR-16
AJXDG1_0025                    NORMAL        46477      46475 /dev/rhdiskpower67             21-APR-16
AJXDG1_0026                    NORMAL        46477      46475 /dev/rhdiskpower68             21-APR-16
AJXDG1_0027                    NORMAL        46477      46475 /dev/rhdiskpower69             21-APR-16
AJXDG1_0028                    NORMAL        46477      46475 /dev/rhdiskpower70             21-APR-16
AJXDG1_0029                    NORMAL        46477      46475 /dev/rhdiskpower71             21-APR-16
AJXDG1_0030                    NORMAL        46477      46475 /dev/rhdiskpower72             21-APR-16
AJXDG1_0031                    NORMAL        46477      46475 /dev/rhdiskpower73             21-APR-16
AJXDG1_0016                    NORMAL        46477      46475 /dev/rhdiskpower58             21-APR-16


---3.2 發起rebalance,檢視rebalance進度
alter diskgroup ajxdg1 rebalance power 500;


select * from v$asm_operation;
GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           1 REBAL RUN         500        500      46040     393494      15207          22    <---est_rate=15207表示每分鐘正在被move的AU個數,AU_SIZE=1M時表示每分鐘15GB/分鐘,摺合250MB/s

***調高優先順序,觀察同步速度是否加快

alter diskgroup ajxdg1 rebalance power 999;

select * from v$asm_operation;


GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           1 REBAL RUN         999        999      13462     236076      15482          14     <---改變了優先順序,相當於將原來優先順序為500的rebalance先停止然後以新的優先順序發起所以EST_WORK值比一開始時小,刨掉了已經完成同步的AU


select * from v$asm_operation;


GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           1 REBAL RUN         999        999      13822     236076      15469          14


看出調整為999之後速率並未明顯上升,可能是因為IO請求已經出現了排隊
powermt display every=1
Symmetrix logical device count=118
==============================================================================
----- Host Bus Adapters ---------  ------ I/O Paths -----  ------ Stats ------
###  HW Path                       Summary   Total   Dead  IO/Sec Q-IOs Errors
==============================================================================
   0 fscsi0                        optimal     118      0     117     1      0   <---Q-IOs!=0
   1 fscsi2                        optimal     118      0     132     2      0
   2 fscsi4                        optimal     118      0     127     2      0
   3 fscsi6                        optimal     118      0     124     1      0


---3.3 rebalance完成後的結果檢查
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG1');


NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG1_0017                    NORMAL        46477      21882 /dev/rhdiskpower59             21-APR-16
AJXDG1_0018                    NORMAL        46477      21882 /dev/rhdiskpower60             21-APR-16
AJXDG1_0019                    NORMAL        46477      21882 /dev/rhdiskpower61             21-APR-16
AJXDG1_0020                    NORMAL        46477      21881 /dev/rhdiskpower62             21-APR-16
AJXDG1_0021                    NORMAL        46477      21881 /dev/rhdiskpower63             21-APR-16
AJXDG1_0022                    NORMAL        46477      21882 /dev/rhdiskpower64             21-APR-16
AJXDG1_0023                    NORMAL        46477      21882 /dev/rhdiskpower65             21-APR-16
AJXDG1_0024                    NORMAL        46477      21881 /dev/rhdiskpower66             21-APR-16
AJXDG1_0025                    NORMAL        46477      21882 /dev/rhdiskpower67             21-APR-16
AJXDG1_0026                    NORMAL        46477      21882 /dev/rhdiskpower68             21-APR-16
AJXDG1_0027                    NORMAL        46477      21882 /dev/rhdiskpower69             21-APR-16
AJXDG1_0028                    NORMAL        46477      21882 /dev/rhdiskpower70             21-APR-16
AJXDG1_0029                    NORMAL        46477      21882 /dev/rhdiskpower71             21-APR-16
AJXDG1_0030                    NORMAL        46477      21881 /dev/rhdiskpower72             21-APR-16
AJXDG1_0031                    NORMAL        46477      21880 /dev/rhdiskpower73             21-APR-16
AJXDG1_0016                    NORMAL        46477      21882 /dev/rhdiskpower58             21-APR-16


4、將hdiskpower74~hdiskpower89新增到AJXDG2
AJXDG1: 透過shell指令碼獲取SQL指令碼
i=74
while [ $i -lt 90 ]
do
echo "alter diskgroup AJXDG2 add disk '/dev/rhdiskpower${i}' rebalance power 0;";
(( i=i+1 ))
done


得到如下新增新盤的命令直接執行:
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower74' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower75' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower76' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower77' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower78' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower79' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower80' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower81' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower82' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower83' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower84' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower85' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower86' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower87' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower88' rebalance power 0;
alter diskgroup AJXDG2 add disk '/dev/rhdiskpower89' rebalance power 0;


執行如下SQL:
select 'alter diskgroup '||d.name||' drop disk '||s.name ||' rebalance power 0;' from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2');


得到drop old disk的command
alter diskgroup AJXDG2 drop disk AJXDG2_0000 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0001 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0002 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0003 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0004 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0005 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0006 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0007 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0008 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0009 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0010 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0011 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0012 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0013 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0014 rebalance power 0;
alter diskgroup AJXDG2 drop disk AJXDG2_0015 rebalance power 0;


  ---4.1 觀察rebalance前的磁碟狀態
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2');
NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG2_0000                    DROPPING      46477      20092 /dev/rhdiskpower28             02-AUG-14
AJXDG2_0001                    DROPPING      46477      20091 /dev/rhdiskpower29             02-AUG-14
AJXDG2_0002                    DROPPING      46477      20088 /dev/rhdiskpower30             02-AUG-14
AJXDG2_0003                    DROPPING      46477      20090 /dev/rhdiskpower31             02-AUG-14
AJXDG2_0004                    DROPPING      46477      20092 /dev/rhdiskpower32             02-AUG-14
AJXDG2_0005                    DROPPING      46477      20097 /dev/rhdiskpower33             02-AUG-14
AJXDG2_0006                    DROPPING      46477      20092 /dev/rhdiskpower34             02-AUG-14
AJXDG2_0007                    DROPPING      46477      20089 /dev/rhdiskpower35             02-AUG-14
AJXDG2_0008                    DROPPING      46477      20092 /dev/rhdiskpower36             02-AUG-14
AJXDG2_0009                    DROPPING      46477      20089 /dev/rhdiskpower37             02-AUG-14
AJXDG2_0010                    DROPPING      46477      20089 /dev/rhdiskpower38             02-AUG-14
AJXDG2_0011                    DROPPING      46477      20089 /dev/rhdiskpower39             02-AUG-14
AJXDG2_0012                    DROPPING      46477      20091 /dev/rhdiskpower40             02-AUG-14
AJXDG2_0013                    DROPPING      46477      20094 /dev/rhdiskpower41             02-AUG-14
AJXDG2_0014                    DROPPING      46477      20089 /dev/rhdiskpower42             02-AUG-14
AJXDG2_0015                    DROPPING      46477      20087 /dev/rhdiskpower43             02-AUG-14
AJXDG2_0016                    NORMAL        46477      46475 /dev/rhdiskpower74             22-APR-16
AJXDG2_0028                    NORMAL        46477      46475 /dev/rhdiskpower86             22-APR-16
AJXDG2_0029                    NORMAL        46477      46475 /dev/rhdiskpower87             22-APR-16
AJXDG2_0030                    NORMAL        46477      46475 /dev/rhdiskpower88             22-APR-16
AJXDG2_0031                    NORMAL        46477      46475 /dev/rhdiskpower89             22-APR-16
AJXDG2_0024                    NORMAL        46477      46475 /dev/rhdiskpower82             22-APR-16
AJXDG2_0025                    NORMAL        46477      46475 /dev/rhdiskpower83             22-APR-16
AJXDG2_0026                    NORMAL        46477      46475 /dev/rhdiskpower84             22-APR-16
AJXDG2_0027                    NORMAL        46477      46475 /dev/rhdiskpower85             22-APR-16
AJXDG2_0021                    NORMAL        46477      46475 /dev/rhdiskpower79             22-APR-16
AJXDG2_0022                    NORMAL        46477      46475 /dev/rhdiskpower80             22-APR-16
AJXDG2_0023                    NORMAL        46477      46475 /dev/rhdiskpower81             22-APR-16
AJXDG2_0017                    NORMAL        46477      46475 /dev/rhdiskpower75             22-APR-16
AJXDG2_0018                    NORMAL        46477      46475 /dev/rhdiskpower76             22-APR-16
AJXDG2_0019                    NORMAL        46477      46475 /dev/rhdiskpower77             22-APR-16
AJXDG2_0020                    NORMAL        46477      46475 /dev/rhdiskpower78             22-APR-16


  ---4.2 發起rebalance,檢視rebalance進度
alter diskgroup ajxdg2 rebalance power 1011;


SQL> select * from v$asm_operation


GROUP_NUMBER OPERA STAT      POWER     ACTUAL      SOFAR   EST_WORK   EST_RATE EST_MINUTES ERROR_CODE
------------ ----- ---- ---------- ---------- ---------- ---------- ---------- ----------- --------------------------------------------
           2 REBAL RUN        1011       1011       7834     422149      14768          28


  ---4.3 rebalance完成後的結果檢查
col path format a30
col header_status format a10
col mode_status format a20
set linesize 130 pagesize 130
select s.name,s.state,s.total_mb,s.free_mb,path,create_date from v$asm_disk_stat s,v$asm_diskgroup_stat d where s.group_number=d.group_number and d.name in ('AJXDG2') order by s.name;
NAME                           STATE      TOTAL_MB    FREE_MB PATH                           CREATE_DA
------------------------------ -------- ---------- ---------- ------------------------------ ---------
AJXDG2_0016                    NORMAL        46477      20090 /dev/rhdiskpower74             22-APR-16
AJXDG2_0017                    NORMAL        46477      20091 /dev/rhdiskpower75             22-APR-16
AJXDG2_0018                    NORMAL        46477      20091 /dev/rhdiskpower76             22-APR-16
AJXDG2_0019                    NORMAL        46477      20091 /dev/rhdiskpower77             22-APR-16
AJXDG2_0020                    NORMAL        46477      20091 /dev/rhdiskpower78             22-APR-16
AJXDG2_0021                    NORMAL        46477      20090 /dev/rhdiskpower79             22-APR-16
AJXDG2_0022                    NORMAL        46477      20091 /dev/rhdiskpower80             22-APR-16
AJXDG2_0023                    NORMAL        46477      20091 /dev/rhdiskpower81             22-APR-16
AJXDG2_0024                    NORMAL        46477      20090 /dev/rhdiskpower82             22-APR-16
AJXDG2_0025                    NORMAL        46477      20091 /dev/rhdiskpower83             22-APR-16
AJXDG2_0026                    NORMAL        46477      20091 /dev/rhdiskpower84             22-APR-16
AJXDG2_0027                    NORMAL        46477      20091 /dev/rhdiskpower85             22-APR-16
AJXDG2_0028                    NORMAL        46477      20090 /dev/rhdiskpower86             22-APR-16
AJXDG2_0029                    NORMAL        46477      20090 /dev/rhdiskpower87             22-APR-16
AJXDG2_0030                    NORMAL        46477      20091 /dev/rhdiskpower88             22-APR-16
AJXDG2_0031                    NORMAL        46477      20091 /dev/rhdiskpower89             22-APR-16


5、(可選),將原先的Votedisk從external redundancy的DG遷移到normal redundancy的DG
grid@qc740702a:/home/grid>crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   a48d1185b7be4f17bf06f6dd01e84d88 (/dev/rhdiskpower106) [SYSDG2]    <---原來voting disk只有一塊盤
Located 1 voting disk(s).


***建立normal redundancy的diskgroup
create diskgroup sysdg3 normal redundancy failgroup vdfg1 disk '/dev/rhdiskpower110','/dev/rhdiskpower111' failgroup vdfg2 disk '/dev/rhdiskpower112','/dev/rhdiskpower113' failgroup vdfg3 disk '/dev/rhdiskpower114','/dev/rhdiskpower115' attribute 'compatible.asm'='11.2.0.3','compatible.rdbms'='11.2.0.0.0' ;


***遷移到sysdg3
grid@qc740702a:/home/grid>crsctl replace votedisk +sysdg3
Successful addition of voting disk 72d81026f64d4f86bf7a7ec0845201db.
Successful addition of voting disk 2fa55993ec494fa8bf839661e2c43e89.
Successful addition of voting disk 1127e47a60084f8ebf6f2292143710c0.
Successful deletion of voting disk a48d1185b7be4f17bf06f6dd01e84d88.
Successfully replaced voting disk group with +sysdg3.
CRS-4266: Voting file(s) successfully replaced


grid@qc740702a:/home/grid>crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   72d81026f64d4f86bf7a7ec0845201db (/dev/rhdiskpower110) [SYSDG3] <---現在voting disk有三塊了
 2. ONLINE   2fa55993ec494fa8bf839661e2c43e89 (/dev/rhdiskpower112) [SYSDG3]
 3. ONLINE   1127e47a60084f8ebf6f2292143710c0 (/dev/rhdiskpower114) [SYSDG3]
Located 3 voting disk(s).

來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/53956/viewspace-2087627/,如需轉載,請註明出處,否則將追究法律責任。

相關文章