oracle RAC 更換儲存遷移資料
我們利用ASM rebalance特性實現更換儲存遷移資料的需求
基本零當機時間（操作步驟總結）
1）保證新儲存和RAC當前節點間的可用性；
2）新儲存劃分LUN，可以重新規劃儲存方案；
3）遷移OCR和表決盤
4）給現有ASM磁碟組新增ASM磁碟(新儲存分配的)，充分利用ASM REBALANCE技術
5）刪除原儲存的ASM磁碟
6）觀察期

1 當前儲存資訊
以下ASM磁碟組、OCR、VOTE資訊：
ASM磁碟組：
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 4096 1048576 3071982 3071091 298 1535396 0 N BACK/
MOUNTED NORMAL N 512 4096 1048576 4095976 1561759 633568 464095 0 N DATA/
MOUNTED NORMAL N 512 4096 1048576 102396 101470 326 50572 0 N OCR/

當前ASM有BACK，DATA，OCR三個磁碟組，總大小7TB，磁碟組主要存放資料檔案和歸檔日誌檔案，COR檔案，以下是各磁碟組磁碟資訊：
SQL> select NAME,PATH,total_mb,free_mb from v$asm_disk;
NAME PATH TOTAL_MB FREE_MB
------------------------------ ------------------------------ ---------- ----------
BACK_VOL1 ORCL:BACK_VOL1 1023994 390436
DATA_VOL1 ORCL:DATA_VOL1 1023994 390450
DATA_VOL2 ORCL:DATA_VOL2 1023994 390447
DATA_VOL3 ORCL:DATA_VOL3 1023994 390426
DATA_VOL4 ORCL:DATA_VOL4 1023994 1023697
DATA_VOL5 ORCL:DATA_VOL5 1023994 1023698
DATA_VOL6 ORCL:DATA_VOL6 1023994 1023696
OCR_VOL1 ORCL:OCR_VOL1 31376 31075
OCR_VOL2 ORCL:OCR_VOL2 31376 31077
OCR_VOL3 ORCL:OCR_VOL3 39644 39318
10 rows selected.

OCR&VOTE資訊：
[grid@oracle1 bin]$ ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2720
Available space (kbytes) : 259400
ID : 2006438789
Device/File Name : +OCR
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
由於OCR和VOTEDISK都和ASM放在同一個儲存上，所以OCR和VOTEDISK也需要遷移到新的儲存上。

2 新儲存磁碟分割槽
要求：（由儲存工程師操作）
2.1. 共享儲存，兩臺伺服器都可以看到新儲存分配的磁碟空間。
2.2. 如之前ASM磁碟組所有儲存分割槽大小，個數保持一致。

3 劃分後磁碟分割槽
[root@oracle1 sbin]# fdisk -l

Disk /dev/cciss/c0d0: 1000.1 GB, 1000171331584 bytes
255 heads, 63 sectors/track, 121597 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 13 104391 83 Linux
/dev/cciss/c0d0p2 14 121597 976623480 8e Linux LVM

Disk /dev/sda: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 130541 1048570551 83 Linux

Disk /dev/sdb: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 130541 1048570551 83 Linux

Disk /dev/sdc: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 130541 1048570551 83 Linux

Disk /dev/sdd: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdd1 1 130541 1048570551 83 Linux

Disk /dev/sde: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sde1 1 130541 1048570551 83 Linux

Disk /dev/sdf: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdf1 1 130541 1048570551 83 Linux

Disk /dev/sdg: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdg1 1 130541 1048570551 83 Linux

Disk /dev/sdh: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdh1 1 4000 32129968+ 83 Linux
/dev/sdh2 4001 8000 32130000 83 Linux
/dev/sdh3 8001 13054 40596255 83 Linux

WARNING: The size of this disk is2.9 TB (2919504019456 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Disk /dev/sdi: 2919.5 GB, 2919504019456 bytes
255 heads, 63 sectors/track, 354942 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdi1 1 130000 1044224968+ 83 Linux
/dev/sdi2 130001 267349 1103255842+ 83 Linux

Disk /dev/sdj: 1073.7 GB, 1073741824000 bytes --------------------------------新加盤
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdj doesn't contain a valid partition table

Disk /dev/sdk: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdk doesn't contain a valid partition table

Disk /dev/sdl: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdl doesn't contain a valid partition table

Disk /dev/sdm: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdm doesn't contain a valid partition table

Disk /dev/sdn: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdn doesn't contain a validpartition table

Disk /dev/sdo: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdo doesn't conain a valid partition table

Disk /dev/sdp: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdp doesn't contain a valid partition table

Disk /dev/sdq: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdq doesn't contain a valid partition table

Disk /dev/sdr: 2919.5 GB, 2919504019456 bytes
255 heads, 63 sectors/track, 354942 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdr doesn't contain a valid partition table

Disk /dev/sds: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sds1 1 39162 314568733+ 8e Linux LVM

Disk /dev/sdt: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdt1 1 39162 314568733+ 8e Linux LVM

4 配置ASM新磁碟
/etc/init.d/oracleasm createdisk DATA_VOL01 /dev/sdj1
/etc/init.d/oracleasm createdisk DATA_VOL02 /dev/sdk1
/etc/init.d/oracleasm createdisk DATA_VOL03 /dev/sdl1
/etc/init.d/oracleasm createdisk DATA_VOL04 /dev/sdm1
/etc/init.d/oracleasm createdisk DATA_VOL05 /dev/sdn1
/etc/init.d/oracleasm createdisk DATA_VOL06 /dev/sdo1
/etc/init.d/oracleasm createdisk BACK_VOL01 /dev/sdp1
/etc/init.d/oracleasm createdisk OCR_VOL4 /dev/sdq1
/etc/init.d/oracleasm createdisk OCR_VOL5 /dev/sdq2
/etc/init.d/oracleasm createdisk OCR_VOL6 /dev/sdq3

5 建立新的OCRNEW磁碟組
su – grid
sqlplus / as sysasm
CREATE DISKGROUP OCRNEW NORMAL REDUNDANCY
DISK 'ORCL:OCR_VOL4' NAME VOL4
DISK 'ORCL:OCR_VOL5' NAME VOL5
DISK 'ORCL:OCR_VOL6' NAME VOL6 ATTRIBUTE 'compatible.asm'='11.2';

6 新增OCR資訊到OCRNEW
[root@oracle1 bin]# ./ocrconfig -add +OCRNEW
[root@oracle1 bin]# ./ocrcheck -config
Oracle Cluster Registry configuration is :
Device/File Name : +OCR
Device/File Name : +OCRNEW
[root@oracle1 bin]# more /etc/oracle/ocr.loc
#Device/file getting replaced by device +OCRNEW
ocrconfig_loc=+OCR
ocrmirrorconfig_loc=+OCRNEW
local_only=false
[root@oracle1 bin]#

可以看到OCRNEW 磁碟組已經成功新增到OCR磁碟資訊中

遷移vote 檔案
當前votedisk資訊
[grid@oracle1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 14f694d9d4414f9ebf85d3ce6b9aef0b (ORCL:OCR_VOL1) [OCR]
2. ONLINE 9f9ee7281c954f8abfcc6e88c33257ac (ORCL:OCR_VOL2) [OCR]
3. ONLINE 38114fd602194fa9bf4d05655b3d89b7 (ORCL:OCR_VOL3) [OCR]
Located 3 voting disk(s).

[grid@oracle1 ~]$ crsctl replace votedisk +OCRNEW
Successful addition of voting disk 00634ef593ee4f92bf48e8c089cb5565.
Successful addition of voting disk 232159722de04f67bf03a78b757e3bec.
Successful addition of voting disk a340d5b23aac4f6fbf9f7b1d59088fa5.
Successful deletion of voting disk 14f694d9d4414f9ebf85d3ce6b9aef0b.
Successful deletion of voting disk 9f9ee7281c954f8abfcc6e88c33257ac.
Successful deletion of voting disk 38114fd602194fa9bf4d05655b3d89b7.
Successfully replaced voting disk group with +OCRNEW.
CRS-4266: Voting file(s) successfully replaced

7 建立ASM例項spfile到OCR_NEW
建立ASM例項spfile到新建立的OCR_NEW ASM磁碟組上（在一個節點grid使用者登入ASM例項執行）
SQL> create pfile='/home/grid/asmpfile.ora' from spfile;
File created.

SQL> create spfile='+OCRNEW' from pfile='/home/grid/asmpfile.ora';
File created.

8 刪除ASM磁碟組OCR
[root@oracle1 bin]# ./ocrconfig -delete +OCR
檢視OCR和VOTE新狀態與位置
[root@oracle1 bin]# ./ocrcheck && ./crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2768
Available space (kbytes) : 259352
ID : 2006438789
Device/File Name : +OCRNEW
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 00634ef593ee4f92bf48e8c089cb5565 (ORCL:OCR_VOL4) [OCRNEW]
2. ONLINE 232159722de04f67bf03a78b757e3bec (ORCL:OCR_VOL5) [OCRNEW]
3. ONLINE a340d5b23aac4f6fbf9f7b1d59088fa5 (ORCL:OCR_VOL6) [OCRNEW]
Located 3 voting disk(s).

SYS@+ASM1> alter diskgroup OCR dismount;

Diskgroup altered.
SYS@+ASM2> drop diskgroup OCR including contents;

Diskgroup dropped.
SYS@+ASM2> SQL> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB V COMPATIBILITY
------------ ------------------------------ ----------- ------ ---------- ---------- - ------------------------------------------------------------
1 BACK MOUNTED NORMAL 3071982 3070675 N 11.2.0.0.0
2 DATA MOUNTED NORMAL 4095976 1561759 N 11.2.0.0.0
3 OCRNEW MOUNTED NORMAL 102396 101470 N 11.2.0.0.0

SYS@+ASM2> SQL> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER;
GROUP_NUMBER DISK_NUMBER STATE REDUNDA TOTAL_MB FREE_MB NAME PATH FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ------------------------------ ------------------------------ ------------------------------
0 0 NORMAL UNKNOWN 0 0 ORCL:OCR_VOL1
0 1 NORMAL UNKNOWN 0 0 ORCL:OCR_VOL2
0 2 NORMAL UNKNOWN 0 0 ORCL:OCR_VOL3
1 1 NORMAL UNKNOWN 1023994 1023559 DATA_VOL5 ORCL:DATA_VOL5 DATA_VOL5
1 0 NORMAL UNKNOWN 1023994 1023559 DATA_VOL4 ORCL:DATA_VOL4 DATA_VOL4
1 2 NORMAL UNKNOWN 1023994 1023557 DATA_VOL6 ORCL:DATA_VOL6 DATA_VOL6
2 2 NORMAL UNKNOWN 1023994 390447 DATA_VOL2 ORCL:DATA_VOL2 DATA_VOL2
2 1 NORMAL UNKNOWN 1023994 390450 DATA_VOL1 ORCL:DATA_VOL1 DATA_VOL1
2 0 NORMAL UNKNOWN 1023994 390436 BACK_VOL1 ORCL:BACK_VOL1 BACK_VOL1
2 3 NORMAL UNKNOWN 1023994 390426 DATA_VOL3 ORCL:DATA_VOL3 DATA_VOL3
3 0 NORMAL UNKNOWN 31376 31075 VOL4 ORCL:OCR_VOL4 VOL4

GROUP_NUMBER DISK_NUMBER STATE REDUNDA TOTAL_MB FREE_MB NAME PATH FAILGROUP
------------ ----------- -------- ------- ---------- ---------- ------------------------------ ------------------------------ ------------------------------
3 1 NORMAL UNKNOWN 31376 31077 VOL5 ORCL:OCR_VOL5 VOL5
3 2 NORMAL UNKNOWN 39644 39318 VOL6 ORCL:OCR_VOL6 VOL6
13 rows selected.
至此整個OCR&VOTING遷移過程結束

9.這裡可以重啟下叢集crs測試OCR&VOTE是否遷移成功，當然你也可以選擇不重啟，個人建議重啟測試下。

檢視OCR&VOTE位置及ASM例項spfile位置
[root@oracle1 bin]# ./ocrcheck && ./crsctl query css votedisk
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2768
Available space (kbytes) : 259352
ID : 2006438789
Device/File Name : +OCRNEW
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 00634ef593ee4f92bf48e8c089cb5565 (ORCL:OCR_VOL4) [OCRNEW]
2. ONLINE 232159722de04f67bf03a78b757e3bec (ORCL:OCR_VOL5) [OCRNEW]
3. ONLINE a340d5b23aac4f6fbf9f7b1d59088fa5 (ORCL:OCR_VOL6) [OCRNEW]
Located 3 voting disk(s).

SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +OCRNEW/oracle-cluster/asmpara
meterfile/registry.253.8456918
87
SQL>

10 遷移資料磁碟組資料
SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL01' rebalance power 11;
Diskgroup altered.
SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL02' rebalance power 11;
Diskgroup altered.
SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL03' rebalance power 11;
Diskgroup altered.
SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL04' rebalance power 11;
Diskgroup altered.
SQL> alter diskgroup back add disk 'ORCL:DATA_VOL05' rebalance power 11;
Diskgroup altered.
SQL> alter diskgroup back add disk 'ORCL:DATA_VOL06' rebalance power 11;
SQL> alter diskgroup BACK add disk 'ORCL:BACK_VOL01' rebalance power 11;
Diskgroup altered.
由於指定rebalance power 11，ASM會自動均衡ASM磁碟組DATA裡面存放的資料在各ASM磁碟的分佈。

當rebalance結束後，查詢V$ASM_OPERATION檢視，將不會返回資訊。
SQL> select * from V$ASM_OPERATION;
no rows selected

11 刪除資料磁碟組上的舊磁碟
alter diskgroup data drop disk 'BACK_VOL1' rebalance power 11;
alter diskgroup data drop disk 'DATA_VOL2' rebalance power 11;
alter diskgroup data drop disk 'DATA_VOL3' rebalance power 11;
alter diskgroup back drop disk 'DATA_VOL4' rebalance power 11;
alter diskgroup back drop disk 'DATA_VOL5' rebalance power 11;
alter diskgroup back drop disk 'DATA_VOL6' rebalance power 11;
ASM不但向磁碟組中加入新磁碟時會做rebalance，在刪除ASM磁碟時也會rebalance，將該磁碟上的資料rebalance到其他該磁碟組的磁碟中。

按照此方法刪除ASM磁碟後，ASM的所有資料都已經存放在新的儲存上。
09:40:38 SQL> select a.NAME GROUP_NAME,a.TOTAL_MB,a.FREE_MB GROUP_FREE_MB,b.OS_MB,b.FREE_MB,b.name,b.path from v$asm_diskgroup a,v$asm_disk b where a.GROUP_NUMBER=b.GROUP_NUMBER;
GROUP_NAME TOTAL_MB GROUP_FREE_MB OS_MB FREE_MB NAME PATH
------------------------------ ---------- ------------- ---------- ---------- ------------------------------ ----------------------------------------
BACK 3071982 3070868 1023994 1023622 DATA_VOL05 ORCL:DATA_VOL05
BACK 3071982 3070868 1023994 1023624 DATA_VOL06 ORCL:DATA_VOL06
OCRNEW 102396 101470 31376 31075 VOL4 ORCL:OCR_VOL4
OCRNEW 102396 101470 31376 31077 VOL5 ORCL:OCR_VOL5
OCRNEW 102396 101470 39644 39318 VOL6 ORCL:OCR_VOL6
DATA 4095976 1561759 1023994 390437 DATA_VOL01 ORCL:DATA_VOL01
DATA 4095976 1561759 1023994 390440 DATA_VOL02 ORCL:DATA_VOL02
DATA 4095976 1561759 1023994 390443 DATA_VOL03 ORCL:DATA_VOL03
DATA 4095976 1561759 1023994 390439 DATA_VOL04 ORCL:DATA_VOL04
BACK 3071982 3070868 1023994 1023622 BACK_VOL01 ORCL:BACK_VOL01

12 刪除舊ASM盤配置資訊
[root@oracle1 bin]# oracleasm listdisks
BACK_VOL01
BACK_VOL1
DATA_VOL01
DATA_VOL02
DATA_VOL03
DATA_VOL04
DATA_VOL05
DATA_VOL06
DATA_VOL1
DATA_VOL2
DATA_VOL3
DATA_VOL4
DATA_VOL5
DATA_VOL6
OCR_VOL4
OCR_VOL5
OCR_VOL6

oracleasm deletedisk DAA_VOL1
oracleasm deletedisk DATA_VOL2
oracleasm deletedisk DATA_VOL3
oracleasm deletedisk DATA_VOL4
oracleasm deletedisk DATA_VOL5
oracleasm deletedisk DATA_VOL6
oracleasm deletedisk BACK_VOL1
oracleasm deletedisk OCR_VOL1
oracleasm deletedisk OCR_VOL2
oracleasm deletedisk OCR_VOL3

[root@oracle2 bin]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Cleaning disk "BACK_VOL1"
Cleaning disk "DATA_VOL1"
Cleaning disk "DATA_VOL2"
Cleaning disk "DATA_VOL3"
Cleaning disk "DATA_VOL4"
Cleaning disk "DATA_VOL5"
Cleaning disk "DATA_VOL6"
Scanning system for ASM disks...
You have new mail in /var/spool/mail/root

[root@oracle2 bin]# oracleasm listdisks
BACK_VOL01
DATA_VOL01
DATA_VOL02
DATA_VOL03
DATA_VOL04
DATA_VOL05
DATA_VOL06
OCR_VOL4
OCR_VOL5
OCR_VOL6
至此整個舊儲存資料全部遷移到新的儲存上了。

oracle RAC 更換儲存遷移資料

相關文章