Oracle ASM ACFS disk group rebalance
從Oracle 11.2開始，一個ASM磁碟組可以被用來建立一個或多個叢集檔案系統。這就是Oracle ASM叢集檔案系統或Oracle ACFS。這個功能透過在ASM磁碟組中建立特定的volume檔案來實現，然後作為塊裝置給作業系統來使用，再在這些塊裝置上建立檔案系統。下面將介紹ACFS volume檔案的rebalance,mirror與extent管理。
測試環境如下:
.64-bit Oracle Linux 5.4
.Oracle Restart and ASM version 11.2.0.4.0 - 64bit
設定ACFS volumes
單例項載入ADVM/ACFS驅動的命令如下，RAC環境不需要，因為已經預設載入
[root@jyrac1 bin]# ./acfsroot install ACFS-9300: ADVM/ACFS distribution files found. ACFS-9118: oracleadvm.ko driver in use - cannot unload. ACFS-9312: Existing ADVM/ACFS installation detected. ACFS-9118: oracleadvm.ko driver in use - cannot unload. ACFS-9314: Removing previous ADVM/ACFS installation. ACFS-9315: Previous ADVM/ACFS components successfully removed. ACFS-9307: Installing requested ADVM/ACFS software. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9321: Creating udev for ADVM/ACFS. ACFS-9323: Creating module dependencies - this may take some time. ACFS-9154: Loading 'oracleacfs.ko' driver. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. ACFS-9156: Detecting control device '/dev/ofsctl'. ACFS-9309: ADVM/ACFS installation correctness verified. [root@jyrac1 bin]# ./acfsload start ACFS-9391: Checking for existing ADVM/ACFS installation. ACFS-9392: Validating ADVM/ACFS installation files for operating system. ACFS-9393: Verifying ASM Administrator setup. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. ACFS-9156: Detecting control device '/dev/ofsctl'. ACFS-9322: completed [root@jyrac1 bin]# ./acfsdriverstate version ACFS-9325: Driver OS kernel version = 2.6.18-8.el5(x86_64). ACFS-9326: Driver Oracle version = 130707.
建立一個用來建立ASM叢集檔案系統的磁碟組
SQL> create diskgroup acfs disk '/dev/raw/raw5','/dev/raw/raw6' attribute 'COMPATIBLE.ASM' = '11.2', 'COMPATIBLE.ADVM' = '11.2'; Diskgroup created.
雖然一個磁碟組可以用來儲存資料庫檔案與ACFS volume files，但是建議為ACFS volume建立一個單獨的磁碟組。這將提供角色/功能分離與對資料庫檔案效能有潛在好處。
檢查所有磁碟組的AU大小
SQL> select group_number "Group#", name "Name", allocation_unit_size "AU size" from v$asm_diskgroup_stat;
Group# Name AU size
---------- ------------------------------------------------------------ ----------
1 ARCHDG 1048576
2 CRSDG 1048576
3 DATADG 1048576
4 ACFS 1048576
對於所有磁碟組來說預設的AU大小為1MB，當後面介紹volume file的區大小時會使用到AU大小。
在磁碟組ACFS中建立三個volume
[grid@jyrac1 ~]$ asmcmd volcreate -G ACFS -s 1G ACFS_VOL1 [grid@jyrac1 ~]$ asmcmd volcreate -G ACFS -s 1G ACFS_VOL2 [grid@jyrac1 ~]$ asmcmd volcreate -G ACFS -s 1G ACFS_VOL3
檢視volume資訊
[grid@jyrac1 ~]$ asmcmd volinfo -a
Diskgroup Name: ACFS
Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-10
State: ENABLED
Size (MB): 1024
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
Volume Name: ACFS_VOL2
Volume Device: /dev/asm/acfs_vol2-10
State: ENABLED
Size (MB): 1024
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
Volume Name: ACFS_VOL3
Volume Device: /dev/asm/acfs_vol3-10
State: ENABLED
Size (MB): 1024
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
在volume建立這後會自動被啟用。當伺服器重啟之後可能需要手動載入ADVM/ACFS驅動(acfsload start)並啟用volume(asmcmd volenable -a)。
對於每個volume，ASM將建立一個volume file。在冗餘磁碟組中，每個卷將有一個dirty region logging(DRL)檔案
SQL> select file_number "File#", volume_name "Volume", volume_device "Device", size_mb "MB", drl_file_number "DRL#" from v$asm_volume; File# Volume Device MB DRL# ----- ---------------------------------------- ---------------------------------------- ---------- ---------- 257 ACFS_VOL1 /dev/asm/acfs_vol1-10 1024 256 259 ACFS_VOL2 /dev/asm/acfs_vol2-10 1024 258 261 ACFS_VOL3 /dev/asm/acfs_vol3-10 1024 260
除了卷名，裝置名與大小之外，還顯示了ASM檔案號257,259,261給卷裝置使用，ASM檔案號256,258,260給DRL檔案使用。
查詢卷檔案的AU分佈情況
SQL> select
2 xnum_kffxp, -- virtual extent number
3 pxn_kffxp, -- physical extent number
4 disk_kffxp, -- disk number
5 au_kffxp -- allocation unit number
6 from x$kffxp
7 where number_kffxp=261-- asm file 256
8 and group_kffxp=4 -- group number 1
9 order by 1,2,3;
XNUM_KFFXP PXN_KFFXP DISK_KFFXP AU_KFFXP
---------- ---------- ---------- ----------
0 0 0 2160
0 1 1 2160
1 2 1 2168
1 3 0 2168
2 4 0 2176
2 5 1 2176
3 6 1 2184
3 7 0 2184
4 8 0 2192
4 9 1 2192
5 10 1 2200
5 11 0 2200
6 12 0 2208
6 13 1 2208
......
124 248 0 3152
124 249 1 3152
125 250 1 3160
125 251 0 3160
126 252 0 3168
126 253 1 3168
127 254 1 3176
127 255 0 3176
2147483648 0 0 2156
2147483648 1 1 2156
2147483648 2 65534 4294967294
259 rows selected.
當在normal冗餘磁碟組中建立卷，那麼卷的每個區同樣也會被映象。可以看到卷檔案261有128個區。卷大小為1GB，這意味著每個區大小為8MB或8個AU。卷檔案有屬於它自己的區大小，不像標準的ASM檔案繼承來自磁碟組AU大小來初始化區大小。
在邏輯卷裝置上建立ASM叢集檔案系統（ACFS）
[grid@jyrac1 ~]$ /sbin/mkfs -t acfs /dev/asm/acfs_vol1-10 mkfs.acfs: version = 11.2.0.4.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfs_vol1-10 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete. [grid@jyrac1 ~]$ /sbin/mkfs -t acfs /dev/asm/acfs_vol2-10 mkfs.acfs: version = 11.2.0.4.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfs_vol2-10 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete. [grid@jyrac1 ~]$ /sbin/mkfs -t acfs /dev/asm/acfs_vol3-10 mkfs.acfs: version = 11.2.0.4.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfs_vol3-10 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete. [root@jyrac1 /]# mkdir /acfs1 [root@jyrac1 /]# mkdir /acfs2 [root@jyrac1 /]# mkdir /acfs3 [root@jyrac1 /]# chown -R grid:oinstall /acfs1 [root@jyrac1 /]# chown -R grid:oinstall /acfs2 [root@jyrac1 /]# chown -R grid:oinstall /acfs3 [root@jyrac1 /]# chmod -R 777 /acfs1 [root@jyrac1 /]# chmod -R 777 /acfs2 [root@jyrac1 /]# chmod -R 777 /acfs3 [root@jyrac1 /]# mount -t acfs /dev/asm/acfs_vol1-10 /acfs1 [root@jyrac1 /]# mount -t acfs /dev/asm/acfs_vol2-10 /acfs2 [root@jyrac1 /]# mount -t acfs /dev/asm/acfs_vol3-10 /acfs3 [root@jyrac1 /]# mount | grep acfs /dev/asm/acfs_vol1-10 on /acfs1 type acfs (rw) /dev/asm/acfs_vol2-10 on /acfs2 type acfs (rw) /dev/asm/acfs_vol3-10 on /acfs3 type acfs (rw)
複製一些檔案到新的檔案系統中
[grid@jyrac1 +asm]$ cp $ORACLE_BASE/diag/asm/+asm/+ASM1/trace/* /acfs1 [grid@jyrac1 +asm]$ cp $ORACLE_BASE/diag/asm/+asm/+ASM1/trace/* /acfs2 [grid@jyrac1 +asm]$ cp $ORACLE_BASE/diag/asm/+asm/+ASM1/trace/* /acfs3
檢查使用空間
[root@jyrac1 /]# df -h /acfs?
Filesystem Size Used Avail Use% Mounted on
/dev/asm/acfs_vol1-10
1.0G 105M 920M 11% /acfs1
/dev/asm/acfs_vol2-10
1.0G 105M 920M 11% /acfs2
/dev/asm/acfs_vol3-10
1.0G 105M 920M 11% /acfs3
現在向ACFS磁碟組新增磁碟組並監控rebalance操作
SQL> alter diskgroup ACFS add disk '/dev/raw/raw7'; Diskgroup altered.
從alert_+ASM1.log檔案中可以找到ARB0程式的PID為1074
[grid@jyrac1 trace]$ tail -f alert_+ASM1.log SQL> alter diskgroup ACFS add disk '/dev/raw/raw7' NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (4,2) to disk (/dev/raw/raw7) NOTE: requesting all-instance membership refresh for group=4 NOTE: initializing header on grp 4 disk ACFS_0002 NOTE: requesting all-instance disk validation for group=4 Thu Jan 12 14:54:45 2017 NOTE: skipping rediscovery for group 4/0xd98640a (ACFS) on local instance. NOTE: requesting all-instance disk validation for group=4 NOTE: skipping rediscovery for group 4/0xd98640a (ACFS) on local instance. Thu Jan 12 14:54:45 2017 GMON updating for reconfiguration, group 4 at 249 for pid 27, osid 18644 NOTE: group 4 PST updated. NOTE: initiating PST update: grp = 4 GMON updating group 4 at 250 for pid 27, osid 18644 NOTE: group ACFS: updated PST location: disk 0000 (PST copy 0) NOTE: group ACFS: updated PST location: disk 0001 (PST copy 1) NOTE: group ACFS: updated PST location: disk 0002 (PST copy 2) NOTE: PST update grp = 4 completed successfully NOTE: membership refresh pending for group 4/0xd98640a (ACFS) GMON querying group 4 at 251 for pid 18, osid 5012 NOTE: cache opening disk 2 of grp 4: ACFS_0002 path:/dev/raw/raw7 GMON querying group 4 at 252 for pid 18, osid 5012 SUCCESS: refreshed membership for 4/0xd98640a (ACFS) NOTE: starting rebalance of group 4/0xd98640a (ACFS) at power 1 SUCCESS: alter diskgroup ACFS add disk '/dev/raw/raw7' Starting background process ARB0 Thu Jan 12 14:54:48 2017 ARB0 started with pid=40, OS id=1074 NOTE: assigning ARB0 to group 4/0xd98640a (ACFS) with 1 parallel I/O cellip.ora not found. NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 2:2 for diskgroup 4 (ACFS) Thu Jan 12 14:55:00 2017 NOTE: Attempting voting file refresh on diskgroup ACFS NOTE: Refresh completed on diskgroup ACFS. No voting file found.
透過命令tail -f +ASM1_arb0_1074.trc來監控rebalance過程
*** 2017-01-12 14:55:18.731 ARB0 relocating file +ACFS.259.933075367 (86 entries) *** 2017-01-12 14:55:38.599 ARB0 relocating file +ACFS.259.933075367 (1 entries) ARB0 relocating file +ACFS.260.933075373 (17 entries) *** 2017-01-12 14:55:39.617 ARB0 relocating file +ACFS.261.933075373 (86 entries) *** 2017-01-12 14:55:59.106 ARB0 relocating file +ACFS.261.933075373 (1 entries) *** 2017-01-12 14:55:59.274 ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.257.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) *** 2017-01-12 14:56:00.201 ARB0 relocating file +ACFS.1.1 (1 entries) ARB0 relocating file +ACFS.7.1 (1 entries) ARB0 relocating file +ACFS.5.1 (1 entries) ARB0 relocating file +ACFS.8.1 (1 entries) ARB0 relocating file +ACFS.9.1 (1 entries) ARB0 relocating file +ACFS.6.1 (1 entries) ARB0 relocating file +ACFS.4.1 (1 entries) ARB0 relocating file +ACFS.4.1 (1 entries) ARB0 relocating file +ACFS.3.1 (1 entries) .....
可以看到每個ASM檔案的rebalance操作過程。這種操作行為與資料庫檔案是一樣的，ASM對每個檔案執行rebalance操作。ASM後設資料檔案(1-9)最先被rebalance。ASM然後對卷檔案號257,259,261，ASM檔案號256,258,260執行rebalance等等。
可以看到對卷檔案(與其它ASM檔案）執行rebalance操作並不是對儲存在相關檔案系統中的使用者檔案進行操作，而是對每個卷檔案執行rebalance操作。
ACFS磁碟組中的磁碟聯機操作
當一個ASM磁碟離線時，ASM將建立staleness registry and staleness directory來跟蹤磁碟聯機時需要修改的區。一旦磁碟聯機，ASM使用這些資訊來執行快速映象重新同步。這個功能對於ASM 11.2中的卷檔案是不可用的。相反，對於聯機的磁碟，ASM將重建整個磁碟內容。這就是為什麼對於儲存卷檔案的磁碟組執行磁碟聯機的效能要比儲存標準資料庫檔案的磁碟組執行磁碟聯機的效能差的原因。 對於卷檔案執行快速映象重新同步在ASM 12.1及以後的版本中是可以使用的。
小結:
ASM磁碟組可以被用來建立一般目錄的叢集檔案系統。ASM透過在磁碟組中建立卷檔案來實現，並將它們作為塊裝置提供給作業系統使用。現有的ASM磁碟組映象功能(normal與high冗餘)可以被用來在檔案系統級別保護使用者檔案。ASM透過映象卷檔案區來實現，這種方式也用於任何其它的ASM檔案。卷檔案有它自己的區大小，不像標準資料庫檔案繼承來自磁碟組的AU大小來初始化區大小。對儲存ASM叢集檔案系統卷的ASM磁碟組執行rebalance操作，實際上是對每個卷檔案執行rebalance操作，而不是對儲存在相關檔案系統中的單個使用者檔案執行rebalance操作。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26015009/viewspace-2132471/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle ASM spfile in a disk groupOracleASM
- Oracle ASM Disk Group AttributesOracleASM
- ASM Disk Group TemplateASM
- Oracle ACFS ( ASM Cluster File System )OracleASM
- Oracle ASM Rebalance執行過程OracleASM
- ASM disk group mount fails with ORA-15036ASMAI
- Oracle ASM Disk DirectoryOracleASM
- Oracle ASM Disk PartnerOracleASM
- ORA-15260: permission denied on ASM disk groupASM
- Oracle ASM Disk HeaderOracleASMHeader
- How to move ASM spfile to a different disk group [ID 1082943.1]ASM
- ASM disk group mount fails with ORA-15036: disk is truncated [ID 1077175.1]ASMAI
- Oracle ASM Disk Used Space DirectoryOracleASM
- ASM Rebalance(摘自Blogspot)ASM
- 移動ASM的spfile到一個新的disk groupASM
- drop asm disk、撤銷drop asm diskASM
- [Oracle Script] ASM Disk Groups UsedOracleASM
- Oracle ASM User Directory and Group DirectoryOracleASM
- 規劃ASM DISK GROUP、檢視asm 磁碟當前狀態、mount or dismount 磁碟組ASM
- 手工建立ASM Disk Groups、為 ASM Disk Groups 新增 diskASM
- Oracle ASM disk誤被格式化OracleASM
- ORACLE 10.2.0.5 RAC OCR&vote disk 磁碟遷移 DG Rebalance測試Oracle
- Asm disk managerASM
- 12c ASM中建立ACFSASM
- ASM DISK Group載入ORA-15183錯誤一例ASM
- [zt] Oracle10g ASM Auto IO Rebalance新特性及ASM體系結構OracleASM
- ASM叢集檔案系統ACFS（ASM Cluster File System）ASM
- oracle10g_asm_v$asm_disk之header_statusOracleASMHeader
- ASM: Device is already labeled for ASM diskASMdev
- Oracle OCP 1Z0 053 Q659(ASM Rebalance Trigger)OracleASM
- Oracle 使用BBED 檢視 ASM Disk Header 內容OracleASMHeader
- oracle 11g acfsOracle
- ASM DISK HEADER CORRUPTION & REPAIRASMHeaderAI
- ASM Normal需要3個disk ?ASMORM
- asm disk 的結構（1）ASM
- oracleasm createdisk ASM: Device is already labeled for ASM diskOracleASMdev
- Oracle ASM (10) - ASM中優先讀failure group配置ASM_PREFERRED_READ_FAILURE_GROUPSOracleASMAI
- Oracle ASM (11) - ASM中優先讀failure group配置ASM_PREFERRED_READ_FAILURE_GROUPSOracleASMAI