Oracle ASM ACFS disk group rebalance
從Oracle 11.2開始，一個ASM磁碟組可以被用來建立一個或多個叢集檔案系統。這就是Oracle ASM叢集檔案系統或Oracle ACFS。這個功能透過在ASM磁碟組中建立特定的volume檔案來實現，然後作為塊裝置給作業系統來使用，再在這些塊裝置上建立檔案系統。下面將介紹ACFS volume檔案的rebalance,mirror與extent管理。
測試環境如下:
.64-bit Oracle Linux 5.4
.Oracle Restart and ASM version 11.2.0.4.0 - 64bit
設定ACFS volumes
單例項載入ADVM/ACFS驅動的命令如下，RAC環境不需要，因為已經預設載入
[root@jyrac1 bin]# ./acfsroot install ACFS-9300: ADVM/ACFS distribution files found. ACFS-9118: oracleadvm.ko driver in use - cannot unload. ACFS-9312: Existing ADVM/ACFS installation detected. ACFS-9118: oracleadvm.ko driver in use - cannot unload. ACFS-9314: Removing previous ADVM/ACFS installation. ACFS-9315: Previous ADVM/ACFS components successfully removed. ACFS-9307: Installing requested ADVM/ACFS software. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9321: Creating udev for ADVM/ACFS. ACFS-9323: Creating module dependencies - this may take some time. ACFS-9154: Loading 'oracleacfs.ko' driver. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. ACFS-9156: Detecting control device '/dev/ofsctl'. ACFS-9309: ADVM/ACFS installation correctness verified. [root@jyrac1 bin]# ./acfsload start ACFS-9391: Checking for existing ADVM/ACFS installation. ACFS-9392: Validating ADVM/ACFS installation files for operating system. ACFS-9393: Verifying ASM Administrator setup. ACFS-9308: Loading installed ADVM/ACFS drivers. ACFS-9327: Verifying ADVM/ACFS devices. ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'. ACFS-9156: Detecting control device '/dev/ofsctl'. ACFS-9322: completed [root@jyrac1 bin]# ./acfsdriverstate version ACFS-9325: Driver OS kernel version = 2.6.18-8.el5(x86_64). ACFS-9326: Driver Oracle version = 130707.
建立一個用來建立ASM叢集檔案系統的磁碟組
SQL> create diskgroup acfs disk '/dev/raw/raw5','/dev/raw/raw6' attribute 'COMPATIBLE.ASM' = '11.2', 'COMPATIBLE.ADVM' = '11.2'; Diskgroup created.
雖然一個磁碟組可以用來儲存資料庫檔案與ACFS volume files，但是建議為ACFS volume建立一個單獨的磁碟組。這將提供角色/功能分離與對資料庫檔案效能有潛在好處。
檢查所有磁碟組的AU大小
SQL> select group_number "Group#", name "Name", allocation_unit_size "AU size" from v$asm_diskgroup_stat;
Group# Name AU size
---------- ------------------------------------------------------------ ----------
1 ARCHDG 1048576
2 CRSDG 1048576
3 DATADG 1048576
4 ACFS 1048576
對於所有磁碟組來說預設的AU大小為1MB，當後面介紹volume file的區大小時會使用到AU大小。
在磁碟組ACFS中建立三個volume
[grid@jyrac1 ~]$ asmcmd volcreate -G ACFS -s 1G ACFS_VOL1 [grid@jyrac1 ~]$ asmcmd volcreate -G ACFS -s 1G ACFS_VOL2 [grid@jyrac1 ~]$ asmcmd volcreate -G ACFS -s 1G ACFS_VOL3
檢視volume資訊
[grid@jyrac1 ~]$ asmcmd volinfo -a
Diskgroup Name: ACFS
Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-10
State: ENABLED
Size (MB): 1024
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
Volume Name: ACFS_VOL2
Volume Device: /dev/asm/acfs_vol2-10
State: ENABLED
Size (MB): 1024
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
Volume Name: ACFS_VOL3
Volume Device: /dev/asm/acfs_vol3-10
State: ENABLED
Size (MB): 1024
Resize Unit (MB): 32
Redundancy: MIRROR
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
在volume建立這後會自動被啟用。當伺服器重啟之後可能需要手動載入ADVM/ACFS驅動(acfsload start)並啟用volume(asmcmd volenable -a)。
對於每個volume，ASM將建立一個volume file。在冗餘磁碟組中，每個卷將有一個dirty region logging(DRL)檔案
SQL> select file_number "File#", volume_name "Volume", volume_device "Device", size_mb "MB", drl_file_number "DRL#" from v$asm_volume; File# Volume Device MB DRL# ----- ---------------------------------------- ---------------------------------------- ---------- ---------- 257 ACFS_VOL1 /dev/asm/acfs_vol1-10 1024 256 259 ACFS_VOL2 /dev/asm/acfs_vol2-10 1024 258 261 ACFS_VOL3 /dev/asm/acfs_vol3-10 1024 260
除了卷名，裝置名與大小之外，還顯示了ASM檔案號257,259,261給卷裝置使用，ASM檔案號256,258,260給DRL檔案使用。
查詢卷檔案的AU分佈情況
SQL> select
2 xnum_kffxp, -- virtual extent number
3 pxn_kffxp, -- physical extent number
4 disk_kffxp, -- disk number
5 au_kffxp -- allocation unit number
6 from x$kffxp
7 where number_kffxp=261-- asm file 256
8 and group_kffxp=4 -- group number 1
9 order by 1,2,3;
XNUM_KFFXP PXN_KFFXP DISK_KFFXP AU_KFFXP
---------- ---------- ---------- ----------
0 0 0 2160
0 1 1 2160
1 2 1 2168
1 3 0 2168
2 4 0 2176
2 5 1 2176
3 6 1 2184
3 7 0 2184
4 8 0 2192
4 9 1 2192
5 10 1 2200
5 11 0 2200
6 12 0 2208
6 13 1 2208
......
124 248 0 3152
124 249 1 3152
125 250 1 3160
125 251 0 3160
126 252 0 3168
126 253 1 3168
127 254 1 3176
127 255 0 3176
2147483648 0 0 2156
2147483648 1 1 2156
2147483648 2 65534 4294967294
259 rows selected.
當在normal冗餘磁碟組中建立卷，那麼卷的每個區同樣也會被映象。可以看到卷檔案261有128個區。卷大小為1GB，這意味著每個區大小為8MB或8個AU。卷檔案有屬於它自己的區大小，不像標準的ASM檔案繼承來自磁碟組AU大小來初始化區大小。
在邏輯卷裝置上建立ASM叢集檔案系統（ACFS）
[grid@jyrac1 ~]$ /sbin/mkfs -t acfs /dev/asm/acfs_vol1-10 mkfs.acfs: version = 11.2.0.4.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfs_vol1-10 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete. [grid@jyrac1 ~]$ /sbin/mkfs -t acfs /dev/asm/acfs_vol2-10 mkfs.acfs: version = 11.2.0.4.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfs_vol2-10 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete. [grid@jyrac1 ~]$ /sbin/mkfs -t acfs /dev/asm/acfs_vol3-10 mkfs.acfs: version = 11.2.0.4.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfs_vol3-10 mkfs.acfs: volume size = 1073741824 mkfs.acfs: Format complete. [root@jyrac1 /]# mkdir /acfs1 [root@jyrac1 /]# mkdir /acfs2 [root@jyrac1 /]# mkdir /acfs3 [root@jyrac1 /]# chown -R grid:oinstall /acfs1 [root@jyrac1 /]# chown -R grid:oinstall /acfs2 [root@jyrac1 /]# chown -R grid:oinstall /acfs3 [root@jyrac1 /]# chmod -R 777 /acfs1 [root@jyrac1 /]# chmod -R 777 /acfs2 [root@jyrac1 /]# chmod -R 777 /acfs3 [root@jyrac1 /]# mount -t acfs /dev/asm/acfs_vol1-10 /acfs1 [root@jyrac1 /]# mount -t acfs /dev/asm/acfs_vol2-10 /acfs2 [root@jyrac1 /]# mount -t acfs /dev/asm/acfs_vol3-10 /acfs3 [root@jyrac1 /]# mount | grep acfs /dev/asm/acfs_vol1-10 on /acfs1 type acfs (rw) /dev/asm/acfs_vol2-10 on /acfs2 type acfs (rw) /dev/asm/acfs_vol3-10 on /acfs3 type acfs (rw)
複製一些檔案到新的檔案系統中
[grid@jyrac1 +asm]$ cp $ORACLE_BASE/diag/asm/+asm/+ASM1/trace/* /acfs1 [grid@jyrac1 +asm]$ cp $ORACLE_BASE/diag/asm/+asm/+ASM1/trace/* /acfs2 [grid@jyrac1 +asm]$ cp $ORACLE_BASE/diag/asm/+asm/+ASM1/trace/* /acfs3
檢查使用空間
[root@jyrac1 /]# df -h /acfs?
Filesystem Size Used Avail Use% Mounted on
/dev/asm/acfs_vol1-10
1.0G 105M 920M 11% /acfs1
/dev/asm/acfs_vol2-10
1.0G 105M 920M 11% /acfs2
/dev/asm/acfs_vol3-10
1.0G 105M 920M 11% /acfs3
現在向ACFS磁碟組新增磁碟組並監控rebalance操作
SQL> alter diskgroup ACFS add disk '/dev/raw/raw7'; Diskgroup altered.
從alert_+ASM1.log檔案中可以找到ARB0程式的PID為1074
[grid@jyrac1 trace]$ tail -f alert_+ASM1.log SQL> alter diskgroup ACFS add disk '/dev/raw/raw7' NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (4,2) to disk (/dev/raw/raw7) NOTE: requesting all-instance membership refresh for group=4 NOTE: initializing header on grp 4 disk ACFS_0002 NOTE: requesting all-instance disk validation for group=4 Thu Jan 12 14:54:45 2017 NOTE: skipping rediscovery for group 4/0xd98640a (ACFS) on local instance. NOTE: requesting all-instance disk validation for group=4 NOTE: skipping rediscovery for group 4/0xd98640a (ACFS) on local instance. Thu Jan 12 14:54:45 2017 GMON updating for reconfiguration, group 4 at 249 for pid 27, osid 18644 NOTE: group 4 PST updated. NOTE: initiating PST update: grp = 4 GMON updating group 4 at 250 for pid 27, osid 18644 NOTE: group ACFS: updated PST location: disk 0000 (PST copy 0) NOTE: group ACFS: updated PST location: disk 0001 (PST copy 1) NOTE: group ACFS: updated PST location: disk 0002 (PST copy 2) NOTE: PST update grp = 4 completed successfully NOTE: membership refresh pending for group 4/0xd98640a (ACFS) GMON querying group 4 at 251 for pid 18, osid 5012 NOTE: cache opening disk 2 of grp 4: ACFS_0002 path:/dev/raw/raw7 GMON querying group 4 at 252 for pid 18, osid 5012 SUCCESS: refreshed membership for 4/0xd98640a (ACFS) NOTE: starting rebalance of group 4/0xd98640a (ACFS) at power 1 SUCCESS: alter diskgroup ACFS add disk '/dev/raw/raw7' Starting background process ARB0 Thu Jan 12 14:54:48 2017 ARB0 started with pid=40, OS id=1074 NOTE: assigning ARB0 to group 4/0xd98640a (ACFS) with 1 parallel I/O cellip.ora not found. NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 2:2 for diskgroup 4 (ACFS) Thu Jan 12 14:55:00 2017 NOTE: Attempting voting file refresh on diskgroup ACFS NOTE: Refresh completed on diskgroup ACFS. No voting file found.
透過命令tail -f +ASM1_arb0_1074.trc來監控rebalance過程
*** 2017-01-12 14:55:18.731 ARB0 relocating file +ACFS.259.933075367 (86 entries) *** 2017-01-12 14:55:38.599 ARB0 relocating file +ACFS.259.933075367 (1 entries) ARB0 relocating file +ACFS.260.933075373 (17 entries) *** 2017-01-12 14:55:39.617 ARB0 relocating file +ACFS.261.933075373 (86 entries) *** 2017-01-12 14:55:59.106 ARB0 relocating file +ACFS.261.933075373 (1 entries) *** 2017-01-12 14:55:59.274 ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.258.933075367 (1 entries) ARB0 relocating file +ACFS.257.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) ARB0 relocating file +ACFS.256.933075361 (1 entries) *** 2017-01-12 14:56:00.201 ARB0 relocating file +ACFS.1.1 (1 entries) ARB0 relocating file +ACFS.7.1 (1 entries) ARB0 relocating file +ACFS.5.1 (1 entries) ARB0 relocating file +ACFS.8.1 (1 entries) ARB0 relocating file +ACFS.9.1 (1 entries) ARB0 relocating file +ACFS.6.1 (1 entries) ARB0 relocating file +ACFS.4.1 (1 entries) ARB0 relocating file +ACFS.4.1 (1 entries) ARB0 relocating file +ACFS.3.1 (1 entries) .....
可以看到每個ASM檔案的rebalance操作過程。這種操作行為與資料庫檔案是一樣的，ASM對每個檔案執行rebalance操作。ASM後設資料檔案(1-9)最先被rebalance。ASM然後對卷檔案號257,259,261，ASM檔案號256,258,260執行rebalance等等。
可以看到對卷檔案(與其它ASM檔案）執行rebalance操作並不是對儲存在相關檔案系統中的使用者檔案進行操作，而是對每個卷檔案執行rebalance操作。
ACFS磁碟組中的磁碟聯機操作
當一個ASM磁碟離線時，ASM將建立staleness registry and staleness directory來跟蹤磁碟聯機時需要修改的區。一旦磁碟聯機，ASM使用這些資訊來執行快速映象重新同步。這個功能對於ASM 11.2中的卷檔案是不可用的。相反，對於聯機的磁碟，ASM將重建整個磁碟內容。這就是為什麼對於儲存卷檔案的磁碟組執行磁碟聯機的效能要比儲存標準資料庫檔案的磁碟組執行磁碟聯機的效能差的原因。 對於卷檔案執行快速映象重新同步在ASM 12.1及以後的版本中是可以使用的。
小結:
ASM磁碟組可以被用來建立一般目錄的叢集檔案系統。ASM透過在磁碟組中建立卷檔案來實現，並將它們作為塊裝置提供給作業系統使用。現有的ASM磁碟組映象功能(normal與high冗餘)可以被用來在檔案系統級別保護使用者檔案。ASM透過映象卷檔案區來實現，這種方式也用於任何其它的ASM檔案。卷檔案有它自己的區大小，不像標準資料庫檔案繼承來自磁碟組的AU大小來初始化區大小。對儲存ASM叢集檔案系統卷的ASM磁碟組執行rebalance操作，實際上是對每個卷檔案執行rebalance操作，而不是對儲存在相關檔案系統中的單個使用者檔案執行rebalance操作。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26015009/viewspace-2132471/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle ASM Rebalance執行過程OracleASM
- ASM叢集檔案系統ACFS（ASM Cluster File System）ASM
- Oracle 11.2.0.4 rac for aix acfs異常環境的克隆環境ASM磁碟組掛載緩慢OracleAIASM
- ORA-15020:discoverd duplicate ASM disk "XXX_CJCDB_OCR_0002"ASM
- Oracle ASM神書《撥雲見日 解密Oracle ASM核心》出版了OracleASM解密
- Oracle ASM擴容(NFS)OracleASMNFS
- 【ASM】Oracle asm刪除磁碟組注意事項ASMOracle
- oracle資料庫的ACFS圖形介面不可選擇Oracle資料庫
- Oracle：ASM & 密碼檔案OracleASM密碼
- Oracle ASM Cluster File Systems (ACOracleASM
- Oracle ASM AMDU工具的使用OracleASM
- oracle partition by group by,詳解partition by和group by對比Oracle
- Oracle 12C Database File Mapping for Oracle ASM FilesOracleDatabaseAPPASM
- 12C Oracle ASM Filter DriverOracleASMFilter
- Overview of Oracle Flex ASM In Oracle 19c RAC-20220111ViewOracleFlexASM
- 【ASM】Oracle asm磁碟被格式化，如何掛載該磁碟組ASMOracle
- ORACLE ASM磁碟組空間溢位OracleASM
- 【Oracle】ASM例項安裝入門OracleASM
- 2.10.3 使用 Oracle Automatic Storage Management (Oracle ASM) 克隆資料庫OracleASM資料庫
- Oracle asm磁碟損壞異常恢復OracleASM
- ORACLE ASM的SSD磁碟空間回收分析OracleASM
- Oracle RAC日常運維-ASM磁碟擴容Oracle運維ASM
- Oracle ASM磁碟組擴容(AIX7.1)OracleASMAI
- oracle asm 資料塊重構恢復OracleASM
- 【BUILD_ORACLE】使用ASMLib包搭建ASM磁碟UIOracleASM
- Oracle 12C ASM asmcmd amdu_extractOracleASM
- oracle中distinct和group by的區別Oracle
- Kafka Consumer 的 Rebalance 機制Kafka
- Kafka 消費者組 RebalanceKafka
- 【ASK_ORACLE】Oracle RAC配置ASM Filter Driver（ASMFD）（一）ASMFD簡介OracleASMFilter
- Oracle Linux 7使用syslog來管理Oracle ASM的審計檔案OracleLinuxASM
- Oracle資料庫 ASM磁碟線上擴容Oracle資料庫ASM
- Oracle ASM故障資料恢復解決方案OracleASM資料恢復
- Oracle RAC環境下ASM磁碟組擴容OracleASM
- Oracle OCP(42)：ASM 儲存結構簡介OracleASM
- 通過ORACLE VM virtualbox環境安裝oracle 11G RAC(ASM)OracleASM
- ORACLE RAC 11.2.0.4 ASM加盤導致叢集重啟之ASM sga設定過小OracleASM
- 關於Oracle 10g ASM磁碟大小的限制Oracle 10gASM
- Oracle 12c新特性--ASMFD（ASM Filter Driver）特性OracleASMFilter