說明

從 12.1.0.2 開始， Oracle 引入了 ASMFD(ASM Filter Driver) ， ASMFD 支援 Linux x86_64 和 Oracle Solaris 平臺。安裝完 Grid Infrastructure 後，你可以決定是否配置它。如果之前使用了 ASMLIB 或者 UDEV ，遷移到 ASMFD 之後，需要解除安裝 ASMLIB 或禁用 udev 的規則。透過 Filter driver 可以過濾無效的請求，避免因為非 oracle 的 I/O 請求造成意外的覆寫，進而保證了系統的安全和穩定。

測試環境描述

作業系統版本： Red Hat Enterprise Linux Server release 7.5 (Maipo)

資料庫版本： 19.3.0.0.0

資料庫架構： RAC

ASM 磁碟繫結方式： UDEV

 

原 UDEV 繫結磁碟資訊：

select name,path,state from v$asm_disk;

NAME             PATH               STATE

------------------------------------------------------------------------

DATA2_0000       /dev/asmdisk1        NORMAL

DATA_0001        /dev/asmdisk3        NORMAL

DATA_0000        /dev/asmdisk4        NORMAL

DATA2_0001       /dev/asmdisk2        NORMAL

DATA_0002        /dev/asmdisk5        NORMAL

 

sdb sdc sdd 為測試新增的磁碟：

Disk /dev/sdc: 10.7 GB, 10737418240 bytes, 20971520 sectors

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors

Disk /dev/sdf: 85.9 GB, 85899345920 bytes, 167772160 sectors

Disk /dev/sdh: 21.5 GB, 21474836480 bytes, 41943040 sectors

Disk /dev/sdd: 10.7 GB, 10737418240 bytes, 20971520 sectors

Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors

Disk /dev/sdg: 21.5 GB, 21474836480 bytes, 41943040 sectors

Disk /dev/sde: 85.9 GB, 85899345920 bytes, 167772160 sectors

Disk /dev/sdi: 21.5 GB, 21474836480 bytes, 41943040 sectors

ASMFD

檢視ASMFD是否安裝

[root@rac1 ~]# su – grid

[grid@rac1 ~]$ asmcmd afd_state

ASMCMD-9530: The AFD state is 'NOT INSTALLED'

 

安裝ASMFD

一二節點均需安裝

1 使用 root 使用者關閉 CRS ： crsctl stop crs

2 使用 root 使用者安裝 ASMFD ：

[root@rac1 ~]# export ORACLE_BASE=/oracle/gridbase

[root@rac1 ~]# export ORACLE_HOME=/oracle/grid/crs_1

[root@rac1 ~]# /oracle/grid/crs_1/bin/asmcmd afd_configure

AFD-627: AFD distribution files found.

AFD-634: Removing previous AFD installation.

AFD-635: Previous AFD components successfully removed.

AFD-9294: updating file /etc/sysconfig/oracledrivers.conf

AFD-636: Installing requested AFD software.

AFD-637: Loading installed AFD drivers.

AFD-9321: Creating udev for AFD.

AFD-9323: Creating module dependencies - this may take some time.

AFD-9154: Loading 'oracleafd.ko' driver.

AFD-649: Verifying AFD devices.

AFD-9156: Detecting control device '/dev/oracleafd/admin'.

AFD-638: AFD installation correctness verified.

Modifying resource dependencies - this may take some time.

檢查ASMFD的狀態

[root@rac1 ~]# /oracle/grid/crs_1/bin/asmcmd afd_state

ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac1'

 

[grid@rac1 ~]$ crsctl status res -t -init

ora.driver.afd

      1        ONLINE  ONLINE       rac1                     STABLE

afd_label 標識磁碟

檢視afd_diskstring

[root@rac1 dev]# more /etc/oracleafd.conf

afd_diskstring='/dev/asm*'

 

使用udev繫結磁碟

vi 99-oracle-asmdevices.rules

KERNEL=="sd*[!0-9]", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="0QEMU_QEMU_HARDDISK_drive-scsi8", RUN+="/bin/sh -c 'mknod /dev/asmfd1 b $major $minor; chown grid:oinstall /dev/asmfd1; chmod 0660 /dev/asmfd1'"

KERNEL=="sd*[!0-9]", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="0QEMU_QEMU_HARDDISK_drive-scsi7", RUN+="/bin/sh -c 'mknod /dev/asmfd2 b $major $minor; chown grid:oinstall /dev/asmfd2; chmod 0660 /dev/asmfd2'"

KERNEL=="sd*[!0-9]", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="0QEMU_QEMU_HARDDISK_drive-scsi6", RUN+="/bin/sh -c 'mknod /dev/asmfd3 b $major $minor; chown grid:oinstall /dev/asmfd3; chmod 0660 /dev/asmfd3'"

 

重啟udev

/sbin/udevadm trigger --type=devices --action=change

 

使用afd_label標識磁碟

[grid@rac1 ~]$ asmcmd afd_label ASMFD1 /dev/asmfd1

[grid@rac1 ~]$ asmcmd afd_label ASMFD2 /dev/asmfd2

[grid@rac1 ~]$ asmcmd afd_label ASMFD3 /dev/asmfd3

[grid@rac1 ~]$ asmcmd afd_lsdsk

------------------------------------------------------------------------

Label                     Filtering   Path

=========================================================================

ASMFD1                      ENABLED   /dev/asmfd1

ASMFD2                      ENABLED   /dev/asmfd2

ASMFD3                      ENABLED   /dev/asmfd3

/etc/udev/rules.d/53-afd.rules

[root@rac1 rules.d]# more 53-afd.rules

#

# AFD devices

KERNEL=="oracleafd/.*", OWNER="grid", GROUP="asmadmin", MODE="0770"

KERNEL=="oracleafd/*", OWNER="grid", GROUP="asmadmin", MODE="0770"

KERNEL=="oracleafd/disks/*", OWNER="grid", GROUP="asmadmin", MODE="0660"

 

若標識錯誤，使用afd_unlabel 移除標識的磁碟

[grid@rac1 ~]$ asmcmd afd_unlabel ASMFD1

[grid@rac1 ~]$ asmcmd afd_unlabel ASMFD2

[grid@rac1 ~]$ asmcmd afd_unlabel ASMFD3

 

在另外的節點中，不再需要作 label ，而是直接 scan 即可。

[grid@rac1 ~]$ asmcmd afd_scan

[grid@rac1 ~]$ asmcmd afd_lsdsk

 

ASM 磁碟組

圖形化介面方式

 

 

 

 

修改 asm_diskstring

 

 

 

 

命令列方式

[grid@rac1 ~]$ sqlplus / as sysasm

show parameter string

 

NAME                                 TYPE        VALUE

------------------------------------ --------------------------

asm_diskstring                       string      /dev/asm*, AFD:*

 

select name,path,state from v$asm_disk;

 

NAME         PATH               STATE

------------ ------------------ --------

             /dev/asmfd2        NORMAL

             AFD:ASMFD1         NORMAL

             AFD:ASMFD2         NORMAL

             /dev/asmfd1        NORMAL

             /dev/asmfd3        NORMAL

             AFD:ASMFD3         NORMAL

DATA2_0000   /dev/asmdisk1      NORMAL

DATA_0001    /dev/asmdisk3      NORMAL

DATA_0000    /dev/asmdisk4      NORMAL

DATA2_0001   /dev/asmdisk2      NORMAL

DATA_0002    /dev/asmdisk5      NORMAL

 

11 rows selected.

 

CREATE DISKGROUP DATA3 EXTERNAL REDUNDANCY DISK 'AFD:ASMFD1';

alter diskgroup DATA3 add disk 'AFD:ASMFD2';

alter diskgroup DATA3 add disk 'AFD:ASMFD3';

 

select name,path,state from v$asm_disk;

 

NAME          PATH               STATE

------------- ------------------ --------

              /dev/asmfd2        NORMAL

              /dev/asmfd1        NORMAL

              /dev/asmfd3        NORMAL

DATA2_0000    /dev/asmdisk1      NORMAL

DATA_0001     /dev/asmdisk3      NORMAL

DATA_0000     /dev/asmdisk4      NORMAL

DATA2_0001    /dev/asmdisk2      NORMAL

DATA_0002     /dev/asmdisk5      NORMAL

ASMFD1        AFD:ASMFD1         NORMAL

ASMFD2        AFD:ASMFD2         NORMAL

ASMFD3        AFD:ASMFD3         NORMAL

 

Filter 功能

檢視 filter 功能是否啟用

[grid@rac1 ~]$ asmcmd afd_lsdsk

--------------------------------------------------------------------

Label                     Filtering   Path

===================================================================

ASMFD1                      ENABLED   /dev/asmfd1

ASMFD2                      ENABLED   /dev/asmfd2

ASMFD3                      ENABLED   /dev/asmfd3

 

在啟用 Filter 功能下， dd 磁碟測試

[root@rac1 dev]# dd if=/dev/zero of=/dev/asmfd1

dd: writing to ‘/dev/asmfd1’: No space left on device

20971521+0 records in

20971520+0 records out

10737418240 bytes (11 GB) copied, 931.568 s, 11.5 MB/s

 

透過 strings 檢視 /dev/sdb ，可以發現，裡面的內容並沒有被清空

[root@rac1 dev]# strings /dev/asmfd1

ORCLDISKASMFD1

ASMFD1

DATA3

ASMFD1

ORCLDISKASMFD1

ASMFD1

DATA3

ASMFD1

……

 

解除安裝、掛載磁碟組測試

SQL> select NAME,STATE from v$asm_diskgroup;

NAME                           STATE

------------------------------ -----------

DATA2                          MOUNTED

DATA                           MOUNTED

DATA3                          MOUNTED

 

SQL> alter diskgroup DATA3 dismount;

Diskgroup altered.

 

SQL> alter diskgroup DATA3 mount;

Diskgroup altered.

 

/var/log/messages 裡顯示的錯誤資訊

Mar  4 14:31:43 rac1 kernel: buffer_io_error: 17144 callbacks suppressed

Mar  4 14:31:43 rac1 kernel: Buffer I/O error on dev sdb, logical block 2560823, lost async page write

Mar  4 14:31:43 rac1 kernel: Buffer I/O error on dev sdb, logical block 2560824, lost async page write

Mar  4 14:31:43 rac1 kernel: Buffer I/O error on dev sdb, logical block 2560825, lost async page write

 

參考文件：

Grid Infrastructure Installation and Upgrade Guide for Linux

 

Automatic Storage Management Administrator's Guide