RHEL5 multipath多路徑聚合到RAC的RAW裝置繫結中不能用raw raw1 /dev/dm-*來實現
Name不會變化,但dm-*會重新識別,在RAC的RAW裝置繫結中,不能用raw raw1 /dev/dm-*來實現。
兩個節點,分別連線兩臺SAN交換,接到同一個儲存上生成48條通道,為了能讓48條通道冗餘,用了linux下的multipath。
配置multipath.conf
cat /etc/multipath.conf
.........
# The defaults section
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio const
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
no_path_retry 12
user_friendly_names yes
}
.........
blacklist {
# wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][[0-9]*]"
devnode "^cciss!c[0-9]d[0-9]*"
.......
multipaths {
# based on its wwid
multipath {
wwid 36001438005de9b800000600000140000
alias PSIFile
}
multipath {
wwid 36001438005de9b800000600000280000
alias PSIDataRaw01 //給光纖的WWID起個別名
}
multipath {
wwid 36001438005de9b8000006000002e0000
alias PSIDataRaw02
}
}
device {
vendor "HP"
product "HSV2[01]0|HSV300|HSV4[05]0"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio alua
hardware_handler "0"
path_selector "round-robin 0"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
path_checker tur
}
重啟服務：
/etc/init.d/multipath start
/etc/init.d/boot.multipathd start
這樣會在/dev下生成/dev/dm-*，/dev/disk/by-name和/dev/mapper的裝置名稱。
ls -l /dev/disk/
total 0
drwxr-xr-x 2 root root 820 Dec 4 2009 by-id
drwxr-xr-x 2 root root 240 Dec 4 2009 by-name
drwxr-xr-x 2 root root 3460 Dec 4 2009 by-path
drwxr-xr-x 2 root root 180 Dec 4 2009 by-uuid
ls -l /dev/mapper/
total 0
brw------- 1 root root 253, 2 Dec 4 2009 PSIDataRaw01
brw------- 1 root root 253, 3 Dec 4 2009 PSIDataRaw01-part1
brw------- 1 root root 253, 4 Dec 4 2009 PSIDataRaw01-part2
brw------- 1 root root 253, 7 Dec 4 2009 PSIDataRaw01-part3
brw------- 1 root root 253, 0 Dec 4 2009 PSIDataRaw02
brw------- 1 root root 253, 5 Dec 4 2009 PSIDataRaw02-part1
brw------- 1 root root 253, 6 Dec 4 2009 PSIDataRaw02-part2
brw-r----- 1 oracle oinstall 253, 8 Dec 3 22:18 PSIDataRaw02-part3
brw------- 1 root root 253, 1 Dec 4 2009 PSIFile
brw------- 1 root root 253, 9 Dec 4 2009 PSIFile-part1
lrwxrwxrwx 1 root root 16 Dec 4 2009 control -> ../device-mapper
ls -l /dev/dm*
brw-r----- 1 root disk 253, 0 Dec 4 2009 /dev/dm-0
brw-r----- 1 root disk 253, 1 Dec 4 2009 /dev/dm-1
brw-r----- 1 root disk 253, 2 Dec 4 2009 /dev/dm-2
brw-r----- 1 root disk 253, 3 Dec 4 2009 /dev/dm-3 -----做multipath時,重啟後這些裝置名稱會重新識別
brw-r----- 1 root disk 253, 4 Dec 4 2009 /dev/dm-4
brw-r----- 1 root disk 253, 5 Dec 4 2009 /dev/dm-5
brw-r----- 1 root disk 253, 6 Dec 4 2009 /dev/dm-6
brw-r----- 1 root disk 253, 7 Dec 4 2009 /dev/dm-7
brw-r----- 1 root disk 253, 8 Dec 4 2009 /dev/dm-8
brw-r----- 1 root disk 253, 9 Dec 4 2009 /dev/dm-9
ls -l /dev/disk/by-name/
total 0
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw01 -> ../../dm-2
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw01-part1 -> ../../dm-3
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw01-part2 -> ../../dm-4
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw01-part3 -> ../../dm-7 ----重啟後,前面的alias name不會變化,但dm-*會重新識別.
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw02 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw02-part1 -> ../../dm-5
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw02-part2 -> ../../dm-6
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIDataRaw02-part3 -> ../../dm-8
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIFile -> ../../dm-1
lrwxrwxrwx 1 root root 10 Dec 4 2009 PSIFile-part1 -> ../../dm-9
在RAC的RAW裝置繫結中,不能用raw raw1 /dev/dm-*來實現.
cat /etc/raw
raw1:mapper/PSIDataRaw02-part1 --OCR
raw2:mapper/PSIDataRaw02-part2 --OCR
raw3:mapper/PSIDataRaw01-part1 --VOTEDISK
raw4:mapper/PSIDataRaw01-part2 --VOTEDISK
raw5:mapper/PSIDataRaw01-part3 --VOTEDISK
在SUSE 10中.RAW裝置分割槽屬主和許可權不能寫在/etc/rc.d/boot.local中.要用udev的rules來單獨定義,單獨在/etc/udev/rules.d下建個rules來定義這些.
[PSIDB01:~ ]# cat /etc/udev/rules.d/99-raw.rules
KERNEL=="raw[1-2]*", OWNER="root", GROUP="oinstall", MODE="640"
KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="660"
# /etc/init.d/boot.udev stop
# /etc/init.d/boot.udev start --重啟udev
用作ASM磁碟的分割槽屬主和許可權在/etc/rc.d/boot.local下新增:
/bin/chown oracle:oinstall /dev/mapper/PSIDataRaw02-part3
/bin/chmod 640 /dev/mapper/PSIDataRaw02-part3
# chkconfig raw on
# chkconfig multipathd on
# chkconfig boot.multipath boot.
重啟節點後,要重新執行
# /oracle/product/crs/root.sh
(今天早上ORACLE回覆,可以執行/etc/init.d/init.cssd startcheck 去檢視相關問題,root.sh可以fix一些配置問題.印象中我也執行了這個命令,好象是報對crsctl沒有許可權.問題雖然解決,錯過一次學習的機會)
檢視crs程式
# ps -ef |grep init
root 1 0 0 17:02 ? 00:00:09 init [5]
root 6888 25389 0 22:38 pts/2 00:00:00 grep init
root 10113 1 0 17:03 ? 00:00:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
root 31640 1 0 22:26 ? 00:00:01 /bin/sh /etc/init.d/init.cssd fatal
root 31837 1 0 22:26 ? 00:00:00 /bin/sh /etc/init.d/init.evmd run
root 31946 1 0 22:26 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
root 32317 31640 0 22:27 ? 00:00:00 /bin/sh /etc/init.d/init.cssd oprocd
root 32324 31640 0 22:27 ? 00:00:00 /bin/sh /etc/init.d/init.cssd oclsomon
root 32340 31640 0 22:27 ? 00:00:00 /bin/sh /etc/init.d/init.cssd daemon
# ps -ef |grep crs
oracle 586 32340 0 22:27 ? 00:00:00 /oracle/product/crs/bin/ocssd.bin
oracle 644 31837 0 22:27 ? 00:00:00 /bin/su -l oracle -c sh -c 'ulimit -c unlimited; cd /oracle/product/crs/log/psidb02/evmd; exec /oracle/product/crs/bin/evmd '
oracle 645 644 0 22:27 ? 00:00:00 /oracle/product/crs/bin/evmd.bin
root 736 31946 0 22:27 ? 00:00:01 /oracle/product/crs/bin/crsd.bin reboot
oracle 841 645 0 22:27 ? 00:00:00 /oracle/product/crs/bin/evmlogger.bin -o /oracle/product/crs/evm/log/evmlogger.info -l /oracle/product/crs/evm/log/evmlogger.log
oracle 1638 1 0 22:28 ? 00:00:00 /oracle/product/crs/opmn/bin/ons -d
oracle 1639 1638 0 22:28 ? 00:00:00 /oracle/product/crs/opmn/bin/ons -d
root 7161 25389 0 22:39 pts/2 00:00:00 grep crs
root 31946 1 0 22:26 ? 00:00:00 /bin/sh /etc/init.d/init.crsd run
root 32630 32317 0 22:27 ? 00:00:00 /oracle/product/crs/bin/oprocd.bin run -t 1000 -m 500 -f
oracle 32641 32324 0 22:27 ? 00:00:00 /bin/su -l oracle -c /bin/sh -c 'cd /oracle/product/crs/log/psidb02/cssd/oclsomon; ulimit -c unlimited; /oracle/product/crs/bin/oclsomon || exit $?'
oracle 32642 32641 0 22:27 ? 00:00:00 /bin/sh -c cd /oracle/product/crs/log/psidb02/cssd/oclsomon; ulimit -c unlimited; /oracle/product/crs/bin/oclsomon || exit $?
oracle 32660 32642 0 22:27 ? 00:00:00 /oracle/product/crs/bin/oclsomon.bin
CRS在啟動時出現問題,會在tmp下生成ocrctl.*的檔案,檢視這些檔案資訊,參考crs/log/nodename/cssd和crsd的日誌可以進行分析.
# ll -l /tmp/crs*
-rw-r--r-- 1 oracle oinstall 0 Dec 3 20:35 /tmp/crsctl.14126
CRS正常,在節點服務啟動時出現的問題,可以到下面路徑裡檢視相關資訊
PSIDB02:/oracle/product/db/log/psidb02/racg
# ll
total 268
-rw-r--r-- 1 oracle users 784 Nov 30 09:58 imon.log
-rw-r--r-- 1 oracle users 123401 Dec 3 22:28 imon_psiccic.log
-rw-r--r-- 1 oracle users 50897 Dec 3 22:28 imonpsiccic.log
-rw-r--r-- 1 oracle users 10560 Dec 3 22:28 mdb.log
-rw-r--r-- 1 oracle users 16855 Dec 3 22:28 ora.psiccic.psiccic2.inst.log
-rw-r--r-- 1 oracle users 14744 Dec 3 22:28 ora.psidb02.ASM2.asm.log
-rw-r--r-- 1 oracle users 12790 Dec 3 22:28 ora.psidb02.LISTENER_PSIDB02.lsnr.log
drwxr-xr-t 2 oracle users 4096 Nov 24 17:01 racgeut
drwxr-xr-t 2 oracle users 4096 Nov 24 17:03 racgimon
drwxr-xr-t 2 oracle users 4096 Nov 24 17:01 racgmain
drwxr-xr-t 2 oracle users 4096 Nov 24 17:03 racgmdb
先裝的ASM,磁碟組指向了/dev/dm-*,發現multipath的dem-*變化的現象後,ASM磁碟組也需要進行對應的更改,將原來磁碟組對應r的/dev/dm-5改為/dev/mapper/PSIDataRaw02-part3'
:~> export ORACLE_SID=+ASM1
:~> sqlplus /nolog
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Dec 3 21:59:55 2009
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
SQL> connect /as sysdba
Connected.
SQL> show parameter asm;
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
asm_diskgroups string DATA
asm_diskstring string /dev/dm*
asm_power_limit integer 1
SQL> select * from v$asm_diskgroup;
no rows selected
SQL> select * from v$asm_disk;
no rows selected
SQL> alter system set asm_diskstring='/dev/mapper/PSIDataRaw02-part3';
System altered.
SQL> select path,mount_status from v$asm_disk;
PATH MOUNT_STATUS
-------------------------------------------------------------------------------- --------------
/dev/mapper/PSIDataRaw02-part3 CLOSED
SQL> alter diskgroup DATA mount;
Diskgroup altered.
SQL> select path,mount_status from v$asm_disk;
PATH MOUNT_STATUS
-------------------------------------------------------------------------------- --------------
/dev/mapper/PSIDataRaw02-part3 CACHED
SQL> create pfile from spfile;
File created.
SQL> shutdown immediate ---關閉ASM例項
$ ./srvctl start asm -n psidb01 ---重啟例項
SQL> show parameter asm;
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
asm_diskgroups string DATA
asm_diskstring string /dev/mapper/PSIDataRaw02-part3
asm_power_limit integer
SQL> select group_number,name,allocation_unit_size alloc_unit_size,state,type, total_mb, usable_file_mb from v$asm_diskgroup;
GROUP_NUMBER NAME ALLOC_UNIT_SIZE STATE TYPE TOTAL_MB USABLE_FILE_MB
------------ ---- --------------- ------- ------ -------- --------------
1 DATA 1048576 MOUNTED EXTERN 1330210 930221
SQL> select name,path,header_status,total_mb free_mb,trunc(bytes_read/1024/1024) read_mb, trunc(bytes_written/1024/1024) write_mb from v$asm_disk;
NAME PATH HEADER_STATUS FREE_MB READ_MB WRITE_MB
--------- ------------------------------ ------------- ------- ------- --------
DATA_0000 /dev/mapper/PSIDataRaw02-part3 MEMBER 1330210 0 0
$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.psiccic.db application ONLINE ONLINE psidb02
ora....c1.inst application OFFLINE OFFLINE
ora....c2.inst application ONLINE ONLINE psidb02
ora....SM1.asm application ONLINE ONLINE psidb01
ora....01.lsnr application ONLINE ONLINE psidb01
ora....b01.gsd application ONLINE ONLINE psidb01
ora....b01.ons application ONLINE ONLINE psidb01
ora....b01.vip application ONLINE ONLINE psidb01
ora....SM2.asm application ONLINE ONLINE psidb02
ora....02.lsnr application ONLINE ONLINE psidb02
ora....b02.gsd application ONLINE ONLINE psidb02
ora....b02.ons application ONLINE ONLINE psidb02
ora....b02.vip application ONLINE ONLINE psidb02
PSIDB01:/oracle/product/crs/bin # ./srvctl start instance -d psiccic -i psiccic1
PSIDB01:/oracle/product/crs/bin # ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.psiccic.db application ONLINE ONLINE psidb02
ora....c1.inst application ONLINE ONLINE psidb01
ora....c2.inst application ONLINE ONLINE psidb02
ora....SM1.asm application ONLINE ONLINE psidb01
ora....01.lsnr application ONLINE ONLINE psidb01
ora....b01.gsd application ONLINE ONLINE psidb01
ora....b01.ons application ONLINE ONLINE psidb01
ora....b01.vip application ONLINE ONLINE psidb01
ora....SM2.asm application ONLINE ONLINE psidb02
ora....02.lsnr application ONLINE ONLINE psidb02
ora....b02.gsd application ONLINE ONLINE psidb02
ora....b02.ons application ONLINE ONLINE psidb02
ora....b02.vip application ONLINE ONLINE psidb02
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/9034054/viewspace-1987939/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Redhat7 Udev繫結（DM multipath）多路徑裝置Redhatdev
- '+CRS' in '/etc/oracle/ocr.loc' and '/dev/raw/raw1' does not matchOracledev
- [zt] LINUX裸裝置(raw)取消繫結Linux
- Oracle Linux 7.8 多路徑（Multipath）+Udev繫結磁碟OracleLinuxdev
- Linux配置和管理裝置對映多路徑multipathLinux
- Error:The specified shared raw partition /dev/raw/raw1...Errordev
- multipath多路徑配置
- Redhat 5 中裸裝置(raw) 的配置Redhat
- raw 裸裝置資料檔案更改其他raw裸裝置位置
- redhat multipath多路徑 DMRedhat
- [轉]RedHat 5中裸裝置(raw)的配置Redhat
- 字元裝置、塊裝置、裸裝置、RAW裝置 .字元
- 使用OpenFiler來模擬儲存配置RAC中ASM共享盤及多路徑（multipath）的測試ASM
- linux之 multipath 多路徑Linux
- linux多路徑multipath配置Linux
- RAW+ASM 的RAC 安裝文件ASM
- oracle靜默安裝raw裝置Oracle
- RedHat 5.4上iscsi+multipath多路徑儲存配置獲取裝置的wwidRedhat
- RAW裝置與ORACLE的問答 （zt）Oracle
- Linux-建立裸裝置RawLinux
- RAW+ASM 的RAC 安裝文件 收藏(ZF)ASM
- Linux Multipath多路徑配置與使用案例Linux
- 如何遷移裸裝置raw device資料檔案到另一個raw devicedev
- Multipath實現LUN裝置名稱的持久化持久化
- Configuring non-raw multipath devices for Oracle Clusterware 11g (11.1.0) on RHEL5/OEL5devOracle
- 【Raw Device】OEL4.8裸裝置對映方法dev
- Linux平臺的多路徑軟體multipath的使用案例Linux
- rhel 6 對映多路徑(DM Multipath)配置及管理
- Redhat7 Udev繫結單路徑裝置方法Redhatdev
- AIX5.3ML05 ->ML07補丁升級,可能造成Oracle RAW裝置檔案不能用AIOracle
- AIX 共享磁碟中的字元裝置(/dev/rhdiskn)及塊裝置(/dev/hdiskn)AI字元dev
- redhat5中修改raw裝置的預設使用者和許可權Redhat
- LINUX 多路徑對映Multipath與正規表示式Linux
- linux iscsi multipath多路徑及名稱持久化配置Linux持久化
- centos上iscsi+multipath多路徑儲存配置手冊CentOS
- Configuring non-raw Multipath Devices for Oracle Clusterware 11gdevOracle
- Redhat 5.4 + ASM + RAW+ Oracle 10g RAC 安裝文件RedhatASMOracle 10g
- RAW影像處理工具——RAW Power for MacMac