VMWARE+OCFS2+ASM+10GRAC實驗

花近有10天的時間做了10G的RAC。由於OCFS2和ASM都熟悉時間長的點，不要笑我呀。

OS：Linux node1pub 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 i686 i386 GNU/Linux

ORACLE：10.2.0.1

儲存的佈局：ocfs2 for OCR, CRS voting, ASM for data.

VMWARE:Server 1.0.5 build-80187

[@more@]

實驗過程中嚴重參考：

http://blog.chinaunix.net/u/31/showart_518611.html

http://tolywang.itpub.net/post/48/454440

vmware-vdiskmanager 建立 pre-allocated並且是lsi contoller的硬碟 1GB一個 for ocfs2 for OCR +

CRS voting
vmware-vdiskmanager 建立 pre-allocated並且是lsi contoller的硬碟 1GB 四個for ASM for Oracle

data & flash recovery area (同上)

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:ocfs.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:ocfs.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm1.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm1.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm2.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm2.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>
C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm3.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm3.vmdk'
Create: 100% done.
Virtual disk creation successful.

C:Program FilesVMwareVMware Server>vmware-vdiskmanager.exe -c -s 1Gb -a lsilo
gic -t 2 d:asm4.vmdk
Using log file C:DOCUME~1L5MLOCALS~1Tempvmware-L5Mvdiskmanager.log
Creating monolithic preallocated disk 'd:asm4.vmdk'
Create: 100% done.
Virtual disk creation successful.

分別在兩個虛擬機器的Red Hat Enterprise Linux 4.vmx檔案裡增加以下內容
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "virtual"

scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.filename = "D:Virtual Machinesocfs.vmdk"
scsi1:1.deviceType = "plainDisk"

scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.filename = "D:Virtual Machinesasm1.vmdk"
scsi1:2.deviceType = "plainDisk"

scsi1:3.present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.filename = "D:Virtual Machinesasm2.vmdk"
scsi1:3.deviceType = "plainDisk"

scsi1:4.present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.filename = "D:Virtual Machinesasm3.vmdk"
scsi1:4.deviceType = "plainDisk"

scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.filename = "D:Virtual Machinesasm4.vmdk"
scsi1:5.deviceType = "plainDisk"

disk.locking = "false"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.DataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"

2.配置兩臺機的IP及HOSTS
[root@node1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost

172.17.61.130 node1pub
172.17.61.131 node2pub

10.1.1.1 node1prv
10.1.1.2 node2prv

172.17.61.140 node1vip
172.17.61.141 node2vip

3.建立組和使用者（兩個node)
[root@node2 .ssh]# groupadd oinstall
[root@node2 .ssh]# groupadd dba
[root@node2 .ssh]# useradd oracle -g oinstall -G dba
[root@node2 .ssh]# mkdir -p /u01/app/oracle
[root@node2 .ssh]# chown -R oracle /u01/app/oracle
[root@node2 .ssh]# chmod -R 775 /u01/app/oracle
[root@node2 .ssh]# passwd oracle
這個目錄給 ocfs2用來裝OCR, CRS voting 的

4.配置SSH（rsa/dsa都要配）
而且必須SSH透過自己（注意是ORACLE使用者）
測試一下：
ssh node1pub date
ssh node2pub date
ssh node1prv date
ssh node2prv date

5.修改/etc/sysctl.conf,新增引數
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144

kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000

6.修改/etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

7.修改 pam.d login
vi /etc/pam.d/login
最後新增 session required /lib/security/pam_limits.so

8. 增加 hancheck-timer 模組在2個node上

先看一下有沒有 find /lib/modules -name "hangcheck-timer.ko"

然後加上去 echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >>

/etc/modprobe.conf
然後把這個模組調起來 modprobe hangcheck-timer
然後檢查是否工作正常了 grep Hangcheck /var/log/messages | tail -2
看到類似這個的資訊就表明工作正常了 Jan 18 01:08:19 n1pub kernel: Hangcheck: starting hangcheck

timer 0.5.0 (tick is 30 seconds, margin is 180 seconds).

9.安裝ocfs2
[root@node1 rac package]# rpm -ivh ocfs2-tools-1.2.7-1.el4.i386.rpm
Preparing... ########################################### [100%]
1:ocfs2-tools ########################################### [100%]
[root@node1 rac package]# rpm -ivh ocfs2-2.6.9-42.ELsmp-1.2.8-2.el4.i686.rpm
Preparing... ########################################### [100%]
1:ocfs2-2.6.9-42.ELsmp ########################################### [100%]
[root@node1 rac package]# rpm -ivh ocfs2console-1.2.7-1.el4.i386.rpm
Preparing... ########################################### [100%]
1:ocfs2console ########################################### [100%]

10.進入X, 然後執行 ocfs2console, 把你的2個node都新增進去（兩個節點）
[root@node1pub ocfs2]# cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 172.17.61.130
number = 0
name = node1pub
cluster = ocfs2

node:
ip_port = 7777
ip_address = 172.17.61.131
number = 1
name = node2pub
cluster = ocfs2

cluster:
node_count = 2
name = ocfs2

11.開啟ocfs2(兩個節點)
[root@node1pub ocfs2]# /etc/init.d/o2cb enable
Writing O2CB configuration: OK
O2CB cluster ocfs2 already online
[root@node1pub ocfs2]# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
O2CB cluster ocfs2 already online
[root@node1pub ocfs2]# /etc/init.d/o2cb stop
Stopping O2CB cluster ocfs2: OK
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK
[root@node1pub ocfs2]# /etc/init.d/o2cb start
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

12.格式化ocfs 磁碟（一個節點）
[root@node1pub init.d]# mkfs.ocfs2 -b 4k -C 32k -L oradatafiles /dev/sdb
mkfs.ocfs2 1.2.7
Filesystem label=oradatafiles
Block size=4096 (bits=12)
Cluster size=32768 (bits=15)
Volume size=1073741824 (32768 clusters) (262144 blocks)
2 cluster groups (tail covers 512 clusters, rest cover 32256 clusters)
Journal size=67108864
Initial number of node slots: 4
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 0 block(s)
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful

13.mount ocfs2 (兩個節點)
mkdir -p /u02/oradata/orcl
mount -t ocfs2 -o datavolume /dev/sdb /u02/oradata/orcl
修改你的 /etc/fstab , 新增 類似這樣的行
/dev/sdb /u02/oradata/orcl ocfs2 _netdev,datavolume 0 0
chown -R oracle.oinstall /u02/oradata/orcl/
chmod -R 775 /u02/oradata/orcl/
到這裡，我們的ocfs2 for OCR, CRS voting 就OK了

14. 修改 /etc/sysconfig/o2cb(兩個節點)
把threshhold 的 值設定成 601
O2CB_HEARTBEAT_THRESHOLD=601

15.安裝ASM包(兩個節點)
[root@node1pub rac package]# rpm -ivh oracleasm-support-2.0.3-1.i386.rpm
Preparing... ########################################### [100%]
1:oracleasm-support ########################################### [100%]
[root@node1pub rac package]# rpm -ivh oracleasm-2.6.9-42.ELsmp-2.0.3-1.i686.rpm
Preparing... ########################################### [100%]
1:oracleasm-2.6.9-42.ELsm############################################### [100%]

[root@node1pub rac package]# rpm -ivh oracleasmlib-2.0.2-1.i386.rpm
Preparing... ########################################### [100%]
1:oracleasmlib ########################################### [100%]

16.然後執行 /etc/init.d/oracleasm configure(兩個節點)

回答 oracle , dba, y, y 就可以了

[root@node1pub rac package]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Creating /dev/oracleasm mount point: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]
[root@node1pub rac package]#

17.建立ASM
建立ASM之前，需要將幾個磁碟fdisk成分割槽，否則ASM不認。
[root@node1pub ~]# fdisk /dev/sde
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-130, default 130):
Using default value 130

Command (m for help): w
The partition table has been altered!

[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdc1
Marking disk "/dev/sdc1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL2 /dev/sdd1
Marking disk "/dev/sdd1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL3 /dev/sde1
Marking disk "/dev/sde1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm createdisk VOL4 /dev/sdf1
Marking disk "/dev/sdf1" as an ASM disk: [ OK ]
[root@node1pub ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

在另外一個節點上
/etc/init.d/oracleasm scandisks
[root@node2pub ~]# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks: [ OK ]
[root@node2pub ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

18.修改 oracle使用者目錄下的 .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/crs
export ORACLE_SID=orcl1

export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp

第二個節點的 ORACLE_SID=orcl2 其他都一樣

19.安裝CRS
1）用oracle 使用者login , 然後執行你的unzip 之後的那個clusterware目錄下的runInsaller 進入圖形安

裝
2）確認你的安裝目錄是/u01/app/oracle/product/crs
3）修改 你的clustername 從crs到一個隨便你想要的名字
然後增加
node1pub node1prv node1vip
node2pub node2prv node2vip
4）然後指定 eth0 的型別時public
5）然後制定你的OCR 和mirror
/u02/oradata/orcl/OCRFile
/u02/oradata/orcl/OCRFile_mirror
6）然後指定你的voting
/u02/oradata/orcl/VotingFile
/u02/oradata/orcl/VotingFile_mirror1
/u02/oradata/orcl/VotingFile_mirror2
7）然後就開始安裝了，當眾會讓你用完全的root身份在每個節點上執行orainstRoot.sh，你就一個節點一

個節點執行，不要搶時間，一個個來
8）然後要求你在每個節點上用完全的root許可權執行 root.sh,你也一樣，一個個來，一定要等一個做好了，

然後做另外一個
9）做第二個時候，如果告訴你你的eth0不是public,你就在第一個節點上用ROOT帳戶執行圖形介面工

具/01/app/product/crs/bin/vipca
next--&gt選擇eth0(public)--&gt加入兩個VIP就可以了。
10)如安裝時出現Remote 'AttachHome' failed on 'node2pub'.的錯誤，需要在第node2pub的節點上ORACLE

使用者執行以下commond就可以了。

[oracle@node2pub bin]$ pwd
/u01/app/oracle/product/crs/oui/bin
[oracle@node2pub bin]$ ./runInstaller -attachHome -noClusterEnabled

ORACLE_HOME=/u01/app/oracle/product/crs ORACLE_HOME_NAME=OraCrs10g_home

CLUSTER_NODES=node1pub,node2pub CRS=true "INVENTORY_LOCATION=/u01/app/oracle/oraInventory"

LOCAL_NODE=node2pub

至此CRS安裝完畢！
測試一下：
[root@node1pub oracle]# su - oracle
[oracle@node1pub ~]$ cd /u01/app/oracle/product/crs/bin
[oracle@node1pub bin]$ olsnodes -n
node1pub 1
node2pub 2

20.安裝Oracle 10gR2 database
1)用oracle使用者，執行你unzip之後的那個database目錄下的runInstaller
2) ORACLE安裝目錄指定到 /u01/app/oracle/product/10.2.0/db_1
3) 把2個node選擇上
4) 選擇 Install database Software only
5) 會要求你用完全的root許可權執行 root.sh ,分別在2個node上一一執行，不要搶時間
6) 安裝完畢

21.配置netca
1) oracle 使用者在一個node上執行 netca
2) 選擇所有node
3) 選擇 Listener configuration
4)新增一個LISTEN, 1521 port
5) 回到開始介面，選擇Naming Methods configuration
6) 把Local Naming和Easy Naming 新增進去.然後結束配置
7. 確認一下
[root@node1pub logs]# ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_NODE1PUB

[root@node2pub db_1]# ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_NODE2PUB

22.安裝database

1) 用oracle使用者執行 dbca
2) 選擇custom database
3) 輸入資料庫的全域性名，比如orcl
4) 給你的系統的不同角色建立密碼，我都選擇same as ....
5) 選擇用ASM管理資料庫
6) 輸入你前面設定的管理員密碼，並把SPFILE 設定成/u02/oradata/orcl/dbs/spfile+ASM.ora
7)進入ASM配置介面, create NEW, 名字叫做DATA
然後下面你會看到你前面建立的4個ASM VOL1-4, 你選擇 1 和2, "Redundancy" 選擇 Normal.
這樣 DATA diskgroup就建立好了
8) 回到ASM介面後. 仍舊選擇create new, 然後名字叫做FLASH_RECOVERY_AREA, 選擇第3和4兩個卷，然

後"Redundancy" 選擇External
9) 然後OK結束ASM配置
10)Database File Locations 選擇DATA
11) Recovery Configuration 選擇剛才建立的 FLASH_RECOVERY_AREA diskgroup
12) 接下來預設的就可以了
13. Database Services 這裡，你選擇Add你一個新的service, 隨便叫名字，比如ractest
然後選擇 TAF Policy,是Basic
14. 開始建立資料庫

===================步驟六: 校驗RAC===============
1.用oracle使用者login, 執行

$ srvctl status database -d orcl
Instance orcl1 is running on node node1pub
Instance orcl2 is running on node node2pub

2.
$ srvctl status service -d orcl -s ractest
Service orcltest is running on instance(s) orcl2, orcl1

3.
$ srvctl status nodeapps -n node1pub
VIP is running on node: node1pub
GSD is running on node: node1pub
Listener is running on node: node1pub
ONS daemon is running on node: node1pub

4.
$ srvctl status asm -n node1pub
ASM instance +ASM1 is running on node node1pub.

5.
$ srvctl config nodeapps -n node1pub -a -g -s -l
VIP exists.: /node1vip/192.168.1.111/255.255.255.0/eth0:eth1
GSD exists.
ONS daemon exists.
Listener exists.

6. 執行 sqlplus /nolog

SQL> connect /as sysdba
SQL>

SELECT
inst_id
, instance_number inst_no
, instance_name inst_name
, parallel
, status
, database_status db_status
, active_state state
, host_name host
FROM gv$instance
ORDER BY inst_id;

INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST
-------- -------- ---------- --- ------- ------------ --------- -------
1 1 orcl1 YES OPEN ACTIVE NORMAL node1pub
2 2 orcl2 YES OPEN ACTIVE NORMAL node2pub

7.其他的測試，測試方法n多，你用sqlplus, asmcmd, 或者透過Web進 enterprise manager都可以.