DBA強化實戰系列第四期：在企業私有云vmware上部署oracle11.2.0.4 rac

    隨著企業私有云的普及和發展，現在越來越多的人開始在雲平臺上建立大型應用。有的企業也開始把oracle rac搭建在企業私有云上面，我們也接到了這樣的一個活，到現場為客戶實施。現在我們就在私有云上面搭建rac的詳細步驟進行說明。

    本案例是基於vmware5.1虛擬平臺，作業系統採用的centos6.4搭建一個oracle11.2.0.4 rac
       
1、在虛擬機器上安裝作業系統，centos6.4，這個問題都不是很大

    安裝的時候，特別要注意，配置dns。雖然dns不是必須的，但我們還是建議，在安裝的過程中，把這個配置進去，這樣可以配置多個scan-ip，便於單ip故障！
    在後面配置也沒有問題，
   
主要設定如下：

[root@mytest1 etc]# cat resolv.conf
# Generated by NetworkManager
search hdjt.hdad.local
nameserver 10.10.100.8
nameserver 10.10.100.9

[root@mytest2 network-scripts]# cat ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
UUID=46b7400b-8087-4298-8f10-4a5532ce42b4
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
HWADDR=00:50:56:95:45:B8
IPADDR=10.10.92.182
PREFIX=24
GATEWAY=10.10.92.254
DNS1=10.10.100.8
DNS2=10.10.100.9
DOMAIN=hdjt.hdad.local
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"

注意：基本上配置這兩個地方就沒問題了，本小節中兩臺機器均要配置！

2、配置作業系統環境

--編輯語言環境
[root@mytest1 etc]# cd /etc/sysconfig
[root@mytest1 sysconfig]# vi i18n

#LANG="zh_CN.UTF-8"
LANG="en_US.UTF-8"
通常情況下，配置成英文環境較好。

--編輯hosts檔案

[root@mytest1 bin]# vi /etc/hosts
#public ip
10.10.92.181  mytest1
10.10.92.182  mytest2

#priv ip
10.10.100.181  mytest1-private
10.10.100.182  mytest2-private

#vip ip
10.10.92.151  mytest1-vip
10.10.92.152  mytest2-vip

#scan ip
10.10.92.100      mytest-scan

--編輯系統引數檔案

編輯系統引數

[root@mytest1 ~]# vi /etc/sysctl.conf
# Controls the maximum shared segment size, in bytes
#kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
#kernel.shmall = 4294967296

fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 8388608
kernel.shmmax = 19327352832
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
vm.min_free_kbytes = 102400

[root@mytest1 ~]# vi /etc/security/limits.conf

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle hard stack 10240
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid hard stack 10240

這裡的兩個使用者均需要編輯，否則在切換使用者時會報錯

[root@mytest1 ~]# vi /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

[root@mytest1 ~]# vi /etc/pam.d/login

#%PAM-1.0
auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so
auth       include      system-auth
account    required     pam_nologin.so
account    include      system-auth
password   include      system-auth
# pam_selinux.so close should be the first session rule
session    required     pam_selinux.so close
session    required     pam_loginuid.so
session    optional     pam_console.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session    required     pam_selinux.so open
session    required     pam_namespace.so
session    optional     pam_keyinit.so force revoke
session    include      system-auth
-session   optional     pam_ck_connector.so
session    required     pam_limits.so

注意：在本小節中，兩臺機上都進行相同的配置即可

3、安裝必須的補丁包

--配置yum源
[root@mytest1 u02]# mv /etc/yum.repos.d/CentOS-Base.repo

--檢視光碟掛載是否有問題
[root@mytest1 u02]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              48G  6.6G   39G  15% /
tmpfs                  16G   76K   16G   1% /dev/shm
/dev/sr0              4.1G  4.1G     0 100% /media/CentOS_6.4_Final

--編輯yum配置檔案，如下所示
-bash: /etc/yum.repos.d/CentOS-Media.repo: Permission denied
[root@mytest1 u02]# vi /etc/yum.repos.d/CentOS-Media.repo

       
       
# CentOS-Media.repo
#
#  This repo can be used with mounted DVD media, verify the mount point for
#  CentOS-6.  You can use this repo and yum to install items directly off the
#  DVD ISO that we release.
#
# To use this repo, put in your DVD and use it with the other repos too:
#  yum --enablerepo=c6-media [command]
#
# or for ONLY the media repo, do this:
#
#  yum --disablerepo=\* --enablerepo=c6-media [command]
[c6-media]
name=CentOS-$releasever - Media
baseurl=file:///media/CentOS_6.4_Final
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

以下補丁包是必須安裝的
compat-libstdc++-33-3.2.3-69.el6.i686.rpm 
compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm
gcc-4.4.4-13.el6                          
gcc-4.4.7-3.el6.x86_64.rpm                
gcc-c++-4.4.7-3.el6.x86_64.rpm            
glibc-2.12-1.107.el6.i686.rpm             
glibc-devel-2.12-1.107.el6.x86_64.rpm     
glibc-devel-2.12-1.107.el6.i686.rpm       
ksh-20100621-19.el6.x86_64.rpm            
libgcc-4.4.7-3.el6.i686.rpm               
libgcc-4.4.7-3.el6.x86_64.rpm             
libstdc++-docs-4.4.7-3.el6.x86_64.rpm     
libstdc++-devel-4.4.7-3.el6.x86_64.rpm    
libstdc++-4.4.7-3.el6.x86_64.rpm          
libstdc++-devel-4.4.7-3.el6.i686.rpm      
libaio-0.3.107-10.el6.i686.rpm            
libaio-0.3.107-10.el6.x86_64.rpm          
libaio-devel-0.3.107-10.el6.i686.rpm      
libaio-devel-0.3.107-10.el6.x86_64.rpm    
elfutils-libelf-0.152-1.el6.i686.rpm

另外，pdksh-5.2.14-36.el5.i386.rpm，也要安裝，如果找不到，也可以不安裝，與ksh是衝突的！
我是兩個都安裝的，在預檢查的時候，會提示到這個包！

注意：安裝補丁包，兩臺機均要進行操作

4、建立rac使用者組及使用者  
兩臺機器上建立相同的使用者，命令及id如下：
[root@mytest1 ~] groupadd -g 501 oinstall
[root@mytest1 ~] groupadd -g 502 dba
[root@mytest1 ~] groupadd -g 503 oper
[root@mytest1 ~] groupadd -g 504 asmadmin
[root@mytest1 ~] groupadd -g 505 asmoper
[root@mytest1 ~] groupadd -g 506 asmdba
[root@mytest1 ~] useradd -g oinstall -G dba,asmdba,oper oracle
[root@mytest1 ~] useradd -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid

[root@mytest1 ~]# passwd oracle
Changing password for user oracle.
New password:
BAD PASSWORD: it is based on a dictionary word
Retype new password:
passwd: all authentication tokens updated successfully.
[root@mytest1 ~]# passwd grid
Changing password for user grid.
New password:
BAD PASSWORD: it is based on a dictionary word
Retype new password:
passwd: all authentication tokens updated successfully.

5、建目錄並授權

在兩臺機器上建成相同的目錄

[root@mytest1 ~]# mkdir -p /u01/app/oracle/product/11.2.0/db1
[root@mytest1 ~]# mkdir -p /u01/app/grid/product/11.2.0/crs
[root@mytest1 ~]# chown -R grid:oinstall /u01/app/grid
[root@mytest1 ~]# chown -R oracle:oinstall /u01/app/oracle
[root@mytest1 ~]# chmod -R 775 /u01/app

6、編輯叢集使用者引數

引數編輯，要求在兩臺機器上分別對oracle\grid進行編輯配置
[root@mytest1 ~]# su - grid
[grid@mytest1 ~]$ vi .bash_profile

PATH=$PATH:$HOME/bin
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db1; export ORACLE_HOME
ORACLE_SID=+ASM; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
NLS_LANG=AMERICAN_AMERICA.ZHS16GBK;export NLS_LANG
export PATH
if [ $USER = "oracle" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
         ulimit -p 16384
              ulimit -n 65536
        else
              ulimit -u 16384 -n 65536
        fi
        umask 022
fi

[root@mytest1 ~]# su - oracle
[oracle@mytest1 ~]$ vi .bash_profile
PATH=$PATH:$HOME/bin
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db1; export ORACLE_HOME
ORACLE_SID=+ASM; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
NLS_LANG=AMERICAN_AMERICA.ZHS16GBK;export NLS_LANG
export PATH
if [ $USER = "oracle" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
         ulimit -p 16384
              ulimit -n 65536
        else
              ulimit -u 16384 -n 65536
        fi
        umask 022
fi

7、配置oracle\grid信任

--停防火牆
[root@mytest1 ~]# service iptables stop
iptables: Flushing firewall rules: [  OK  ]
iptables: Setting chains to policy ACCEPT: nat mangle filter [  OK  ]
iptables: Unloading modules: [  OK  ]
[root@mytest1 ~]# chkconfig iptables off
在兩臺機器上都關閉防火牆

--配置grid信任
[root@mytest1 ~]# su - grid
[grid@mytest1 ~]$ mkdir ~/.ssh
[grid@mytest1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
55:28:6c:8b:d2:4a:ed:7b:54:70:52:7e:32:2b:71:fa
The key's randomart image is:
+--[ RSA 2048]----+
|       . ....    |
|        *.o.     |
|     o o.** .    |
|    o + .=.=     |
|   . +  S..      |
|    . . .o       |
|       o  E      |
|      . .        |
|       .         |
+-----------------+
[grid@mytest1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
64:1d:fa:6c:94:44:23:64:07:cc:01:51:52:0e:00:13
The key's randomart image is:
+--[ DSA 1024]----+
|  Eo..=OO+*      |
|   .   =o* +     |
|        = +      |
|       o +       |
|        S +      |
|         .       |
|                 |
|                 |
|                 |
+-----------------+

以上兩步，分別在兩臺機器上進行配置

[grid@mytest1 ~]$ touch ~/.ssh/authorized_keys
[grid@mytest1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[grid@mytest1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[grid@mytest1 ~]$ ssh mytest2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'mytest2 (10.10..92.182)' can't be established.
RSA key fingerprint is 6f:ef:66:b3:32:dd:c1:19:62:b1:4c:dc:d6:cd:ed:3d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mytest2,10.10..92.182' (RSA) to the list of known hosts.
password:
[grid@mytest1 ~]$ ssh mytest2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
password:
[grid@mytest1 ~]$ scp ~/.ssh/authorized_keys mytest2:~/.ssh/authorized_keys
password:
authorized_keys                               100% 1988     1.9KB/s   00:00

這幾步，只需要在一臺機器上操作就行了

[grid@mytest1 ~]$ ssh mytest1 date
The authenticity of host 'mytest1 (10.10..92.181)' can't be established.
RSA key fingerprint is be:43:7f:84:3f:5a:8c:b9:20:43:be:8d:b4:4e:eb:b6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mytest1,10.10..92.181' (RSA) to the list of known hosts.
Fri Apr 11 16:55:02 CST 2014
[grid@mytest1 ~]$ ssh mytest2 date
Fri Apr 11 16:55:06 CST 2014
[grid@mytest1 ~]$ ssh mytest2-private date
The authenticity of host 'mytest2-private (10.10..100.182)' can't be established.
RSA key fingerprint is 6f:ef:66:b3:32:dd:c1:19:62:b1:4c:dc:d6:cd:ed:3d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mytest2-private,10.10..100.182' (RSA) to the list of known hosts.
Fri Apr 11 16:55:16 CST 2014
[grid@mytest1 ~]$ ssh mytest1-private date
The authenticity of host 'mytest1-private (10.10..100.181)' can't be established.
RSA key fingerprint is be:43:7f:84:3f:5a:8c:b9:20:43:be:8d:b4:4e:eb:b6.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mytest1-private,10.10..100.181' (RSA) to the list of known hosts.
Fri Apr 11 16:55:22 CST 2014

--配置oracle信任

與上面的方法完全相同，就不再列舉出來了

8、配置共享盤

   首先要規劃一下。需要三個盤來建crsdg，每個盤分配4g，然後歸檔、重做日誌、資料、索引、系統

這一很重要，需要關閉虛擬機器，因為涉及到對虛擬機器的引數的配置問題，是由圖形介面配置的，我把步驟說一下：

--選中你要編輯的虛擬機器，點選編輯虛擬機器
--在新的介面中，點選新增
--在新的介面中，選擇磁碟，點選下一步
--在新的介面中，選擇建立新的虛擬磁碟，點選下一步
--在新的介面中，輸入磁碟大小，選擇厚置備置零，選擇你的磁碟所要放置的底層儲存
--在新的介面中，選擇虛擬裝置節點。如果系統盤選擇的是scsi(0:n），那麼這裡建議選擇1:n的模式，這樣會生成一個新的scsi控制器。
--到最後完成，就建成了一個虛擬磁碟，這個需要點時間

回到上述步驟第一個介面，點選新生成的scsi控制器，可以看到scsi共享匯流排編輯，選擇物理，虛擬磁碟可以在任何伺服器上的虛擬機器共享

在另一臺虛擬機器上共享磁碟

--選中你要編輯的虛擬機器，點選編輯虛擬機器
--在新的介面中，點選新增
--在新的介面中，選擇磁碟，點選下一步
--在新的介面中，選擇使用現有的虛擬磁碟
--在彈出來的選項框中，根據上面所選的儲存及虛擬機器名稱，找到相應的目錄，然後把磁碟新增進去
--在新的介面中，選擇虛擬裝置節點。這裡要與上面的對應起來，保持一致。

完成了磁碟新增後，需要在配置引數中加入引數

--選中你要編輯的虛擬機器，點選編輯虛擬機器
--在彈出的介面中，左上角，點選選項
--在選項的範圍內，點選常規
--然後你會看到配置引數
--點選配置引數，加入以下內容，分別填到每行的兩個格中，有多少個盤，填多少！兩臺虛擬機器均要新增
scsi1:0.sharing  "multi-writer"
scsi1:1.sharing  "multi-writer"
scsi1:2.sharing  "multi-writer"
scsi1:3.sharing  "multi-writer"
scsi1:4.sharing  "multi-writer"
scsi1:5.sharing  "multi-writer"

注意，要跳過scsi1:7.sharing

9、繫結裸裝置

我們使用的是udev，具體操作如下：

--磁碟分割槽
[root@mytest1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
 Building a new DOS disklabel with disk identifier 0x646bfece.
 Changes will remain in memory only, until you decide to write them.
 After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
          switch off the mode (command 'c') and change display units to
          sectors (command 'u').

Command (m for help): n
 Command action
    e   extended
    p   primary partition (1-4)
 p
 Partition number (1-4): 1
 First cylinder (1-1017, default 1):
 Using default value 1
 Last cylinder, +cylinders or +size{K,M,G} (1-1017, default 1017):
 Using default value 1017

Command (m for help): w
 The partition table has been altered!

Calling ioctl() to re-read partition table.
 Syncing disks.

其它盤也進行如此操作即可

--繫結裸裝置
[root@mytest1 ~]# cd /etc/udev
[root@mytest1 udev]# cd rules.d/
[root@mytest1 rules.d]# vi 60-raw.rules

# Enter raw device bindings here.
#
# An example would be:
#   ACTION=="add", KERNEL=="sda", RUN+="/bin/raw /dev/raw/raw1 %N"
# to bind /dev/raw/raw1 to /dev/sda, or
#   ACTION=="add", ENV{MAJOR}=="8", ENV{MINOR}=="1", RUN+="/bin/raw /dev/raw/raw2 %M %m"
# to bind /dev/raw/raw2 to the device with major 8, minor 1.
ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N"
..............
KERNEL=="raw[1-8]", OWNER="grid", GROUP="dba", MODE="660"

[root@mytest1 yum.repos.d]# start_udev
[root@mytest1 yum.repos.d]# raw -qa
/dev/raw/raw1:  bound to major 8, minor 17
/dev/raw/raw2:  bound to major 8, minor 33
/dev/raw/raw3:  bound to major 8, minor 49
/dev/raw/raw4:  bound to major 8, minor 65
/dev/raw/raw5:  bound to major 8, minor 81
/dev/raw/raw6:  bound to major 8, minor 97
/dev/raw/raw7:  bound to major 8, minor 113
/dev/raw/raw8:  bound to major 8, minor 129

--在另一個節點執行，對磁碟進行確認

[root@mytest2 ~]# partprobe

執行此命令後，可以看到在另一個節點磁碟也認到了

然後像第一個節點，編輯60-raw.rules檔案

[root@mytest2 ~]# start_udev

至此，準備工作基本結束

10、安裝叢集軟體

--上傳安裝檔案
p13390677_112040_Linux-x86-64_3of7.zip
p13390677_112040_Linux-x86-64_2of7.zip
p13390677_112040_Linux-x86-64_1of7.zip

3是叢集安裝包，1和2都是rdbms安裝包
解壓軟體，rdbms安裝包，先解壓第一個，再解壓第二個

--驗證叢集安裝環境
[grid@mytest1 grid]$ ./runcluvfy.sh stage -pre crsinst -n mytest1,mytest2 -fixup -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "mytest1"
  Destination Node                      Reachable?             
  ------------------------------------  ------------------------
  mytest2                                yes                    
  mytest1                                yes                    
Result: Node reachability check passed from node "mytest1"

Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Status                 
  ------------------------------------  ------------------------
  mytest2                                passed                 
  mytest1                                passed                 
Result: User equivalence check passed for user "grid"
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "hdjt.hdad.local" as found on node "mytest2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status                 
  ------------------------------------  ------------------------
  mytest2                                passed                 
  mytest1                                passed                 
The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

Check: Time zone consistency
Result: Time zone consistency check passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

到最後，說我的沒有成功，一個是dns，主要是因為可能域伺服器忙，另一個是ntp服務，但這個並不影響的

--安裝叢集軟體
開啟xmanager的passive
[grid@mytest1 u02]$ export DISPLAY=172.17.255.25:0.0
[grid@mytest1 grid]$ ./runInstaller

具體步驟如下：
--選擇skip software updates
--選擇install and configure oracle grid infrastructure for a Cluster
--選擇Typical installation
--新增第二個節點的資訊
--輸入密碼，點選setup建立信任
--在oracle base這地方，會提示不能在同一目錄，修改base為：/u01/app/oracle
--在接下來的介面中，先選擇三個盤，建成normal的crsdg，其它的盤可以等以後再建。這些需要事先規劃好。兩個盤只能是external
--接下來的是驗證系統及變數，發現了磁碟的屬主不是asm，這個不影響，可以忽略
--下一步進行安裝
--分別執行兩個指令碼，在執行root.sh時，一定要等一臺完成後，再執行另一臺。
--到最後，有一點小錯誤，是關於ntp的，這個不影響，因為我們沒配

檢查服務可以看到已經啟動

[grid@mytest1 bin]$ ./crsctl stat resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
ora.CRSDG.dg
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
ora.DATADG.dg
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
ora.REDODG1.dg
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
ora.REDODG2.dg
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
ora.asm
               ONLINE  ONLINE       mytest1                   Started            
               ONLINE  ONLINE       mytest2                   Started            
ora.gsd
               ONLINE  OFFLINE      mytest1                                      
               ONLINE  OFFLINE      mytest2                                      
ora.net1.network
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
ora.ons
               ONLINE  ONLINE       mytest1                                      
               ONLINE  ONLINE       mytest2                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       mytest2                                                   
ora.mytest1.vip
      1        ONLINE  ONLINE       mytest1                                      
ora.mytest2.vip
      1        ONLINE  ONLINE       mytest2                                      
ora.cvu
      1        ONLINE  ONLINE       mytest1                                      
ora.oc4j
      1        ONLINE  ONLINE       mytest1                                      
ora.scan1.vip
      1        ONLINE  ONLINE       mytest2

--安裝rdbms軟體

如果能夠成功安裝叢集軟體，安裝這個應該沒什麼問題，直接按提示走就行了

--建立規劃好的dg
[grid@mytest1 bin]$ ./asmca

圖形介面操作，問題不大

--建立資料庫

在這一步，遇到了麻煩，開始始終認不到dg，感覺挺奇怪，經檢查其它的安裝都沒有問題。後來確認到是組不對！

[root@mytest1 ~]# id oracle
uid=500(oracle) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),506(asmdba)
[root@mytest1 ~]# id grid
uid=501(grid) gid=501(oinstall) groups=501(oinstall),502(dba),503(oper),504(asmadmin),505(asmoper),506(asmdba)
[root@mytest1 ~]# usermod -a -G asmadmin oracle
[root@mytest1 ~]# cd /u01/app/grid/product/11.2.0/crs/bin
[root@mytest1 bin]# chmod +s oracle

再次執行時，問題得到解決！