RAC11g搭建-centos7+openfiler+multipath+udev

hellohf123發表於2021-09-16
CentOSdev

環境：centos7.9

          openfiler

          oracle11.2.0.4


  1. 1IP規劃

    Hostname

    Host-alias

    type

    ip

    interface

    rac1

    rac1

    Public ip

    10.8.98.103

    Eth0

    rac1-vip

    rac1-vip

    Virtual ip

    10.8.98.105

    Eth0:1

    rac1-priv

    rac1-priv

    Private ip

    192.168.1.103

    Eth1



    rac2

    rac2

    Public ip

    10.8.98.104

    Eth0

    rac2-vip

    rac2-vip

    Virtual ip

    10.8.98.106

    Eth0:1

    rac2-priv

    rac2-priv

    Private ip

    192.168.1.104

    Eth1



    cluster-scan

    cluster-scan

    Scan ip

    10.8.98.107

    Eth0



    openfiler

    openfiler

    10.8.98.100

    Eth0

    10.8.98.101

    Eth1


   倆節點hosts檔案

[root@rac1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#public ip eth0
10.8.98.103         rac1
10.8.98.104         rac2
#priv ip eth1
192.168.1.103     rac1-priv
192.168.1.104     rac2-priv
#vip ip
10.8.98.105         rac1-vip
10.8.98.106         rac2-vip
#scan ip
10.8.98.107         cluster-scan

1.2 目錄規劃

Grid infrastructure

Oracle database

HOME=/home/oracle

HOME=/home/oracle

ORACLE_BASE=/u01/app/grid

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/11.2.0/grid

ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

 

1.3 ORACLE SID

DATABASE SID

prod

Node 1 instance sid

prod1

Node 2 instance sid

prod2

 

 

1.4 儲存規劃

儲存LUN （分割槽）規劃原則（生產環境）

1.        資料檔案和歸檔檔案底層屬於不同的raid 組，屬於不通的DG( 磁碟組)

2.        線上日誌檔案同一組中不同成員分別儲存在兩個DG 中

3.        控制檔案不同成員分別儲存在兩個DG 中

4.        Ocr 和votedisk 存放在資料之外單獨的ASM 磁碟組

 

ASM 磁碟劃分

Rac 節點

Lun name

Asm 磁碟名稱/alias

Size

ASM 磁碟組名稱

冗餘型別

Rac1

asmocrvote1

2G

SYSTEMDG

High/normal

asmocrvote2

2G

asmocrvote3

2G

asmdata01

200G

DATADG1

external

asmdata02

200G

Asmarch01

200G

FRADG

external




Rac2

asmocrvote1

2G

SYSTEMDG

High/normal

asmocrvote2

2G

asmocrvote3

2G

asmdata01

200G

DATADG1

external

asmdata02

200G

Asmarch01

200G

FRADG

external

 

此處我做實驗，用的空間為1G 1G 1G 20G 20G 10G

 

openfile的搭建參考 http://blog.itpub.net/70004783/viewspace-2791487/

本地yum源的搭建參考   http://blog.itpub.net/70004783/viewspace-2789438/  



安裝需要的軟體列表

yum install binutils -y
yum install compat-libcap1 -y
yum install compat-libstdc -y
yum install gcc -y
yum install gcc-c++ -y
yum install glibc -y
yum install glibc-devel -y
yum install ksh -y
yum install libgcc -y
yum install libstdc++ -y
yum install libstdc++-devel -y
yum install libaio -y
yum install libaio-devel -y
yum install libXext -y
yum install libXtst -y
yum install libX11 -y
yum install libXau -y
yum install libxcb -y
yum install libXi -y
yum install make -y
yum install sysstat -y
yum install unixODBC -y
yum install unixODBC-devel –y
yum install elfutils-libelf-devel* -y
yum install compat-libcap* -y
yum install libaio-devel* -y
yum install compat-libstdc++-* -y


2.1  建立使用者組，使用者和目錄

在兩個節點增加使用者與組：

/usr/sbin/groupadd -g 501 oinstall
/usr/sbin/groupadd -g 502 dba
/usr/sbin/groupadd -g 503 oper
/usr/sbin/groupadd -g 504 asmadmin
/usr/sbin/groupadd -g 505 asmoper
/usr/sbin/groupadd -g 506 asmdba
/usr/sbin/useradd -g oinstall -G dba,asmdba,oper oracle
/usr/sbin/useradd -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid


修改使用者密碼：

passwd grid

passwd oracle

登陸測試（必須做），分別用oracle,grid使用者透過ssh登入，按提示修改密碼


目錄

mkdir  -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oraInventory
chown -R grid:oinstall /u01
 
mkdir  -p /u01/app/oracle
chown     oracle:oinstall    /u01/app/oracle
 
chmod -R 775 /u01


2.修改作業系統引數2節點

修改/etc/security/limits.conf檔案，增加內容如下：

vi /etc/security/limits.conf

#ORACLE SETTING
grid                 soft    nproc   2047 
grid                 hard    nproc   16384
grid                 soft    nofile  1024
grid                 hard    nofile  65536
grid   				soft   stack    10240
grid   				hard   stack    32768
oracle               soft    nproc   2047
oracle               hard    nproc   16384
oracle               soft    nofile  1024
oracle               hard    nofile  65536
oracle   			soft   stack    10240
oracle  			 hard   stack    32768


3.禁用SELINUX

vi /etc/selinux/config      修改下面

SELINUX=disabled

 

讓SELINUX生效：

setenforce 0


4.關閉防火牆

systemctl stop firewalld
systemctl disable firewalld

 

 

5、修改/etc/pam.d/login檔案

vi /etc/pam.d/login  新增如下 

#ORACLE SETTING
session    required     pam_limits.so



6、修改sysctl.conf配置檔案

 vi /etc/sysctl.conf    新增如下

#ORACLE SETTING
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmax = 2147483648
kernel.shmall = 524288
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586

#kernel.shmmax = 2147483648  (大於SGA，小於實體記憶體)

#kernel.shmall = 524288   （實體記憶體/4K）

如果以上引數設定不正確，可能導致以下報錯：

ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device .

讓配置生效：

sysctl -p


7.禁止NTP




8.配置grid和oracle使用者的環境變數

節點1 配置grid

[grid@rac1:/home/grid]$cat .bash_profile 
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
PS1="[`whoami`@`hostname`:"'$PWD]$'
export PS1
umask 022
alias sqlplus="rlwrap sqlplus"
export TMP=/tmp
export
export TMPDIR=$TMP
export ORACLE_HOSTNAME=rac1
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
PATH=.:$PATH:$HOME/bin:$ORACLE_HOME/bin; export PATH
THREADS_FLAG=native; export THREADS_FLAG
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
            ulimit -p 16384
              ulimit -n 65536
  else
   ulimit -u 16384 -n 65536
      fi
    umask 022
fi
[grid@rac1:/home/grid]$

節點2 一樣，只需要更改以下2行

export ORACLE_HOSTNAME=rac2
ORACLE_SID=+ASM2; export ORACLE_SID


節點1配置oracle

[oracle@rac1:/home/oracle]$cat .bash_profile 
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
PS1="[`whoami`@`hostname`:"'$PWD]$'
alias sqlplus="rlwrap sqlplus"
alias rman="rlwrap rman"
export PS1
export TMP=/tmp
export
export TMPDIR=$TMP
export ORACLE_HOSTNAME=rac1
export ORACLE_UNQNAME=prod
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; export ORACLE_HOME
ORACLE_SID=prod1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
NLS_LANG=AMERICAN_AMERICA.ZHS16GBK;export NLS_LANG
PATH=.:$PATH:$HOME/bin:$ORACLE_BASE/product/11.2.0/db_1/bin:$ORACLE_HOME/bin; export PATH
THREADS_FLAG=native; export THREADS_FLAG
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
            ulimit -p 16384
              ulimit -n 65536
  else
   ulimit -u 16384 -n 65536
      fi
    umask 022
fi

節點2一樣，需要更改2行 

export ORACLE_HOSTNAME=rac2
ORACLE_SID=prod2; export ORACLE_SID



9. 安裝配置iscsi

yum install iscsi-initiator-utils* -y 
檢查是否開機啟動
systemctl list-unit-files | grep iscsi
如果不是,則這樣設定
systemctl enable iscsid.service
systemctl enable iscsi.service


啟動iscsi服務

systemctl restart iscsi
systemctl restart iscsid
發現儲存上的iscsi目標
iscsiadm -m discovery -t st -p 10.8.98.100:3260
登陸目標
iscsiadm -m node --loginall=all


察看iscsi資訊，是不是2條鏈路


iscsiadm -m session

[root@rac1 ntp]# iscsiadm -m session

tcp: [1] 10.8.98.100:3260,1 iqn.2006-01.com.openfiler:tsn.e7c9f1f8f2df (non-flash)

tcp: [2] 10.8.98.101:3260,1 iqn.2006-01.com.openfiler:tsn.e7c9f1f8f2df (non-flash)


ls -l /dev/disk/by-path/


因為我們是手動發現的iscsi，所以如果重啟則會消失，所以將iscsi目標寫入iscsi配置檔案中

vi /etc/iscsi/initiatorname.iscsi 

InitiatorName=iqn.2006-01.com.openfiler:tsn.e7c9f1f8f2df


重啟iscsi服務

systemctl restart iscsi
systemctl restart iscsid

重新整理新增加iscsi 儲存目標

iscsiadm -m session -R

察看iscsi通道

cat /proc/scsi/scsi

刪除/登出target連線

iscsiadm -m node --logoutall=all




10.安裝多路徑

yum install device-mapper*


檢查device-mapper有沒有安裝

rpm -qa | grep device-mapper

安裝未安裝的rpm包


設定maltipath開機啟動

[root@rac1 ntp]# systemctl list-unit-files | grep multipath

multipathd.service                            enabled 


把multipath模組加入核心

modprobe dm-multipath

 modprobe dm-round-robin


/sbin/mpathconf -help

/sbin/mpathconf --enable

啟動服務

systemctl restart multipathd


配置multipath -ll

[root@rac1 ~]# multipath -ll
mapthf (14f504e46494c45524153744b37302d556f616c2d4f746342) dm-4 OPNFILER,VIRTUAL-DISK    
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:5 sdg 8:96  active ready running
  `- 4:0:0:5 sdm 8:192 active ready running
mapthe (14f504e46494c455279736a6244732d756565502d6755736d) dm-5 OPNFILER,VIRTUAL-DISK    
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:4 sdf 8:80  active ready running
  `- 4:0:0:4 sdl 8:176 active ready running
mapthc (14f504e46494c45524e335577524e2d534a57432d32627371) dm-6 OPNFILER,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:2 sdd 8:48  active ready running
  `- 4:0:0:2 sdj 8:144 active ready running
mapthd (14f504e46494c45524d6b6f46624a2d634450442d30484c4a) dm-3 OPNFILER,VIRTUAL-DISK    
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:3 sde 8:64  active ready running
  `- 4:0:0:3 sdk 8:160 active ready running
mapthb (14f504e46494c45526a4158746b782d584b59772d6c4f4f76) dm-2 OPNFILER,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:1 sdc 8:32  active ready running
  `- 4:0:0:1 sdi 8:128 active ready running
maptha (14f504e46494c45526776575844302d74637a682d51784646) dm-7 OPNFILER,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:0 sdb 8:16  active ready running
  `- 4:0:0:0 sdh 8:112 active ready running


根據上面的磁碟大小以及wwid號

統計以及規劃如下

/14f504e46494c45526776575844302d74637a682d51784646/   1G  asmvote01
/14f504e46494c45526a4158746b782d584b59772d6c4f4f76/   1G  asmvote02
/14f504e46494c45524e335577524e2d534a57432d32627371/   1G  asmvote03
/14f504e46494c45524d6b6f46624a2d634450442d30484c4a/   20  asmdata01
/14f504e46494c45524153744b37302d556f616c2d4f746342/   20  asmarch01
/14f504e46494c455279736a6244732d756565502d6755736d/   10  asmdata02


還有兩個命令可以查詢uuid

multipath -v3 

more /etc/multipath/wwids 

根據以上得到的資料修改多路徑的配置檔案



備份配置檔案模板

[root@rac1 ~]# cp /etc/multipath.conf /etc/multipath.conf.bak2021

根據上面的規劃，改後的配置檔案如下

cat  /etc/multipath.conf

defaults {
        user_friendly_names yes
        find_multipaths yes
}
blacklist {
       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
#我這裡sda是本地硬碟，所以加黑名單
         devnode "^sda"
}
multipaths {
        multipath {
                wwid                 14f504e46494c45526776575844302d74637a682d51784646
                alias                   asmvote01
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
        }
        multipath {
                wwid                 14f504e46494c45526a4158746b782d584b59772d6c4f4f76
                alias                   asmvote02
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
        }
		multipath {
                wwid                 14f504e46494c45524e335577524e2d534a57432d32627371
                alias                   asmvote03
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
        }
		multipath {
                wwid                 14f504e46494c45524d6b6f46624a2d634450442d30484c4a
                alias                   asmdata01
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
        }
        multipath {
                wwid                 14f504e46494c455279736a6244732d756565502d6755736d
                alias                    asmdata02
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
        }
        multipath {
                wwid                 14f504e46494c45524153744b37302d556f616c2d4f746342
                alias                    asmarch01
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
        }
}
devices {
       device {
               vendor                  "openfiler  "
               product                 "virtual disk"
               path_grouping_policy    multibus
               path_checker            readsector0
               path_selector           "round-robin 0"
               hardware_handler        "0"
       }
}


節點2同理



刪除現有多路徑

multipath -F

格式化多路徑

multipath -v2

倆節點

multipath -ll

[root@rac1 ~]# multipath -ll
asmarch01 (14f504e46494c45524153744b37302d556f616c2d4f746342) dm-4 OPNFILER,VIRTUAL-DISK    
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:5 sdg 8:96  active ready running
  `- 4:0:0:5 sdm 8:192 active ready running
asmdata02 (14f504e46494c455279736a6244732d756565502d6755736d) dm-5 OPNFILER,VIRTUAL-DISK    
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:4 sdf 8:80  active ready running
  `- 4:0:0:4 sdl 8:176 active ready running
asmvote03 (14f504e46494c45524e335577524e2d534a57432d32627371) dm-6 OPNFILER,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:2 sdd 8:48  active ready running
  `- 4:0:0:2 sdj 8:144 active ready running
asmdata01 (14f504e46494c45524d6b6f46624a2d634450442d30484c4a) dm-3 OPNFILER,VIRTUAL-DISK    
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:3 sde 8:64  active ready running
  `- 4:0:0:3 sdk 8:160 active ready running
asmvote02 (14f504e46494c45526a4158746b782d584b59772d6c4f4f76) dm-2 OPNFILER,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:1 sdc 8:32  active ready running
  `- 4:0:0:1 sdi 8:128 active ready running
asmvote01 (14f504e46494c45526776575844302d74637a682d51784646) dm-7 OPNFILER,VIRTUAL-DISK    
size=1.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 3:0:0:0 sdb 8:16  active ready running
  `- 4:0:0:0 sdh 8:112 active ready running
[root@rac1 ~]#



11.配置udev

cd /usr/share/doc/device-mapper-1.02.170/

cp 12-dm-permissions.rules /etc/udev/rules.d/

[root@rac1 ~]# vi /etc/udev/rules.d/12-dm-permissions.rules

在最後新增如下

# OWNER:="root", GROUP:="root", MODE:="660"
LABEL="dm_end"
ENV{DM_NAME}=="asmvote01", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmvote02", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmvote03", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmdata01", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmdata02", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_NAME}=="asmarch01", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
[root@rac1 ~]#


啟動udev

[root@rac1 rules.d]#  /sbin/udevadm trigger --type=devices --action=change

檢查是否正確

[root@rac1 ~]# ll /dev/dm* 
brw-rw---- 1 root disk     253, 0 Sep 15 16:57 /dev/dm-0
brw-rw---- 1 root disk     253, 1 Sep 15 16:57 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 2 Sep 15 16:58 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 3 Sep 15 16:58 /dev/dm-3
brw-rw---- 1 grid asmadmin 253, 4 Sep 15 16:58 /dev/dm-4
brw-rw---- 1 grid asmadmin 253, 5 Sep 15 16:58 /dev/dm-5
brw-rw---- 1 grid asmadmin 253, 6 Sep 15 16:58 /dev/dm-6
brw-rw---- 1 grid asmadmin 253, 7 Sep 15 16:58 /dev/dm-7
[root@rac1 ~]#


不重啟，重新應用udev規則

/sbin/udevadm control --reload-rules && udevadm trigger



2節點重啟系統，檢查如上則可以了

[root@rac1 ~]# ll /dev/dm*



12.安裝叢集軟體（只擷取重要步驟）

[root@rac1 ~]# yum install tigervnc*
[grid@rac1:/home/grid]$vncserver

grid使用者vnc圖形介面安裝（之前遇到子視窗無法拉開的情況）可以加如下引數


./runInstaller -jreLoc /etc/alternatives/jre_1.8.0


倆節點安裝

[root@rac1 grid]# yum install elfutils-libelf-devel
 
[root@rac1 rpm]# pwd
/home/grid/grid/rpm
[root@rac1 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm



執行第二個指令碼的時候卡住不動了（2節點都需要）

解決方案參考 https://blog.csdn.net/shayuwei/article/details/90483029



此處報錯可以忽視，繼續



13.建立rac，只安裝軟體不建庫（只擷取部分）

同上，加引數，子圖形視窗可以調整

[oracle@rac1:/home/oracle/database]$./runInstaller -jreLoc /etc/alternatives/jre_1.8.0


解決方案參照 http://blog.itpub.net/70004783/viewspace-2789569/


14，asmca新增磁碟


15.dbca建庫



建庫完成


16.檢視資料庫是否正常。

oracle使用者檢視

[oracle@rac1:/home/oracle]$sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Thu Sep 16 08:57:30 2021
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select name from v$datafile;
NAME
--------------------------------------------------------------------------------
+DATADG01/prod/datafile/system.265.1083366415
+DATADG01/prod/datafile/sysaux.266.1083366415
+DATADG01/prod/datafile/undotbs1.267.1083366415
+DATADG01/prod/datafile/users.268.1083366415
+DATADG01/prod/datafile/example.273.1083366513
+DATADG01/prod/datafile/undotbs2.274.1083366691
6 rows selected.
SQL>


grid使用者檢視

[grid@rac1:/home/grid/grid]$crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....DG01.dg ora....up.type ONLINE    ONLINE    rac1        
ora....DG01.dg ora....up.type ONLINE    ONLINE    rac1        
ora....DG02.dg ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....DG01.dg ora....up.type ONLINE    ONLINE    rac1        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac1        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac1        
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora.prod.db    ora....se.type ONLINE    ONLINE    rac1        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1        
[grid@rac1:/home/grid/grid]$

root使用者環境變數新增下面紅色部分

PATH=$PATH:$HOME/bin :/u01/app/11.2.0/grid/bin

測試叢集啟停，生產環境，建議手動停止資料庫

[root@rac1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.SYSTEMDG01.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.prod.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2676: Start of 'ora.cvu' on 'rac2' succeeded
CRS-2677: Stop of 'ora.prod.db' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ARCHDG01.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.DATADG01.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.DATADG02.dg' on 'rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2677: Stop of 'ora.DATADG02.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATADG01.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.ARCHDG01.dg' on 'rac1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rac2'
CRS-2677: Stop of 'ora.SYSTEMDG01.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 ~]#

所有服務都跑到節點2上了 

[grid@rac2:/home/grid]$crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....DG01.dg ora....up.type ONLINE    ONLINE    rac2        
ora....DG01.dg ora....up.type ONLINE    ONLINE    rac2        
ora....DG02.dg ora....up.type ONLINE    ONLINE    rac2        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora....DG01.dg ora....up.type ONLINE    ONLINE    rac2        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac2        
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac2        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    rac2        
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    rac2        
ora.ons        ora.ons.type   ONLINE    ONLINE    rac2        
ora.prod.db    ora....se.type ONLINE    ONLINE    rac2        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac2        
[grid@rac2:/home/grid]$


來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/70004783/viewspace-2792218/，如需轉載，請註明出處，否則將追究法律責任。

相關文章