Oracle 無備份情況恢復ocr和olr

Oracle 11.2 GI中OLR損壞的修復方法

一、先關閉GI，命令不再貼出來了，參見：
http://blog.csdn.net/msdnchina/article/details/49402299

二、模擬OLR損壞

[root@hostb cdata]# ls -lrt
total 2884
drwxr-xr-x 2 grid oinstall      4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06  drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 hostb.olr
[root@hostb cdata]# mv hostb.olr  back_hostb.olr ----------->  mv掉OLR
[root@hostb cdata]# ls -lrt
total 2884
drwxr-xr-x 2 grid oinstall      4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 back_hostb.olr
[root@hostb cdata]# /u01/app/11.2.0/grid/bin/crsctl start crs   --------->起不來。
CRS-4124:  High Availability Services startup failed.
CRS-4000:  Start failed, or completed with errors.
[root@hostb cdata]# pwd
/u01/app/11.2.0/grid/cdata
[root@hostb cdata]#

[oracle@hostb ohasd]$ pwd
/u01/app/11.2.0/grid/log/hostb/ohasd
[oracle@hostb ohasd]$ ls -lrt
total 95720
-rw-r--r-- 1 root root 10540929 Nov  4 03:51 ohasd.l09
-rw-r--r-- 1 root root 10556558 Nov  8 16:28 ohasd.l08
-rw-r--r-- 1 root root 10558423 Nov 13 12:39 ohasd.l07
-rw-r--r-- 1 root root 10558697 Nov 18 05:44 ohasd.l06
-rw-r--r-- 1 root root 10557602 Nov 22 18:15 ohasd.l05
-rw-r--r-- 1 root root 10532670 Nov 27 08:24 ohasd.l04
-rw-r--r-- 1 root root 10556869 Dec  2 01:30 ohasd.l03
-rw-r--r-- 1 root root 10557883 Dec  6 23:49 ohasd.l02
-rw-r--r-- 1 root root 10542530 Dec 21 11:49 ohasd.l01
-rw-r--r-- 1 root root     6700 Jan  8 13:42 ohasdOUT.log
-rw-r--r-- 1 root root  2986628 Jan  8 13:42 ohasd.log
[oracle@hostb ohasd]$ vi ohasd.log

2016-01-08 13:42:59.562: [ default][3149834016] Initializing OLR
2016-01-08 13:42:59.564: [  OCROSD][3149834016]utopen:6m': failed in stat OCR file/disk /u01/app/11.2.0/grid/cdata/hostb.olr, errno=2, os err string=No such file or directory
2016-01-08 13:42:59.564: [  OCROSD][3149834016]utopen:7: failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2016-01-08 13:42:59.564: [  OCRRAW][3149834016]proprinit: Could not open raw device
2016-01-08 13:42:59.564: [  OCRAPI][3149834016]a_init:16!: Backend init unsuccessful : [26]
2016-01-08 13:42:59.564: [  CRSOCR][3149834016] OCR context init failure.  Error: PROCL-26: Error while accessing the physical storage Operating  error [No such file or directory] [2]
2016-01-08 13:42:59.565: [ default][3149834016] Created alert : (:OHAS00106:) :  OLR initialization failed, error: PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
2016-01-08 13:42:59.565: [ default][3149834016][PANIC] OHASD exiting; Could not init OLR
2016-01-08 13:42:59.565: [ default][3149834016] Done.

開始對本地節點（主機名為hostb）進行OLR的recreate,參考文章：
Top 5 Grid Infrastructure Startup Issues (文件 ID 1368382.1) ，搜尋關鍵字為"OLR"

[root@hostb bin]# ./crsctl stop crs
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.

第一步：# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force

[root@hostb install]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle clusterware stack on this node
[root@hostb install]#

此時的/u01/app/11.2.0/grid/cdata目錄情況：

[grid@hostb cdata]$ ls -lrt
total 13124
drwxr-xr-x 2 grid oinstall      4096 Oct 30 13:21 hostb
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 back_hostb.olr
-rw-r--r-- 1 root root      10485760 Jan 12 18:08 hostb.olr
[grid@hostb cdata]$

第二步：# /u01/app/11.2.0/grid/root.sh

[root@hostb install]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by  Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-2672: Attempting to start 'ora.mdnsd' on 'hostb'
CRS-2676: Start of 'ora.mdnsd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hostb'
CRS-2676: Start of 'ora.gpnpd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hostb'
CRS-2672: Attempting to start 'ora.gipcd' on 'hostb'
CRS-2676: Start of 'ora.cssdmonitor' on 'hostb' succeeded
CRS-2676: Start of 'ora.gipcd' on 'hostb' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hostb'
CRS-2672: Attempting to start 'ora.diskmon' on 'hostb'
CRS-2676: Start of 'ora.diskmon' on 'hostb' succeeded
CRS-2676: Start of 'ora.cssd' on 'hostb' succeeded
PRKO-2190 : VIP exists for node hostb, VIP name hostb-vip
PRCS-1037 : Single Client Access Name VIPs already exist
PRCS-1028 : Single Client Access Name listeners already exist
OC4J could not be created as it already exists
PRCR-1086 : resource ora.oc4j is already registered
PRCR-1086 : resource ora.cvu is already registered
PRCC-1014 : scan1 was already running
PRCR-1004 : Resource ora.scan1.vip is already running
PRCR-1079 : Failed to start resource ora.scan1.vip
CRS-5702: Resource 'ora.scan1.vip' is already running on 'hostb'

PRCC-1014 : LISTENER_SCAN1 was already running
PRCR-1004 : Resource ora.LISTENER_SCAN1.lsnr is already running
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN1.lsnr
CRS-5702: Resource 'ora.LISTENER_SCAN1.lsnr' is already running on 'hostb'

PRCC-1014 : oc4j was already running
PRCR-1004 : Resource ora.oc4j is already running
PRCR-1079 : Failed to start resource ora.oc4j
CRS-5702: Resource 'ora.oc4j' is already running on 'hostb'
CRS-2672: Attempting to start 'ora.registry.acfs' on 'hostb'
CRS-2676: Start of 'ora.registry.acfs' on 'hostb' succeeded
PRCC-1014 : cvu was already running
PRCR-1004 : Resource ora.cvu is already running
PRCR-1079 : Failed to start resource ora.cvu
CRS-5702: Resource 'ora.cvu' is already running on 'hostb'
Preparing s for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@hostb install]#

完成後的/u01/app/11.2.0/grid/cdata目錄情況：

[grid@hostb cdata]$ ls -lrt
total 5612
drwxr-xr-x 2 grid oinstall      4096 Oct 30 19:06 localhost
drwxrwxr-x 2 grid oinstall      4096 Dec 22 13:27 hosta-cluster
-rw------- 1 root oinstall 272756736 Jan  8 13:38 back_hostb.olr
drwxr-xr-x 2 grid oinstall      4096 Jan 12 18:19 hostb
-rw------- 1 root oinstall 272756736 Jan 12 18:19 hostb.olr
[grid@hostb cdata]$

[grid@hostb cdata]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       hostb                                      
ora.DATAFILE.dg
               ONLINE  ONLINE       hostb                                      
ora.FRA.dg
               ONLINE  ONLINE       hostb                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       hostb                                      
ora.asm
               ONLINE  ONLINE       hostb                   Started            
ora.gsd
               OFFLINE OFFLINE      hostb                                      
ora.net1. ONLINE  ONLINE       hostb                                      
ora.ons
               ONLINE  ONLINE       hostb                                      
ora.registry.acfs
               ONLINE  ONLINE       hostb                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       hostb                                      
ora.cvu
      1        ONLINE  ONLINE       hostb                                      
ora.orcl.db
      1        ONLINE  OFFLINE                                                  
      2        ONLINE  ONLINE       hostb                   Open               
ora.orcl.orclxa.svc
      1        ONLINE  ONLINE       hostb                                      
ora.orcla.vip
      1        ONLINE  INTERMEDIATE hostb                   FAILED OVER        
ora.hostb.vip
      1        ONLINE  ONLINE       hostb                                      
ora.oc4j
      1        ONLINE  ONLINE       hostb                                      
ora.scan1.vip
      1        ONLINE  ONLINE       hostb                                      
[grid@hostb cdata]$

[grid@hostb cdata]$ lsnrctl status

LSNRCTL for :  11.2.0.3.0 - Production on 12-JAN-2016 18:21:34

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                12-JAN-2016 18:18:21
Uptime                    0 days 0 hr. 3 min. 13 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/hostb/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.182)(PORT=1528)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.188.184)(PORT=1528)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "orcl" has 1 instance(s).
  Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclXDB" has 1 instance(s).
  Instance "orcl2", status READY, has 1 handler(s) for this service...
Service "orclxa" has 1 instance(s).
  Instance "orcl2", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@hostb cdata]$

[oracle@hostb ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Jan 12 18:29:52 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

Oracle RAC無備份情況恢復ocr和olr

原文地址：

今天是2014-04-01，忙碌了一天終於有時間寫點東西了。前段時間寫了RAc中ocr存在備份的情況下如何恢復，今天寫一下在沒有備份情況下如何去重建ocr和olr呢？
最大的悲劇莫過於資料庫壞了備份沒了。讓我想起了前幾年流行的一句話：“最大的悲劇是人活著，錢沒了”。總之備份重於一切。
情景介紹：
在11G中ocr的asm磁碟組被使用者不小心刪了且所有備份都誤刪，olr呢也損壞了。至此悲劇重演。
恢復過程：（其實就是使用root.sh重建ocr的過程，重建之後可能需要重新註冊相關資源如listener or database 例項等！）
1、清空所有節點cluster配置資訊：（11G中存在rootcrs.sh指令碼，該指令碼預設存在於$GRID_HOME/crs/install）。
注，該指令碼需要在root下執行，否則會有錯誤提示
[grid@rac-one install]$ ./rootcrs.pl
You must be logged in as root to run this script.
Log in as root and rerun this script.
2014-04-01 17:08:12: Not running as authorized user
Insufficient privileges to execute this script.
root or administrative privileges needed to run the script.
瞭解該指令碼的功能可以檢視-help。如升級、回退、重新配置等待》
[root@rac-two install]# ./rootcrs.pl -help
Usage:
rootcrs.pl [-verbose] [-upgrade [-force] | -patch]
[-paramfile ]
[-deconfig [-deinstall] [-keepdg] [-force] [-lastnode]]
[-downgrade -oldcrshome -version [-force] [-lastnode]]
[-unlock [-crshome ] [-nocrsstop]]
[-init]

Options:
-verbose Run this script in verbose mode
-upgrade Oracle HA is being upgraded from previous version
-patch Oracle HA is being upgraded to a patch version
-paramfile Complete path of file specifying HA parameter values
-lastnode Force the node this script is executing on to be considered
as the last node of deconfiguration or downgrade, and perform
actions associated with deconfiguring or downgrading the
last node
-downgrade Downgrade the clusterware
-version For use with downgrade; special handling is required if
downgrading to 9i. This is the old crs version in the format
A.B.C.D.E (e.g 11.1.0.6.0).
-deconfig Remove Oracle Clusterware to allow it to be uninstalled or reinstalled
-force Force the execution of steps in delete or dwongrade that cannot
be verified to be safe
-deinstall Reset the permissions on CRS home during de-configuration
-keepdg Keep existing diskgroups during de-configuration
-unlock Unlock CRS home
-crshome Complete path of crs home. Use with unlock option
-oldcrshome For use with downgrade. Complete path of the old crs home
-nocrsstop used with unlock option to reset permissions on an inactive grid home
-init Reset the permissions of all files and directories under CRS home

If neither -upgrade nor -patch is supplied, a new install is performed

To see the full manpage for this program, execute:
perldoc rootcrs.pl

[root@rac-two install]#
在root下清除所有節點cluster 資訊：
節點2：
[grid@rac-one install]$ su
Password:
[root@rac-one install]# ./rootcrs.pl -deconfig -force
Using configuration parameter file: ./crsconfig_params
Network exists: 1/192.168.4.0/255.255.255.0/eth0, type static
VIP exists: /rac-one-vip/192.168.4.113/192.168.4.0/255.255.255.0/eth0, hosting node rac-one
VIP exists: /rac-two-vip/192.168.4.114/192.168.4.0/255.255.255.0/eth0, hosting node rac-two
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac-one'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac-one' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-one'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac-one'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac-one'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac-one'
CRS-2673: Attempting to stop 'ora.GIDG.dg' on 'rac-one'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac-one'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.GIDG.dg' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac-one'
CRS-2677: Stop of 'ora.oc4j' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rac-two'
CRS-2677: Stop of 'ora.asm' on 'rac-one' succeeded
CRS-2676: Start of 'ora.oc4j' on 'rac-two' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac-one' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac-one'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-one'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac-one'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-one'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac-one'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac-one'
CRS-2677: Stop of 'ora.crf' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-one'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-one'
CRS-2677: Stop of 'ora.cssd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac-one'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac-one' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac-one'
CRS-2677: Stop of 'ora.gpnpd' on 'rac-one' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-one' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
[root@rac-one install]#
節點1，因為我的RAC是雙節點的，那麼在清除最後一個使用lastnode引數：
[grid@rac-two crs]$ cd install/
[grid@rac-two install]$ su
Password:
[root@rac-two install]# ./rootcrs.pl -deconfig -force -lastnode
Using configuration parameter file: ./crsconfig_params
CRS resources for listeners are still configured
Network exists: 1/192.168.4.0/255.255.255.0/eth0, type static
VIP exists: /rac-two-vip/192.168.4.114/192.168.4.0/255.255.255.0/eth0, hosting node rac-two
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac-two'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'rac-two'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac-two'
CRS-2673: Attempting to stop 'ora.GIDG.dg' on 'rac-two'
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac-two'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.GIDG.dg' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac-two' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.evmd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-two'
CRS-2677: Stop of 'ora.cssd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded
CRS-4611: Successful deletion of voting disk +GIDG.
ASM de-configuration trace file location: /tmp/asmcadc_clean2014-04-01_05-14-52-PM.log
ASM Clean Configuration START
ASM Clean Configuration END

ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2014-04-01_05-14-52-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-two'
CRS-2677: Stop of 'ora.cssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac-two'
CRS-2677: Stop of 'ora.crf' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac-two'
CRS-2677: Stop of 'ora.gipcd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac-two'
CRS-2677: Stop of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-two' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
[root@rac-two install]#
二、重建ocr和olr,使用root.sh指令碼完成重建，其實這就是在安裝RAC中執行的指令碼，預設位置為：$GRID_HOME/;
eg：
節點1：
[root@rac-two grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac-two'
CRS-2676: Start of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac-two'
CRS-2676: Start of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded

ASM created and started successfully.

Disk Group GIDG created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Successful addition of voting disk 498646ba39604f86bf697c9748a67697.
Successful addition of voting disk 2e1bd16f9e6d4f36bf93550dc8268725.
Successful addition of voting disk 3fbd31a0b2634feabfa1115a504cbbe6.
Successfully replaced voting disk group with +GIDG.
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 498646ba39604f86bf697c9748a67697 (/dev/asm-diske) [GIDG]
2. ONLINE 2e1bd16f9e6d4f36bf93550dc8268725 (/dev/asm-diskd) [GIDG]
3. ONLINE 3fbd31a0b2634feabfa1115a504cbbe6 (/dev/asm-diskf) [GIDG]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac-two'
CRS-2676: Start of 'ora.asm' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.GIDG.dg' on 'rac-two'
CRS-2676: Start of 'ora.GIDG.dg' on 'rac-two' succeeded
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac-two grid]#
另外注意，如果asm磁碟損壞了那麼需要重新修復磁碟，然後會自動重新建立之前的diskgroup.
至此olr和ocr建立成功
節點2：
[root@rac-one grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac-two, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac-one grid]#
3、檢查資源資訊：
現在呢開始使用srvctl和crsctl工具，這是oracle RAC 經常使用的，另外還有一個工具是oifcfg配置網路卡資訊等。不在介紹了。

[root@rac-one bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac-one bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-two
ora.cvu
1 ONLINE ONLINE rac-two
ora.oc4j
1 ONLINE ONLINE rac-two
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.scan1.vip
1 ONLINE ONLINE rac-one
ora.scan2.vip
1 ONLINE ONLINE rac-two
ora.scan3.vip
1 ONLINE ONLINE rac-two
[root@rac-one bin]#
[root@rac-one bin]# su - grid
[grid@rac-one ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Tue Apr 1 17:40:53 2014

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select name,state from v$asm_diskgroup;

NAME STATE
------------------------------ -----------
GIDG MOUNTED
DATADG DISMOUNTED

SQL> alter diskgroup datadg mount;

Diskgroup altered.

SQL>
4、新增資源
可以看到目前啟動的缺少本地監聽和database資源和例項，那麼下面開始重新註冊這些資訊到ocr中，
註冊listener：
注：使用grid使用者：

[grid@rac-one ~]$ srvctl add listener -l listener
PRCN-2061 : Failed to add listener ora.LISTENER.lsnr
PRCN-2065 : Port(s) 1521 are not available on the nodes given
PRCN-2067 : Port 1521 is not available across node(s) "rac-two-vip"
[grid@rac-one ~]$
提示1521該埠被ora.rac-two.vip所使用，暫且跳過。

[grid@rac-one ~]$ crsctl stop resource ora.rac-two.vip
CRS-2673: Attempting to stop 'ora.rac-two.vip' on 'rac-two'
CRS-2677: Stop of 'ora.rac-two.vip' on 'rac-two' succeeded
[grid@rac-one ~]$ srvctl add listener -l listener
[grid@rac-one ~]$ crsctl start resource ora.rac-two.vip
CRS-2672: Attempting to start 'ora.rac-two.vip' on 'rac-two'
CRS-2676: Start of 'ora.rac-two.vip' on 'rac-two' succeeded
[grid@rac-one ~]$ srvctl config listener
Name: LISTENER
Network: 1, Owner: grid
Home:
End points: TCP:1521
[grid@rac-one ~]$
[grid@rac-one ~]$
[grid@rac-one ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
OFFLINE OFFLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-two
ora.cvu
1 ONLINE ONLINE rac-two
ora.oc4j
1 ONLINE ONLINE rac-two
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.scan1.vip
1 ONLINE ONLINE rac-one
ora.scan2.vip
1 ONLINE ONLINE rac-two
ora.scan3.vip
1 ONLINE ONLINE rac-two
[grid@rac-one ~]$
至此監聽新增到ocr完畢。下面開始新增db，注意使用oracle使用者：
繼續新增：

[grid@rac-one ~]$ su - oracle
Password:
[oracle@rac-one ~]$ srvctl add database -h

Adds a database configuration to the Oracle Clusterware.

Usage: srvctl add database -d -o [-c {RACONENODE | RAC | SINGLE} [-e ] [-i ] [-w ]] [-m ] [-p ] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY | SNAPSHOT_STANDBY}] [-s ] [-t ] [-n ] [-y {AUTOMATIC | MANUAL | NORESTART}] [-g ""] [-x ] [-a ""] [-j ""]
-d   Unique name for the database
-o   ORACLE_HOME path
-c   Type of database: RAC One Node, RAC, or Single Instance
-e   Candidate server list for RAC One Node database
-i   Instance name prefix for administrator-managed RAC One Node database (default first 12 characters of )
-w   Online relocation timeout in minutes
-x   Node name. -x option is specified for single-instance databases
-m   Domain for database. Must be set if database has DB_DOMAIN set.
-p   Server parameter file path
-r   Role of the database (primary, physical_standby, logical_standby, snapshot_standby)
-s   Startup options for the database. Examples of startup options are OPEN, MOUNT, or 'READ ONLY'.
-t   Stop options for the database. Examples of shutdown options are NORMAL, TRANSACTIONAL, IMMEDIATE, or ABORT.
-n   Database name (DB_NAME), if different from the unique name given by the -d option
-y   Management policy for the database (AUTOMATIC, MANUAL, or NORESTART)
-g "" Comma separated list of database server pool names
-a "" Comma separated list of disk groups
-j "" Comma separated list of ACFS paths where database's dependency will be set
-h Print usage
[oracle@rac-one ~]$ srvctl add database -d Rac -o /u01/app/oracle/product/11.2.0/db_1/ -c RAC
[oracle@rac-one ~]$
[oracle@rac-one ~]$ srvctl add instance -h

Adds a database instance configuration to the Oracle Clusterware.

Usage: srvctl add instance -d -i -n [-f]
-d   Unique name for the database
-i   Instance name
-n   Node name
-f Force the add operation even though some resource(s) will be stopped
-h Print usage
[oracle@rac-one ~]$ srvctl add instance -d Rac -i Rac1 -n rac-two
[oracle@rac-one ~]$ srvctl add instance -d Rac -i Rac2 -n rac-one
[oracle@rac-one ~]$
[oracle@rac-one ~]$ srvctl config database -d Rac
Database unique name: Rac
Database name:
Oracle home: /u01/app/oracle/product/11.2.0/db_1/
Oracle user: oracle
Spfile:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: Rac
Database instances: Rac2,Rac1
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@rac-one ~]$

重啟crs。

[root@rac-one bin]# ./crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-one'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-one' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.cssd' on 'rac-one'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-one'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-one' succeeded
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac-one'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac-one'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2676: Start of 'ora.ctssd' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac-one'CRS-2676: Start of 'ora.ctssd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac-two'
CRS-2676: Start of 'ora.evmd' on 'rac-one' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac-one'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac-two'
CRS-2676: Start of 'ora.asm' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac-two'
CRS-2676: Start of 'ora.crsd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.asm' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac-one'
CRS-2676: Start of 'ora.crsd' on 'rac-one' succeeded
再次檢視（等2分鐘）：

[root@rac-one bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
ONLINE ONLINE rac-one
ONLINE INTERMEDIATE rac-two Not All Endpoints R
egistered
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-two
ora.cvu
1 ONLINE ONLINE rac-two
ora.oc4j
1 ONLINE ONLINE rac-two
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.rac.db
1 ONLINE ONLINE rac-two Open
2 ONLINE ONLINE rac-one Open
ora.scan1.vip
1 ONLINE ONLINE rac-one
ora.scan2.vip
1 ONLINE ONLINE rac-two
ora.scan3.vip
1 ONLINE ONLINE rac-two

為什麼會存在一個監聽在rac-two上無法註冊呢？
檢視該節點監聽：

[grid@rac-two admin]$ ps -ef | grep LISTENER
grid 6120 1 0 17:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 8511 1 0 18:28 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 11010 1 0 18:42 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 11246 9191 0 18:44 pts/1 00:00:00 grep LISTENER
原來是啟動了兩個，全部停掉再次重啟：

[grid@rac-two admin]$ id
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1100(asmadmin),1200(dba),1300(asmdba),1301(asmoper)
[grid@rac-two admin]$ lsnrctl stop

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 01-APR-2014 18:45:11

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
The command completed successfully
[grid@rac-two admin]$ ps -ef | grep LISTENER
grid 6120 1 0 17:02 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid 8511 1 0 18:28 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 11452 9191 0 18:45 pts/1 00:00:00 grep LISTENER
[grid@rac-two admin]$ exit
logout
[root@rac-two bin]# kill -9 6120
[root@rac-two bin]# ps -ef | grep LISTENER
grid 8511 1 0 18:28 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
root 11518 11280 0 18:46 pts/1 00:00:00 grep LISTENER
[root@rac-two bin]# su - grid
[grid@rac-two ~]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
ONLINE ONLINE rac-one
OFFLINE OFFLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-one
ora.cvu
1 ONLINE ONLINE rac-one
ora.oc4j
1 ONLINE ONLINE rac-one
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.rac.db
1 ONLINE ONLINE rac-two Open
2 ONLINE ONLINE rac-one Open
ora.scan1.vip
1 ONLINE ONLINE rac-two
ora.scan2.vip
1 ONLINE ONLINE rac-one
ora.scan3.vip
1 ONLINE ONLINE rac-one
[grid@rac-two ~]$ crsctl start resource ora.LISTENER.lsnr
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rac-two'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rac-two' succeeded
[grid@rac-two ~]$
[grid@rac-two ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.GIDG.dg
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.LISTENER.lsnr
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.asm
ONLINE ONLINE rac-one Started
ONLINE ONLINE rac-two Started
ora.gsd
OFFLINE OFFLINE rac-one
OFFLINE OFFLINE rac-two
ora.net1.network
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.ons
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
ora.registry.acfs
ONLINE ONLINE rac-one
ONLINE ONLINE rac-two
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac-two
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac-one
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac-one
ora.cvu
1 ONLINE ONLINE rac-one
ora.oc4j
1 ONLINE ONLINE rac-one
ora.rac-one.vip
1 ONLINE ONLINE rac-one
ora.rac-two.vip
1 ONLINE ONLINE rac-two
ora.rac.db
1 ONLINE ONLINE rac-two Open
2 ONLINE ONLINE rac-one Open
ora.scan1.vip
1 ONLINE ONLINE rac-two
ora.scan2.vip
1 ONLINE ONLINE rac-one
ora.scan3.vip
1 ONLINE ONLINE rac-one
[grid@rac-two ~]$

至此所有問題得到徹底解決。

Oracle database 11g rac損壞ocr和votedisk恢復實驗
本人的操作環境：oracle database rdbms 11g rac on OEL5.5
檢視一下表決磁碟和ocr的資訊：
[root@rac1 bin]# pwd
/u01/app/11.2.0/grid/bin
[root@rac1 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 5122b184495d4fe9bf1fad29647807ba (ORCL:VOL1) [OCRVOTI]
Located 1 voting disk(s).

[root@rac1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2700
         Available space (kbytes) :     259420
         ID                       :  268167937
         Device/File Name         :   +OCRVOTI
                                    Device/File integrity check succeeded
                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

			檢視當前ocr備份情況，ocr
		
[root@rac1 bin]# ./ocrconfig -showbackup

rac2     2014/03/25 12:04:28     /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr

rac2     2014/03/21 16:16:32     /u01/app/11.2.0/grid/cdata/rac-cluster/backup01.ocr

rac2     2014/03/21 12:16:31     /u01/app/11.2.0/grid/cdata/rac-cluster/backup02.ocr

rac2     2014/03/25 12:04:28     /u01/app/11.2.0/grid/cdata/rac-cluster/day.ocr

rac2     2014/03/19 14:26:16     /u01/app/11.2.0/grid/cdata/rac-cluster/week.ocr
可以進行手工備份：
[root@rac1 bin]# ./ocrconfig -local -manualbackup
rac1     2014/03/25 14:33:39     /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_143339.olr

rac1     2014/03/25 10:34:33     /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_103433.olr

rac1     2014/03/25 09:53:32     /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_095332.olr

rac1     2014/03/25 09:53:18     /u01/app/11.2.0/grid/cdata/rac1/backup_20140325_095318.olr

rac1     2014/03/18 10:57:51     /u01/app/11.2.0/grid/cdata/rac1/backup_20140318_105751.olr

				

			
在asmcmd的md_backup命令備份磁碟組，順便檢視該磁碟組都存放什麼？？
[grid@rac1 ~]$ asmcmd -p
ASMCMD [+] > md_backup /home/grid/ocrvote2.bak -G OCRVOTI
Disk group metadata to be backed up: OCRVOTI
Current alias directory path: rac-cluster
Current alias directory path: rac-cluster/ASMPARAMETERFILE
Current alias directory path: rac-cluster/OCRFILE

			也可以手工匯出ocr內容

[root@rac1 bin]# ./ocrconfig -export /home/grid/ocr2.bak
		

我們可以破壞存放ocr的裝置檔案
[root@rac1 bin]# dd if=/dev/zero of=/dev/sdg bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.002366 seconds, 443 MB/s

然哈停止叢集：
[root@rac1 bin]# ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCRVOTI.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'
CRS-2673: Attempting to stop 'ora.test.db' on 'rac1'
CRS-2673: Attempting to stop 'ora.gsd' on 'rac1'
CRS-2677: Stop of 'ora.gsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' su 忽略。。。。。
我們在啟動clusterware 發現無法啟動了
[root@rac1 bin]# ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

				ocr和vote disk損壞恢復步驟大致如下：
			

				1）停止所有節點clusterware

# crsctl stop crs

# crsctl stop crs -f

2）以root使用者在其中一個節點獨佔模式啟動clusterware

# crsctl start crs -excl -nocrs

備註：如果發現crsd在執行，那麼透過如下命令將之停止。

# crsctl stop resource ora.crsd -init

3）建立新的存放ocr和vote disk的磁碟組，磁碟組名和原有的一致(如果想改變位置，需修改/etc/oracle/ocr.loc檔案)

備註：如發現無法建立等情況，可以採用如下刪除磁碟組等排錯思路

SQL> drop diskgroup disk_group_name force including contents;

4）還原ocr，並檢查

# ocrconfig -restore file_name

# ocrcheck

5）恢復表決磁碟，並檢查

# crsctl replace votedisk +asm_disk_group

# crsctl query css votedisk

6）停止獨佔模式執行的clusterware

# crsctl stop crs -f

7）所有節點正常啟動clusterware

# crsctl start crs

8）CVU驗證所有RAC節點OCR的完整性

$ cluvfy comp ocr -n all -verbose
			

			

		

About Me

...............................................................................................................................

● 本文整理自網路

● 本文在itpub（http://blog.itpub.net/26736162）、部落格園（http://www.cnblogs.com/lhrbest）和個人微信公眾號（xiaomaimiaolhr）上有同步更新

● 本文itpub地址：http://blog.itpub.net/26736162/abstract/1/

● 本文部落格園地址：http://www.cnblogs.com/lhrbest

● 本文pdf版及小麥苗雲盤地址：http://blog.itpub.net/26736162/viewspace-1624453/

● 資料庫筆試面試題庫及解答：http://blog.itpub.net/26736162/viewspace-2134706/

● QQ群：230161599 微信群：私聊

● 聯絡我請加QQ好友(646634621)，註明新增緣由

● 於 2017-06-02 09:00 ~ 2017-06-30 22:00 在魔都完成

● 文章內容來源於小麥苗的學習筆記，部分整理自網路，若有侵權或不當之處還請諒解

● 版權所有，歡迎分享本文，轉載請保留出處

...............................................................................................................................

拿起手機使用微信客戶端掃描下邊的左邊圖片來關注小麥苗的微信公眾號：xiaomaimiaolhr，掃描右邊的二維碼加入小麥苗的QQ群，學習最實用的資料庫技術。

Oracle 無備份情況恢復ocr和olr

Oracle 無備份情況恢復ocr和olr

Oracle RAC無備份情況恢復ocr和olr

相關文章