Oracle Database 12c RAC損壞ocr和votedisk恢復實驗
今天在Oracle 12c RAC下進行了破壞ocr和votedisk之後恢復的實驗，基本和11g RAC相差無異，下面將實驗過程分享一下。
實驗環境：2-NODES Oracle Database 12c RAC on Linux6(OEL 6.4)
檢視錶決磁碟和Ocr相關資訊
[root@12crac1 ~]# cd /u01/app/12.1.0/grid/bin/[root@12crac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE d883c23a7bfc4fdcbf418c9f631bd0af (/dev/asm-crs) [RACCRS]Located 1 voting disk(s).[root@12crac1 bin]# ./ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 4 Total space (kbytes) : 409568 Used space (kbytes) : 1608 Available space (kbytes) : 407960 ID : 1658916461 Device/File Name : +RACCRS Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded
檢視當前ocr備份情況
[root@12crac1 bin]# ./ocrconfig -showbackup12crac1 2013/07/02 23:21:22 /u01/app/12.1.0/grid/cdata/scan12c/backup00.ocr12crac1 2013/07/02 19:21:21 /u01/app/12.1.0/grid/cdata/scan12c/backup01.ocr12crac1 2013/07/01 04:52:41 /u01/app/12.1.0/grid/cdata/scan12c/backup02.ocr12crac1 2013/07/01 04:52:41 /u01/app/12.1.0/grid/cdata/scan12c/day.ocr12crac1 2013/07/01 04:52:41 /u01/app/12.1.0/grid/cdata/scan12c/week.ocr12crac1 2013/07/01 00:48:56 /u01/app/12.1.0/grid/cdata/scan12c/backup_20130701_004856.ocr12crac1 2013/07/01 00:39:40 /u01/app/12.1.0/grid/cdata/scan12c/backup_20130701_003940.ocr
可以如下方式進行手工備份
[root@12crac1 bin]# ./ocrconfig -local -manualbackup12crac1 2013/07/21 17:55:10 /u01/app/12.1.0/grid/cdata/12crac1/backup_20130721_175510.olr12crac1 2013/07/01 00:39:39 /u01/app/12.1.0/grid/cdata/12crac1/backup_20130701_003939.olr
檢視RAC資源服務狀態
[grid@12crac1 ~]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.LISTENER.lsnr ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACCRS.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACDATA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACFRA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.asm ONLINE ONLINE 12crac1 Started,STABLE ONLINE ONLINE 12crac2 Started,STABLE ora.net1.network ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.ons ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12crac1.vip 1 ONLINE ONLINE 12crac1 STABLE ora.12crac2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.MGMTLSNR 1 ONLINE ONLINE 12crac1 169.254.88.173 192.1 68.80.150,STABLE ora.cvu 1 ONLINE ONLINE 12crac1 STABLE ora.luocs12c.db 1 ONLINE ONLINE 12crac2 Open,STABLE 2 ONLINE ONLINE 12crac1 Open,STABLE ora.mgmtdb 1 ONLINE ONLINE 12crac1 Open,STABLE ora.oc4j 1 ONLINE ONLINE 12crac1 STABLE ora.scan1.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan2.vip 1 ONLINE ONLINE 12crac1 STABLE ora.scan3.vip 1 ONLINE ONLINE 12crac1 STABLE--------------------------------------------------------------------------------
用ASMCMD的md_backup命令備份磁碟組，順便檢視該磁碟組裡都存放什麼。
[root@12crac2 ~]# su - grid[grid@12crac2 ~]$ asmcmd -p ASMCMD [+] > md_backup /home/grid/ocrvote.bak -G RACCRSDisk group metadata to be backed up: RACCRSCurrent alias directory path: scan12cCurrent alias directory path: ASMCurrent alias directory path: _MGMTDB/CONTROLFILECurrent alias directory path: _MGMTDB/TEMPFILECurrent alias directory path: ASM/PASSWORDCurrent alias directory path: _MGMTDB/ONLINELOGCurrent alias directory path: _MGMTDBCurrent alias directory path: scan12c/OCRFILECurrent alias directory path: _MGMTDB/DATAFILECurrent alias directory path: scan12c/ASMPARAMETERFILECurrent alias directory path: _MGMTDB/PARAMETERFILE-- 從這裡可以看出,在Oracle 12c RAC中，存放ocr的磁碟組裡多了不少檔案，有_MGMTDB相關檔案以及ASM的PASSWORD。下面是11g RAC中存放OCR的磁碟組內容ASMCMD [+] > md_backup /home/grid/ocrvote.bak -G hk_crsDisk group metadata to be backed up: HK_CRSCurrent alias directory path: racscan/OCRFILECurrent alias directory path: racscanCurrent alias directory path: racscan/ASMPARAMETERFILE
也可以匯出ocr的內容
[root@12crac1 bin]# ./ocrconfig -export /home/grid/ocr.bak
以下方式都無法刪除當前使用的ocr內容
ASMCMD [+] > rm -rf /raccrs/scan12c/ocrfile ORA-29261: bad argument ORA-15178: directory 'ocrfile' is not empty; cannot drop this directory ORA-15028: ASM file '+RACCRS.255.819592481' not dropped; currently being accessed ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)ASMCMD [+] > cd /raccrs/scan12c/ocrfile ASMCMD [+raccrs/scan12c/ocrfile] > ls REGISTRY.255.819592481ASMCMD [+raccrs/scan12c/ocrfile] > rm -rf REGISTRY.255.819592481ORA-15032: not all alterations performed ORA-15028: ASM file '+raccrs/scan12c/ocrfile/REGISTRY.255.819592481' not dropped; currently being accessed (DBD ERROR: OCIStmtExecute)
那我們可以破壞存放ocr的裝置檔案
[root@12crac1 bin]# dd if=/dev/zero of=/dev/sdg bs=1024k count=11+0 records in1+0 records out1048576 bytes (1.0 MB) copied, 0.00335689 s, 312 MB/s
然後停止叢集：
[root@12crac1 bin]# ./crsctl stop has [root@12crac2 bin]# ./crsctl stop has -f
或crsctl stop crs [-f]也可以
嘗試啟動clusterware，發現clusterware無法正常啟動
[root@12crac1 bin]# ./crsctl start hasCRS-4123: Oracle High Availability Services has been started.[grid@12crac1 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager
檢視叢集日誌：
2013-07-20 23:46:01.413: [ohasd(18692)]CRS-0714:Oracle Clusterware Release 12.1.0.1.0 - Production Copyright 1996, 2010 Oracle. All rights reserved.2013-07-20 23:46:01.451: [ohasd(18692)]CRS-2112:The OLR service started on node 12crac1.2013-07-20 23:46:01.494: [ohasd(18692)]CRS-1301:Oracle High Availability Service started on node 12crac1.2013-07-20 23:46:01.498: [ohasd(18692)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred2013-07-20 23:46:10.768: [gpnpd(19041)]CRS-2328:GPNPD started on node 12crac1. 2013-07-20 23:46:42.712: [cssd(19212)]CRS-1713:CSSD daemon is started in hub mode2013-07-20 23:46:43.221: [cssd(19212)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/12.1.0/grid/log/12crac1/cssd/ocssd.log2013-07-20 23:46:44.142: [ohasd(18692)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE2013-07-20 23:46:44.143: [ohasd(18692)]CRS-2769:Unable to failover resource 'ora.diskmon'.2013-07-20 23:46:58.280: [cssd(19212)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/12.1.0/grid/log/12crac1/cssd/ocssd.log
檢視/u01/app/12.1.0/grid/log/12crac1/cssd/ocssd.log日誌
2013-07-20 23:48:13.450: [ GPNP][1105622784]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2207] get-profile call to url "ipc://GPNPD_12crac1" disco "" [f=0 claimed- host: cname: cguid: cli:gpnp p:19212 role: seq: ep: auth: diag:[]] 2013-07-20 23:48:13.476: [ GPNP][1105622784]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2360] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_12crac1" disco ""2013-07-20 23:48:13.477: [ CSSD][1105622784]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)2013-07-20 23:48:13.478: [ CSSD][1105622784]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery 2013-07-20 23:48:13.478: [ SKGFD][1105622784]Discovery with str:/dev/asm*:2013-07-20 23:48:13.478: [ SKGFD][1105622784]UFS discovery with :/dev/asm*:2013-07-20 23:48:13.491: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-data:2013-07-20 23:48:13.491: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-fra:2013-07-20 23:48:13.492: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-crs:2013-07-20 23:48:13.492: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-extcrs:2013-07-20 23:48:13.492: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm:2013-07-20 23:48:13.492: [ SKGFD][1105622784]OSS discovery with :/dev/asm*:2013-07-20 23:48:13.495: [ SKGFD][1105622784]Handle 0x7f8c10170500 from lib :UFS:: for disk :/dev/asm-data:2013-07-20 23:48:13.498: [ SKGFD][1105622784]Handle 0x7f8c1016e8a0 from lib :UFS:: for disk :/dev/asm-fra:2013-07-20 23:48:13.500: [ SKGFD][1105622784]Handle 0x7f8c1016f0d0 from lib :UFS:: for disk :/dev/asm-crs:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Handle 0x7f8c1011c4e0 from lib :UFS:: for disk :/dev/asm-extcrs:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c10170500 for disk :/dev/asm-data:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c1016e8a0 for disk :/dev/asm-fra:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c1016f0d0 for disk :/dev/asm-crs:2013-07-20 23:48:13.502: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c1011c4e0 for disk :/dev/asm-extcrs:2013-07-20 23:48:13.503: [ CSSD][1105622784]clssnmvDiskVerify: Successful discovery of 0 disks2013-07-20 23:48:13.503: [ CSSD][1105622784]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery2013-07-20 23:48:13.503: [ CSSD][1105622784]clssnmvFindInitialConfigs: No voting files found2013-07-20 23:48:13.503: [ CSSD][1105622784](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
– 我們可以看出表決磁碟無法找到等報錯資訊。
下面將叢集關閉，嘗試恢復。
[root@12crac1 bin]# ./crsctl stop has -f
ocr和vote disk損壞恢復步驟大致如下：
1）停止所有節點clusterware
# crsctl stop crs
# crsctl stop crs -f
2）以root使用者在其中一個節點度扎模式啟動clusterware
# crsctl start crs -excl -nocrs
備註：如果發現crsd在執行，那麼透過如下命令將之停止。
# crsctl stop resource ora.crsd -init
3）建立新的存放ocr和vote disk的磁碟組，磁碟組名和原有的一致(如果想改變位置，需修改/etc/oracle/ocr.loc檔案)
備註：如發現無法建立等情況，可以採用如下刪除磁碟組等排錯思路
SQL> drop diskgroup disk_group_name force including contents;
4）還原ocr，並檢查
# ocrconfig -restore file_name
# ocrcheck
5）恢復表決磁碟，並檢查
# crsctl replace votedisk +asm_disk_group
# crsctl query css votedisk
6）停止獨佔模式執行的clusterware
# crsctl stop crs -f
7）所有節點正常啟動clusterware
# crsctl start crs
8）CVU驗證所有RAC節點OCR的完整性
$ cluvfy comp ocr -n all -verbose
下面開始演示操作，獨佔模式執行clusterware[root@12crac1 bin]# ./crsctl start crs -excl -nocrsCRS-4123: Oracle High Availability Services has been started.CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12crac2'CRS-2677: Stop of 'ora.drivers.acfs' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.evmd' on '12crac2'CRS-2672: Attempting to start 'ora.mdnsd' on '12crac2'CRS-2676: Start of 'ora.evmd' on '12crac2' succeeded CRS-2676: Start of 'ora.mdnsd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on '12crac2'CRS-2676: Start of 'ora.gpnpd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on '12crac2'CRS-2672: Attempting to start 'ora.gipcd' on '12crac2'CRS-2676: Start of 'ora.cssdmonitor' on '12crac2' succeeded CRS-2676: Start of 'ora.gipcd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.cssd' on '12crac2'CRS-2672: Attempting to start 'ora.diskmon' on '12crac2'CRS-2676: Start of 'ora.diskmon' on '12crac2' succeeded CRS-2676: Start of 'ora.cssd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on '12crac2'CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12crac2'CRS-2672: Attempting to start 'ora.ctssd' on '12crac2'CRS-2676: Start of 'ora.drivers.acfs' on '12crac2' succeeded CRS-2676: Start of 'ora.ctssd' on '12crac2' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.asm' on '12crac2'CRS-2676: Start of 'ora.asm' on '12crac2' succeeded透過grid使用者登入sqlplus建立ASM磁碟組[grid@12crac2 ~]$ sqlplus / as sysasm SQL*Plus: Release 12.1.0.1.0 Production on Sun Jul 21 00:11:46 2013Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management options SQL> create diskgroup raccrs external redundancy disk '/dev/asm-extcrs' attribute 'compatible.asm' = '12.1.0.0.0';Diskgroup created.透過ocrconfig還原Ocr[root@12crac1 bin]# ./ocrconfig -import /home/grid/ocr.bak 或者[root@12crac1 bin]# ./ocrconfig -restore /u01/app/12.1.0/grid/cdata/scan12c/backup00.ocr檢視錶決磁碟資訊，當前無法找到[root@12crac1 bin]# ./crsctl query css votediskLocated 0 voting disk(s).恢復表決磁碟，可能會遇到如下問題[root@12crac1 bin]# ./crsctl replace votedisk +RACCRSCRS-4602: Failed 27 to add voting file 3782393479bf4f07bf313dc5a8f4c58a.Failed to replace voting disk group with +RACCRS.CRS-4000: Command Replace failed, or completed with errors.此問題需要重新配置一下ASM的引數並重啟ASM來解決。[grid@12crac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 12.1.0.1.0 Production on Sun Jul 21 00:40:01 2013Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management options SQL> alter system set asm_diskstring='/dev/asm*';System altered.SQL> create spfile from memory;File created.SQL> startup force mount;ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance startedTotal System Global Area 1135747072 bytesFixed Size 2297344 bytesVariable Size 1108283904 bytes ASM Cache 25165824 bytes ASM diskgroups mounted ASM diskgroups volume enabled重新恢復表決磁碟[root@12crac1 bin]# ./crsctl replace votedisk +RACCRSSuccessful addition of voting disk 1499cddff03a4f86bf01599718febcb1.Successfully replaced voting disk group with +RACCRS.CRS-4266: Voting file(s) successfully replaced[root@12crac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 1499cddff03a4f86bf01599718febcb1 (/dev/asm-extcrs) [RACCRS]Located 1 voting disk(s).退出獨佔模式：[root@12crac1 bin]# ./crsctl stop crs -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '12crac1'CRS-2673: Attempting to stop 'ora.ctssd' on '12crac1'CRS-2673: Attempting to stop 'ora.mdnsd' on '12crac1'CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12crac1'CRS-2673: Attempting to stop 'ora.gpnpd' on '12crac1'CRS-2677: Stop of 'ora.drivers.acfs' on '12crac1' succeeded CRS-2677: Stop of 'ora.mdnsd' on '12crac1' succeeded CRS-2677: Stop of 'ora.gpnpd' on '12crac1' succeeded CRS-2677: Stop of 'ora.ctssd' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.evmd' on '12crac1'CRS-2673: Attempting to stop 'ora.asm' on '12crac1'CRS-2677: Stop of 'ora.evmd' on '12crac1' succeeded CRS-2677: Stop of 'ora.asm' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12crac1'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on '12crac1'CRS-2677: Stop of 'ora.cssd' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on '12crac1'CRS-2677: Stop of 'ora.gipcd' on '12crac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '12crac1' has completed CRS-4133: Oracle High Availability Services has been stopped.所有節點都正常啟動：[root@12crac1 bin]# ./crsctl start hasCRS-4123: Oracle High Availability Services has been started.[root@12crac2 bin]# ./crsctl start hasCRS-4123: Oracle High Availability Services has been started.檢視clusterware執行狀態[grid@12crac1 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online[grid@12crac2 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online檢視所有資源狀態[grid@12crac1 ~]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.LISTENER.lsnr ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACCRS.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACDATA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACFRA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.asm ONLINE ONLINE 12crac1 Started,STABLE ONLINE ONLINE 12crac2 Started,STABLE ora.net1.network ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.ons ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12crac1.vip 1 ONLINE ONLINE 12crac1 STABLE ora.12crac2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.MGMTLSNR 1 ONLINE ONLINE 12crac2 169.254.171.71 192.1 68.80.154,STABLE ora.cvu 1 ONLINE ONLINE 12crac1 STABLE ora.luocs12c.db 1 ONLINE ONLINE 12crac2 Open,STABLE 2 ONLINE ONLINE 12crac1 Open,STABLE ora.mgmtdb 1 ONLINE OFFLINE Instance Shutdown,ST ABLE ora.oc4j 1 ONLINE ONLINE 12crac1 STABLE ora.scan1.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan2.vip 1 ONLINE ONLINE 12crac1 STABLE ora.scan3.vip 1 ONLINE ONLINE 12crac1 STABLE--------------------------------------------------------------------------------
在這裡我們會發現mgmtdb沒有正常啟動，手動嘗試啟動，會遇到問題。
[grid@12crac1 ~]$ srvctl start mgmtdb PRCR-1079 : Failed to start resource ora.mgmtdb CRS-5017: The resource action "ora.mgmtdb start" encountered the following error: ORA-01078: failure in processing system parameters LRM-00109: could not open parameter file '/u01/app/12.1.0/grid/dbs/init-MGMTDB.ora'. For details refer to "(:CLSN00107:)" in "/u01/app/12.1.0/grid/log/12crac2/agent/crsd/oraagent_grid/oraagent_grid.log".CRS-2674: Start of 'ora.mgmtdb' on '12crac2' failed CRS-5017: The resource action "ora.mgmtdb start" encountered the following error: ORA-01078: failure in processing system parameters ORA-01565: error in identifying file '+RACCRS/_mgmtdb/spfile-MGMTDB.ora'ORA-17503: ksfdopn:2 Failed to open file +RACCRS/_mgmtdb/spfile-MGMTDB.ora ORA-15056: additional error message ORA-17503: ksfdopn:2 Failed to open file +RACCRS/_mgmtdb/spfile-mgmtdb.ora ORA-15173: entry '_mgmtdb' does not exist in directory '/'ORA-06512: at line 4. For details refer to "(:CLSN00107:)" in "/u01/app/12.1.0/grid/log/12crac1/agent/crsd/oraagent_grid/oraagent_grid.log".CRS-2674: Start of 'ora.mgmtdb' on '12crac1' failed CRS-2632: There are no more servers to try to place resource 'ora.mgmtdb' on that would satisfy its placement policy
– 此問題造成原因是，我們dd了存放ocr的ASM磁碟組之後，裡面的_MGMTDB相關檔案也都將損壞丟失。從報錯資訊可見無法找到引數檔案。
檢視mgmtdb配置資訊
[grid@12crac1 ~]$ srvctl config mgmtdb -all -verboseDatabase unique name: _mgmtdbDatabase name: Oracle home: /u01/app/12.1.0/gridOracle user: gridSpfile: +RACCRS/_mgmtdb/spfile-MGMTDB.oraPassword file: Domain: Start options: openStop options: immediateDatabase role: PRIMARYManagement policy: AUTOMATICDatabase instance: -MGMTDBType: ManagementDatabase is enabled
目前還不知如何修復mgmtdb的方法，因此我remove了下
[grid@12crac1 ~]$ srvctl remove mgmtdbRemove the database _mgmtdb? (y/[n]) y[grid@12crac1 ~]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.LISTENER.lsnr ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACCRS.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACDATA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACFRA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.asm ONLINE ONLINE 12crac1 Started,STABLE ONLINE ONLINE 12crac2 Started,STABLE ora.net1.network ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.ons ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12crac1.vip 1 ONLINE ONLINE 12crac1 STABLE ora.12crac2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.MGMTLSNR 1 ONLINE ONLINE 12crac2 169.254.171.71 192.1 68.80.154,STABLE ora.cvu 1 ONLINE ONLINE 12crac1 STABLE ora.luocs12c.db 1 ONLINE ONLINE 12crac2 Open,STABLE 2 ONLINE ONLINE 12crac1 Open,STABLE ora.oc4j 1 ONLINE ONLINE 12crac1 STABLE ora.scan1.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan3.vip 1 ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/14377/viewspace-2286731/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle database 11g rac損壞ocr和votedisk恢復實驗OracleDatabase
- Oracle 12c rac ocr和votedisk管理Oracle
- Oracle OCR和VOTEDISK故障修復Oracle
- oracle 12c裡如何恢復votediskOracle
- oracle10g rac 表決盤損壞、ocr損壞處理Oracle
- 聯機日誌損壞恢復實驗
- 針對11.2 RAC丟失OCR和Votedisk所在ASM Diskgroup的恢復手段ASM
- 存貯在asm中的ocr/votedisk/asm磁碟組損害的恢復ASM
- ASM之OCR所在磁碟組損壞後的恢復ASM
- RAC 11G ASM磁碟損壞恢復ASM
- index損壞恢復Index
- Oracle RAC OCR 和 VotingDisk 的備份與恢復Oracle
- [zt]Oracle rac 11g線上新增ocr,votediskOracle
- Oracle 12c 備份恢復OCR(Oracle Cluster Registry)Oracle
- Oracle asm磁碟損壞異常恢復OracleASM
- Oracle10g 10.2.0.5 RAC一次恢復ORC、votedisk實戰Oracle
- Oracle塊損壞恢復（有rman備份）Oracle
- Oracle 10G RAC的ocr備份恢復Oracle 10g
- Oracle資料庫UNDO損壞後的恢復Oracle資料庫
- Oracle 10g RAC OCR 和 VotingDisk 的備份與恢復Oracle 10g
- 11g RAC OCR,VOTING DISK儲存全部損壞，利用自動備份，恢復OCR,VOTING DISK到新儲存。
- Oracle 12c RAC OCR和voting files介紹Oracle
- oracle database 例項恢復和介質恢復OracleDatabase
- 轉載：Oracle資料塊損壞恢復總結Oracle
- Oracle不同檔案丟失/損壞的恢復方法Oracle
- 損壞聯機日誌 恢復
- Oracle RAC CRS、OCR、Voting破壞重建Oracle
- Oracle資料檔案損壞恢復例項二則Oracle
- oracle 11g CSS 和OCR 的恢復OracleCSS
- 【12c 庫異機恢復】實驗
- Oracle 11g RAC ASM VOTEDISK所在磁碟全部丟失重大故障恢復OracleASM
- 損壞控制檔案的恢復方法
- SQL Server ldf 檔案損壞恢復SQLServer
- u盤檔案損壞怎麼恢復資料 u盤恢復損壞資料的有效方法
- Oracle恢復實驗（一）Oracle
- Oracle恢復實驗（二）Oracle
- Oracle恢復實驗（三）Oracle
- Oracle恢復實驗（四）Oracle