Oracle Database 12c RAC損壞ocr和votedisk恢復實驗
今天在Oracle 12c RAC下進行了破壞ocr和votedisk之後恢復的實驗，基本和11g RAC相差無異，下面將實驗過程分享一下。
實驗環境：2-NODES Oracle Database 12c RAC on Linux6(OEL 6.4)
檢視錶決磁碟和Ocr相關資訊
[root@12crac1 ~]# cd /u01/app/12.1.0/grid/bin/[root@12crac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE d883c23a7bfc4fdcbf418c9f631bd0af (/dev/asm-crs) [RACCRS]Located 1 voting disk(s).[root@12crac1 bin]# ./ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 4 Total space (kbytes) : 409568 Used space (kbytes) : 1608 Available space (kbytes) : 407960 ID : 1658916461 Device/File Name : +RACCRS Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded
檢視當前ocr備份情況
[root@12crac1 bin]# ./ocrconfig -showbackup12crac1 2013/07/02 23:21:22 /u01/app/12.1.0/grid/cdata/scan12c/backup00.ocr12crac1 2013/07/02 19:21:21 /u01/app/12.1.0/grid/cdata/scan12c/backup01.ocr12crac1 2013/07/01 04:52:41 /u01/app/12.1.0/grid/cdata/scan12c/backup02.ocr12crac1 2013/07/01 04:52:41 /u01/app/12.1.0/grid/cdata/scan12c/day.ocr12crac1 2013/07/01 04:52:41 /u01/app/12.1.0/grid/cdata/scan12c/week.ocr12crac1 2013/07/01 00:48:56 /u01/app/12.1.0/grid/cdata/scan12c/backup_20130701_004856.ocr12crac1 2013/07/01 00:39:40 /u01/app/12.1.0/grid/cdata/scan12c/backup_20130701_003940.ocr
可以如下方式進行手工備份
[root@12crac1 bin]# ./ocrconfig -local -manualbackup12crac1 2013/07/21 17:55:10 /u01/app/12.1.0/grid/cdata/12crac1/backup_20130721_175510.olr12crac1 2013/07/01 00:39:39 /u01/app/12.1.0/grid/cdata/12crac1/backup_20130701_003939.olr
檢視RAC資源服務狀態
[grid@12crac1 ~]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.LISTENER.lsnr ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACCRS.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACDATA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACFRA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.asm ONLINE ONLINE 12crac1 Started,STABLE ONLINE ONLINE 12crac2 Started,STABLE ora.net1.network ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.ons ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12crac1.vip 1 ONLINE ONLINE 12crac1 STABLE ora.12crac2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.MGMTLSNR 1 ONLINE ONLINE 12crac1 169.254.88.173 192.1 68.80.150,STABLE ora.cvu 1 ONLINE ONLINE 12crac1 STABLE ora.luocs12c.db 1 ONLINE ONLINE 12crac2 Open,STABLE 2 ONLINE ONLINE 12crac1 Open,STABLE ora.mgmtdb 1 ONLINE ONLINE 12crac1 Open,STABLE ora.oc4j 1 ONLINE ONLINE 12crac1 STABLE ora.scan1.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan2.vip 1 ONLINE ONLINE 12crac1 STABLE ora.scan3.vip 1 ONLINE ONLINE 12crac1 STABLE--------------------------------------------------------------------------------
用ASMCMD的md_backup命令備份磁碟組，順便檢視該磁碟組裡都存放什麼。
[root@12crac2 ~]# su - grid[grid@12crac2 ~]$ asmcmd -p ASMCMD [+] > md_backup /home/grid/ocrvote.bak -G RACCRSDisk group metadata to be backed up: RACCRSCurrent alias directory path: scan12cCurrent alias directory path: ASMCurrent alias directory path: _MGMTDB/CONTROLFILECurrent alias directory path: _MGMTDB/TEMPFILECurrent alias directory path: ASM/PASSWORDCurrent alias directory path: _MGMTDB/ONLINELOGCurrent alias directory path: _MGMTDBCurrent alias directory path: scan12c/OCRFILECurrent alias directory path: _MGMTDB/DATAFILECurrent alias directory path: scan12c/ASMPARAMETERFILECurrent alias directory path: _MGMTDB/PARAMETERFILE-- 從這裡可以看出,在Oracle 12c RAC中，存放ocr的磁碟組裡多了不少檔案，有_MGMTDB相關檔案以及ASM的PASSWORD。下面是11g RAC中存放OCR的磁碟組內容ASMCMD [+] > md_backup /home/grid/ocrvote.bak -G hk_crsDisk group metadata to be backed up: HK_CRSCurrent alias directory path: racscan/OCRFILECurrent alias directory path: racscanCurrent alias directory path: racscan/ASMPARAMETERFILE
也可以匯出ocr的內容
[root@12crac1 bin]# ./ocrconfig -export /home/grid/ocr.bak
以下方式都無法刪除當前使用的ocr內容
ASMCMD [+] > rm -rf /raccrs/scan12c/ocrfile ORA-29261: bad argument ORA-15178: directory 'ocrfile' is not empty; cannot drop this directory ORA-15028: ASM file '+RACCRS.255.819592481' not dropped; currently being accessed ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)ASMCMD [+] > cd /raccrs/scan12c/ocrfile ASMCMD [+raccrs/scan12c/ocrfile] > ls REGISTRY.255.819592481ASMCMD [+raccrs/scan12c/ocrfile] > rm -rf REGISTRY.255.819592481ORA-15032: not all alterations performed ORA-15028: ASM file '+raccrs/scan12c/ocrfile/REGISTRY.255.819592481' not dropped; currently being accessed (DBD ERROR: OCIStmtExecute)
那我們可以破壞存放ocr的裝置檔案
[root@12crac1 bin]# dd if=/dev/zero of=/dev/sdg bs=1024k count=11+0 records in1+0 records out1048576 bytes (1.0 MB) copied, 0.00335689 s, 312 MB/s
然後停止叢集：
[root@12crac1 bin]# ./crsctl stop has [root@12crac2 bin]# ./crsctl stop has -f
或crsctl stop crs [-f]也可以
嘗試啟動clusterware，發現clusterware無法正常啟動
[root@12crac1 bin]# ./crsctl start hasCRS-4123: Oracle High Availability Services has been started.[grid@12crac1 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager
檢視叢集日誌：
2013-07-20 23:46:01.413: [ohasd(18692)]CRS-0714:Oracle Clusterware Release 12.1.0.1.0 - Production Copyright 1996, 2010 Oracle. All rights reserved.2013-07-20 23:46:01.451: [ohasd(18692)]CRS-2112:The OLR service started on node 12crac1.2013-07-20 23:46:01.494: [ohasd(18692)]CRS-1301:Oracle High Availability Service started on node 12crac1.2013-07-20 23:46:01.498: [ohasd(18692)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred2013-07-20 23:46:10.768: [gpnpd(19041)]CRS-2328:GPNPD started on node 12crac1. 2013-07-20 23:46:42.712: [cssd(19212)]CRS-1713:CSSD daemon is started in hub mode2013-07-20 23:46:43.221: [cssd(19212)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/12.1.0/grid/log/12crac1/cssd/ocssd.log2013-07-20 23:46:44.142: [ohasd(18692)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE2013-07-20 23:46:44.143: [ohasd(18692)]CRS-2769:Unable to failover resource 'ora.diskmon'.2013-07-20 23:46:58.280: [cssd(19212)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/12.1.0/grid/log/12crac1/cssd/ocssd.log
檢視/u01/app/12.1.0/grid/log/12crac1/cssd/ocssd.log日誌
2013-07-20 23:48:13.450: [ GPNP][1105622784]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2207] get-profile call to url "ipc://GPNPD_12crac1" disco "" [f=0 claimed- host: cname: cguid: cli:gpnp p:19212 role: seq: ep: auth: diag:[]] 2013-07-20 23:48:13.476: [ GPNP][1105622784]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2360] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_12crac1" disco ""2013-07-20 23:48:13.477: [ CSSD][1105622784]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)2013-07-20 23:48:13.478: [ CSSD][1105622784]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery 2013-07-20 23:48:13.478: [ SKGFD][1105622784]Discovery with str:/dev/asm*:2013-07-20 23:48:13.478: [ SKGFD][1105622784]UFS discovery with :/dev/asm*:2013-07-20 23:48:13.491: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-data:2013-07-20 23:48:13.491: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-fra:2013-07-20 23:48:13.492: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-crs:2013-07-20 23:48:13.492: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm-extcrs:2013-07-20 23:48:13.492: [ SKGFD][1105622784]Fetching UFS disk :/dev/asm:2013-07-20 23:48:13.492: [ SKGFD][1105622784]OSS discovery with :/dev/asm*:2013-07-20 23:48:13.495: [ SKGFD][1105622784]Handle 0x7f8c10170500 from lib :UFS:: for disk :/dev/asm-data:2013-07-20 23:48:13.498: [ SKGFD][1105622784]Handle 0x7f8c1016e8a0 from lib :UFS:: for disk :/dev/asm-fra:2013-07-20 23:48:13.500: [ SKGFD][1105622784]Handle 0x7f8c1016f0d0 from lib :UFS:: for disk :/dev/asm-crs:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Handle 0x7f8c1011c4e0 from lib :UFS:: for disk :/dev/asm-extcrs:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c10170500 for disk :/dev/asm-data:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c1016e8a0 for disk :/dev/asm-fra:2013-07-20 23:48:13.501: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c1016f0d0 for disk :/dev/asm-crs:2013-07-20 23:48:13.502: [ SKGFD][1105622784]Lib :UFS:: closing handle 0x7f8c1011c4e0 for disk :/dev/asm-extcrs:2013-07-20 23:48:13.503: [ CSSD][1105622784]clssnmvDiskVerify: Successful discovery of 0 disks2013-07-20 23:48:13.503: [ CSSD][1105622784]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery2013-07-20 23:48:13.503: [ CSSD][1105622784]clssnmvFindInitialConfigs: No voting files found2013-07-20 23:48:13.503: [ CSSD][1105622784](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
– 我們可以看出表決磁碟無法找到等報錯資訊。
下面將叢集關閉，嘗試恢復。
[root@12crac1 bin]# ./crsctl stop has -f
ocr和vote disk損壞恢復步驟大致如下：
1）停止所有節點clusterware
# crsctl stop crs
# crsctl stop crs -f
2）以root使用者在其中一個節點度扎模式啟動clusterware
# crsctl start crs -excl -nocrs
備註：如果發現crsd在執行，那麼透過如下命令將之停止。
# crsctl stop resource ora.crsd -init
3）建立新的存放ocr和vote disk的磁碟組，磁碟組名和原有的一致(如果想改變位置，需修改/etc/oracle/ocr.loc檔案)
備註：如發現無法建立等情況，可以採用如下刪除磁碟組等排錯思路
SQL> drop diskgroup disk_group_name force including contents;
4）還原ocr，並檢查
# ocrconfig -restore file_name
# ocrcheck
5）恢復表決磁碟，並檢查
# crsctl replace votedisk +asm_disk_group
# crsctl query css votedisk
6）停止獨佔模式執行的clusterware
# crsctl stop crs -f
7）所有節點正常啟動clusterware
# crsctl start crs
8）CVU驗證所有RAC節點OCR的完整性
$ cluvfy comp ocr -n all -verbose
下面開始演示操作，獨佔模式執行clusterware[root@12crac1 bin]# ./crsctl start crs -excl -nocrsCRS-4123: Oracle High Availability Services has been started.CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12crac2'CRS-2677: Stop of 'ora.drivers.acfs' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.evmd' on '12crac2'CRS-2672: Attempting to start 'ora.mdnsd' on '12crac2'CRS-2676: Start of 'ora.evmd' on '12crac2' succeeded CRS-2676: Start of 'ora.mdnsd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on '12crac2'CRS-2676: Start of 'ora.gpnpd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on '12crac2'CRS-2672: Attempting to start 'ora.gipcd' on '12crac2'CRS-2676: Start of 'ora.cssdmonitor' on '12crac2' succeeded CRS-2676: Start of 'ora.gipcd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.cssd' on '12crac2'CRS-2672: Attempting to start 'ora.diskmon' on '12crac2'CRS-2676: Start of 'ora.diskmon' on '12crac2' succeeded CRS-2676: Start of 'ora.cssd' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on '12crac2'CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on '12crac2'CRS-2672: Attempting to start 'ora.ctssd' on '12crac2'CRS-2676: Start of 'ora.drivers.acfs' on '12crac2' succeeded CRS-2676: Start of 'ora.ctssd' on '12crac2' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on '12crac2' succeeded CRS-2672: Attempting to start 'ora.asm' on '12crac2'CRS-2676: Start of 'ora.asm' on '12crac2' succeeded透過grid使用者登入sqlplus建立ASM磁碟組[grid@12crac2 ~]$ sqlplus / as sysasm SQL*Plus: Release 12.1.0.1.0 Production on Sun Jul 21 00:11:46 2013Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management options SQL> create diskgroup raccrs external redundancy disk '/dev/asm-extcrs' attribute 'compatible.asm' = '12.1.0.0.0';Diskgroup created.透過ocrconfig還原Ocr[root@12crac1 bin]# ./ocrconfig -import /home/grid/ocr.bak 或者[root@12crac1 bin]# ./ocrconfig -restore /u01/app/12.1.0/grid/cdata/scan12c/backup00.ocr檢視錶決磁碟資訊，當前無法找到[root@12crac1 bin]# ./crsctl query css votediskLocated 0 voting disk(s).恢復表決磁碟，可能會遇到如下問題[root@12crac1 bin]# ./crsctl replace votedisk +RACCRSCRS-4602: Failed 27 to add voting file 3782393479bf4f07bf313dc5a8f4c58a.Failed to replace voting disk group with +RACCRS.CRS-4000: Command Replace failed, or completed with errors.此問題需要重新配置一下ASM的引數並重啟ASM來解決。[grid@12crac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 12.1.0.1.0 Production on Sun Jul 21 00:40:01 2013Copyright (c) 1982, 2013, Oracle. All rights reserved.Connected to:Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management options SQL> alter system set asm_diskstring='/dev/asm*';System altered.SQL> create spfile from memory;File created.SQL> startup force mount;ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance startedTotal System Global Area 1135747072 bytesFixed Size 2297344 bytesVariable Size 1108283904 bytes ASM Cache 25165824 bytes ASM diskgroups mounted ASM diskgroups volume enabled重新恢復表決磁碟[root@12crac1 bin]# ./crsctl replace votedisk +RACCRSSuccessful addition of voting disk 1499cddff03a4f86bf01599718febcb1.Successfully replaced voting disk group with +RACCRS.CRS-4266: Voting file(s) successfully replaced[root@12crac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 1499cddff03a4f86bf01599718febcb1 (/dev/asm-extcrs) [RACCRS]Located 1 voting disk(s).退出獨佔模式：[root@12crac1 bin]# ./crsctl stop crs -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on '12crac1'CRS-2673: Attempting to stop 'ora.ctssd' on '12crac1'CRS-2673: Attempting to stop 'ora.mdnsd' on '12crac1'CRS-2673: Attempting to stop 'ora.drivers.acfs' on '12crac1'CRS-2673: Attempting to stop 'ora.gpnpd' on '12crac1'CRS-2677: Stop of 'ora.drivers.acfs' on '12crac1' succeeded CRS-2677: Stop of 'ora.mdnsd' on '12crac1' succeeded CRS-2677: Stop of 'ora.gpnpd' on '12crac1' succeeded CRS-2677: Stop of 'ora.ctssd' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.evmd' on '12crac1'CRS-2673: Attempting to stop 'ora.asm' on '12crac1'CRS-2677: Stop of 'ora.evmd' on '12crac1' succeeded CRS-2677: Stop of 'ora.asm' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on '12crac1'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on '12crac1'CRS-2677: Stop of 'ora.cssd' on '12crac1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on '12crac1'CRS-2677: Stop of 'ora.gipcd' on '12crac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on '12crac1' has completed CRS-4133: Oracle High Availability Services has been stopped.所有節點都正常啟動：[root@12crac1 bin]# ./crsctl start hasCRS-4123: Oracle High Availability Services has been started.[root@12crac2 bin]# ./crsctl start hasCRS-4123: Oracle High Availability Services has been started.檢視clusterware執行狀態[grid@12crac1 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online[grid@12crac2 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online檢視所有資源狀態[grid@12crac1 ~]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.LISTENER.lsnr ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACCRS.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACDATA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACFRA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.asm ONLINE ONLINE 12crac1 Started,STABLE ONLINE ONLINE 12crac2 Started,STABLE ora.net1.network ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.ons ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12crac1.vip 1 ONLINE ONLINE 12crac1 STABLE ora.12crac2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE 12crac1 STABLE ora.MGMTLSNR 1 ONLINE ONLINE 12crac2 169.254.171.71 192.1 68.80.154,STABLE ora.cvu 1 ONLINE ONLINE 12crac1 STABLE ora.luocs12c.db 1 ONLINE ONLINE 12crac2 Open,STABLE 2 ONLINE ONLINE 12crac1 Open,STABLE ora.mgmtdb 1 ONLINE OFFLINE Instance Shutdown,ST ABLE ora.oc4j 1 ONLINE ONLINE 12crac1 STABLE ora.scan1.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan2.vip 1 ONLINE ONLINE 12crac1 STABLE ora.scan3.vip 1 ONLINE ONLINE 12crac1 STABLE--------------------------------------------------------------------------------
在這裡我們會發現mgmtdb沒有正常啟動，手動嘗試啟動，會遇到問題。
[grid@12crac1 ~]$ srvctl start mgmtdb PRCR-1079 : Failed to start resource ora.mgmtdb CRS-5017: The resource action "ora.mgmtdb start" encountered the following error: ORA-01078: failure in processing system parameters LRM-00109: could not open parameter file '/u01/app/12.1.0/grid/dbs/init-MGMTDB.ora'. For details refer to "(:CLSN00107:)" in "/u01/app/12.1.0/grid/log/12crac2/agent/crsd/oraagent_grid/oraagent_grid.log".CRS-2674: Start of 'ora.mgmtdb' on '12crac2' failed CRS-5017: The resource action "ora.mgmtdb start" encountered the following error: ORA-01078: failure in processing system parameters ORA-01565: error in identifying file '+RACCRS/_mgmtdb/spfile-MGMTDB.ora'ORA-17503: ksfdopn:2 Failed to open file +RACCRS/_mgmtdb/spfile-MGMTDB.ora ORA-15056: additional error message ORA-17503: ksfdopn:2 Failed to open file +RACCRS/_mgmtdb/spfile-mgmtdb.ora ORA-15173: entry '_mgmtdb' does not exist in directory '/'ORA-06512: at line 4. For details refer to "(:CLSN00107:)" in "/u01/app/12.1.0/grid/log/12crac1/agent/crsd/oraagent_grid/oraagent_grid.log".CRS-2674: Start of 'ora.mgmtdb' on '12crac1' failed CRS-2632: There are no more servers to try to place resource 'ora.mgmtdb' on that would satisfy its placement policy
– 此問題造成原因是，我們dd了存放ocr的ASM磁碟組之後，裡面的_MGMTDB相關檔案也都將損壞丟失。從報錯資訊可見無法找到引數檔案。
檢視mgmtdb配置資訊
[grid@12crac1 ~]$ srvctl config mgmtdb -all -verboseDatabase unique name: _mgmtdbDatabase name: Oracle home: /u01/app/12.1.0/gridOracle user: gridSpfile: +RACCRS/_mgmtdb/spfile-MGMTDB.oraPassword file: Domain: Start options: openStop options: immediateDatabase role: PRIMARYManagement policy: AUTOMATICDatabase instance: -MGMTDBType: ManagementDatabase is enabled
目前還不知如何修復mgmtdb的方法，因此我remove了下
[grid@12crac1 ~]$ srvctl remove mgmtdbRemove the database _mgmtdb? (y/[n]) y[grid@12crac1 ~]$ crsctl stat res -t--------------------------------------------------------------------------------Name Target State Server State details --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.LISTENER.lsnr ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACCRS.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACDATA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.RACFRA.dg ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.asm ONLINE ONLINE 12crac1 Started,STABLE ONLINE ONLINE 12crac2 Started,STABLE ora.net1.network ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE ora.ons ONLINE ONLINE 12crac1 STABLE ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------Cluster Resources--------------------------------------------------------------------------------ora.12crac1.vip 1 ONLINE ONLINE 12crac1 STABLE ora.12crac2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE 12crac2 STABLE ora.MGMTLSNR 1 ONLINE ONLINE 12crac2 169.254.171.71 192.1 68.80.154,STABLE ora.cvu 1 ONLINE ONLINE 12crac1 STABLE ora.luocs12c.db 1 ONLINE ONLINE 12crac2 Open,STABLE 2 ONLINE ONLINE 12crac1 Open,STABLE ora.oc4j 1 ONLINE ONLINE 12crac1 STABLE ora.scan1.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan2.vip 1 ONLINE ONLINE 12crac2 STABLE ora.scan3.vip 1 ONLINE ONLINE 12crac2 STABLE--------------------------------------------------------------------------------
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/14377/viewspace-2286731/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle 12c rac ocr和votedisk管理Oracle
- Oracle asm磁碟損壞異常恢復OracleASM
- Oracle資料庫不同損壞級別的恢復詳解Oracle資料庫
- 【12c 庫異機恢復】實驗
- 記一次Oracle RAC for aix 儲存雙控鎖盤導致ASM控制檔案損壞恢復OracleAIASM
- u盤檔案損壞怎麼恢復資料 u盤恢復損壞資料的有效方法
- Networker恢復oracle rac到單機Oracle
- 【Oracle 恢復表空間】 實驗Oracle
- oracle控制檔案的損壞或完全丟失的恢復辦法Oracle
- Oracle 12c 備份與恢復Oracle
- 資料底層損壞的恢復方法—拼碎片恢復資料
- u盤檔案損壞怎麼恢復資料 u盤損壞無法讀取怎麼恢復資料
- Oracle 12C新特性-RMAN恢復表Oracle
- 【伺服器資料恢復】IBM儲存伺服器硬碟壞道離線、oracle資料庫損壞的資料恢復伺服器資料恢復IBM硬碟Oracle資料庫
- 隨身碟顆粒損壞資料恢復資料恢復
- 伺服器Oracle資料庫損壞修復伺服器Oracle資料庫
- 深入解析：段頭塊損壞bbed異常恢復
- Oracle 12C Database File Mapping for Oracle ASM FilesOracleDatabaseAPPASM
- oracle資料庫損壞的恢復過程-基於IBM伺服器儲存Oracle資料庫IBM伺服器
- 資料恢復工具Recoverit使用教程:如何修復損壞的影片資料恢復
- WPS文件損壞如何修復？WPS文件損壞的修復方法
- Oracle RAC 遷移替換 OCR 盤Oracle
- 電腦進水導致硬碟損壞資料恢復硬碟資料恢復
- RAC磁碟頭損壞問題處理
- 學習這篇Oracle資料庫檔案壞塊損壞的恢復方法,擴充你的知識面Oracle資料庫
- ORACLE 11.2.0.4 RAC RMAN異機恢復之ORA-15001Oracle
- oracle 增量備份恢復驗證Oracle
- 資料庫資料恢復—NTFS分割槽損壞如何恢復SqlServer資料庫資料資料庫資料恢復SQLServer
- 【資料庫資料恢復】MongoDB資料庫檔案損壞的資料恢復案例資料庫資料恢復MongoDB
- [20181018]Oracle Database 12c: Data Redaction.txtOracleDatabase
- Oracle 12c 使用RMAN搭建物理備庫(RAC to RAC)Oracle
- 【LINUX】Oracle資料庫 linux磁碟頭資料損壞修復LinuxOracle資料庫
- 【儲存資料恢復】IBM儲存檔案NTFS系統損壞的資料恢復案例資料恢復IBM
- PostgreSQL DBA(30) - Backup&Recovery#3(資料檔案損壞恢復)SQL
- ibdata1檔案損壞時恢復InnoDB單表測試
- 伺服器硬碟磁頭損壞，碟片劃傷資料恢復伺服器硬碟資料恢復
- 伺服器系統癱瘓系統損壞資料恢復伺服器資料恢復
- 【RECO_ORACLE】Oracle 12c之CDB與PDB的備份與恢復（四）PDB的幾種恢復方式Oracle