RAC 11G ASM磁碟損壞恢復
一個儲存檔案的rac資料庫起不來了，生產環境是linux rac 11.2.0.4,原因是因為用工具測試磁碟IO時損壞了ocr所在磁碟組與儲存資料ASM磁碟的磁碟頭。下面是恢復過程:
1.檢查crs的狀態:
[grid@darac1 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager [root@darac1 crsd]# ps -ef|grep crs root 3126 1 1 10:34 ? 00:00:31 /u01/app/product/11.2.0/crs/bin/ohasd.bin reboot grid 3514 1 0 10:34 ? 00:00:07 /u01/app/product/11.2.0/crs/bin/oraagent.bin grid 3525 1 0 10:34 ? 00:00:00 /u01/app/product/11.2.0/crs/bin/mdnsd.bin grid 3537 1 0 10:34 ? 00:00:16 /u01/app/product/11.2.0/crs/bin/gpnpd.bin grid 3549 1 1 10:34 ? 00:00:33 /u01/app/product/11.2.0/crs/bin/gipcd.bin root 4128 1 0 10:54 ? 00:00:02 /u01/app/product/11.2.0/crs/bin/cssdmonitor root 4144 1 0 10:54 ? 00:00:01 /u01/app/product/11.2.0/crs/bin/cssdagent grid 4167 1 2 10:55 ? 00:00:14 /u01/app/product/11.2.0/crs/bin/ocssd.bin root 4354 3680 0 11:04 pts/1 00:00:00 grep crs
2.強制關閉crs
[root@darac1 bin]# ./crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'darac1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'darac1' CRS-2673: Attempting to stop 'ora.gipcd' on 'darac1' CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'darac1' CRS-2677: Stop of 'ora.cssdmonitor' on 'darac1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'darac1' succeeded CRS-2677: Stop of 'ora.gipcd' on 'darac1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'darac1' CRS-2677: Stop of 'ora.gpnpd' on 'darac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'darac1' has completed CRS-4133: Oracle High Availability Services has been stopped.
3.以exclusive模式啟動crs
[root@darac1 bin]# ./crsctl start crs -excl -nocrs CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'darac1' CRS-2676: Start of 'ora.mdnsd' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'darac1' CRS-2676: Start of 'ora.gpnpd' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'darac1' CRS-2672: Attempting to start 'ora.gipcd' on 'darac1' CRS-2676: Start of 'ora.gipcd' on 'darac1' succeeded CRS-2676: Start of 'ora.cssdmonitor' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'darac1' CRS-2672: Attempting to start 'ora.diskmon' on 'darac1' CRS-2676: Start of 'ora.diskmon' on 'darac1' succeeded CRS-2676: Start of 'ora.cssd' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on 'darac1' CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'darac1' CRS-2672: Attempting to start 'ora.ctssd' on 'darac1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'darac1' CRS-2676: Start of 'ora.ctssd' on 'darac1' succeeded CRS-2676: Start of 'ora.drivers.acfs' on 'darac1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'darac1' CRS-2676: Start of 'ora.asm' on 'darac1' succeeded
4.檢視GI相關的alert.log日誌檔案如何
[ohasd(5040)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2016-10-13 11:20:47.302:
[gpnpd(5215)]CRS-2328:GPNPD started on node darac1.
2016-10-13 11:20:58.388:
[ohasd(5040)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2016-10-13 11:21:00.608:
[cssd(5318)]CRS-1713:CSSD daemon is started in clustered mode
2016-10-13 11:21:01.521:
[/u01/app/product/11.2.0/crs/bin/orarootagent.bin(5304)]CRS-5013:Agent "/u01/app/product/11.2.0/crs/bin/orarootagent.bin" failed to start process
"/u01/app/product/11.2.0/crs/bin/osysmond" for action "start": details at "(:CLSN00008:)" in
"/u01/app/product/11.2.0/crs/log/darac1/agent/ohasd/orarootagent_root//orarootagent_root.log"
2016-10-13 11:21:03.585:
[ohasd(5040)]CRS-2878:Failed to restart resource 'ora.crf'
2016-10-13 11:21:05.399:
[/u01/app/product/11.2.0/crs/bin/orarootagent.bin(5340)]CRS-5013:Agent "/u01/app/product/11.2.0/crs/bin/orarootagent.bin" failed to start process
"/u01/app/product/11.2.0/crs/bin/osysmond" for action "start": details at "(:CLSN00008:)" in
"/u01/app/product/11.2.0/crs/log/darac1/agent/ohasd/orarootagent_root//orarootagent_root.log"
2016-10-13 11:21:10.703:
[ohasd(5040)]CRS-2878:Failed to restart resource 'ora.crf'
2016-10-13 11:21:23.464:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:21:38.698:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:21:53.925:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:09.463:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:24.804:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:40.252:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:56.722:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:12.009:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:27.290:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:42.872:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:58.198:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:13.500:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:28.786:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:43.488:
[client(5394)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/product/11.2.0/crs/log/darac1/client/ocrcheck_5394.log.
2016-10-13 11:24:43.959:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:51.823:
[client(5424)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/product/11.2.0/crs/log/darac1/client/crsctl_grid.log.
2016-10-13 11:24:59.345:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:25:14.526:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:25:29.696:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:25:44.860:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:00.042:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:15.218:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:30.409:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:45.577:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:49.031:
[client(5460)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/product/11.2.0/crs/log/darac1/client/ocrconfig_5460.log.
2016-10-13 11:27:00.766:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:27:15.951:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:27:31.142:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:27:46.339:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:01.530:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:16.733:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:32.008:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:47.191:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:02.389:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:17.610:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:32.832:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:48.035:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:03.229:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:18.434:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:33.679:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:48.876:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:31:01.534:
[/u01/app/product/11.2.0/crs/bin/cssdagent(5284)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in
/u01/app/product/11.2.0/crs/log/darac1/agent/ohasd/oracssdagent_root//oracssdagent_root.log.
2016-10-13 11:31:01.540:
[cssd(5318)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:31:01.541:
[cssd(5318)]CRS-1603:CSSD on node darac1 shutdown by user.
從上面的資訊可以看到找不到voting files
5.檢查ASM的alert.log可以找如下建立CRSDG,DATADG磁碟組的建立語句:
Wed Dec 02 16:09:01 2015 SQL> CREATE DISKGROUP CRSDG EXTERNAL REDUNDANCY DISK '/dev/raw/raw1' ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */
6.檢查磁碟頭
[grid@darac1 ~]$ kfed read /dev/raw/raw1 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 300392945 ; 0x00c: 0x11e7a1f1 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 B7F46200 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
7.使用kfed恢復CRSDG的磁碟頭，但因為備份資訊也被損壞所以恢復時報錯，而且沒有手動備份
[grid@darac1 ~]$ kfed repair /dev/raw/raw1 KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]
沒有透過自動備份的磁碟頭資訊來進行恢復，只能使用自動備份的ocr資訊來恢復了操作如下。
8.建立磁碟組
[grid@darac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Thu Oct 13 13:00:42 2016 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production With the Real Application Clusters and Automatic Storage Management options SQL> select * from v$asm_diskgroup; no rows selected SQL> create diskgroup CRSDG external redundancy disk '/dev/raw/raw1' attribute 'COMPATIBLE.ASM' = '11.2.0.0.0'; Diskgroup created.
9.檢視自動備份的ocr檔案
[root@darac1 bin]# ./ocrconfig -showbackup PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy darac2 2016/10/13 06:29:53 /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup00.ocr darac2 2016/10/13 02:29:45 /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup01.ocr darac2 2016/10/12 22:29:37 /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup02.ocr darac2 2016/10/12 02:27:20 /u01/app/product/11.2.0/crs/cdata/darac-cluster/day.ocr darac2 2016/10/11 22:27:10 /u01/app/product/11.2.0/crs/cdata/darac-cluster/week.ocr
10.還原ocr
[root@darac1 bin]# ./ocrconfig -restore /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup00.ocr
11.處理votedisk
[root@darac1 bin]# ./ocrconfig -restore /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup00.ocr [root@darac1 bin]# ./crsctl replace votedisk +CRSDG Successful addition of voting disk 44eaf86504ea4f76bfb43cb7931a3fc7. Successfully replaced voting disk group with +CRSDG. CRS-4266: Voting file(s) successfully replaced
12.建立asm spfile
[grid@darac1 ~]$ vi /tmp/asm.txt instance_type='asm' large_pool_size=12M remote_login_passwordfile= 'EXCLUSIVE' asm_diskstring = '/dev/raw/raw*' asm_power_limit =1 [grid@darac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Thu Oct 13 13:40:02 2016 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production With the Real Application Clusters and Automatic Storage Management options SQL> create spfile='+CRSDG' FROM pfile='/tmp/asm.txt'; File created.
13.重啟crs
[root@darac1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'darac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'darac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'darac1'
CRS-2673: Attempting to stop 'ora.asm' on 'darac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'darac1'
CRS-2677: Stop of 'ora.ctssd' on 'darac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'darac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'darac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'darac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'darac1'
CRS-2677: Stop of 'ora.cssd' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'darac1'
CRS-2677: Stop of 'ora.gipcd' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'darac1'
CRS-2677: Stop of 'ora.gpnpd' on 'darac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'darac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@darac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[grid@darac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.DATADG.dg
ONLINE OFFLINE darac1
ONLINE OFFLINE darac2
ora.LISTENER.lsnr
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.asm
ONLINE ONLINE darac1 Started
ONLINE ONLINE darac2 Started
ora.gsd
OFFLINE OFFLINE darac1
OFFLINE OFFLINE darac2
ora.net1.network
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.ons
ONLINE ONLINE darac1
ONLINE OFFLINE darac2
ora.registry.acfs
ONLINE ONLINE darac1
ONLINE ONLINE darac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE darac1
ora.cvu
1 ONLINE ONLINE darac1
ora.darac.db
1 ONLINE OFFLINE Corrupted Controlfi
le
2 ONLINE OFFLINE Corrupted Controlfi
le
ora.darac1.vip
1 ONLINE ONLINE darac1
ora.darac2.vip
1 ONLINE ONLINE darac2
ora.darac3.vip
1 ONLINE OFFLINE
ora.oc4j
1 ONLINE OFFLINE STARTING
ora.scan1.vip
1 ONLINE ONLINE darac1
從上面的資訊可以看到DATADG磁碟組沒有載入，資料庫darac也沒有啟動，並且顯示錯誤的控制檔案。alert_asm1.log中，有建立磁碟組的資訊:
Wed Dec 02 18:27:46 2015 SQL> CREATE DISKGROUP DATADG EXTERNAL REDUNDANCY DISK '/dev/raw/raw3' SIZE 10240M ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */
14.檢視磁碟組的狀態
SQL> select name,state from v$asm_diskgroup; NAME STATE -------------------------------------------------- ---------------------- CRSDG MOUNTED ARCH MOUNTED
15.手動載入DATADG磁碟報錯
SQL> alter diskgroup DATADG mount; alter diskgroup DATADG mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15017: diskgroup "DATADG" cannot be mounted ORA-15040: diskgroup is incomplete
16.檢視磁碟組磁碟頭的狀態,可以看到/dev/raw/raw3為candidate
SQL> select name,path,header_status from v$asm_disk;
NAME PATH HEADER_STATUS
-------------------------------------------------- -------------------------------------------------- ------------------------------
/dev/raw/raw3 CANDIDATE
ARCH_0000 /dev/raw/raw2 MEMBER
CRSDG_0000 /dev/raw/raw1 MEMBER
17.嘗試使用自動備份的磁碟頭資訊來恢復磁碟頭,這個DATADG磁碟恢復成功。
[grid@darac1 ~]$ kfed repair /dev/raw/raw3
SQL> select name,state from v$asm_diskgroup;
NAME STATE
-------------------------------------------------- ----------------------
CRSDG MOUNTED
DATADG DISMOUNTED
ARCH MOUNTED
SQL> select name,path,header_status from v$asm_disk;
NAME PATH HEADER_STATUS
-------------------------------------------------- -------------------------------------------------- ------------------------------
/dev/raw/raw3 MEMBER
ARCH_0000 /dev/raw/raw2 MEMBER
CRSDG_0000 /dev/raw/raw1 MEMBER
18.手動載入DATADG磁碟報錯
SQL> alter diskgroup DATADG mount; Diskgroup altered. SQL> select name,state from v$asm_diskgroup; NAME STATE -------------------------------------------------- ---------------------- CRSDG MOUNTED DATADG MOUNTED ARCH MOUNTED
19.檢視磁碟組磁碟頭的狀態,可以看到/dev/raw/raw3為member
SQL> select name,path,header_status from v$asm_disk; NAME PATH HEADER_STATUS -------------------------------------------------- -------------------------------------------------- ------------------------------ ARCH_0000 /dev/raw/raw2 MEMBER DATADG_0000 /dev/raw/raw3 MEMBER CRSDG_0000 /dev/raw/raw1 MEMBER
20.啟動資料庫darac
[grid@darac1 ~]$ srvctl start database -d darac
[grid@darac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.CRSDG.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.DATADG.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.LISTENER.lsnr
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.asm
ONLINE ONLINE darac1 Started
ONLINE ONLINE darac2 Started
ora.gsd
OFFLINE OFFLINE darac1
OFFLINE OFFLINE darac2
ora.net1.network
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.ons
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.registry.acfs
ONLINE ONLINE darac1
ONLINE ONLINE darac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE darac1
ora.cvu
1 ONLINE ONLINE darac1
ora.darac.db
1 ONLINE ONLINE darac1 Open
2 ONLINE ONLINE darac2 Open
ora.darac1.vip
1 ONLINE ONLINE darac1
ora.darac2.vip
1 ONLINE ONLINE darac2
ora.darac3.vip
1 ONLINE OFFLINE
ora.oc4j
1 ONLINE ONLINE darac1
ora.scan1.vip
1 ONLINE ONLINE darac1
到此資料庫恢復成功。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26015009/viewspace-2126295/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle asm磁碟損壞異常恢復OracleASM
- ASM之OCR所在磁碟組損壞後的恢復ASM
- ASM磁碟頭資訊損壞和修復（kfed/dd）ASM
- Oracle database 11g rac損壞ocr和votedisk恢復實驗OracleDatabase
- index損壞恢復Index
- Oracle 11g RAC ASM VOTEDISK所在磁碟全部丟失重大故障恢復OracleASM
- 磁碟損壞造成RMAN備份檔案有壞塊的恢復案例
- 存貯在asm中的ocr/votedisk/asm磁碟組損害的恢復ASM
- RAC磁碟頭損壞問題處理
- linux下修復磁碟損壞Linux
- 磁碟損壞導致資料檔案丟失的恢復
- 探索ORACLE之RMAN_07 磁碟損壞資料丟失恢復Oracle
- 損壞聯機日誌 恢復
- 介質恢復，即磁碟或作業系統檔案損壞的恢復-非RMAN方法作業系統
- Oracle Database 12c RAC損壞ocr和votedisk恢復實驗OracleDatabase
- 記一次Oracle RAC for aix 儲存雙控鎖盤導致ASM控制檔案損壞恢復OracleAIASM
- 損壞控制檔案的恢復方法
- SQL Server ldf 檔案損壞恢復SQLServer
- 11G RAC 為 ASM 磁碟組增加一個磁碟。（AIX）ASMAI
- u盤檔案損壞怎麼恢復資料 u盤恢復損壞資料的有效方法
- RAC asm恢復到單例項ASM單例
- Oracle 11g 資料庫恢復-場景7：部分檔案損壞Oracle資料庫
- Oracle 10g rac+asm 磁碟頭備份與恢復實驗Oracle 10gASM
- oracle 11g rac 共享儲存壞掉後資料庫恢復Oracle資料庫
- 聯機日誌損壞恢復實驗
- Oracle塊損壞恢復（有rman備份）Oracle
- 恢復之單個控制檔案損壞
- 損壞聯機日誌的恢復方法
- 單個控制檔案損壞的恢復
- 當前日誌組全部損壞的恢復
- 線上日誌檔案損壞恢復方法
- Oracle 11g 資料庫恢復-場景5：部分檔案損壞恢復,開庫狀態,低可用恢復方式Oracle資料庫
- Oracle 11g 資料庫恢復-場景4：部分檔案損壞恢復,開庫狀態, 高可用恢復方式Oracle資料庫
- Oralce 11g資料庫恢復-場景3：部分檔案損壞恢復,關庫狀態,高可用恢復方式資料庫
- oracle crs voting disk損壞一例（asm+rac）OracleASM
- rac asm 恢復到 單例項 1ASM單例
- rac asm 恢復到 單例項 2ASM單例
- u盤檔案損壞怎麼恢復資料 u盤損壞無法讀取怎麼恢復資料