RAC 11G ASM磁碟損壞恢復
一個儲存檔案的rac資料庫起不來了，生產環境是linux rac 11.2.0.4,原因是因為用工具測試磁碟IO時損壞了ocr所在磁碟組與儲存資料ASM磁碟的磁碟頭。下面是恢復過程:
1.檢查crs的狀態:
[grid@darac1 ~]$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager [root@darac1 crsd]# ps -ef|grep crs root 3126 1 1 10:34 ? 00:00:31 /u01/app/product/11.2.0/crs/bin/ohasd.bin reboot grid 3514 1 0 10:34 ? 00:00:07 /u01/app/product/11.2.0/crs/bin/oraagent.bin grid 3525 1 0 10:34 ? 00:00:00 /u01/app/product/11.2.0/crs/bin/mdnsd.bin grid 3537 1 0 10:34 ? 00:00:16 /u01/app/product/11.2.0/crs/bin/gpnpd.bin grid 3549 1 1 10:34 ? 00:00:33 /u01/app/product/11.2.0/crs/bin/gipcd.bin root 4128 1 0 10:54 ? 00:00:02 /u01/app/product/11.2.0/crs/bin/cssdmonitor root 4144 1 0 10:54 ? 00:00:01 /u01/app/product/11.2.0/crs/bin/cssdagent grid 4167 1 2 10:55 ? 00:00:14 /u01/app/product/11.2.0/crs/bin/ocssd.bin root 4354 3680 0 11:04 pts/1 00:00:00 grep crs
2.強制關閉crs
[root@darac1 bin]# ./crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'darac1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'darac1' CRS-2673: Attempting to stop 'ora.gipcd' on 'darac1' CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'darac1' CRS-2677: Stop of 'ora.cssdmonitor' on 'darac1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'darac1' succeeded CRS-2677: Stop of 'ora.gipcd' on 'darac1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'darac1' CRS-2677: Stop of 'ora.gpnpd' on 'darac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'darac1' has completed CRS-4133: Oracle High Availability Services has been stopped.
3.以exclusive模式啟動crs
[root@darac1 bin]# ./crsctl start crs -excl -nocrs CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'darac1' CRS-2676: Start of 'ora.mdnsd' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'darac1' CRS-2676: Start of 'ora.gpnpd' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'darac1' CRS-2672: Attempting to start 'ora.gipcd' on 'darac1' CRS-2676: Start of 'ora.gipcd' on 'darac1' succeeded CRS-2676: Start of 'ora.cssdmonitor' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'darac1' CRS-2672: Attempting to start 'ora.diskmon' on 'darac1' CRS-2676: Start of 'ora.diskmon' on 'darac1' succeeded CRS-2676: Start of 'ora.cssd' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on 'darac1' CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'darac1' CRS-2672: Attempting to start 'ora.ctssd' on 'darac1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'darac1' CRS-2676: Start of 'ora.ctssd' on 'darac1' succeeded CRS-2676: Start of 'ora.drivers.acfs' on 'darac1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'darac1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'darac1' CRS-2676: Start of 'ora.asm' on 'darac1' succeeded
4.檢視GI相關的alert.log日誌檔案如何
[ohasd(5040)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2016-10-13 11:20:47.302:
[gpnpd(5215)]CRS-2328:GPNPD started on node darac1.
2016-10-13 11:20:58.388:
[ohasd(5040)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2016-10-13 11:21:00.608:
[cssd(5318)]CRS-1713:CSSD daemon is started in clustered mode
2016-10-13 11:21:01.521:
[/u01/app/product/11.2.0/crs/bin/orarootagent.bin(5304)]CRS-5013:Agent "/u01/app/product/11.2.0/crs/bin/orarootagent.bin" failed to start process
"/u01/app/product/11.2.0/crs/bin/osysmond" for action "start": details at "(:CLSN00008:)" in
"/u01/app/product/11.2.0/crs/log/darac1/agent/ohasd/orarootagent_root//orarootagent_root.log"
2016-10-13 11:21:03.585:
[ohasd(5040)]CRS-2878:Failed to restart resource 'ora.crf'
2016-10-13 11:21:05.399:
[/u01/app/product/11.2.0/crs/bin/orarootagent.bin(5340)]CRS-5013:Agent "/u01/app/product/11.2.0/crs/bin/orarootagent.bin" failed to start process
"/u01/app/product/11.2.0/crs/bin/osysmond" for action "start": details at "(:CLSN00008:)" in
"/u01/app/product/11.2.0/crs/log/darac1/agent/ohasd/orarootagent_root//orarootagent_root.log"
2016-10-13 11:21:10.703:
[ohasd(5040)]CRS-2878:Failed to restart resource 'ora.crf'
2016-10-13 11:21:23.464:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:21:38.698:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:21:53.925:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:09.463:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:24.804:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:40.252:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:22:56.722:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:12.009:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:27.290:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:42.872:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:23:58.198:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:13.500:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:28.786:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:43.488:
[client(5394)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/product/11.2.0/crs/log/darac1/client/ocrcheck_5394.log.
2016-10-13 11:24:43.959:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:24:51.823:
[client(5424)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/product/11.2.0/crs/log/darac1/client/crsctl_grid.log.
2016-10-13 11:24:59.345:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:25:14.526:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:25:29.696:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:25:44.860:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:00.042:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:15.218:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:30.409:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:45.577:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:26:49.031:
[client(5460)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/product/11.2.0/crs/log/darac1/client/ocrconfig_5460.log.
2016-10-13 11:27:00.766:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:27:15.951:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:27:31.142:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:27:46.339:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:01.530:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:16.733:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:32.008:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:28:47.191:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:02.389:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:17.610:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:32.832:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:29:48.035:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:03.229:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:18.434:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:33.679:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:30:48.876:
[cssd(5318)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in
/u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:31:01.534:
[/u01/app/product/11.2.0/crs/bin/cssdagent(5284)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in
/u01/app/product/11.2.0/crs/log/darac1/agent/ohasd/oracssdagent_root//oracssdagent_root.log.
2016-10-13 11:31:01.540:
[cssd(5318)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/product/11.2.0/crs/log/darac1/cssd/ocssd.log
2016-10-13 11:31:01.541:
[cssd(5318)]CRS-1603:CSSD on node darac1 shutdown by user.
從上面的資訊可以看到找不到voting files
5.檢查ASM的alert.log可以找如下建立CRSDG,DATADG磁碟組的建立語句:
Wed Dec 02 16:09:01 2015 SQL> CREATE DISKGROUP CRSDG EXTERNAL REDUNDANCY DISK '/dev/raw/raw1' ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */
6.檢查磁碟頭
[grid@darac1 ~]$ kfed read /dev/raw/raw1 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 300392945 ; 0x00c: 0x11e7a1f1 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 B7F46200 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
7.使用kfed恢復CRSDG的磁碟頭，但因為備份資訊也被損壞所以恢復時報錯，而且沒有手動備份
[grid@darac1 ~]$ kfed repair /dev/raw/raw1 KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]
沒有透過自動備份的磁碟頭資訊來進行恢復，只能使用自動備份的ocr資訊來恢復了操作如下。
8.建立磁碟組
[grid@darac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Thu Oct 13 13:00:42 2016 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production With the Real Application Clusters and Automatic Storage Management options SQL> select * from v$asm_diskgroup; no rows selected SQL> create diskgroup CRSDG external redundancy disk '/dev/raw/raw1' attribute 'COMPATIBLE.ASM' = '11.2.0.0.0'; Diskgroup created.
9.檢視自動備份的ocr檔案
[root@darac1 bin]# ./ocrconfig -showbackup PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy darac2 2016/10/13 06:29:53 /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup00.ocr darac2 2016/10/13 02:29:45 /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup01.ocr darac2 2016/10/12 22:29:37 /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup02.ocr darac2 2016/10/12 02:27:20 /u01/app/product/11.2.0/crs/cdata/darac-cluster/day.ocr darac2 2016/10/11 22:27:10 /u01/app/product/11.2.0/crs/cdata/darac-cluster/week.ocr
10.還原ocr
[root@darac1 bin]# ./ocrconfig -restore /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup00.ocr
11.處理votedisk
[root@darac1 bin]# ./ocrconfig -restore /u01/app/product/11.2.0/crs/cdata/darac-cluster/backup00.ocr [root@darac1 bin]# ./crsctl replace votedisk +CRSDG Successful addition of voting disk 44eaf86504ea4f76bfb43cb7931a3fc7. Successfully replaced voting disk group with +CRSDG. CRS-4266: Voting file(s) successfully replaced
12.建立asm spfile
[grid@darac1 ~]$ vi /tmp/asm.txt instance_type='asm' large_pool_size=12M remote_login_passwordfile= 'EXCLUSIVE' asm_diskstring = '/dev/raw/raw*' asm_power_limit =1 [grid@darac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Thu Oct 13 13:40:02 2016 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - Production With the Real Application Clusters and Automatic Storage Management options SQL> create spfile='+CRSDG' FROM pfile='/tmp/asm.txt'; File created.
13.重啟crs
[root@darac1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'darac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'darac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'darac1'
CRS-2673: Attempting to stop 'ora.asm' on 'darac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'darac1'
CRS-2677: Stop of 'ora.ctssd' on 'darac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'darac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'darac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'darac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'darac1'
CRS-2677: Stop of 'ora.cssd' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'darac1'
CRS-2677: Stop of 'ora.gipcd' on 'darac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'darac1'
CRS-2677: Stop of 'ora.gpnpd' on 'darac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'darac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@darac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[grid@darac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.DATADG.dg
ONLINE OFFLINE darac1
ONLINE OFFLINE darac2
ora.LISTENER.lsnr
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.asm
ONLINE ONLINE darac1 Started
ONLINE ONLINE darac2 Started
ora.gsd
OFFLINE OFFLINE darac1
OFFLINE OFFLINE darac2
ora.net1.network
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.ons
ONLINE ONLINE darac1
ONLINE OFFLINE darac2
ora.registry.acfs
ONLINE ONLINE darac1
ONLINE ONLINE darac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE darac1
ora.cvu
1 ONLINE ONLINE darac1
ora.darac.db
1 ONLINE OFFLINE Corrupted Controlfi
le
2 ONLINE OFFLINE Corrupted Controlfi
le
ora.darac1.vip
1 ONLINE ONLINE darac1
ora.darac2.vip
1 ONLINE ONLINE darac2
ora.darac3.vip
1 ONLINE OFFLINE
ora.oc4j
1 ONLINE OFFLINE STARTING
ora.scan1.vip
1 ONLINE ONLINE darac1
從上面的資訊可以看到DATADG磁碟組沒有載入，資料庫darac也沒有啟動，並且顯示錯誤的控制檔案。alert_asm1.log中，有建立磁碟組的資訊:
Wed Dec 02 18:27:46 2015 SQL> CREATE DISKGROUP DATADG EXTERNAL REDUNDANCY DISK '/dev/raw/raw3' SIZE 10240M ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M' /* ASMCA */
14.檢視磁碟組的狀態
SQL> select name,state from v$asm_diskgroup; NAME STATE -------------------------------------------------- ---------------------- CRSDG MOUNTED ARCH MOUNTED
15.手動載入DATADG磁碟報錯
SQL> alter diskgroup DATADG mount; alter diskgroup DATADG mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15017: diskgroup "DATADG" cannot be mounted ORA-15040: diskgroup is incomplete
16.檢視磁碟組磁碟頭的狀態,可以看到/dev/raw/raw3為candidate
SQL> select name,path,header_status from v$asm_disk;
NAME PATH HEADER_STATUS
-------------------------------------------------- -------------------------------------------------- ------------------------------
/dev/raw/raw3 CANDIDATE
ARCH_0000 /dev/raw/raw2 MEMBER
CRSDG_0000 /dev/raw/raw1 MEMBER
17.嘗試使用自動備份的磁碟頭資訊來恢復磁碟頭,這個DATADG磁碟恢復成功。
[grid@darac1 ~]$ kfed repair /dev/raw/raw3
SQL> select name,state from v$asm_diskgroup;
NAME STATE
-------------------------------------------------- ----------------------
CRSDG MOUNTED
DATADG DISMOUNTED
ARCH MOUNTED
SQL> select name,path,header_status from v$asm_disk;
NAME PATH HEADER_STATUS
-------------------------------------------------- -------------------------------------------------- ------------------------------
/dev/raw/raw3 MEMBER
ARCH_0000 /dev/raw/raw2 MEMBER
CRSDG_0000 /dev/raw/raw1 MEMBER
18.手動載入DATADG磁碟報錯
SQL> alter diskgroup DATADG mount; Diskgroup altered. SQL> select name,state from v$asm_diskgroup; NAME STATE -------------------------------------------------- ---------------------- CRSDG MOUNTED DATADG MOUNTED ARCH MOUNTED
19.檢視磁碟組磁碟頭的狀態,可以看到/dev/raw/raw3為member
SQL> select name,path,header_status from v$asm_disk; NAME PATH HEADER_STATUS -------------------------------------------------- -------------------------------------------------- ------------------------------ ARCH_0000 /dev/raw/raw2 MEMBER DATADG_0000 /dev/raw/raw3 MEMBER CRSDG_0000 /dev/raw/raw1 MEMBER
20.啟動資料庫darac
[grid@darac1 ~]$ srvctl start database -d darac
[grid@darac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.CRSDG.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.DATADG.dg
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.LISTENER.lsnr
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.asm
ONLINE ONLINE darac1 Started
ONLINE ONLINE darac2 Started
ora.gsd
OFFLINE OFFLINE darac1
OFFLINE OFFLINE darac2
ora.net1.network
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.ons
ONLINE ONLINE darac1
ONLINE ONLINE darac2
ora.registry.acfs
ONLINE ONLINE darac1
ONLINE ONLINE darac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE darac1
ora.cvu
1 ONLINE ONLINE darac1
ora.darac.db
1 ONLINE ONLINE darac1 Open
2 ONLINE ONLINE darac2 Open
ora.darac1.vip
1 ONLINE ONLINE darac1
ora.darac2.vip
1 ONLINE ONLINE darac2
ora.darac3.vip
1 ONLINE OFFLINE
ora.oc4j
1 ONLINE ONLINE darac1
ora.scan1.vip
1 ONLINE ONLINE darac1
到此資料庫恢復成功。
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/26015009/viewspace-2126295/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- Oracle asm磁碟損壞異常恢復OracleASM
- ASM磁碟頭資訊損壞和修復（kfed/dd）ASM
- RAC磁碟頭損壞問題處理
- linux下修復磁碟損壞Linux
- 記一次Oracle RAC for aix 儲存雙控鎖盤導致ASM控制檔案損壞恢復OracleAIASM
- Oracle Database 12c RAC損壞ocr和votedisk恢復實驗OracleDatabase
- u盤檔案損壞怎麼恢復資料 u盤恢復損壞資料的有效方法
- RAID10磁碟陣列損壞修復操作AI陣列
- linux RAID10磁碟陣列損壞的修復LinuxAI陣列
- 資料底層損壞的恢復方法—拼碎片恢復資料
- u盤檔案損壞怎麼恢復資料 u盤損壞無法讀取怎麼恢復資料
- 隨身碟顆粒損壞資料恢復資料恢復
- Oracle RAC日常運維-ASM磁碟擴容Oracle運維ASM
- 深入解析：段頭塊損壞bbed異常恢復
- 【ASM】ASM磁碟頭被重寫，如何修復ASM
- 資料恢復工具Recoverit使用教程:如何修復損壞的影片資料恢復
- Oracle RAC環境下ASM磁碟組擴容OracleASM
- 【LINUX】Oracle資料庫 linux磁碟頭資料損壞修復LinuxOracle資料庫
- WPS文件損壞如何修復？WPS文件損壞的修復方法
- 【資料庫資料恢復】Oracle資料庫ASM磁碟組掉線如何恢復資料？資料庫資料恢復OracleASM
- 電腦進水導致硬碟損壞資料恢復硬碟資料恢復
- 【資料庫資料恢復】ASM磁碟組掉線的Oracle資料庫資料恢復案例資料庫資料恢復ASMOracle
- RAC之grid叢集安裝及ASM磁碟組配置ASM
- Oracle資料庫不同損壞級別的恢復詳解Oracle資料庫
- 【伺服器資料恢復】EqualLogic儲存磁碟出現壞道的資料恢復案例伺服器資料恢復
- 資料庫資料恢復—NTFS分割槽損壞如何恢復SqlServer資料庫資料資料庫資料恢復SQLServer
- 【資料庫資料恢復】MongoDB資料庫檔案損壞的資料恢復案例資料庫資料恢復MongoDB
- Ceph 磁碟損壞現象和解決方法
- Oracle RAC ASM磁碟組擴容時遇到的VIP漂移OracleASM
- 【儲存資料恢復】IBM儲存檔案NTFS系統損壞的資料恢復案例資料恢復IBM
- PostgreSQL DBA(30) - Backup&Recovery#3(資料檔案損壞恢復)SQL
- ibdata1檔案損壞時恢復InnoDB單表測試
- 伺服器硬碟磁頭損壞，碟片劃傷資料恢復伺服器硬碟資料恢復
- 伺服器系統癱瘓系統損壞資料恢復伺服器資料恢復
- RAC恢復到單機
- 通過ORACLE VM virtualbox環境安裝oracle 11G RAC(ASM)OracleASM
- goldengate + asm + racGoASM
- 【伺服器資料恢復】IBM儲存伺服器硬碟壞道離線、oracle資料庫損壞的資料恢復伺服器資料恢復IBM硬碟Oracle資料庫
- 資料庫資料恢復-SQL SERVER資料庫MDF (NDF)或LDF損壞如何恢復資料？資料庫資料恢復SQLServer