ASM 仲裁盤OCR DG損壞，重建步驟

startay發表於2016-01-25
ASM

主機啟動後發現少了ocrsd.bin程式，檢查grid alert日誌，發現下面資訊。 ASM OCR DG 無法啟動。

  1. GMON updating disk modes for group 2 at 10 for pid 27, osid 7121
  2. ERROR: no read quorum in group: required 2, found 0 disks
  3. Fri Jan 15 10:33:21 2016
  4. NOTE: cache dismounting (not clean) group 2/0x676844DF (OCRDG)
  5. NOTE: messaging CKPT to quiesce pins Unix process pid: 7124, image: oracle@cntl202 (B001)
  6. Fri Jan 15 10:33:21 2016
  7. NOTE: halting all I/Os to diskgroup 2 (OCRDG)      >>>>>>>>>>>>>>>OCRDG有IO故障。
  8. Fri Jan 15 10:33:21 2016
  9. NOTE: LGWR doing non-clean dismount of group 2 (OCRDG)
  10. NOTE: LGWR sync ABA=6.60 last written ABA 6.60
  11. WARNING: Offline for disk OCRDG_0000 in mode 0x7f failed.
  12. WARNING: Offline for disk OCRDG_0001 in mode 0x7f failed.
  13. WARNING: Offline for disk OCRDG_0002 in mode 0x7f failed.
  14. Fri Jan 15 10:33:21 2016
  15. NOTE: No asm libraries found in the system

ASM使用ASMLIB，"oracleasm listdisks" 發現少了一個ASM DISKS，
  1. --缺少了一個ASMDISK1
  2. [/root]# oracleasm listdisks
  3. ASMDISK2 
  4. ASMDISK3 
  5. DATADISK1
 [/root]# oracleasm querydisk -v -p -d ASMDISK1
 Disk "ASMDISK1" does not exist or is not instantiated
重建ASMDISK1
  1. --無法建立，提示已存在。
  2. [/root]#oracleasm createdisk ASMDISK1 /dev/sdd1
  3. Device "/dev/sdd1" is already labeled for ASM disk ""

  4.
  5. [/root]#oracleasm deletedisk ASMDISK1
  6. Disk "ASMDISK1" does not exist or is not instantiated
 --清空磁碟頭。
  1. [/root]#dd if=/dev/zero of=/dev/sdd1 bs=1024 count=100
  2. 100+0 records in
  3. 100+0 records out
  4. 102400 bytes (102 kB) copied, 0.0812539 s, 1.3 MB/s

  6. [/root]#oracleasm createdisk ASMDISK1 /dev/sdd1
  7. Writing disk header: done
  8. Instantiating disk: done


1. [ALL NODES]停止CRS，crsd.bin程式不存在，無法正常停，需要 "-f" 強制停止
  1. [/root]#crsctl stop crs -f
  2. CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cntl069'
  3. CRS-2673: Attempting to stop 'ora.mdnsd' on 'cntl069'
  4. CRS-2673: Attempting to stop 'ora.ctssd' on 'cntl069'
  5. CRS-2673: Attempting to stop 'ora.asm' on 'cntl069'
  6. CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cntl069'
  7. CRS-2677: Stop of 'ora.mdnsd' on 'cntl069' succeeded
  8. CRS-2677: Stop of 'ora.asm' on 'cntl069' succeeded
  9. CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'cntl069'
  10. CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'cntl069' succeeded
  11. CRS-2677: Stop of 'ora.drivers.acfs' on 'cntl069' succeeded
  12. CRS-2677: Stop of 'ora.ctssd' on 'cntl069' succeeded
  13. CRS-2673: Attempting to stop 'ora.cssd' on 'cntl069'
  14. CRS-2677: Stop of 'ora.cssd' on 'cntl069' succeeded
  15. CRS-2673: Attempting to stop 'ora.gipcd' on 'cntl069'
  16. CRS-2677: Stop of 'ora.gipcd' on 'cntl069' succeeded
  17. CRS-2673: Attempting to stop 'ora.gpnpd' on 'cntl069'
  18. CRS-2677: Stop of 'ora.gpnpd' on 'cntl069' succeeded
  19. CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cntl069' has completed
  20. CRS-4133: Oracle High Availability Services has been stopped.

2. 刪除CRS  <--root user
  1. [/root]#$CRS_HOME/crs/install/rootcrs.pl -deconfig -force               >>>>>>>>>其他節點
  2. Using configuration parameter file: /grid/11.2/product/crs/install/crsconfig_params
  3. PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
  4. PRCR-1068 : Failed to query resources
  5. Cannot communicate with crsd
  6. PRCR-1070 : Failed to check if resource ora.gsd is registered
  7. Cannot communicate with crsd
  8. PRCR-1070 : Failed to check if resource ora.ons is registered
  9. Cannot communicate with crsd
  10. ...

  12. [/root]#$CRS_HOME/crs/install/rootcrs.pl -deconfig -force -lastnode        >>>>>>>>>最後一個節點
  13. CRS-4535: Cannot communicate with Cluster Ready Services
  14. CRS-4000: Command Stop failed, or completed with errors.
  15. CRS-4544: Unable to connect to OHAS
  16. CRS-4000: Command Stop failed, or completed with errors.
  17. Removing Trace File Analyzer
  18. error: package cvuqdisk is not installed
  19. Successfully deconfigured Oracle clusterware stack on this node

3. [ALL NODES] 執行root.sh 配置CS  <-- root user
  1. [/root]#CRS_HOME/root.sh
  2. Performing root user operation for Oracle 11g

  4. The following environment variables are set as:
  5.     ORACLE_OWNER= grid
  6.     ORACLE_HOME= /grid/11.2/product

  8. Enter the full pathname of the local bin directory: [/usr/local/bin]:
  9. The contents of "dbhome" have not changed. No need to overwrite.
  10. The contents of "oraenv" have not changed. No need to overwrite.
  11. The contents of "coraenv" have not changed. No need to overwrite.

  13. Entries will be added to the /etc/oratab file as needed by
  14. Database Configuration Assistant when a database is created
  15. Finished running generic part of root script.
  16. Now product-specific root actions will be performed.
  17. Using configuration parameter file: /grid/11.2/product/crs/install/crsconfig_params
  18. User ignored Prerequisites during installation
  19. Installing Trace File Analyzer
  20. OLR initialization - successful
  21. Adding Clusterware entries to upstart
  22. CRS-2672: Attempting to start 'ora.mdnsd' on 'cntl069'
  23. CRS-2676: Start of 'ora.mdnsd' on 'cntl069' succeeded
  24. CRS-2672: Attempting to start 'ora.gpnpd' on 'cntl069'
  25. CRS-2676: Start of 'ora.gpnpd' on 'cntl069' succeeded
  26. CRS-2672: Attempting to start 'ora.cssdmonitor' on 'cntl069'
  27. CRS-2672: Attempting to start 'ora.gipcd' on 'cntl069'
  28. CRS-2676: Start of 'ora.cssdmonitor' on 'cntl069' succeeded
  29. CRS-2676: Start of 'ora.gipcd' on 'cntl069' succeeded
  30. CRS-2672: Attempting to start 'ora.cssd' on 'cntl069'
  31. CRS-2672: Attempting to start 'ora.diskmon' on 'cntl069'
  32. CRS-2676: Start of 'ora.diskmon' on 'cntl069' succeeded
  33. CRS-2676: Start of 'ora.cssd' on 'cntl069' succeeded

  35. ASM created and started successfully.

  37. Disk Group CRS created successfully.

  39. clscfg: -install mode specified
  40. Successfully accumulated necessary OCR keys.
  41. Creating OCR keys for user 'root', privgrp 'root'..
  42. Operation successful.
  43. Successful addition of voting disk 966344bb8c054f61bf7627175f0d53e8.
  44. Successful addition of voting disk 3f3735b4d5f44f79bf27624845c0c81b.
  45. Successful addition of voting disk 8bf524ee376b4ff1bf5c3b7e69e5c98b.
  46. Successfully replaced voting disk group with +CRS.
  47. CRS-4266: Voting file(s) successfully replaced
  48. ## STATE File Universal Id File Name Disk group
  49. -- ----- ----------------- --------- ---------
  50.  1. ONLINE 966344bb8c054f61bf7627175f0d53e8 (ORCL:ASMDISK1) [CRS]
  51.  2. ONLINE 3f3735b4d5f44f79bf27624845c0c81b (ORCL:ASMDISK2) [CRS]
  52.  3. ONLINE 8bf524ee376b4ff1bf5c3b7e69e5c98b (ORCL:ASMDISK3) [CRS]
  53. Located 3 voting disk(s).
  54.   CRS-2672: Attempting to start 'ora.asm' on 'cntl069'
  55. CRS-2676: Start of 'ora.asm' on 'cntl069' succeeded
  56. CRS-2672: Attempting to start 'ora.CRS.dg' on 'cntl069'
  57. CRS-2676: Start of 'ora.CRS.dg' on 'cntl069' succeeded
  58. Preparing packages for installation...
  59. cvuqdisk-1.0.9-1
  60. Configure Oracle Grid Infrastructure for a Cluster ... succeeded

4. 檢查CRS配置正常。
  1. [/grid/11.2/product]#crsctl status res -t
  2. --------------------------------------------------------------------------------
  3. NAME TARGET STATE SERVER STATE_DETAILS
  4. --------------------------------------------------------------------------------
  5. Local Resources
  6. --------------------------------------------------------------------------------
  7. ora.CRS.dg
  8.                ONLINE ONLINE cntl069
  9.                ONLINE ONLINE cntl070
  10. ora.asm
  11.                ONLINE ONLINE cntl069 Started
  12.                ONLINE ONLINE cntl070 Started
  13. ora.gsd
  14.                OFFLINE OFFLINE cntl069
  15.                OFFLINE OFFLINE cntl070
  16. ora.net1.network
  17.                ONLINE ONLINE cntl069
  18.                ONLINE ONLINE cntl070
  19. ora.ons
  20.                ONLINE ONLINE cntl069
  21.                ONLINE ONLINE cntl070
  22. ora.registry.acfs
  23.                ONLINE ONLINE cntl069
  24.                ONLINE ONLINE cntl070
  25. --------------------------------------------------------------------------------
  26. Cluster Resources
  27. --------------------------------------------------------------------------------
  28. ora.LISTENER_SCAN1.lsnr
  29.       1 ONLINE ONLINE cntl069
  30. ora.cntl069.vip
  31.       1 ONLINE ONLINE cntl069
  32. ora.cntl070.vip
  33.       1 ONLINE ONLINE cntl070
  34. ora.cvu
  35.       1 ONLINE ONLINE cntl069
  36. ora.oc4j
  37.       1 ONLINE ONLINE cntl069
  38. ora.scan1.vip
  39.       1 ONLINE ONLINE cntl069

5. 確實LISTENER資源，netca建立。

6. 如果已經有資料庫，可使用下面命令加入資源
  1. 新增資料庫(oracle使用者下執行)
  2.  [oracle@node1 ~]$ srvctl add database -d racdb -o /u01/product/oracle/11.2.0/db_1
  3.  [oracle@node1 ~]$
  4.  
  5. 新增例項(oracle使用者下執行)
  6.  [oracle@node1 ~]$ srvctl add instance -d racdb -i racdb1 -n node1
  7.  [oracle@node1 ~]$ srvctl add instance -d racdb -i racdb2 -n node2
  8.  
  9. 新增之前資料庫的服務(oracle使用者下執行)
  10.  [oracle@node1 ~]$ srvctl add service -d racdb -s kettle -r racdb1 -a racdb2 -P BASIC
若還沒資料庫，可使用dbca重建。                                                                                                                                                                                                                                                                                             

再次檢查CRS狀態
  1. [/grid/11.2/product]#crsctl status res -t
  2. --------------------------------------------------------------------------------
  3. NAME TARGET STATE SERVER STATE_DETAILS
  4. --------------------------------------------------------------------------------
  5. Local Resources
  6. --------------------------------------------------------------------------------
  7. ora.CRS.dg
  8.                ONLINE ONLINE cntl069
  9.                ONLINE ONLINE cntl070
  10. ora.DATA.dg
  11.                ONLINE ONLINE cntl069
  12.                ONLINE ONLINE cntl070
  13. ora.LISTENER.lsnr
  14.                ONLINE ONLINE cntl069
  15.                ONLINE ONLINE cntl070
  16. ora.asm
  17.                ONLINE ONLINE cntl069 Started
  18.                ONLINE ONLINE cntl070 Started
  19. ora.gsd
  20.                OFFLINE OFFLINE cntl069
  21.                OFFLINE OFFLINE cntl070
  22. ora.net1.network
  23.                ONLINE ONLINE cntl069
  24.                ONLINE ONLINE cntl070
  25. ora.ons
  26.                ONLINE ONLINE cntl069
  27.                ONLINE ONLINE cntl070
  28. ora.registry.acfs
  29.                ONLINE ONLINE cntl069
  30.                ONLINE ONLINE cntl070
  31. --------------------------------------------------------------------------------
  32. Cluster Resources
  33. --------------------------------------------------------------------------------
  34. ora.LISTENER_SCAN1.lsnr
  35.       1 ONLINE ONLINE cntl069
  36. ora.cntl069.vip
  37.       1 ONLINE ONLINE cntl069
  38. ora.cntl070.vip
  39.       1 ONLINE ONLINE cntl070
  40. ora.cvu
  41.       1 ONLINE ONLINE cntl069
  42. ora.ecard2pt.db
  43.       1 ONLINE ONLINE cntl069 Open
  44.       2 ONLINE ONLINE cntl070 Open
  45. ora.oc4j
  46.       1 ONLINE ONLINE cntl069
  47. ora.scan1.vip
  48.       1 ONLINE ONLINE cntl069




來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/17086096/viewspace-1982031/，如需轉載，請註明出處，否則將追究法律責任。

相關文章