RAC環境利用備份恢復RAC資料庫（五）

由於篇幅的問題，上一篇文章只是描述了問題的現象，以及嘗試的一些解決方法，但是這些方法並不奏效。

看來只能透過TRACE的方法來檢查導致問題的原因了：

bash-3.00$ rman target / log=rman.log trace=rman.trc
RMAN> debug on;
RMAN> RUN
2> {
3> ALLOCATE CHANNEL C1 DEVICE TYPE DISK;
4> SET NEWNAME FOR DATAFILE 1 TO '+DATA/tradedb/tradedb_system_1_4g';
5> SET NEWNAME FOR DATAFILE 2 TO '+DATA/tradedb/tradedb_undotbs1_1_4g';
6> SET NEWNAME FOR DATAFILE 3 TO '+DATA/tradedb/tradedb_undotbs1_2_4g';
7> SET NEWNAME FOR DATAFILE 4 TO '+DATA/tradedb/tradedb_undotbs1_3_4g';
8> SET NEWNAME FOR DATAFILE 5 TO '+DATA/tradedb/tradedb_undotbs1_4_4g';
9> SET NEWNAME FOR DATAFILE 6 TO '+DATA/tradedb/tradedb_undotbs1_5_4g';
10> SET NEWNAME FOR DATAFILE 7 TO '+DATA/tradedb/tradedb_sysaux_1_1g';
11> SET NEWNAME FOR DATAFILE 8 TO '+DATA/tradedb/tradedb_undotbs2_1_4g';
12> SET NEWNAME FOR DATAFILE 9 TO '+DATA/tradedb/tradedb_undotbs2_2_4g';
13> SET NEWNAME FOR DATAFILE 10 TO '+DATA/tradedb/tradedb_undotbs2_3_4g';
14> SET NEWNAME FOR DATAFILE 11 TO '+DATA/tradedb/tradedb_undotbs2_4_4g';
15> SET NEWNAME FOR DATAFILE 12 TO '+DATA/tradedb/tradedb_undotbs2_5_4g';
16> SET NEWNAME FOR DATAFILE 13 TO '+DATA/tradedb/tradedb_users_1_200m';
17> SET NEWNAME FOR DATAFILE 14 TO '+DATA/tradedb/tradedb_ndmain_1_4g';
18> SET NEWNAME FOR DATAFILE 15 TO '+DATA/tradedb/tradedb_ndmain_2_4g';
19> SET NEWNAME FOR DATAFILE 16 TO '+DATA/tradedb/tradedb_ndmain_3_4g';
20> SET NEWNAME FOR DATAFILE 17 TO '+DATA/tradedb/tradedb_anhui_1';
21> SET NEWNAME FOR DATAFILE 18 TO '+DATA/tradedb/tradedb_anhui_2';
22> SET NEWNAME FOR DATAFILE 19 TO '+DATA/tradedb/tradedb_anhui_3';
23> SET NEWNAME FOR DATAFILE 20 TO '+DATA/tradedb/tradedb_anhui_4';
24> SET NEWNAME FOR DATAFILE 21 TO '+DATA/tradedb/tradedb_zhejiang_1_4g';
25> SET NEWNAME FOR DATAFILE 22 TO '+DATA/tradedb/tradedb_zhejiang_2_4g';
26> SET NEWNAME FOR DATAFILE 23 TO '+DATA/tradedb/tradedb_zhejiang_3_4g';
27> SET NEWNAME FOR DATAFILE 24 TO '+DATA/tradedb/tradedb_zhejiang_4_4g';
28> SET NEWNAME FOR DATAFILE 25 TO '+DATA/tradedb/tradedb_zhejiang_5_4g';
29> SET NEWNAME FOR DATAFILE 26 TO '+DATA/tradedb/tradedb_zhejiang_6_4g';
30> SET NEWNAME FOR DATAFILE 27 TO '+DATA/tradedb/tradedb_zhejiang_7_4g';
31> SET NEWNAME FOR DATAFILE 28 TO '+DATA/tradedb/tradedb_zhejiang_8_4g';
32> SET NEWNAME FOR DATAFILE 29 TO '+DATA/tradedb/tradedb_zhejiang_9_4g';
33> SET NEWNAME FOR DATAFILE 30 TO '+DATA/tradedb/tradedb_zhejiang_10_4g';
34> SET NEWNAME FOR DATAFILE 31 TO '+DATA/tradedb/tradedb_zhejiang_11_4g';
35> SET NEWNAME FOR DATAFILE 32 TO '+DATA/tradedb/tradedb_zhejiang_12_4g';
36> SET NEWNAME FOR DATAFILE 33 TO '+DATA/tradedb/tradedb_zhejiang_13_4g';
37> SET NEWNAME FOR DATAFILE 34 TO '+DATA/tradedb/tradedb_zhejiang_14_4g';
38> SET NEWNAME FOR DATAFILE 35 TO '+DATA/tradedb/tradedb_zhejiang_15_4g';
39> SET NEWNAME FOR DATAFILE 36 TO '+DATA/tradedb/tradedb_zhejiang_16_4g';
40> SET NEWNAME FOR DATAFILE 37 TO '+DATA/tradedb/tradedb_zhejiang_17_4g';
41> SET NEWNAME FOR DATAFILE 38 TO '+DATA/tradedb/tradedb_zhejiang_18_4g';
42> SET NEWNAME FOR DATAFILE 39 TO '+DATA/tradedb/tradedb_sysaux_2_1g';
43> SET NEWNAME FOR DATAFILE 40 TO '+DATA/tradedb/tradedb_gpo_1_4g';
44> SET NEWNAME FOR DATAFILE 41 TO '+DATA/tradedb/tradedb_gpo_2_4g';
45> SET NEWNAME FOR DATAFILE 42 TO '+DATA/tradedb/tradedb_gpo_3_4g';
46> SET NEWNAME FOR DATAFILE 43 TO '+DATA/tradedb/tradedb_gpo_4_4g';
47> SET NEWNAME FOR DATAFILE 44 TO '+DATA/tradedb/tradedb_zhejiang_19_4g';
48> SET NEWNAME FOR DATAFILE 45 TO '+DATA/tradedb/tradedb_zhejiang_20_4g';
49> SET NEWNAME FOR DATAFILE 46 TO '+DATA/tradedb/tradedb_beijing_1_4g';
50> SET NEWNAME FOR DATAFILE 47 TO '+DATA/tradedb/tradedb_beijing_2_4g';
51> SET NEWNAME FOR DATAFILE 48 TO '+DATA/tradedb/tradedb_beijing_3_4g';
52> SET NEWNAME FOR DATAFILE 49 TO '+DATA/tradedb/tradedb_beijing_4_4g';
53> SET NEWNAME FOR DATAFILE 50 TO '+DATA/tradedb/tradedb_beijing_5_4g';
54> SET NEWNAME FOR DATAFILE 51 TO '+DATA/tradedb/tradedb_beijing_6_4g';
55> SET NEWNAME FOR DATAFILE 52 TO '+DATA/tradedb/tradedb_beijing_7_4g';
56> SET NEWNAME FOR DATAFILE 53 TO '+DATA/tradedb/tradedb_beijing_8_4g';
57> SET NEWNAME FOR DATAFILE 54 TO '+DATA/tradedb/tradedb_beijing_9_4g';
58> SET NEWNAME FOR DATAFILE 55 TO '+DATA/tradedb/tradedb_beijing_10_4g';
59> SET NEWNAME FOR DATAFILE 56 TO '+DATA/tradedb/tradedb_beijing_11_4g';
60> SET NEWNAME FOR DATAFILE 57 TO '+DATA/tradedb/tradedb_beijing_12_4g';
61> SET NEWNAME FOR DATAFILE 58 TO '+DATA/tradedb/tradedb_beijing_13_4g';
62> SET NEWNAME FOR DATAFILE 59 TO '+DATA/tradedb/tradedb_beijing_14_4g';
63> SET NEWNAME FOR DATAFILE 60 TO '+DATA/tradedb/tradedb_beijing_15_4g';
64> RESTORE DATABASE;
65> SWITCH DATAFILE ALL;
66> SWITCH TEMPFILE ALL;
67> RELEASE CHANNEL C1;
68> }
RMAN> debug off;
RMAN> exit

下面檢查RMAN生成的TRACE檔案，由於檔案包含了大量的資訊，這裡只列出和問題相關的部分資訊：

RMAN-08003: 通道 C1: 正在讀取備份段 /data/backup/tradedb/a7kmlkem_1_1

DBGSQL:        EXEC SQL AT TARGET select decode(open_mode,'MOUNTED',0,'READ WRITE',1,'READ ONLY',1,0) into :b1  from v$database  [11
:15:27.012]
DBGSQL:           sqlcode=0 [11:15:27.014]
DBGSQL:              :b1 = 0
DBGMISC:      EXITED krmzlog [11:15:27.015] elapsed time [00:00:00:00.002]
DBGRPC:       krmxrpc: xc=4311199728 kpurpc2 rc=3123 db=target proc=DBMS_BACKUP_RESTORE.RESTOREBACKUPPIECE
DBGRPC:       krmxrpc: xc=4311199728 chid=C1 rpc count=78
DBGRPC:       krmxrpc: xc=4311199728 starting longrunning RPC #78 to target: DBMS_BACKUP_RESTORE.RESTOREBACKUPPIECE
DBGRPC:       krmxr: xc=4311199728 started long running rpc
DBGRPC:       krmxpoq: xc=4311199728, action="0000078 STARTED40", col_l=17, ind=0, sid=161
DBGRPC:       krmxr: callback returned TRUE, skipping sleep
DBGRPC:       krmxpoq: xc=4311199728, action="0000078 STARTED40", col_l=17, ind=0, sid=161
DBGRPC:       krmxr: sleeping for 1 seconds
DBGRPC:       krmxpoq: xc=4311199728, action="0000078 FINISHED40", col_l=18, ind=0, sid=161
DBGRPC:       krmxpod: xc=4311199728 has finished remote rpc #78
DBGRPC:       krmxr: xc=4311199728 chid=C1 calling peicnt
DBGRPC:       krmxrpc: xc=4311199728 kpurpc2 rc=19624 db=target proc=DBMS_BACKUP_RESTORE.RESTOREBACKUPPIECE
DBGRPC:       krmxrpc: xc=4311199728 kpurpc2 rc=0 db=target proc=DBMS_BACKUP_RESTORE.FETCHFILERESTORED
DBGRPC:       krmxrpc: xc=4311199728 chid=C1 rpc count(non-blocking)=79
DBGRPC:       krmxrpc: xc=4311199728 RPC #79 completed immediately
DBGMISC:      ENTERED krmzlog [11:15:28.022]
DBGMISC:       Message (len=0) before truncation=ORA-19624: 操作失敗, 如果可能請重試 [11:15:28.022] (krmzlog)
DBGMISC:      ORA-19870: 讀取備份段 /data/backup/tradedb/a7kmlkem_1_1 時出錯 (krmzlog)
DBGMISC:      ORA-19504: 無法建立檔案"+DATA/tradedb/tradedb_system_1_4g" (krmzlog)
DBGMISC:      ORA-17502: ksfdcre: 3 未能建立檔案 +DATA/tradedb/tradedb_system_1_4g (krmzlog)
DBGMISC:      ORA-15001: 磁碟組 "DATA" 不存在或尚未裝載 (krmzlog)
DBGMISC:      ORA-15001: diskgroup "DATA" does not exist or is not mounted (krmzlog)
DBGMISC:       Unwanted messages removed [11:15:28.022] (krmzlog)
RMAN-01005:

根據這裡顯示的資訊，問題一目瞭然了。ORA-15001：磁碟組DATA不存在或沒有裝載。由於前面碰到的一個錯誤，導致在ASM例項在建立DATA磁碟組的時候報錯，而且最終DATA磁碟組無法被刪除，只能處於OFFLINE狀態。這個錯誤的描述可以參考：建立ASM磁碟組報錯ORA-15063：http://yangtingkun.itpub.net/post/468/492945

正是由於這個錯誤，導致這個環境新建的磁碟組名稱為MEMBER，而在重用單例項恢復的指令碼的時候，沒有將DATA磁碟組名稱替換為MEMBER。正是這個低階的錯誤，引發了整個的錯誤。

不過RMAN的恢復操作沒有將真正有價值的錯誤資訊返回，而只是返回了一些言之不詳的錯誤資訊，也是導致這個錯誤很難被定位的一個原因。