Standby 庫, 根盤暴滿,虛驚一場 [zt]

突然收到備庫上以下錯誤，嚇了一大跳,hehe
根盤空間有3多個G呀，怎麼會突然暴滿?
哎，別想了，先解決問題要緊

Jul 28 16:20:13 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1895 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:20:28 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1897 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:20:56 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1899 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:21:15 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1901 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)
Jul 28 16:21:30 hpdb2 vmunix: vxfs: NOTICE: msgcnt 1903 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block ext
ent)

bdf確認了一下，根盤使用100%了.汗一個。

#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    3145728 3145728       0  100% /
/dev/vg00/lvol1     505392  129008  325840   28% /stand
/dev/vg00/lvol8    5242880 1534784 3679208   29% /var
/dev/vg00/lvol7    5242880 2020824 3196896   39% /usr
/dev/vg00/u01      14680064 8635816 5998560   59% /u01
/dev/vg00/lvol6    2097152 1050976 1038128   50% /tmp
/dev/vg00/lvol5    5996544 3900936 2079272   65% /opt
/dev/vg00/lvol4     524288   83336  437552   16% /home
/dev/vgdgehrdata/lvolvgdgehrdata
                   819200000 435723320 380480784   53% /dgehrdata
/dev/vgdgehrarch/lvolvgdgehrarch
                   256000000 165329936 89961952   65% /dgehrarch
/dev/vgdba01/lvoldba01
                   1331232768 8767432 1312133584    1% /dba01
/dev/vgdba02/lvoldba02
                   1331232768   61856 1320771152    0% /dba02
/dev/vgdgjbarch/lvolvgdgjbarch
                   363528192 112804792 248764768   31% /dgjbarch
/dev/vgdgjbdata/lvolvgdgjbdata
                   1873936384 1122876504 745192272   60% /dgjbdata

糊亂找了一通，終於發現以下目錄下多出了兩個的資料檔案.

#ll
total 5596240
-rw-r-----   1 oracle     dba        2097160192 Jul 28 09:24 rsm_idx80.dbf
-rw-r-----   1 oracle     dba        771751936 Jul 28 09:24 rsm_idx81.dbf

嘿嘿,發現了就趕快刪除吧,別急哦,不能刪除? 為什麼呢,請聽我慢慢道來.(這裡有點繞)

先說明介紹一下系統架構.

這個出問題的主機(簡稱Q主機),這個主機既是其他三臺主機的HA(MC/SG),又是這臺三臺主機上資料的standby(Oracle/DG).

上面各位看到的那個/jbindx02/oraindx/indx 目錄,其實是給HA/Clustering用的，一個沒有mount任何儲存的目錄。

而其 實是用得另外一套目錄，由於在一臺主庫上有一個存放索引的目錄/jbindx02/oraindx/indx 一直沒有存放資料檔案進去，所以就忘記了加在 db_file_name_convert中，今天加表空間時，看到其他目錄快滿了，就向這個目錄放資料檔案了，所以就造成了以上問題。

問題找到了，想辦法解決備庫問題吧.

解決過程如下:

1.先檢視一下alert.log

WARNING: File being created with same name as in Primary
Existing file may be overwritten
Mon Jul 28 09:24:10
File #546 added to control file as 'UNNAMED00546'.
Originally created as:
'/jbindx02/oraindx/indx/rsm_idx81.dbf'
Recovery was unable to create the file as:
'/jbindx02/oraindx/indx/rsm_idx81.dbf'
Errors with log /dgjbarch/dgjb_1_27376_638807858.log
MRP0: Background Media Recovery terminated with error 19502
Mon Jul 28 09:24:10 2008
Errors in file /u01/oracle/admin/hpjob/bdump/dghpjob_mrp0_21025.trc:
ORA-19502: write error on file "/jbindx02/oraindx/indx/rsm_idx81.dbf", blockno 93696 (blocksize=8192)
ORA-27072: File I/O error
HP-UX Error: 2: No such file or directory
Additional information: 4
Additional information: 93696
Additional information: 540672
Mon Jul 28 09:24:51 2008
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
Mon Jul 28 09:24:51 2008
Errors in file /u01/oracle/admin/hpjob/bdump/dghpjob_mrp0_21025.trc:
ORA-19502: write error on file "/jbindx02/oraindx/indx/rsm_idx81.dbf", blockno 93696 (blocksize=8192)
ORA-27072: File I/O error
HP-UX Error: 2: No such file or directory
Additional information: 4
Additional information: 93696
Additional information: 540672
Mon Jul 28 09:24:51 2008
MRP0: Background Media Recovery process shutdown (dghpjob)

2.先停止恢復
alter database recover managed standby database cancel ;

3.把需要轉用的目錄加進去
ALTER SYSTEM SET db_file_name_convert='xxx,xxx' scope=both;

4.關閉
shtudown immediate;

5.移動資料檔案
mv rsm_idx80.dbf /xxxx
mv rsm_idx81.dbf /xxxx

6.啟動資料庫到mount
ALTER SYSTEM SET standby_file_management='MANUAL' SCOPE=BOTH;
alter database rename file '/jbindx02/oraindx/indx/rsm_idx80.dbf' to '/dgjbdata/indx/rsm_idx80.dbf' ;
---成功

alter database rename file '/jbindx02/oraindx/indx/rsm_idx80.dbf' to '/dgjbdata/indx/rsm_idx80.dbf' ;

---失敗(錯誤資訊如下)

ORA-01111: name for data file 546 is unknown - rename to correct file
ORA-01110: data file 546: '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546'
ORA-01157: cannot identify/lock data file 546 - see DBWR trace file
ORA-01111: name for data file 546 is unknown - rename to correct file
ORA-01110: data file 546: '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546'

其實從上面看這個檔案的大小和alert.log中的資訊，已經知道了，這個檔案沒有建立成功。

7.對這個損壞的檔案進行恢復

SQL> select name from v$datafile where file# = 546;

NAME
--------------------------------------------------------------------------------
/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546

SQL> alter database create datafile '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546' as '/dgjbdata/indx/rsm_idx81.dbf';

Database altered.

SQL> select name from v$datafile where file# = 546;

NAME
--------------------------------------------------------------------------------
/dgjbdata/indx/rsm_idx81.dbf

-------過來了^_^

SQL>  alter system set standby_file_management=AUTO scope=both;

SQL> alter database create datafile '/u01/oracle/product/10.2.0.3/dbs/UNNAMED00546' as '/dgjbdata/indx/rsm_idx81.dbf';

Database altered.

8.切換到恢復狀態下

ALTER DATABASE RECOVER  managed standby database disconnect from session

9.完工了，回家鳥 ^_^

http://space.itpub.net/7364032/viewspace-414942