10G RAC 增加資料檔案需要注意下許可權

故障經過:
          2012-03-20,17:00 下班後,客戶電話通知,應用出現異常,報如下錯誤,應用不能插入資料.
           java.io.IOException: ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
          ORA-01110: data file 336: '/dev/rdb04_0036'
處理經過:
          剛到家,穿拖鞋準備吃飯,沒辦法,來事了,趕緊開啟電腦,連上vpn看下吧,先看下aler日誌
    節點1:              
    Tue Mar 20 16:52:11 2012
    Thread 1 advanced to log sequence 35890 (LGWR switch)
     Current log# 3 seq# 35890 mem# 0: /dev/redo_1_3
    Tue Mar 20 16:58:52 2012
    alter tablespace TBS_IOM_PART03 add datafile '/dev/rdb04_0036'  size 8191m
    Tue Mar 20 16:59:24 2012
    Completed: alter tablespace TBS_IOM_PART03 add datafile '/dev/rdb04_0036'  size 8191m
    Tue Mar 20 17:05:34 2012
    Thread 1 advanced to log sequence 35891 (LGWR switch)
     Current log# 2 seq# 35891 mem# 0: /dev/redo_1_2
    節點2:
    Tue Mar 20 16:50:52 2012
    Thread 2 advanced to log sequence 21668 (LGWR switch)
    Current log# 5 seq# 21668 mem# 0: /dev/redo_2_2
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
    ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
    ORA-01110: data file 336: '/dev/rdb04_0036'
    ORA-27041: unable to open file
   IBM AIX RISC System/6000 Error: 13: Permission denied
   Additional information: 11
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
   ORA-01186: file 336 failed verification tests
   ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
   ORA-01110: data file 336: '/dev/rdb04_0036'
   Tue Mar 20 16:59:24 2012
   File 336 not verified due to error ORA-01157
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
   ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
   ORA-01110: data file 336: '/dev/rdb04_0036'
   ORA-27041: unable to open file
   IBM AIX RISC System/6000 Error: 13: Permission denied
   Additional information: 11
   Tue Mar 20 16:59:24 2012
   Errors in file /oracle/admin/fwkt/bdump/fwkt2_dbw0_925932.trc:
   ORA-01186: file 336 failed verification tests
   ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
   ORA-01110: data file 336: '/dev/rdb04_0036'
   File 336 not verified due to error ORA-01157
   Tue Mar 20 17:13:26 2012
   Thread 2 advanced to log sequence 21669 (LGWR switch)
   Current log# 6 seq# 21669 mem# 0: /dev/redo_2_3
   Tue Mar 20 17:26:28 2012
結論: 
    應該是有人在節點1上增加了個資料檔案,'/dev/rdb04_0036' ,但是這個Raw Device在節點2上許可權應該是不對的,
    去節 點2上看下
    # ls -tlr /dev/rdb04_0036
     crw-rw----   1 root     system       48, 36 Mar 20 16:41 /dev/rdb04_0036
   我暈, 果然沒有修改許可權,趕緊修改吧,chown oracle:dba  /dev/rdb04_0036
    修改之後,不要以為就完事了,這個時候,在節點2上當執行如下select count(1) from dba_data_files 還是報同樣的
    錯誤.
    SQL> select count(1) from dba_data_files;
    select count(1) from dba_data_files
     ERROR at line 1:
    ORA-01157: cannot identify/lock data file 336 - see DBWR trace file
    ORA-01110: data file 336: '/dev/rdb04_0036'
     修改完許可權之後,還要加一步,'alter system check datafiles',這才算完事.繼續吃飯了,暈,飯都涼了.