ORACLE RAC 裸裝置資料庫一節點表空間擴容錯誤新增資料檔案到本地的處理

    下面是針對這一問題做的故障模擬及其處理方法。
    處理這個問題有2種思路：
   1、透過備份或匯入匯出將問題表空間的資料從錯誤新增資料檔案的節點匯出，重建表空間，然後將資料重新匯入；這種方法的缺點是，如果表空間資料量大，將會耗空間、消耗時間。
   2、使用rman工具的copy命令，將資料檔案複製到共享儲存的裸裝置中，然後進行恢復裸裝置資料檔案，這樣速度比較快、比較省空間。
    建立業務模擬使用者test，並設定其預設表空間是users表空間，然後建立業務模擬表，插入資料。
    rawrac1建立測試資料：
[oracle@rawrac1 ~]$ sqlplus test/test
SQL*Plus: Release 10.2.0.1.0 - Production on Wed Sep 2 18:39:28 2015
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
SQL> select * from tab;
TNAME       TABTYPE CLUSTERID
------------------------------ ------- ----------
TEST       TABLE
SQL> select * from test;
ID
----------
1
2
3
SQL> select file_name from dba_data_files;
FILE_NAME
--------------------------------------------------------------------------------
/dev/raw/raw14
/dev/raw/raw12
/dev/raw/raw10
/dev/raw/raw13
/dev/raw/raw17
/dev/raw/raw11

6 rows selected.
SQL> !ls -l /dev/raw
total 0
crw-rw---- 1 oracle oinstall 162,  1 Sep  2 18:31 raw1
crw-rw---- 1 oracle oinstall 162, 10 Sep  2 18:36 raw10
crw-rw---- 1 oracle oinstall 162, 11 Sep  2 18:25 raw11
crw-rw---- 1 oracle oinstall 162, 12 Sep  2 18:40 raw12
crw-rw---- 1 oracle oinstall 162, 13 Sep  2 18:40 raw13
crw-rw---- 1 oracle oinstall 162, 14 Sep  2 18:25 raw14
crw-rw---- 1 oracle oinstall 162, 15 Sep  2 18:24 raw15
crw-rw---- 1 oracle oinstall 162, 16 Sep  2 18:24 raw16
crw-rw---- 1 oracle oinstall 162, 17 Sep  2 18:24 raw17
crw-rw---- 1 oracle oinstall 162, 18 Sep  2 18:24 raw18
crw-rw---- 1 oracle oinstall 162,  2 Sep  2 18:40 raw2
crw-rw---- 1 oracle oinstall 162,  3 Sep  2 18:25 raw3
crw-rw---- 1 oracle oinstall 162,  4 Sep  2 18:40 raw4
crw-rw---- 1 oracle oinstall 162,  5 Sep  2 18:40 raw5
crw-rw---- 1 oracle oinstall 162,  6 Sep  2 18:40 raw6
crw-rw---- 1 oracle oinstall 162,  7 Sep  2 18:25 raw7
crw-rw---- 1 oracle oinstall 162,  8 Sep  2 18:25 raw8
crw-rw---- 1 oracle oinstall 162,  9 Sep  2 18:40 raw9
    rawrac2驗證資料
[oracle@rawrac2 ~]$ sqlplus test/test
SQL*Plus: Release 10.2.0.1.0 - Production on Wed Sep 2 18:39:47 2015
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
TNAME       TABTYPE CLUSTERID
------------------------------ ------- ----------
TEST       TABLE
SQL> select * from test;
ID
----------
1
2
3
    rawrac1模擬故障，對users表空間擴容，新增資料檔案到本地目錄
SQL> alter tablespace users add datafile '/opt/oracle/oradata/rawrac/raw18' size 500M;
Tablespace altered.
    rawrac2測試寫入，這一點跟故障現象不一致（原因可能是測試插入的資料量太小，沒有使用到剛新增的新資料檔案）可以寫入，但是dba_data_files檢視無法檢視：
SQL> insert into test values(4);
1 row created.
SQL> commit;
Commit complete.
SQL> select file_name from dba_data_files;
ERROR:
ORA-01157: cannot identify/lock data file 7 - see DBWR trace file
ORA-01110: data file 7: '/opt/oracle/oradata/rawrac/raw18'
no rows selected
    rawrac1檢視rawrac2插入並提交的資料：
SQL> select * from test;
ID
----------
1
2
3
4
SQL> select file_name from dba_data_files;
FILE_NAME
--------------------------------------------------------------------------------
/dev/raw/raw14
/dev/raw/raw12
/dev/raw/raw10
/dev/raw/raw13
/dev/raw/raw17
/dev/raw/raw11
/opt/oracle/oradata/rawrac/raw18
7 rows selected.
    故障處理要點，準備比新增的資料檔案提交要大的裸裝置，如果裸裝置比新增的檔案系統檔案小，當使用rman進行copy時會報錯如下：
[oracle@rawrac1 ~]$ rman target /
Recovery Manager: Release 10.2.0.1.0 - Production on Wed Sep 2 18:52:25 2015
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
connected to target database: RAWRAC (DBID=1916399231)
RMAN> copy datafile '/opt/oracle/oradata/rawrac/raw18' to '/dev/raw/raw18';
Starting backup at 02-SEP-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=152 instance=rawrac1 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00007 name=/opt/oracle/oradata/rawrac/raw18
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 09/02/2015 18:52:42
ORA-19504: failed to create file "/dev/raw/raw18"
ORA-27042: not enough space on raw partition to fullfill request
Additional information: 3
RMAN> quit
Recovery Manager complete.
    如果故障發現的比較早，可以在資料檔案新增的節點rawrac1使用resize命令將資料檔案體積縮小：
SQL> alter database datafile '/opt/oracle/oradata/rawrac/raw18' resize 300M;
Database altered.
    故障處理第一步、rawrac1將問題資料檔案下線
SQL> alter database datafile '/opt/oracle/oradata/rawrac/raw18'  offline;
Database altered.
    故障處理第二步、rawrac1節點使用rman進行資料檔案複製到裸裝置
[oracle@rawrac1 ~]$ rman target /
Recovery Manager: Release 10.2.0.1.0 - Production on Wed Sep 2 18:58:58 2015
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
connected to target database: RAWRAC (DBID=1916399231)
RMAN> copy datafile '/opt/oracle/oradata/rawrac/raw18' to '/dev/raw/raw18';
Starting backup at 02-SEP-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=133 instance=rawrac1 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00007 name=/opt/oracle/oradata/rawrac/raw18
output filename=/dev/raw/raw18 tag=TAG20150902T185912 recid=2 stamp=889383568
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
Finished backup at 02-SEP-15
RMAN> quit
Recovery Manager complete.
    故障處理第三步、rawrac1將問題資料檔案重定向到裸裝置
[oracle@rawrac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.1.0 - Production on Wed Sep 2 19:06:07 2015
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
SQL> alter database rename file '/opt/oracle/oradata/rawrac/raw18' to '/dev/raw/raw18';
Database altered.
    故障處理第四步、rawrac1對裸裝置檔案進行恢復（此時如果是生產環境，可能需要rawrac2節點的歸檔檔案，如有需要則需要將rawrac2節點相關的歸檔檔案上傳到rawrac1歸檔目錄即可）
SQL> recover datafile '/dev/raw/raw18';
Media recovery complete.
    故障處理第五步、rawrac1將恢復完畢的裸裝置檔案上線
SQL> alter database datafile '/dev/raw/raw18' online;
Database altered.
SQL> select file_name from dba_data_files;
FILE_NAME
--------------------------------------------------------------------------------
/dev/raw/raw14
/dev/raw/raw12
/dev/raw/raw10
/dev/raw/raw13
/dev/raw/raw17
/dev/raw/raw11
/dev/raw/raw18
7 rows selected.
    rawrac1節點處理完故障後，rawrac2節點的dba_data_files檢視就可以正常檢視了：
SQL> select file_name from dba_data_files;
FILE_NAME
--------------------------------------------------------------------------------
/dev/raw/raw14
/dev/raw/raw12
/dev/raw/raw10
/dev/raw/raw13
/dev/raw/raw17
/dev/raw/raw11
/dev/raw/raw18
7 rows selected.