使用nfs作為歸檔目錄的問題

環境9iRAC，兩個節點
我的archive_dest有兩個，所以在archive01目錄應該可以看到兩個執行緒生產的歸檔

SQL> show parameter archive

NAME                                 TYPE
------------------------------------ --------------------------------
VALUE
------------------------------
archive_lag_target                   integer
0
log_archive_dest                     string

log_archive_dest_1                   string
LOCATION=/opt/oracle/oradata/ylerp/archive01
log_archive_dest_10                  string

NAME                                 TYPE
------------------------------------ --------------------------------
VALUE
------------------------------
log_archive_dest_2                   string
LOCATION=/opt/oracle/oradata/ylerp/archive02

使用了NFS，在節點1 archive02為nfs
> df   -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3     406G  294G  113G  73% /
tmpfs                 2.0G  4.0K  2.0G   1% /dev/shm
/dev/cciss/c0d0p1     118M   42M   76M  36% /boot
10.98.132.2:/opt/oracle/oradata/ylerp/archive02
                      400G  182G  198G  48% /opt/oracle/oradata/ylerp/archive02

但   rman的list archivelog all        檢視到執行緒2的archive  只有一個
821762  1    160778  A 11-DEC-14 /opt/oracle/oradata/ylerp/archive01/1_160778.dbf
821763  1    160778  A 11-DEC-14 /opt/oracle/oradata/ylerp/archive02/1_160778.dbf
821768  1    160779  A 11-DEC-14 /opt/oracle/oradata/ylerp/archive01/1_160779.dbf
821769  1    160779  A 11-DEC-14 /opt/oracle/oradata/ylerp/archive02/1_160779.dbf
821772  1    160780  A 11-DEC-14 /opt/oracle/oradata/ylerp/archive01/1_160780.dbf
821773  1    160780  A 11-DEC-14 /opt/oracle/oradata/ylerp/archive02/1_160780.dbf
821680  2    134757  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134757.dbf
821683  2    134758  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134758.dbf
821686  2    134759  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134759.dbf
821687  2    134760  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134760.dbf
821692  2    134761  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134761.dbf
821693  2    134762  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134762.dbf
821696  2    134763  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive01/2_134763.dbf
821697  2    134763  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134763.dbf
821700  2    134764  A 10-DEC-14 /opt/oracle/oradata/ylerp/archive02/2_134764.dbf

但在檔案系統可以看到在目錄archive01存線上程2的archive
-rw-r-----  1 oracle oinstall 20710400 2014-12-09 04:31 2_134726.dbf
-rw-r-----  1 oracle oinstall 20966912 2014-12-09 05:06 2_134732.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-09 09:20 2_134736.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-09 10:16 2_134737.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-09 11:18 2_134738.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-09 12:44 2_134739.dbf
-rw-r-----  1 oracle oinstall 20967424 2014-12-09 14:30 2_134740.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-09 15:34 2_134741.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-09 17:00 2_134742.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-09 19:17 2_134743.dbf
-rw-r-----  1 oracle oinstall 20962816 2014-12-10 00:58 2_134744.dbf
-rw-r-----  1 oracle oinstall 20963328 2014-12-10 00:58 2_134745.dbf
-rw-r-----  1 oracle oinstall 20969984 2014-12-10 02:43 2_134757.dbf
-rw-r-----  1 oracle oinstall  5375488 2014-12-10 04:09 2_134758.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-10 05:11 2_134764.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-10 09:11 2_134768.dbf
-rw-r-----  1 oracle oinstall 20970496 2014-12-10 10:12 2_134769.dbf
-rw-r-----  1 oracle oinstall 20969984 2014-12-10 11:07 2_134770.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-10 12:40 2_134771.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-10 14:31 2_134772.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-10 15:34 2_134773.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-10 16:58 2_134774.dbf
-rw-r-----  1 oracle oinstall 20967936 2014-12-10 19:13 2_134775.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-11 00:59 2_134776.dbf
-rw-r-----  1 oracle oinstall 20970496 2014-12-11 01:00 2_134777.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-11 01:06 2_134788.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-11 01:58 2_134789.dbf
-rw-r-----  1 oracle oinstall 19174400 2014-12-11 04:28 2_134791.dbf
-rw-r-----  1 oracle oinstall 20968960 2014-12-11 05:16 2_134797.dbf
-rw-r-----  1 oracle oinstall 20970496 2014-12-11 09:19 2_134801.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-11 10:22 2_134802.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-11 11:20 2_134803.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-11 12:56 2_134804.dbf
-rw-r-----  1 oracle oinstall 20971008 2014-12-11 14:37 2_134805.dbf
> cd ..
> cd archive02
> ll
total 793664
-rw-r-----  1 oracle oinstall   471552 2014-12-10 02:44 1_160757.dbf
-rw-r-----  1 oracle oinstall   286208 2014-12-10 04:11 1_160758.dbf
-rw-r-----  1 oracle oinstall     1024 2014-12-10 04:11 1_160759.dbf
-rw-r-----  1 oracle oinstall     2560 2014-12-10 04:11 1_160760.dbf
-rw-r-----  1 oracle oinstall     1536 2014-12-10 04:11 1_160761.dbf
-rw-r-----  1 oracle oinstall     2048 2014-12-10 04:11 1_160762.dbf
-rw-r-----  1 oracle oinstall     3584 2014-12-10 04:11 1_160763.dbf
-rw-r-----  1 oracle oinstall   323584 2014-12-10 05:14 1_160764.dbf
-rw-r-----  1 oracle oinstall  1245696 2014-12-10 09:13 1_160765.dbf
-rw-r-----  1 oracle oinstall  1090560 2014-12-10 12:42 1_160766.dbf
-rw-r-----  1 oracle oinstall  1341440 2014-12-10 17:00 1_160767.dbf
-rw-r-----  1 oracle oinstall  2439680 2014-12-11 01:02 1_160768.dbf
-rw-r-----  1 oracle oinstall     3584 2014-12-11 01:02 1_160769.dbf
-rw-r-----  1 oracle oinstall     5632 2014-12-11 01:04 1_160770.dbf
-rw-r-----  1 oracle oinstall     6144 2014-12-11 01:04 1_160771.dbf
-rw-r-----  1 oracle oinstall   291840 2014-12-11 02:00 1_160772.dbf
-rw-r-----  1 oracle oinstall   448000 2014-12-11 04:30 1_160773.dbf
-rw-r-----  1 oracle oinstall     1536 2014-12-11 04:30 1_160774.dbf
-rw-r-----  1 oracle oinstall     2560 2014-12-11 04:30 1_160775.dbf
-rw-r-----  1 oracle oinstall     1536 2014-12-11 04:31 1_160776.dbf
-rw-r-----  1 oracle oinstall     1536 2014-12-11 04:31 1_160777.dbf
-rw-r-----  1 oracle oinstall     2560 2014-12-11 04:31 1_160778.dbf
-rw-r-----  1 oracle oinstall   246784 2014-12-11 05:18 1_160779.dbf
-rw-r-----  1 oracle oinstall  1587712 2014-12-11 10:24 1_160780.dbf
-rw-r-----  1 oracle oinstall  1326080 2014-12-11 14:39 1_160781.dbf
-rw-r-----  1 oracle oinstall 20969984 2014-12-10 02:45 2_134757.dbf
-rw-r-----  1 oracle oinstall  5375488 2014-12-10 04:11 2_134758.dbf
-rw-r-----  1 oracle oinstall     1536 2014-12-10 04:11 2_134759.dbf
-rw-r-----  1 oracle oinstall     2048 2014-12-10 04:11 2_134760.dbf
-rw-r-----  1 oracle oinstall     4608 2014-12-10 04:11 2_134761.dbf

這樣的話RMAN不會自動清除目錄archive01中的檔案
於是檢視節點2的alert.log
看到如下錯誤資訊
Errors in file /opt/oracle/admin/ylerp/bdump/ylerp2_arc1_29574.trc:
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134804.dbf"
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134804.dbf"
ORA-27086: skgfglk: unable to lock file - already in use
Linux Error: 5: Input/output error
Additional information: 8
ARC1: Completed archiving  log 2 thread 2 sequence 134804
Thu Dec 11 14:39:50 2014
Thread 2 advanced to log sequence 134806
  Current log# 6 seq# 134806 mem# 0: /opt/oracle/oradata/ylerp/redo06.log
Thu Dec 11 14:39:50 2014
ARC0: Evaluating archive   log 4 thread 2 sequence 134805
ARC0: Beginning to archive log 4 thread 2 sequence 134805
Creating archive destination LOG_ARCHIVE_DEST_2: '/opt/oracle/oradata/ylerp/archive02/2_134805.dbf'
Creating archive destination LOG_ARCHIVE_DEST_1: '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
ARC0: Error 19504 Creating archive log file to '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
Thu Dec 11 14:39:50 2014
Errors in file /opt/oracle/admin/ylerp/bdump/ylerp2_arc0_29572.trc:
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-27086: skgfglk: unable to lock file - already in use
Linux Error: 5: Input/output error      
由於目錄archive01是節點1的本地目錄，目錄archive02是節點2的本地目錄，分別使用nfs載入到節點2和節點1
聯想到早幾天節點1由於報ORA錯誤資料庫自動down掉，應該是nfs出了問題

ndb2:~ # umount /opt/oracle/oradata/ylerp/archive01/

ndb2:~ # mount -t nfs 10.98.32.1:/opt/oracle/oradata/ylerp/archive01    /opt/oracle/oradata/ylerp/archive01/
mount server reported tcp not available, falling back to udp
mount: RPC: Remote system error - Connection refused

 
 
 ndb1:~ #  /usr/sbin/rpcinfo -p localhost | grep nfs
rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused

ndb1:/etc/rc.d # ./nfsserver start
Starting kernel based NFS servermount: nfsd already mounted or /proc/fs/nfsd busy
mount: according to mtab, nfsd is already mounted on /proc/fs/nfsd
                                                                                                                                                                      failed
ndb1:/etc/rc.d # ./nfsserver stop
Shutting down kernel based NFS server                                                                                                                                 failed
ndb1:/etc/rc.d # ./nfsserver start
Starting kernel based NFS server

最後沒辦法，只能重啟了節點1的主機，在節點2重新mount,再檢視alert.log，發現歸檔正常
Creating archive destination LOG_ARCHIVE_DEST_2: '/opt/oracle/oradata/ylerp/archive02/2_134805.dbf'
Creating archive destination LOG_ARCHIVE_DEST_1: '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
ARC0: Error 19504 Creating archive log file to '/opt/oracle/oradata/ylerp/archive01/2_134805.dbf'
Thu Dec 11 14:39:50 2014
Errors in file /opt/oracle/admin/ylerp/bdump/ylerp2_arc0_29572.trc:
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-19504: failed to create file "/opt/oracle/oradata/ylerp/archive01/2_134805.dbf"
ORA-27086: skgfglk: unable to lock file - already in use
Linux Error: 5: Input/output error
Additional information: 8
ARC0: Completed archiving  log 4 thread 2 sequence 134805
Thu Dec 11 15:44:13 2014
Reconfiguration started (old inc 7, new inc 8)
List of nodes:
 1
 Global Resource Directory frozen
one node partition
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 Resources and enqueues cleaned out
 Resources remastered 6439
 63268 GCS shadows traversed, 0 cancelled, 4842 closed
 29990 GCS resources traversed, 0 cancelled
 38614 GCS resources on freelist, 68604 on array, 68604 allocated
 set master node info
 Submitted all remote-enqueue requests
 Update rdomain variables
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 63268 GCS shadows traversed, 0 replayed, 4842 unopened
 Submitted all GCS remote-cache requests
 0 write requests issued in 58426 GCS resources
 1 PIs marked suspect, 0 flush PI msgs
Thu Dec 11 15:44:14 2014
Reconfiguration complete
 Post SMON to start 1st pass IR
Thu Dec 11 15:44:14 2014
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Thu Dec 11 15:49:08 2014
Thread 2 advanced to log sequence 134807
  Current log# 2 seq# 134807 mem# 0: /opt/oracle/oradata/ylerp/redo02.log
Thu Dec 11 15:49:08 2014
ARC1: Evaluating archive   log 6 thread 2 sequence 134806
ARC1: Beginning to archive log 6 thread 2 sequence 134806
Creating archive destination LOG_ARCHIVE_DEST_2: '/opt/oracle/oradata/ylerp/archive02/2_134806.dbf'
Creating archive destination LOG_ARCHIVE_DEST_1: '/opt/oracle/oradata/ylerp/archive01/2_134806.dbf'
ARC1: Completed archiving  log 6 thread 2 sequence 134806
Thu Dec 11 16:56:07 2014
Reconfiguration started (old inc 8, new inc 9)
List of nodes:
 0 1
 Global Resource Directory frozen
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps