客戶的4節點RAC環境，其中一個節點例項出現故障，發現無法正常啟動。

檢查CLUSTER和告警日誌資訊，發現節點1心跳超時，被踢出叢集。伺服器重新啟動後，資料庫例項沒有自動啟動。

告警日誌資訊為：

Mon Apr 16 03:42:39 2012
Thread 1 advanced to log sequence 22348 (LGWR switch)
Current log# 16 seq# 22348 mem# 0: +DATA/orcl/onlinelog/group_16.291.766326571
Current log# 16 seq# 22348 mem# 1: +DATA/orcl/onlinelog/group_16.293.766330969
Mon Apr 16 15:02:58 2012
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.0.0.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 192.168.1.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
processes = 1500
sessions = 1655
sga_max_size = 19830669312
pre_page_sga = FALSE
lock_sga = FALSE
__shared_pool_size = 3674210304
__large_pool_size = 16777216
__java_pool_size = 16777216
__streams_pool_size = 33554432
spfile = +DATA/orcl/spfileorcl.ora
sga_target = 19830669312
control_files = +DATA/orcl/controlfile/current.274.720740395
db_block_size = 8192
__db_cache_size = 16072572928
compatible = 10.2.0.3.0
log_archive_dest_1 = LOCATION=+DATA/
log_archive_format = %t_%s_%r.dbf
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 4
db_create_file_dest = +DATA
_gc_affinity_time = 0
_gc_affinity_limit = 10000000
_gc_affinity_minimum = 10000000
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
dispatchers = (PROTOCOL=TCP) (SERVICE=orclXDB)
local_listener = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.1.21)(PORT = 1521))
remote_listener = LISTENERS_ORCL
job_queue_processes = 20
cursor_sharing = FORCE
background_dump_dest = /u01/app/oracle/admin/orcl/bdump
user_dump_dest = /u01/app/oracle/admin/orcl/udump
core_dump_dest = /u01/app/oracle/admin/orcl/cdump
audit_file_dest = /u01/app/oracle/admin/orcl/adump
db_name = orcl
open_cursors = 1000
pga_aggregate_target = 5872025600
Cluster communication is configured to use the following interface(s) for this instance
10.0.0.11
Mon Apr 16 15:02:59 2012
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=20588
DIAG started with pid=3, OS id=20590
PSP0 started with pid=4, OS id=20592
LMON started with pid=5, OS id=20594
LMD0 started with pid=6, OS id=20596
LMS0 started with pid=7, OS id=20603
LMS1 started with pid=8, OS id=20607
LMS2 started with pid=9, OS id=20611
LMS3 started with pid=10, OS id=20615
MMAN started with pid=11, OS id=20619
DBW0 started with pid=12, OS id=20621
DBW1 started with pid=13, OS id=20623
LGWR started with pid=14, OS id=20625
CKPT started with pid=15, OS id=20627
SMON started with pid=16, OS id=20629
RECO started with pid=17, OS id=20631
CJQ0 started with pid=18, OS id=20633
MMON started with pid=19, OS id=20635
Mon Apr 16 15:03:00 2012
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=20, OS id=20637
Mon Apr 16 15:03:00 2012
starting up 1 shared server(s) ...
Mon Apr 16 15:03:02 2012
lmon registered with NM - instance id 1 (internal mem no 0)
Mon Apr 16 15:03:05 2012
Reconfiguration started (old inc 0, new inc 30)
List of nodes:
0 1 2 3
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid according to instance 3
* domain 0 valid = 1 according to instance 1
Mon Apr 16 15:03:05 2012
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Apr 16 15:03:05 2012
LMS 0: 0 GCS shadows cancelled, 0 closed
Mon Apr 16 15:03:05 2012
LMS 1: 0 GCS shadows cancelled, 0 closed
Mon Apr 16 15:03:05 2012
LMS 2: 0 GCS shadows cancelled, 0 closed
Mon Apr 16 15:03:05 2012
LMS 3: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Mon Apr 16 15:03:07 2012
LMS 2: 0 GCS shadows traversed, 0 replayed
Mon Apr 16 15:03:07 2012
LMS 1: 0 GCS shadows traversed, 0 replayed
Mon Apr 16 15:03:07 2012
LMS 3: 0 GCS shadows traversed, 0 replayed
Mon Apr 16 15:03:07 2012
LMS 0: 0 GCS shadows traversed, 0 replayed
Mon Apr 16 15:03:07 2012
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=23, OS id=20699
Mon Apr 16 15:03:12 2012
ALTER DATABASE MOUNT
Mon Apr 16 15:03:12 2012
Starting background process ASMB
ASMB started with pid=25, OS id=20710
Starting background process RBAL
RBAL started with pid=26, OS id=20714
Mon Apr 16 15:03:17 2012
SUCCESS: diskgroup DATA was mounted
Mon Apr 16 15:03:21 2012
Setting recovery target incarnation to 2
Mon Apr 16 15:03:21 2012
SUCCESS: diskgroup DATA was dismounted
Mon Apr 16 15:03:21 2012
ORA-1105 signalled during: ALTER DATABASE MOUNT...

這個ORA-1105錯誤只是說明當前例項的某些引數設定和RAC其他例項設定的不符，並不能說明導致錯誤的真正原因。

為了找到問題只有手工啟動例項：

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 10.2.0.4.0 - Production on 星期一 4月 16 17:04:54 2012

Connected to an idle instance.

SQL> STARTUP MOUNT
ORACLE instance started.

Total System Global Area 1.9831E+10 bytes
Fixed Size 2119216 bytes
Variable Size 3741321680 bytes
Database Buffers 1.6073E+10 bytes
Redo Buffers 14655488 bytes
ORA-01105: mount is incompatible with mounts by other instances
ORA-01606: gc_files_to_locks not identical to that of another mounted instance

透過手工執行，可以瞭解具體導致錯誤產生的原因。不過gc_files_to_locks並沒有設定為不同的值：

SQL> show parameter gc_files_to_locks

NAME TYPE VALUE
------------------- ------------------- -----------------
gc_files_to_locks string
SQL> select sid, name, value from v$spparameter where name = 'gc_files_to_locks';

SID NAME VALUE
---------- ------------------------------ --------------------------------
* gc_files_to_locks

不過導致問題產生的確實與GC設定有關，問題並非是gc_files_to_locks引數導致，而是SPFILE中設定的_gc_affinity_time引數。這個引數是靜態引數，只有重啟後才能生效，而在SPFILE中設定後，會導致重啟的例項1生效了該引數，因此和沒有重啟過的其他例項產生了不相容。

解決方法有兩個，一個是重啟所有的節點，另外一個是去掉SPFILE中這個引數的設定：

SQL> alter system reset "_gc_affinity_time" scope = spfile sid = '*';

System altered.

SQL> shutdown immediate
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 1.9831E+10 bytes
Fixed Size 2119216 bytes
Variable Size 3741321680 bytes
Database Buffers 1.6073E+10 bytes
Redo Buffers 14655488 bytes
Database mounted.
SQL> alter database open;

Database altered.

RAC環境啟動單例項報錯ORA-1105

相關文章