伺服器時間異常造成ORA-00600 [2252]的故障處理

dbasdk發表於2017-04-19

資料庫伺服器重啟之後,檢查Oracle叢集資源發現四個例項啟動了兩個,還有兩個沒有啟動(wu,rl)

[grid@db1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.CWDATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.DADATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.DATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.LISTENER.lsnr
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.OCR.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.KDATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.asm
               ONLINE  ONLINE       db1                      Started             
               ONLINE  ONLINE       db2                      Started             
ora.gsd
               OFFLINE OFFLINE      db1                                          
               OFFLINE OFFLINE      db2                                          
ora.net1.network
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.ons
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       db1                                          
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       db2                                          
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       db2                                          
ora.wu.db
      1        ONLINE  OFFLINE                               Instance Shutdown   
      2        ONLINE  ONLINE       db2                      Open                
ora.dyl.db
      1        ONLINE  OFFLINE                                                   
      2        ONLINE  ONLINE       db2                      Open                
ora.cvu
      1        ONLINE  ONLINE       db2                                          
ora.da.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.db1.vip
      1        ONLINE  ONLINE       db1                                          
ora.db2.vip
      1        ONLINE  ONLINE       db2                                          
ora.oc4j
      1        ONLINE  ONLINE       db2                                          
ora.rl.db
      1        ONLINE  OFFLINE                               Instance Shutdown   
      2        ONLINE  ONLINE       db2                      Open                
ora.scan1.vip
      1        ONLINE  ONLINE       db1                                          
ora.scan2.vip
      1        ONLINE  ONLINE       db2                                          
ora.scan3.vip
      1        ONLINE  ONLINE       db2                               

檢視rlzy例項的alert.log檔案,可以看到以下錯誤資訊“ORA-00600: internal error code, arguments: [2252], [3418], [573259345], [1594], [50675712]”,關於這個ORA-00600 2252在MOS上有相關bug描述,但我這並不是bug引起的,檢視資訊時我們也是需要關注時間的,這裡時間顯示為2001年1月1號了,與當前時間相差了16年2個多月。

Picked broadcast on commit scheme to generate SCNs
Mon Jan 01 08:23:50 2001
Errors in file /u01/app/oracle/diag/rdbms/rl/RL1/trace/RL1_dbw0_19789.trc  (incident=544328):
ORA-00600: internal error code, arguments: [2252], [3418], [573259345], [1594], [50675712], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/rl/RL1/incident/incdir_544216/RL1_diag_19753_i544216.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/rl/RL1/trace/RL1_dbw0_19789.trc:
ORA-01186: file 2 failed verification tests
ORA-00600: internal error code, arguments: [2252], [3418], [573259345], [1594], [50675712], [], [], [], [], [], [], []
DBW0 (ospid: 19789): terminating the instance due to error 1186
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jan 01 08:23:56 2001
ORA-1092 : opitsk aborting process
Mon Jan 01 08:23:56 2001

檢視系統當前時間,居然變成了2001年1月1號,而且奇怪的是並不是重啟時時間就被修改了,因為有兩個例項在重啟之後正常啟動了

[root@db1 ~]# date
Mon Jan  1 08:25:34 CST 2001

手工更新為當前時間後並手動啟動例項(caiwu,rlzy),就能正常啟動

[root@db1 ~]# date
Fri Mar 24 11:26:44 CST 2017

[grid@db1 ~]$ srvctl start database -d caiwu

[grid@db1 ~]$ srvctl start database -d rlzy

[grid@db1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.CWDATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.DADATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.DATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.LISTENER.lsnr
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.OCR.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.KDATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.asm
               ONLINE  ONLINE       db1                      Started             
               ONLINE  ONLINE       db2                      Started             
ora.gsd
               OFFLINE OFFLINE      db1                                          
               OFFLINE OFFLINE      db2                                          
ora.net1.network
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.ons
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       db1                                          
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       db2                                          
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       db2                                          
ora.wu.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.dyl.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.cvu
      1        ONLINE  ONLINE       db2                                          
ora.da.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.db1.vip
      1        ONLINE  ONLINE       db1                                          
ora.db2.vip
      1        ONLINE  ONLINE       db2                                          
ora.oc4j
      1        ONLINE  ONLINE       db2                                          
ora.rl.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.scan1.vip
      1        ONLINE  ONLINE       db1                                          
ora.scan2.vip
      1        ONLINE  ONLINE       db2                                          
ora.scan3.vip
      1        ONLINE  ONLINE       db2       

問題雖然解決了,但是為什麼伺服器會在啟動兩個例項後,系統時間發生改變還是不得而知,需要找浪潮伺服器的工程師來檢查。

來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/29734436/viewspace-2137625/,如需轉載,請註明出處,否則將追究法律責任。

相關文章