模擬一則ORA-600 [4194][][]故障並處理

AlfredZhao發表於2019-05-17

環境：OEL 5.7 + Oracle 11.2.0.3

1.模擬ORA-600 [4194][][]故障

本次實驗環境模擬的是相關的undo/rollback block在system rollback segment上的場景：

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 6670
Session ID: 26 Serial number: 37

注意：

1.這種場景無法通過使用隱藏引數（_offline_rollback_segments和_corrupted_rollback_segments）來繞過問題;
2.我這裡實驗驗證是通過bbed修改system檔案128號塊的ktuxc.ktuxcfbp[0].ktufbuba.kubarec來模擬4194故障;
3.資料庫版本不一樣，所在塊號可能不同，比如9i/10g就都是9號塊;
4.這個實驗我在11.2.0.3這個版本的測試，4194後面引數是沒有任何值的，而在10.2.0.4測試模擬同樣故障後面是有引數值的;
5.請一定不要在任何生產環境嘗試做此類操作。

查詢system的回滾段頭所在塊：

select HEADER_FILE, HEADER_BLOCK, BLOCKS, EXTENTS from dba_segments where segment_name = 'SYSTEM' and SEGMENT_TYPE = 'ROLLBACK';

檢視資料庫的alert日誌：

Fri May 17 04:59:13 2019
SMON: enabling cache recovery
Errors in file /opt/app/oracle/diag/rdbms/test/test/trace/test_ora_6670.trc  (incident=4969):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/app/oracle/diag/rdbms/test/test/incident/incdir_4969/test_ora_6670_i4969.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri May 17 04:59:17 2019
Dumping diagnostic data in directory=[cdmp_20190517045917], requested by (instance=1, osid=6670), summary=[incident=4969].
Block recovery from logseq 2, block 66292 to scn 1006025
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATADG/test/onlinelog/group_2.341.1008434769
Block recovery stopped at EOT rba 2.66294.16
Block recovery completed at rba 2.66294.16, scn 0.1006022
Block recovery from logseq 2, block 66292 to scn 1006021
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATADG/test/onlinelog/group_2.341.1008434769
Block recovery completed at rba 2.66294.16, scn 0.1006022
Undo initialization errored: err:600 serial:0 start:1356834 end:1362534 diff:5700 (57 seconds)
Errors in file /opt/app/oracle/diag/rdbms/test/test/trace/test_ora_6670.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Errors in file /opt/app/oracle/diag/rdbms/test/test/trace/test_ora_6670.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 6670): terminating the instance due to error 600
Instance terminated by USER, pid = 6670
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (6670) as a result of ORA-1092
Fri May 17 04:59:20 2019
ORA-1092 : opitsk aborting process

2.使用bbed處理

需要處理：

ktuxc.ktuxcnfb
ktuxc.ktuxcfbp[0].ktufbuba.kubadba
kcbh.flg_kcbh
kcbh.chkval_kcbh

核心命令：

set offset ktuxc.ktuxcnfb
modify 0x0000

set offset ktuxc.ktuxcfbp[0].ktufbuba
modify 0x00000000

set offset kcbh.flg_kcbh
modify 0x00

set offset kcbh.chkval_kcbh
modify 0x0000

實際處理過程如下：

[oracle@OEL-ASM tmp]$ bbed parfile=bbed.par 
Password: 

BBED: Release 2.0.0.0.0 - Limited Production on Fri May 17 05:05:22 2019

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

************* !!! For Oracle Internal Use only !!! ***************

BBED> map
 File: /tmp/system01.bak (1)
 Block: 1                                     Dba:0x00400001
------------------------------------------------------------
 Data File Header

 struct kcvfh, 860 bytes                    @0       

 ub4 tailchk                                @8188    


BBED> set dba 1,128
        DBA             0x00400080 (4194432 1,128)

BBED> map
 File: /tmp/system01.bak (1)
 Block: 128                                   Dba:0x00400080
------------------------------------------------------------
 Unlimited Undo Segment Header

 struct kcbh, 20 bytes                      @0       

 struct ktech, 72 bytes                     @20      

 struct ktemh, 16 bytes                     @92      

 struct ktetb[6], 48 bytes                  @108     

 struct ktuxc, 104 bytes                    @4148    

 struct ktuxe[255], 10200 bytes             @4252    

 ub4 tailchk                                @8188    


BBED> p kcbh
struct kcbh, 20 bytes                       @0       
   ub1 type_kcbh                            @0        0x0e
   ub1 frmt_kcbh                            @1        0xa2
   ub1 spare1_kcbh                          @2        0x00
   ub1 spare2_kcbh                          @3        0x00
   ub4 rdba_kcbh                            @4        0x00400080
   ub4 bas_kcbh                             @8        0x000f59bb
   ub2 wrp_kcbh                             @12       0x0000
   ub1 seq_kcbh                             @14       0x01
   ub1 flg_kcbh                             @15       0x04 (KCBHFCKV)
   ub2 chkval_kcbh                          @16       0x3626
   ub2 spare3_kcbh                          @18       0x0000

BBED> p ktuxc
struct ktuxc, 104 bytes                     @4148    
   struct ktuxcscn, 8 bytes                 @4148    
      ub4 kscnbas                           @4148     0x000f26af
      ub2 kscnwrp                           @4152     0x0000
   struct ktuxcuba, 8 bytes                 @4156    
      ub4 kubadba                           @4156     0x0040021f
      ub2 kubaseq                           @4160     0x001b
      ub1 kubarec                           @4162     0x03
   sb2 ktuxcflg                             @4164     1 (KTUXCFSK)
   ub2 ktuxcseq                             @4166     0x001b
   sb2 ktuxcnfb                             @4168     1
   ub4 ktuxcinc                             @4172     0x00000000
   sb2 ktuxcchd                             @4176     3
   sb2 ktuxcctl                             @4178     91
   ub2 ktuxcmgc                             @4180     0x8002
   ub4 ktuxcopt                             @4188     0x7ffffffe
   struct ktuxcfbp[0], 12 bytes             @4192    
      struct ktufbuba, 8 bytes              @4192    
         ub4 kubadba                        @4192     0x0040021f
         ub2 kubaseq                        @4196     0x001b
         ub1 kubarec                        @4198     0x0d
      sb2 ktufbext                          @4200     3
      sb2 ktufbspc                          @4202     5510
   struct ktuxcfbp[1], 12 bytes             @4204    
      struct ktufbuba, 8 bytes              @4204    
         ub4 kubadba                        @4204     0x00000000
         ub2 kubaseq                        @4208     0x001a
         ub1 kubarec                        @4210     0x02
      sb2 ktufbext                          @4212     2
      sb2 ktufbspc                          @4214     7958
   struct ktuxcfbp[2], 12 bytes             @4216    
      struct ktufbuba, 8 bytes              @4216    
         ub4 kubadba                        @4216     0x00000000
         ub2 kubaseq                        @4220     0x0018
         ub1 kubarec                        @4222     0x3a
      sb2 ktufbext                          @4224     0
      sb2 ktufbspc                          @4226     2062
   struct ktuxcfbp[3], 12 bytes             @4228    
      struct ktufbuba, 8 bytes              @4228    
         ub4 kubadba                        @4228     0x00000000
         ub2 kubaseq                        @4232     0x0000
         ub1 kubarec                        @4234     0x00
      sb2 ktufbext                          @4236     0
      sb2 ktufbspc                          @4238     0
   struct ktuxcfbp[4], 12 bytes             @4240    
      struct ktufbuba, 8 bytes              @4240    
         ub4 kubadba                        @4240     0x00000000
         ub2 kubaseq                        @4244     0x0000
         ub1 kubarec                        @4246     0x00
      sb2 ktufbext                          @4248     0
      sb2 ktufbspc                          @4250     0

BBED> set offset 4168
        OFFSET          4168

BBED> p
ktuxc.ktuxcnfb
--------------
sb2 ktuxcnfb                                @4168     1

BBED> modify 0x0000
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /tmp/system01.bak (1)
 Block: 128              Offsets: 4168 to 4679           Dba:0x00400080
------------------------------------------------------------------------
 00000000 00000000 03005b00 02800100 68000000 feffff7f 1f024000 1b000d00 
 03008615 00000000 1a000200 0200161f 00000000 18003a00 00000e08 00000000 
 00000000 00000000 00000000 00000000 00000000 14000000 1e024000 28570f00 
 00000000 09005800 00000000 00000000 00000000 01000000 00000000 13000000 
 1a024000 bc260f00 00000000 09005f00 00000000 00000000 00000000 01000000 
 00000000 13000000 1b024000 db260f00 00000000 09005c00 00000000 00000000 
 00000000 01000000 00000000 13000000 1a024000 b2260f00 00000000 09000400 
 00000000 00000000 00000000 01000000 00000000 13000000 1a024000 b4260f00 
 00000000 09000600 00000000 00000000 00000000 01000000 00000000 13000000 
 1b024000 e5260f00 00000000 09000a00 00000000 00000000 00000000 01000000 
 00000000 13000000 1a024000 b7260f00 00000000 09006100 00000000 00000000 
 00000000 01000000 00000000 13000000 1b024000 c2260f00 00000000 09005900 
 00000000 00000000 00000000 02000000 00000000 14000000 1f024000 30570f00 
 00000000 09005b00 00000000 00000000 00000000 01000000 00000000 13000000 
 1b024000 ce260f00 00000000 09001200 00000000 00000000 00000000 01000000 
 00000000 13000000 1b024000 e7260f00 00000000 09001000 00000000 00000000 

 <32 bytes per line>

BBED> p
ktuxc.ktuxcnfb
--------------
sb2 ktuxcnfb                                @4168     0

BBED> set offset 4192
        OFFSET          4192

BBED> p
ktuxc.ktuxcfbp[0].ktufbuba.kubadba
----------------------------------
ub4 kubadba                                 @4192     0x0040021f

BBED> modify 0x00000000
 File: /tmp/system01.bak (1)
 Block: 128              Offsets: 4192 to 4703           Dba:0x00400080
------------------------------------------------------------------------
 00000000 1b000d00 03008615 00000000 1a000200 0200161f 00000000 18003a00 
 00000e08 00000000 00000000 00000000 00000000 00000000 00000000 14000000 
 1e024000 28570f00 00000000 09005800 00000000 00000000 00000000 01000000 
 00000000 13000000 1a024000 bc260f00 00000000 09005f00 00000000 00000000 
 00000000 01000000 00000000 13000000 1b024000 db260f00 00000000 09005c00 
 00000000 00000000 00000000 01000000 00000000 13000000 1a024000 b2260f00 
 00000000 09000400 00000000 00000000 00000000 01000000 00000000 13000000 
 1a024000 b4260f00 00000000 09000600 00000000 00000000 00000000 01000000 
 00000000 13000000 1b024000 e5260f00 00000000 09000a00 00000000 00000000 
 00000000 01000000 00000000 13000000 1a024000 b7260f00 00000000 09006100 
 00000000 00000000 00000000 01000000 00000000 13000000 1b024000 c2260f00 
 00000000 09005900 00000000 00000000 00000000 02000000 00000000 14000000 
 1f024000 30570f00 00000000 09005b00 00000000 00000000 00000000 01000000 
 00000000 13000000 1b024000 ce260f00 00000000 09001200 00000000 00000000 
 00000000 01000000 00000000 13000000 1b024000 e7260f00 00000000 09001000 
 00000000 00000000 00000000 01000000 00000000 13000000 1c024000 15290f00 

 <32 bytes per line>

BBED> p  
ktuxc.ktuxcfbp[0].ktufbuba.kubadba
----------------------------------
ub4 kubadba                                 @4192     0x00000000

BBED> map
 File: /tmp/system01.bak (1)
 Block: 128                                   Dba:0x00400080
------------------------------------------------------------
 Unlimited Undo Segment Header

 struct kcbh, 20 bytes                      @0       

 struct ktech, 72 bytes                     @20      

 struct ktemh, 16 bytes                     @92      

 struct ktetb[6], 48 bytes                  @108     

 struct ktuxc, 104 bytes                    @4148    

 struct ktuxe[255], 10200 bytes             @4252    

 ub4 tailchk                                @8188    


BBED> p kcbh
struct kcbh, 20 bytes                       @0       
   ub1 type_kcbh                            @0        0x0e
   ub1 frmt_kcbh                            @1        0xa2
   ub1 spare1_kcbh                          @2        0x00
   ub1 spare2_kcbh                          @3        0x00
   ub4 rdba_kcbh                            @4        0x00400080
   ub4 bas_kcbh                             @8        0x000f59bb
   ub2 wrp_kcbh                             @12       0x0000
   ub1 seq_kcbh                             @14       0x01
   ub1 flg_kcbh                             @15       0x04 (KCBHFCKV)
   ub2 chkval_kcbh                          @16       0x3626
   ub2 spare3_kcbh                          @18       0x0000

BBED> set offset 15
        OFFSET          15

BBED> p
kcbh.flg_kcbh
-------------
ub1 flg_kcbh                                @15       0x04 (KCBHFCKV)

BBED> modify 0x00
 File: /tmp/system01.bak (1)
 Block: 128              Offsets:   15 to  526           Dba:0x00400080
------------------------------------------------------------------------
 00263600 00000000 00000000 00000000 00000000 00060000 002f0000 00201000 
 00030000 00070000 00080000 001f0240 00000000 00030000 00000000 00000000 
 00000000 00000000 00000000 00060000 00000000 00000000 00000000 40810040 
 00070000 00880040 00080000 00100240 00080000 00180240 00080000 00200240 
 00080000 00280240 00080000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> p
kcbh.flg_kcbh
-------------
ub1 flg_kcbh                                @15       0x00 (NONE)

BBED> set offset 16
        OFFSET          16

BBED> p
kcbh.chkval_kcbh
----------------
ub2 chkval_kcbh                             @16       0x3626

BBED> modify 0x0000
 File: /tmp/system01.bak (1)
 Block: 128              Offsets:   16 to  527           Dba:0x00400080
------------------------------------------------------------------------
 00000000 00000000 00000000 00000000 00000000 06000000 2f000000 20100000 
 03000000 07000000 08000000 1f024000 00000000 03000000 00000000 00000000 
 00000000 00000000 00000000 06000000 00000000 00000000 00000040 81004000 
 07000000 88004000 08000000 10024000 08000000 18024000 08000000 20024000 
 08000000 28024000 08000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> p
kcbh.chkval_kcbh
----------------
ub2 chkval_kcbh                             @16       0x0000

BBED> sum apply
Check value for File 1, Block 128:
current = 0x0000, required = 0x0000

BBED> exit

[oracle@OEL-ASM tmp]$

3.嘗試啟動資料庫

bbed修改完成，再次嘗試啟動資料庫成功：

SQL> startup mount
ORACLE instance started.

Total System Global Area 1703624704 bytes
Fixed Size                  2229024 bytes
Variable Size             989859040 bytes
Database Buffers          704643072 bytes
Redo Buffers                6893568 bytes
Database mounted.
SQL> alter database open;

Database altered.

其他類ORA-600 [4194]的場景，可參考之前的隨筆：

Oracle建立新undo表空間最佳實踐(包含段檢查)

dataguard故障處理一則
2010-03-25
ORA-600 [12700]故障處理一則（線上重建損壞的索引）
2013-11-24
索引
一次ORA-600故障的處理
2008-02-04
【故障處理】ORA-600:[13013],[5001]故障處理
2020-12-03
ORA-600 [2662]故障處理
2017-11-29
Mysql故障處理2則
2009-01-08
MySql
模擬壞塊故障並搶救資料（一.普通表）
2015-05-02
Oracle資料庫 ORA-600 [13013]故障處理
2016-11-03
Oracle資料庫
【ARCHIVE】模擬因無法歸檔導致日誌無法切換故障及一般處理方法兩則
2010-04-13
Hive
【故障處理】一次RAC故障處理過程
2016-12-10
Oracle 11.2.0.4 Dataguard兩則故障處理
2021-03-12
Oracle
OracleORA-03113 ORA-600 [4193]故障處理
2020-08-01
Oracle
【spfile】修改及重置spfile實操 plus 故障處理一則
2012-04-05
cursor: pin S模擬與處理
2013-03-20
ORA-00600 [4194], [55]處理
2017-10-26
一次dataguard故障處理
2015-06-18
oracle壞塊模擬處理（筆記）
2014-03-19
Oracle筆記
MPP(大規模並行處理)簡介
2020-10-16
並行
OGG 故障處理一例
2014-07-10
一次TM ENQ故障處理
2011-06-02
ENQ
goldengate故障處理一例
2012-04-27
Go
MongoDB故障處理
2014-06-06
MongoDB
故障分析 | Greenplum Segment 故障處理
2023-02-02
ASMCMD處理問題一則
2009-03-08
ASM
【Control File】Oracle個別控制檔案丟失故障場景模擬及處理方法
2011-07-01
Oracle
GPON網路故障如何處理？GPON網路故障處理流程
2020-06-06
【故障處理】ORA- 2730*，status 12故障分析與處理
2010-05-11
ORA-00054 故障處理一例
2014-01-08
goldengate故障處理一例（續）
2012-04-28
Go
database link故障處理一例
2008-03-10
Database
Maxwell 磁場模擬場計算後處理
2024-11-10
linux故障處理
2020-07-30
Linux
ora-故障處理
2009-07-10
ORA-600 [15160]的處理
2009-06-12
MySQL OOM問題處理一則
2016-03-05
MySqlOOM
【EM】鎖等待故障模擬及排查
2011-07-21
Gpssworld模擬（二）：並排排隊系統模擬
2023-04-10
MPP大規模並行處理架構詳解
2021-07-11
並行架構

模擬一則ORA-600 [4194][][]故障並處理

1.模擬ORA-600 [4194][][]故障

2.使用bbed處理

3.嘗試啟動資料庫

相關文章