ORA-00600: [kcratr1_lastbwr]錯誤的處理辦法

出現該錯誤是因為系統強制關機造成的!症狀為資料庫無法開啟!解決辦法:
1>shutdown
2>startup mount
3>recover database
4>alter database open

from:

[@more@]

案例1：

資料庫掉電，重啟發現如下錯誤：

檢視ALERT日誌，發現
Errors in file /opt/oracle/admin/billdb/udump/billdb_ora_7186.trc:
ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []

奇怪，於是先把資料庫關閉，然後重新啟動，報錯如下：
SQL> startup
ORACLE instance started.

Total System Global Area 1375731712 bytes
Fixed Size 1260780 bytes
Variable Size 603980564 bytes
Database Buffers 754974720 bytes
Redo Buffers 15515648 bytes
Database mounted.
ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []

metalink上去，查到如下結果：

Changes
There was a disk problem that caused the database to crash.
Cause
Oracle is unable to perform instance recover but it works when is invoked manually.
Solution
Mount the database and issue a recover statement

接著執行恢復：

SQL> startup mount;

SQL> recover database;

SQL> alter database open

於是照搬上面的步驟，手工恢復後問題解決

以下是官方解釋：

Unbreakable Oracle 10g Release 2 : What if you have ORA-600 kcratr1_lastbwr ?
This an interesting story that happened yesterday on one of our customer site. An engineer powered off the wrong rack of equipment containing a Sun Fire X4600 running Oracle 10g Release 2. Almost no transactions were performed at time so when the system came up the customer expected the database to be up and running very quickly.

In reality this is what happened :

Completed: ALTER DATABASE MOUNT
Tue Nov 7 11:19:42 2006
ALTER DATABASE OPEN
Tue Nov 7 11:19:42 2006
Beginning crash recovery of 1 threads
parallel recovery started with 16 processes
Tue Nov 7 11:19:44 2006
Started redo scan
Tue Nov 7 11:19:44 2006
Errors in file /xxx/oracle/oracle/product/10.2.0/db_1/admin/xxx/udump/xxx_ora_947.trc:
ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []
Tue Nov 7 11:19:44 2006
Aborting crash recovery due to error 600
Tue Nov 7 11:19:44 2006
Errors in file /xxx/oracle/oracle/product/10.2.0/db_1/admin/xxxtest/udump/xxxtest_ora_947.trc:
ORA-00600: internal error code, arguments: [kcratr1_lastbwr], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...


Not too pretty ! Checking the ASM configuration and the IO subsystem showed nothing wrong. So what to do if you do not have a backup handy ?

Well, here is the idea .... what would we do if we had a backup that was inconsistent ?
The recover database command will start an Oracle process which will roll forward all transactions stored in the restored archived logs necessary to make the database consistent again. The recovery process must run up to a point that corresponds with the time just before the error occurred after which the log sequence must be reset to prevent any further system changes from being applied to the database.
So we tried :

startup mount

ALTER DATABASE MOUNT
Tue Nov 7 11:54:03 2006
Starting background process ASMB
ASMB started with pid=61, OS id=1070
Starting background process RBAL
RBAL started with pid=67, OS id=1074
Tue Nov 7 11:54:13 2006
SUCCESS: diskgroup xxxTESTDATA was mounted
Tue Nov 7 11:54:17 2006
Setting recovery target incarnation to 2
Tue Nov 7 11:54:17 2006
Successful mount of redo thread 1, with mount id 2364224219
Tue Nov 7 11:54:17 2006
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Tue Nov 7 11:54:32 2006


recover database

ALTER DATABASE RECOVER database
Tue Nov 7 11:54:32 2006
Media Recovery Start
parallel recovery started with 16 processes
Tue Nov 7 11:54:33 2006
Recovery of Online Redo Log: Thread 1 Group 3 Seq 4 Reading mem 0
Mem# 0 errs 0: +xxxTESTDATA/xxxtest/onlinelog/group_3.263.605819131
Tue Nov 7 11:59:25 2006
Media Recovery Complete (xxxtest)
Tue Nov 7 11:59:27 2006
Completed: ALTER DATABASE RECOVER database


alter database open

Tue Nov 7 12:03:01 2006
alter database open
Tue Nov 7 12:03:01 2006
Beginning crash recovery of 1 threads
parallel recovery started with 16 processes
Tue Nov 7 12:03:01 2006
Started redo scan
Tue Nov 7 12:03:01 2006
Completed redo scan
273 redo blocks read, 0 data blocks need recovery
Tue Nov 7 12:03:01 2006
Started redo application at
Thread 1: logseq 4, block 12858574
Tue Nov 7 12:03:01 2006
Recovery of Online Redo Log: Thread 1 Group 3 Seq 4 Reading mem 0
Mem# 0 errs 0: +xxxTESTDATA/xxxtest/onlinelog/group_3.263.605819131
Tue Nov 7 12:03:01 2006
Completed redo application
Tue Nov 7 12:03:01 2006
Completed crash recovery at
Thread 1: logseq 4, block 12858847, scn 824040
0 data blocks read, 0 data blocks written, 273 redo blocks read
Tue Nov 7 12:03:02 2006
Thread 1 advanced to log sequence 5
Thread 1 opened at log sequence 5
Current log# 1 seq# 5 mem# 0: +xxxTESTDATA/xxxtest/onlinelog/group_1.261.605819081
Successful open of redo thread 1
Tue Nov 7 12:03:02 2006
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Nov 7 12:03:02 2006
SMON: enabling cache recovery
Tue Nov 7 12:03:03 2006
Successfully onlined Undo Tablespace 1.
Tue Nov 7 12:03:03 2006
SMON: enabling tx recovery
Tue Nov 7 12:03:03 2006
Database Characterset is UTF8
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=56, OS id=1128
Tue Nov 7 12:03:05 2006
Completed: alter database open

And we are up and running ! The real thing that Oracle should work on is the quality and clarity of their error messages.
At this point this is quite poor ...

Unbreakable database, maybe. Automatic (and simple) , not yet.

案例二：

如果在執行recover時刻出現

SQL> recover database;

ORA-00283: recovery session canceled due to errors

ORA-12801: error signaled in parallel query server P002

ORA-10562: Error occurred while applying redo to data block (file# 1, block#

4568)

ORA-10564: tablespace SYSTEM

ORA-01110: data file 1: '/opt/oracle/oradata/orcl/system01.dbf'

ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 576

ORA-00600: internal error code, arguments: [6101], [

此時，檢查日誌資訊如下：

Mon Nov 19 15:38:50 2007

ALTER DATABASE RECOVER database

Mon Nov 19 15:38:50 2007

Media Recovery Start

parallel recovery started with 3 processes

Mon Nov 19 15:38:50 2007

Recovery of Online Redo Log: Thread 1 Group 3 Seq 16 Reading mem 0

Mem# 0 errs 0: /opt/oracle/oradata/orcl/redo03.log

Mon Nov 19 15:38:50 2007

Errors in file /opt/oracle/admin/orcl/bdump/orcl_p002_7917.trc:

ORA-00600: internal error code, arguments: [6101], [0], [17], [0], [], [], [], []

Mon Nov 19 15:38:50 2007

Errors in file /opt/oracle/admin/orcl/bdump/orcl_p000_7913.trc:

ORA-00600: internal error code, arguments: [3020], [2], [882], [8389490], [], [], [], []

ORA-10567: Redo is inconsistent with data block (file# 2, block# 882)

ORA-10564: tablespace UNDOTBS1

ORA-01110: data file 2: '/opt/oracle/oradata/orcl/undotbs01.dbf'

ORA-10560: block type 'KTU UNDO BLOCK'

Mon Nov 19 15:38:51 2007

Errors in file /opt/oracle/admin/orcl/bdump/orcl_p000_7913.trc:

ORA-00600: internal error code, arguments: [3020], [2], [882], [8389490], [], [], [], []

ORA-10567: Redo is inconsistent with data block (file# 2, block# 882)

ORA-10564: tablespace UNDOTBS1

ORA-01110: data file 2: '/opt/oracle/oradata/orcl/undotbs01.dbf'

ORA-10560: block type 'KTU UNDO BLOCK'

Mon Nov 19 15:38:51 2007

Errors in file /opt/oracle/admin/orcl/bdump/orcl_p002_7917.trc:

ORA-10562: Error occurred while applying redo to data block (file# 1, block# 4568)

ORA-10564: tablespace SYSTEM

ORA-01110: data file 1: '/opt/oracle/oradata/orcl/system01.dbf'

ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 576

ORA-00600: internal error code, arguments: [6101], [0], [17], [0], [], [], [], []

Mon Nov 19 15:38:54 2007

Errors in file /opt/oracle/admin/orcl/bdump/orcl_p001_7915.trc:

ORA-00600: internal error code, arguments: [kddummy_blkchk], [1], [1658], [6101], [], [], [], []

Mon Nov 19 15:38:54 2007

Errors in file /opt/oracle/admin/orcl/bdump/orcl_p001_7915.trc:

ORA-10562: Error occurred while applying redo to data block (file# 1, block# 1658)

ORA-10564: tablespace SYSTEM

ORA-01110: data file 1: '/opt/oracle/oradata/orcl/system01.dbf'

ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 237

ORA-00607: Internal error occurred while making a change to a data block

ORA-00600: internal error code, arguments: [kddummy_blkchk], [1], [1658], [6101], [], [], [], []

Mon Nov 19 15:38:54 2007

Media Recovery failed with error 12801

ORA-283 signalled during: ALTER DATABASE RECOVER database ...

從上面資訊中抓取了一個資訊：

ORA-10562: Error occurred while applying redo to data block (file# 1, block# 1658)

針對這個錯誤解決如下：

ORA-10562: Error occurred while applying redo to data block (file# string, block# string)
Cause: See other errors on error stack.
Action: Investigate why the error occurred and how important is the data block. Media and standby database recovery usually can continue if user allows recovery to corrupt this data block。

從日誌資訊可以基本判斷問題如下：

當前線上日誌損壞，導致undo回滾段出現問題，又由於系統突然掉電，系統表空間在重啟例項後要進行例項恢復，當前線上日誌損壞，系統表空間無法進行recover，因而出現了上面的錯誤。

找到了問題，解決訪問就有了。

由於沒有備份，資料庫也執行在非歸檔模式下，所以恢復如下步驟：

SQL>startup mount

SQL>recover database using backup controlfile until cancel;

SQL>alter database open resetlogs;

SQL> startup mount

SQL> alter system set “_allow_resetlogs_corruption”=true scope=spfile;

SQL>shutdown immediate

SQL> startup mount

SQL> alter database open resetlogs;

SQL> startup

我的基本恢復步驟：

SQL>startup mount

SQL>recover database using backup controlfile until cancel;

Cancel

SQL>alter database open resetlogs;

此時會提示system表空間需要恢復，但是由於當前日誌損壞，無法進行恢復，所以需要加入隱含引數，oracle才不會監測scn的一致性，才能開啟資料庫。

重啟資料庫加入隱含引數

SQL> startup mount

SQL> alter system set “_allow_resetlogs_corruption”=true scope=spfile;

SQL> shutdown immediate

SQL> startup mount

SQL> alter database open resetlogs;

alter database open resetlogs

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

不管這些，再次登入sqlplus起動資料庫

SQL*Plus: Release 10.2.0.1.0 - Production on Fri Nov 16 08:03:43 2007

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production

With the Partitioning, OLAP and Data Mining options

SQL>startup

資料庫正常開啟，exp備份需要的資料，然後重新建庫，匯入資料。