[20180612]刪除bootstrap$記錄無法啟動.txt

[20180612]刪除bootstrap$記錄無法啟動.txt

--//前幾天看連結
--//按照介紹: 有人在資料庫中注入了惡意指令碼，導致資料庫刪除了bootstrap$中資料,關閉之後無法正常啟動delete from bootstrap$;

--//我更多的思考如果我遇到這個問題如何解決:
1.如果有備份只要恢復到delete bootstrap$之前,但是這裡有問題,因為刪除後資料庫還繼續執行.不能繼續應用日誌,這樣有恢復到
  刪除bootstrap$後狀態.

2.如果有備份很好解決,因為bootstrap$的相關塊的資訊不會變動,只要覆蓋對應塊就ok了.
  而且實際上只要oracle版本相同,OS平臺一樣,使用別的資料庫的system表空間檔案中對應的塊替換應該一點問題都沒有.

3.當然最笨的方法就是恢復刪除的記錄.因為執行刪除記錄多,手工恢復感覺還是比較麻煩.

--//我自己也測試看看,演示後2種恢復方法:千萬不要再生產系統做這樣的測試!!

1.環境:
SCOTT@book> @ ver1
PORT_STRING                    VERSION        BANNER
------------------------------ -------------- --------------------------------------------------------------------------------
x86_64/Linux 2.4.xx            11.2.0.4.0     Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

--//最好備份資料庫略.首先看看bootstrap$佔用那些塊.
SCOTT@book> select HEADER_FILE,HEADER_BLOCK,BLOCKS,EXTENTS from dba_segments where owner='SYS' and segment_name='BOOTSTRAP$';
HEADER_FILE HEADER_BLOCK     BLOCKS    EXTENTS
----------- ------------ ---------- ----------
          1          520          8          1

SCOTT@book> column PARTITION_NAME noprint
SCOTT@book> select * from dba_extents where owner='SYS' and segment_name='BOOTSTRAP$';
OWNER  SEGMENT_NAME         SEGMENT_TYPE       TABLESPACE_NAME                 EXTENT_ID    FILE_ID   BLOCK_ID      BYTES     BLOCKS RELATIVE_FNO
------ -------------------- ------------------ ------------------------------ ---------- ---------- ---------- ---------- ---------- ------------
SYS    BOOTSTRAP$           TABLE              SYSTEM                                  0          1        520      65536          8            1

--//在system資料檔案頭部也記錄這個位置.透過bbed觀察:
BBED> p dba 1,1 kcvfh.kcvfhrdb
ub4 kcvfhrdb                                @96       0x00400208

BBED> set dba 0x00400208
        DBA             0x00400208 (4194824 1,520)
--//dba= 0x00400208指向的位置就是1,520,也就是sys.BOOTSTRAP$的段頭.

--//做一個sys.bootstrap$的備份:
SCOTT@book> create table bootstrap$bak as select * from sys.bootstrap$;
Table created.

SCOTT@book> select HEADER_FILE,HEADER_BLOCK,BLOCKS,EXTENTS from dba_segments where owner=OWNER and segment_name='BOOTSTRAP$BAK';
HEADER_FILE HEADER_BLOCK     BLOCKS    EXTENTS
----------- ------------ ---------- ----------
          4          858          8          1
--//開始破壞....
SYS@book> delete from sys.bootstrap$;
60 rows deleted.

SYS@book> commit ;
Commit complete.

--//實際上這個問題最嚴重的是如果你一直不重啟,根本不知道這個問題的存在,知道下次重啟才發現問題,
--//也就是可能備份的檔案一直存在問題的.^_^.

2.重啟資料庫:
SYS@book> shutdown immediate ;
Database closed.
Database dismounted.
ORACLE instance shut down.
SYS@book> startup
ORACLE instance started.

Total System Global Area  634732544 bytes
Fixed Size                  2255792 bytes
Variable Size             197133392 bytes
Database Buffers          427819008 bytes
Redo Buffers                7524352 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 54149
Session ID: 274 Serial number: 3

--//alert顯示
SMON: enabling cache recovery
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x977B258, lmebucp()+24] [flags: 0x0, count: 1]
Thu Jun 14 09:00:48 2018
ARC1 started with pid=46, OS id=54268
Errors in file /u01/app/oracle/diag/rdbms/book/book/trace/book_ora_54218.trc  (incident=2125010):
ORA-07445: exception encountered: core dump [lmebucp()+24] [SIGSEGV] [ADDR:0x0] [PC:0x977B258] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/book/book/incident/incdir_2125010/book_ora_54218_i2125010.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ARC1: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC1: Becoming the heartbeat ARCH
ARC0: Evaluating archive   log 3 thread 1 sequence 922
krss_find_arc: Selecting ARC1 to receive message as last resort
ARC1: Evaluating archive   log 3 thread 1 sequence 922
ARC0: Beginning to archive thread 1 sequence 922 (13280587483-13280607489) (book)
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_1: '/u01/app/oracle/archivelog/book/1_922_896605872.dbf' (thread 1 sequence 922) (book)
ARC1: Unable to archive thread 1 sequence 922
      Log actively being archived by another process
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/u01/app/oracle/archivelog/book/1_922_896605872.dbf' (book)
Committing creation of archivelog '/u01/app/oracle/archivelog/book/1_922_896605872.dbf'
Archived Log entry 16 added for thread 1 sequence 922 ID 0x4fb7d86e dest 1:
ARC0: Completed archiving thread 1 sequence 922 (13280587483-13280607489) (book)
Dumping diagnostic data in directory=[cdmp_20180614090051], requested by (instance=1, osid=54218), summary=[incident=2125010].
Incremental checkpoint up to RBA [0x39b.3.0], current log tail at RBA [0x39b.3.0]
Thu Jun 14 09:00:51 2018
PMON (ospid: 54174): terminating the instance due to error 397
Thu Jun 14 09:00:52 2018
System state dump requested by (instance=1, osid=54174 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/book/book/trace/book_diag_54184_20180614090052.trc
Dumping diagnostic data in directory=[cdmp_20180614090052], requested by (instance=1, osid=54174 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 54174
--//從以上跟蹤檔案很難發現是由於bootstrap$資訊被刪除.

3.使用10046跟蹤分析.
SYS@book> startup mount
ORACLE instance started.

Total System Global Area  634732544 bytes
Fixed Size                  2255792 bytes
Variable Size             197133392 bytes
Database Buffers          427819008 bytes
Redo Buffers                7524352 bytes
Database mounted.
SYS@book> @ &r/10046on 12
old   1: alter session set events '10046 trace name context forever, level &1'
new   1: alter session set events '10046 trace name context forever, level 12'

Session altered.

SYS@book> alter database open ;
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 54336
Session ID: 274 Serial number: 3

SYS@book> @ &r/10046off
ERROR:
ORA-03114: not connected to ORACLE

--//透過跟蹤檔案發現如下錯誤:
=====================
PARSING IN CURSOR #140711922570672 len=188 dep=1 uid=0 oct=1 lid=0 tim=1528938422280523 hv=4006182593 ad='7d2995f0' sqlid='32r4f1brckzq1'
create table bootstrap$ (
END OF STMT
PARSE #140711922570672:c=2000,e=947,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1528938422280521
EXEC #140711922570672:c=0,e=235,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=1528938422280848
CLOSE #140711922570672:c=0,e=8,dep=1,type=0,tim=1528938422280938
=====================
PARSING IN CURSOR #140711922570672 len=55 dep=1 uid=0 oct=3 lid=0 tim=1528938422281543 hv=2111436465 ad='7d297f30' sqlid='6apq2rjyxmxpj'
select line#, sql_text from bootstrap$ where obj# != :1
END OF STMT
PARSE #140711922570672:c=999,e=571,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1528938422281541
BINDS #140711922570672:
Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7ffa0c291168  bln=22  avl=02  flg=05
  value=59
EXEC #140711922570672:c=1000,e=976,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=867914364,tim=1528938422282657
WAIT #140711922570672: nam='db file sequential read' ela= 11 file#=1 block#=520 blocks=1 obj#=59 tim=1528938422282744
WAIT #140711922570672: nam='db file scattered read' ela= 24 file#=1 block#=521 blocks=3 obj#=59 tim=1528938422282974
FETCH #140711922570672:c=0,e=370,p=4,cr=5,cu=0,mis=0,r=0,dep=1,og=4,plh=867914364,tim=1528938422283068
STAT #140711922570672 id=1 cnt=0 pid=0 pos=1 obj=59 op='TABLE ACCESS FULL BOOTSTRAP$ (cr=5 pr=4 pw=0 time=371 us)'

*** 2018-06-14 09:07:02.283
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x977B258, lmebucp()+24] [flags: 0x0, count: 1]
Incident 2127890 created, dump file: /u01/app/oracle/diag/rdbms/book/book/incident/incdir_2127890/book_ora_54445_i2127890.trc
ORA-07445: exception encountered: core dump [lmebucp()+24] [SIGSEGV] [ADDR:0x0] [PC:0x977B258] [Address not mapped to object] []

ssexhd: crashing the process...
Shadow_Core_Dump = partial
ksdbgcra: writing core file to directory '/u01/app/oracle/diag/rdbms/book/book/cdump'

--//可以發現在執行
select line#, sql_text from bootstrap$ where obj# != :1
--// :1 = 59時候就報錯了.

--//透過bbed觀察
BBED> x /rnnc *kdbr[0]
rowdata[6877]                               @8167
-------------
flag@8167: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH)
lock@8168: 0x01
cols@8169:    0

BBED> x /rnnc *kdbr[1]
rowdata[6740]                               @8030
-------------
flag@8030: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH)
lock@8031: 0x01
cols@8032:    0

BBED> dump /v
File: /mnt/ramdisk/book/system01.dbf (1)
Block: 521                               Offsets: 8030 to 8093                            Dba:0x00400209
-----------------------------------------------------------------------------------------------------------
3c010301 80018081 43524541 54452052 4f4c4c42 41434b20 5345474d 454e5420 l <.......CREATE ROLLBACK SEGMENT
53595354 454d2053 544f5241 47452028 2020494e 49544941 4c203131 324b204e l SYSTEM STORAGE (  INITIAL 112K N
<32 bytes per line>
--//可以發現相關記錄被刪除了.我手工修復2條記錄看看:
BBED> set dba 1,521
        DBA             0x00400209 (4194825 1,521)

BBED> x /rnnc *kdbr[0]
rowdata[6877]                               @8167
-------------
flag@8167: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH)
lock@8168: 0x01
cols@8169:    0

BBED> assign  dba 1,521 offset 8167 = 0x2c
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
ub1 rowdata[0]                              @8167     0x2c

BBED> x /rnnc *kdbr[0]
rowdata[6877]                               @8167
-------------
flag@8167: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@8168: 0x01
cols@8169:    3

col    0[3] @8170: -1
col    1[3] @8174: -1
col    2[9] @8178: 8.0.0.0.0

BBED> assign  dba 1,521 offset 8030 = 0x2c
ub1 rowdata[0]                              @8030     0x2c

BBED> x /rnnc *kdbr[1]
rowdata[6740]                               @8030
-------------
flag@8030: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@8031: 0x01
cols@8032:    3

col    0[1] @8033: 0
col    1[1] @8035: 0
col  2[129] @8037: CREATE ROLLBACK SEGMENT SYSTEM STORAGE (  INITIAL 112K NEXT 56K MINEXTENTS 1 MAXEXTENTS 32765 OBJNO 0 EXTENTS (FILE 1 BLOCK 128))

--//注意一個細節:cols@8169:    0的數量我被沒有修改,也就是恢復刪除記錄僅僅修嘎flag從0x3c=>0x2c(多數情況是這樣,除非出現行遷移或者分片之類情況).

4.恢復:
--//藉助備份恢復,我測試環境一直儲存一個冷備份,藉助這個備份恢復看看.

$  dd if=/u01/backup/20170301B/system01.dbf of=/mnt/ramdisk/book/system01.dbf bs=8192 skip=520  seek=520 count=8 conv=notrunc
8+0 records in
8+0 records out
65536 bytes (66 kB) copied, 0.0163988 seconds, 4.0 MB/s

--//再次提醒使用dd命令,我現在每次使用都心存敬畏.就是自己做錯了1次.
--//千萬不要搞錯輸入輸出的物件..
--//skip對應輸入檔案跳過的塊數,seek對應輸出檔案跳過的塊數,注意一定不要忘記加conv=notrunc引數,不然輸出檔案可能被截斷.
--//這裡資料塊大小8192.
--//說明:/u01/app/oracle11g/oradata/test/system01.dbf這個是很久以前的冷備份.
--//再次透過bbed觀察
BBED> set dba 1,521
        DBA             0x00400209 (4194825 1,521)

BBED> x /rnnc *kdbr[1]
rowdata[6740]                               @8030
-------------
flag@8030: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@8031: 0x01
cols@8032:    3

col    0[1] @8033: 0
col    1[1] @8035: 0
col  2[129] @8037: CREATE ROLLBACK SEGMENT SYSTEM STORAGE (  INITIAL 112K NEXT 56K MINEXTENTS 1 MAXEXTENTS 32765 OBJNO 0 EXTENTS (FILE 1 BLOCK 128))
--//ok,現在已經修復.

--//再次重啟資料庫:
SYS@book> startup mount
ORACLE instance started.
Total System Global Area  634732544 bytes
Fixed Size                  2255792 bytes
Variable Size             197133392 bytes
Database Buffers          427819008 bytes
Redo Buffers                7524352 bytes
Database mounted.

SYS@book> alter database open ;
Database altered.

5.測試使用bbed的方法:
--//估計有點長另外寫一篇文章.