[20221020]奇怪的增量備份.txt

[20221020]奇怪的增量備份.txt

--//生產系統做增量備份遇到的怪異問題,給奇葩的運維人員狠狠地涮了一把,做一個記錄:

1.環境:
SYS@192.168.100.235:1521/orcl> @ pr
==============================
PORT_STRING                   : x86_64/Linux 2.4.xx
VERSION                       : 19.0.0.0.0
BANNER                        : Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
BANNER_FULL                   : Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
BANNER_LEGACY                 : Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
CON_ID                        : 0
PL/SQL procedure successfully completed.

2.問題:
--//在rman下檢視:
RMAN> list backupset 7868;
List of Backup Sets
===================
BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
7868    Incr 1  4.47G      SBT_TAPE    00:35:56     2022-10-20 00:26:15
        BP Key: 8615   Status: AVAILABLE  Compressed: NO  Tag: 2022_10_19_23_50_13
        Handle: ORDB_ORCL_7894_1_1118533819   Media:
  List of Datafiles in backup set 7868
  File LV Type Ckp SCN    Ckp Time            Abs Fuz SCN Sparse Name
  ---- -- ---- ---------- ------------------- ----------- ------ ----
  1    1  Incr 42592240372 2022-10-19 23:50:19              NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/system01.dbf
  2    1  Incr 42592240372 2022-10-19 23:50:19 42592296511  NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/lis_max_data.dbf
  3    1  Incr 42592240372 2022-10-19 23:50:19              NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/sysaux01.dbf
  4    1  Incr 42592240372 2022-10-19 23:50:19              NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/undotbs01.dbf
  5    1  Incr 42592240372 2022-10-19 23:50:19 42592295965  NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/lis_data.dbf
  7    1  Incr 42592240372 2022-10-19 23:50:19              NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/users01.dbf
  8    1  Incr 42592240372 2022-10-19 23:50:19              NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/lis_max_data01.dbf
  9    1  Incr 42592240372 2022-10-19 23:50:19 42592244778  NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/lis_data02.dbf
  10   1  Incr 42592240372 2022-10-19 23:50:19 42592295898  NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/lis_max_data02.dbf
  11   1  Incr 42592240372 2022-10-19 23:50:19 42592268142  NO    /u02/app/oracle/oradata/orcl/datafile/ORCL/lis_max_data03.dbf
--//不知道虛擬磁帶庫是否有檔案大小的限制.目前4.47G.

SYS@192.168.100.235:1521/orcl> @ dashtop sql_id,module1 1=1 &day
    Total
  Seconds     AAS %This   SQL_ID        MODULE1              FIRST_SEEN          LAST_SEEN
--------- ------- ------- ------------- -------------------- ------------------- -------------------
     7030      .1   22%                                      2022-10-11 10:56:18 2022-10-12 10:00:29
     5090      .1   16%                 w3wp.exe             2022-10-11 10:56:18 2022-10-12 10:00:19
     2180      .0    7%                 backup incr datafile 2022-10-11 23:50:09 2022-10-12 00:26:24
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     1530      .0    5%                 backup archivelog    2022-10-12 00:27:04 2022-10-12 02:32:11

--//很明顯發現問題在於運維人員沒有開啟塊跟蹤特性,導致要掃描整個資料檔案,不然做增量level=1不會使用這麼長時間.
--//ALTER DATABASE ENABLE BLOCK CHANGE TRACKING;
SYS@192.168.100.235:1521/orcl> SELECT * FROM V$BLOCK_CHANGE_TRACKING
  2  @ pr
==============================
STATUS                        : DISABLED
FILENAME                      :
BYTES                         :
CON_ID                        :
PL/SQL procedure successfully completed.

SYS@192.168.100.235:1521/orcl> alter database enable block change tracking using file '/u01/app/oracle/oradata/orcl/changetracking/block_change_tracking.f' reuse;
Database altered.

SYS@192.168.100.235:1521/orcl> SELECT * FROM V$BLOCK_CHANGE_TRACKING
  2  @pr
==============================
STATUS                        : ENABLED
FILENAME                      : /u01/app/oracle/oradata/orcl/changetracking/block_change_tracking.f
BYTES                         : 11599872
CON_ID                        : 0
PL/SQL procedure successfully completed.

--//幾天後檢查發現增量備份時間並沒有減少,我開始以為我自己禁用了塊跟蹤,但是我清晰的記得當時下班前我是開啟的了.

--//我接著再次執行(我的工作筆記記錄的是上個星期4做的操作2022/10/20):
alter database enable block change tracking using file '/u01/app/oracle/oradata/orcl/changetracking/block_change_tracking.f' reuse;
--//這次應該可以了把.因為星期6,7應該有1次level=0的全備份.

--//可是今天上班檢查(星期1 2022/10/24)發現增量備份還是需要很長時間.難道我對backup incr datafile理解有誤.
--//難道是接著做增量的第1次level=1的增量備份還是無法使用,我給在測試環境測試看看.

SYS@192.168.100.235:1521/orcl> SELECT * FROM V$BLOCK_CHANGE_TRACKING
  2  @ pr
==============================
STATUS                        : DISABLED
FILENAME                      :
BYTES                         :
CON_ID                        :
PL/SQL procedure successfully completed.
--//發現居然變成了DISABLED,難道有人不讓我enable block change tracking嗎?而且這次我不可能犯錯.

3.檢查跟蹤檔案發現:
--//檢查發現實際上對方寫的指令碼有1個alter database disable block change tracking操作,奇葩!!alert*.log有記錄:
2022-10-18T23:50:12.040797+08:00
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
alter database disable block change tracking
2022-10-18T23:50:12.057827+08:00
stopping change tracking
2022-10-18T23:50:12.058898+08:00
Block change tracking service stopping.
Stopping background process CTWR
2022-10-18T23:50:13.112232+08:00
Deleted file /u01/app/oracle/oradata/orcl/changetracking/block_change_tracking.f
Completed: alter database disable block change tracking
2022-10-19T00:25:54.803845+08:00
Control autobackup written to SBT_TAPE device

--//很明顯對方的增量備份指令碼在備份前禁用了塊跟蹤檔案特性,執行時間也能對上2022-10-18T23:50.12.
--//奇葩的運維人員....我根本不知道這位同行是如何想的,難道這位同行遇到什麼bug或者遇到這類增量方式無法恢復的情況.
--//連續浪費好幾天的時間檢查該問題,如果一開始查詢alert檔案,問題很快就可以定位了.