替換OCR和表決磁碟後，重啟叢集，資料庫資源的叢集狀態為OFFLINE

1 系統概述

業務系統需要更換SSD儲存裝置，需要將所有的磁碟替換掉。更換資料磁碟、OCR 、表決磁碟後，在節點1重啟叢集進行驗證，發現資料庫例項的資源狀態為OFFLINE，使用叢集命令servctl也不能啟動，報ORA-15017: diskgroup "OCR" cannot be mounted

2 啟動資料庫，檢視叢集狀態，發現資料庫資源狀態為offline

啟動資料庫

SYS@test1 >startup

ORACLE instance started.

Total System Global Area 2.3517E+11 bytes

Fixed Size 2267464 bytes

Variable Size 5.0466E+10 bytes

Database Buffers 1.8415E+11 bytes

Redo Buffers 553766912 bytes

Database mounted.

Database opened.

檢視資料庫是否啟動

[grid@testdb1 ~]$ ps -ef|grep smon

root 13667 1 8 Mar26 ? 12:56:12 /opt/app/11.2.0.4/grid/bin/osysmond.bin

grid 15764 1 0 Mar26 ? 00:00:17 asm_smon_+ASM1

grid 17875 1 0 Mar26 ? 00:26:00 ora_smon_test1

grid 163774 163708 0 15:22 pts/1 00:00:00 grep smon

檢視叢集狀態，發現節點1資料庫例項的狀態不對

ora.test.db

1 ONLINE OFFLINE Instance Shutdown

2 ONLINE ONLINE testdb2 Open

3 ONLINE ONLINE testdb3 Open

3 關閉資料庫，使用叢集命令進行啟動，發現報錯的原因為沒有OCR磁碟組，此磁碟組在1小時前已經進行替換和刪除。

SYS@test1 >shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SYS@test1 >exit

試著用叢集命令啟動，發現報錯

[grid@testdb1 ~]$ srvctl start instance -d test -i test1

PRCR-1013 : Failed to start resource ora.test.db

PRCR-1064 : Failed to start resource ora.test.db on node testdb1

CRS-5017: The resource action "ora.OCR.dg start" encountered the following error:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "OCR" cannot be mounted

ORA-15040: diskgroup is incomplete

. For details refer to "(:CLSN00107:)" in "/opt/app/11.2.0.4/grid/log/testdb1/agent/crsd/oraagent_grid//oraagent_grid.log".

CRS-2674: Start of 'ora.OCR.dg' on 'testdb1' failed

4 檢視報錯資訊的trace檔案，還是在尋找OCR磁碟組

2022-03-26 02:23:40.443: [ora.asm][1424000768]{1:20237:2} [check] AsmProxyAgent::createAsmOcrKeys host: testdb1 oracleSid_lc:

+asm1 oracleSid_uc: +ASM1 tempHost1: testdb1.+asm1

[ OCRAPI][1424000768]a_is_valid_user_group: User [oracle] does not match with init user

[ OCRUTL][1424000768]u_fill_errorbuf: Error Info : [User [oracle] does not match with initialized user]

[ OCRAPI][1424000768]a_create_n_set_key:THE SECURITY ATTRIBUTE PASSED is invalid

2022-03-26 02:23:40.454: [ OCRAPI][1424000768]a_batch_execute: Check batch exec failed [5]

2022-03-26 02:23:40.454: [ OCRAPI][1424000768]a_batch_execute: Failed to execute batch. Return [5]

2022-03-26 02:23:40.454: [ora.asm][1424000768]{1:20237:2} [check] OcrBatch Execute failed 5

2022-03-26 02:23:40.455: [ora.asm][1424000768]{1:20237:2} [check] AsmProxyAgent::createAsmOcrKeys failed

[ OCRUTL][1424000768]u_check_tag: Tag check failed on 0th comparison for [PROCRKEY]. Ptr passed was [40446f0] Byte was [0]

2022-03-26 02:23:40.455: [ora.asm][1424000768]{1:20237:2} [check] AsmProxyAgent::createAsmOcrKeys exit }

2022-03-26 02:23:40.455: [ora.asm][1424000768]{1:20237:2} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0

2022-03-26 02:23:40.477: [ora.OCR.dg][787871488]{1:20237:207} [start] ORA-15032: not all alterations performed

ORA-15017: diskgroup "OCR" cannot be mounted

ORA-15040: diskgroup is incomplete

2022-03-26 02:23:40.478: [ USRTHRD][1424000768]{1:20237:2} CrsCmd::ClscrsCmdData::destroy

2022-03-26 02:23:40.478: [ora.OCR.dg][787871488]{1:20237:207} [start] clsnUtils::error Exception type=2 string=

CRS-5017: The resource action "ora.OCR.dg start" encountered the following error:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "OCR" cannot be mounted

ORA-15040: diskgroup is incomplete

. For details refer to "(:CLSN00107:)" in "/opt/app/11.2.0.4/grid/log/testdb1/agent/crsd/oraagent_grid//oraagent_grid.log".

5 原因分析

根據如上資訊，可以確定叢集需要OCR磁碟組，但此磁碟組由於更換儲存裝置，已經進行了刪除，遷移到新的磁碟組下。故判斷是由於將資料庫相關資訊註冊到OCR磁碟組下，導致叢集狀態不正常及不能使用叢集命令啟動資料庫。檢視20分鐘官方文件，未找到合適的官方文件，經過思考，認為就是叢集註冊資訊在老的OCR磁碟組下，故可以將資料庫及例項的叢集註冊資訊刪除，並重建，估計就恢復正常了。經測試，一切OK。

5.1 關閉所有資料庫例項

5.2 使用如下命令刪除資料庫資源的相關資訊

刪除：

cd /opt/app/11.2.0.4/grid/bin

./srvctl remove database -d test

./srvctl remove instance -d test -i test1

./srvctl remove instance -d test -i test2

./srvctl remove instance -d test -i test3

5.３ 使用如下命令新增資料庫資源的相關資訊

新增：

srvctl add database -d test -o $ORACLE_HOME

srvctl add instance -d test -i test1 -n testdb1

srvctl add instance -d test -i test2 -n testdb2

srvctl add instance -d test -i test3 -n testdb3

5.４ 使用叢集命令啟動資料庫，並檢視叢集資訊，發現叢集資訊已經恢復正常

ora.test.db

1 ONLINE ONLINE testdb1 Open

2 ONLINE ONLINE testdb2 Open

3 ONLINE ONLINE testdb3 Open