ORACLE RAC 11.2.0.4 ASM加盤導致叢集重啟之ASM sga設定過小

    最近，一同事為一2節點的ORACLE RAC 11.2.0.4叢集ASM加盤，沒有注意到ASM的sga設定過小，加盤reblance時導致叢集重啟。詳細描述如下：

    1、問題描述

    ORACLE RAC 11.2.0.4叢集ASM加盤，沒有注意到ASM的sga設定過小，加盤reblance時導致叢集重啟。

    2、問題分析

    2.1 首先，接到運維監控告警，某某平臺發生嚴重告警

    2.2 登陸相關平臺檢視資料庫和叢集ASM日誌，發現節點1叢集發生重啟，相關報錯日誌如下：

Wed Dec 26 17:20:29 2018

ARB0 started with pid=43, OS id=22557 

NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O

NOTE: Attempting voting file refresh on diskgroup ORADATA

Wed Dec 26 17:22:37 2018

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_22557.trc  (incident=208353):

ORA-04031: unable to allocate 7072 bytes of shared memory ("shared pool","unknown object","sga heap(1,1)","ges resource ")

Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208353/+ASM1_arb0_22557_i208353.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Wed Dec 26 17:22:39 2018

Dumping diagnostic data in directory=[cdmp_20181226172239], requested by (instance=1, osid=22557 (ARB0)), summary=[incident=208353].

Wed Dec 26 17:22:46 2018

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc  (incident=208345):

ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable")

Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208345/+ASM1_m000_22771_i208345.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc:

ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable")

    2.3 資料庫運維人員注意到，此前的主要操作主要就是給磁碟組加盤，觀察ASM告警日誌，相關操作記錄如下：

Wed Dec 26 17:20:16 2018

SQL> ALTER DISKGROUP ORADATA ADD  DISK '/dev/rdisk/VD-DATA21' SIZE 182272M ,

'/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */ 

NOTE: GroupBlock outside rolling migration privileged region

NOTE: Assigning number (2,20) to disk (/dev/rdisk/VD-DATA21)

NOTE: Assigning number (2,21) to disk (/dev/rdisk/VD-DATA22)

NOTE: requesting all-instance membership refresh for group=2

NOTE: initializing header on grp 2 disk ORADATA_0020

NOTE: initializing header on grp 2 disk ORADATA_0021

NOTE: requesting all-instance disk validation for group=2

Wed Dec 26 17:20:19 2018

NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance.

NOTE: requesting all-instance disk validation for group=2

NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance.

NOTE: initiating PST update: grp = 2

Wed Dec 26 17:20:25 2018

GMON updating group 2 at 7 for pid 42, osid 22238

NOTE: PST update grp = 2 completed successfully 

NOTE: membership refresh pending for group 2/0x6b4139d2 (ORADATA)

GMON querying group 2 at 8 for pid 18, osid 5128

NOTE: cache opening disk 20 of grp 2: ORADATA_0020 path:/dev/rdisk/VD-DATA21

NOTE: cache opening disk 21 of grp 2: ORADATA_0021 path:/dev/rdisk/VD-DATA22

Wed Dec 26 17:20:29 2018

NOTE: Attempting voting file refresh on diskgroup ORADATA

GMON querying group 2 at 9 for pid 18, osid 5128

SUCCESS: refreshed membership for 2/0x6b4139d2 (ORADATA)

Wed Dec 26 17:20:29 2018

SUCCESS: ALTER DISKGROUP ORADATA ADD  DISK '/dev/rdisk/VD-DATA21' SIZE 182272M ,

'/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */

NOTE: starting rebalance of group 2/0x6b4139d2 (ORADATA) at power 1

Starting background process ARB0

Wed Dec 26 17:20:29 2018

ARB0 started with pid=43, OS id=22557 

NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O

NOTE: Attempting voting file refresh on diskgroup ORADATA

    2.4 到此，基本上可以確定是ASM加盤，ASM在進行reblance的時候，由於ASM的SGA設定過小導致的叢集重啟

    3、問題處理

    調整相關叢集的ASM的SGA大小到2G，重新啟動叢集，ASM開始正常的reblance:

Wed Dec 26 17:56:32 2018

NOTE: client SYXPTDB1:SYXPTDB registered, osid 1074, mbr 0x1

Wed Dec 26 18:35:33 2018

ALTER SYSTEM SET memory_max_target='2G' SCOPE=SPFILE SID='*';

ALTER SYSTEM SET memory_target='2G' SCOPE=SPFILE SID='*';

ALTER SYSTEM SET sga_max_size='2G' SCOPE=SPFILE SID='*';

Wed Dec 26 20:50:11 2018

NOTE: GroupBlock outside rolling migration privileged region

NOTE: requesting all-instance membership refresh for group=2

Wed Dec 26 20:50:14 2018

NOTE: membership refresh pending for group 2/0x1edaa5d7 (ORADATA)

Wed Dec 26 20:50:17 2018

GMON querying group 2 at 7 for pid 18, osid 26306

SUCCESS: refreshed membership for 2/0x1edaa5d7 (ORADATA)

NOTE: Attempting voting file refresh on diskgroup ORADATA

    到此問題處理完成！