Oracle 12c叢集啟動故障
由於維護人員修改 OracleLinux 7中的/dev/shm大小造成其大小小於Oracle例項的MEMORY_TARGET或者SGA_TARGET而導致叢集不能啟動(CRS-4535,CRS-4000)
[grid@jtp1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
檢查asm磁碟的許可權是否有問題
[root@jtp3 ~]# ls -lrt /dev/asm*
重啟crs
[root@jtp1 bin]# ./crsctl stop crs -f
[root@jtp1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
檢視crs的alert.log發現磁碟組不能載入
[root@jtp1 ~]# tail -f /u01/app/grid/diag/crs/jtp1/crs/trace/alert.log
locations are on ASM disk groups [CRS], and none of these disk groups are mounted
繼續檢視 ohasd_orarootagent_root.trc
[root@jtp1 ~]# more /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc
Trace file /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc
Oracle Database 12c Clusterware Release 12.2.0.1.0 - Production Copyright 1996, 2016 Oracle. All rights reserved.
*** TRACE CONTINUED FROM FILE /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root_93.trc ***
2018-04-02 18:42:09.165 : CSSCLNT:3554666240: clsssterm: terminating context (0x7f03c0229390)
2018-04-02 18:42:09.165 : default:3554666240: clsCredDomClose: Credctx deleted 0x7f03c0459470
2018-04-02 18:42:09.166 : GPNP:3554666240: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'
2018-04-02 18:42:09.253 : CSSCLNT:3554666240: clsssinit: initialized context: (0x7f03c045c2c0) flags 0x115
2018-04-02 18:42:09.253 : CSSCLNT:3554666240: clsssterm: terminating context (0x7f03c045c2c0)
2018-04-02 18:42:09.254 : CLSNS:3554666240: clsns_SetTraceLevel:trace level set to 1.
2018-04-02 18:42:09.254 : GPNP:3554666240: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'
2018-04-02 18:42:09.257 : default:3554666240: Inited LSF context: 0x7f03c04f0420
2018-04-02 18:42:09.260 : CLSCRED:3554666240: clsCredCommonInit: Inited singleton credctx.
2018-04-02 18:42:09.260 : CLSCRED:3554666240: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.
2018-04-02 18:42:09.294 : USRTHRD:3554666240: {0:9:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2018-04-02 18:42:09.300 : USRTHRD:3554666240: {0:9:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2018-04-02 18:42:09.356 : CLSCRED:3554666240: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.5c82286a084bcf37ffa014144074e5dd.root not found
2018-04-02 18:42:09.356 : USRTHRD:3554666240: {0:9:3} 7755 Error 4 opening dom root in 0x7f03c064c980
檢查ASM的alert.log 發現/dev/shm大小小於MEMORY_TARGET大小，並且給出了/dev/shm應該被設定的最小值
[root@jtp1 ~]# tail -f /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
WARNING: ASM does not support ipclw. Switching to skgxp
WARNING: ASM does not support ipclw. Switching to skgxp
WARNING: ASM does not support ipclw. Switching to skgxp
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal) (OS id: 9343)
2018-04-02T18:31:00.187055+08:00
CLI notifier numLatches:7 maxDescs:2301
2018-04-02T18:31:00.193961+08:00
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at least 1140850688 bytes. /dev/shm is either not mounted or is mounted with available space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is 1073573888 and used is 167936 bytes. Ensure that the mount point is /dev/shm for this directory.
修改/dev/shm的大小可以透過修改/etc/fstab來實現，將/dev/shm的大小修改為12G
[root@jtp1 bin]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ol-root 49G 42G 7.9G 85% /
devtmpfs 12G 28K 12G 1% /dev
tmpfs 1.0G 164K 1.0G 1% /dev/shm
tmpfs 1.0G 9.3M 1015M 1% /run
tmpfs 1.0G 0 1.0G 0% /sys/fs/cgroup
/dev/sda1 1014M 141M 874M 14% /boot
[root@jtp1 bin]# vi /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sat Mar 18 15:27:13 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root / xfs defaults 0 0
UUID=ca5854cd-0125-4954-a5c4-1ac42c9a0f70 /boot xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults,size=12G 0 0
tmpfs /run tmpfs defaults,size=12G 0 0
tmpfs /sys/fs/cgroup tmpfs defaults,size=12G 0 0
重啟叢集后，再次檢查叢集資源狀態恢復正常
--------------------------------------------------------------------------------
[grid@jtp1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.CRS.dg
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.DATA.dg
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.FRA.dg
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.TEST.dg
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.chad
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.net1.network
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.ons
ONLINE ONLINE jtp1 STABLE
ONLINE ONLINE jtp2 STABLE
ora.proxy_advm
OFFLINE OFFLINE jtp1 STABLE
OFFLINE OFFLINE jtp2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE jtp1 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE jtp2 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE jtp2 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE jtp2 169.254.237.250 88.8
8.88.2,STABLE
ora.asm
1 ONLINE ONLINE jtp1 Started,STABLE
2 ONLINE ONLINE jtp2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE jtp2 STABLE
ora.jy.db
1 ONLINE OFFLINE STABLE
2 ONLINE OFFLINE STABLE
ora.jtp1.vip
1 ONLINE ONLINE jtp1 STABLE
ora.jtp2.vip
1 ONLINE ONLINE jtp2 STABLE
ora.mgmtdb
1 ONLINE ONLINE jtp2 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE jtp2 STABLE
ora.scan1.vip
1 ONLINE ONLINE jtp1 STABLE
ora.scan2.vip
1 ONLINE ONLINE jtp2 STABLE
ora.scan3.vip
1 ONLINE ONLINE jtp2 STABLE
--------------------------------------------------------------------------------
到此叢集恢復正常
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/31530407/viewspace-2152930/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- 修改/dev/shm大小造成Oracle 12c叢集啟動故障devOracle
- ORACLE 12C 叢集啟動順序圖Oracle
- Oracle叢集技術 | 叢集的自啟動系列（一）Oracle
- 私有IP丟失造成Oracle 12C RAC叢集節點不能啟動Oracle
- windows叢集下 ORACLE EM 的啟動問題WindowsOracle
- oracle 11gR2 grid 叢集資源設定跟隨叢集自動啟動Oracle
- ORACLE 12C 之叢集日誌位置變化Oracle
- 關於Oracle 12c的叢集監控（CHM）Oracle
- 沃趣微講堂 | Oracle叢集技術（三）：被誤傳的叢集自啟動Oracle
- Oracle 叢集的自啟動，OLR與套接字檔案Oracle
- storm叢集啟動停止指令碼ORM指令碼
- hadoop叢集配置和啟動Hadoop
- Oracle RAC日常運維-NetworkManager導致叢集故障Oracle運維
- WebSphere 叢集建立及故障排除Web
- 將12c RAC由標準叢集改為flex叢集Flex
- Oracle 12c啟動時PDBs的自動開啟Oracle
- Hadoop叢集初始化啟動Hadoop
- Sybase IQ 16叢集修改啟動埠
- Oracle RAC啟動失敗(DNS故障)OracleDNS
- oracle 12C RAC 12.1.0.2 叢集日誌（cluster log)目錄Oracle
- Hadoop叢集環境啟動順序Hadoop
- 儲存磁碟名稱不同啟動叢集
- Oracle 12C pluggable database自啟動OracleDatabase
- ORACLE RAC 11.2.0.4 FOR RHEL6叢集無法啟動的處理Oracle
- 【Redis】Redis Cluster-叢集故障轉移Redis
- redis cluster 叢集故障恢復操作思路Redis
- Oracle12c叢集啟動時提示%CRS_LIMIT_OPENFILE%: invalid numberOracleMIT
- Oracle 11gR2 RAC叢集服務啟動與關閉總結Oracle
- Oracle 11gR2 RAC 叢集服務啟動與關閉總結Oracle
- mongodb 啟動故障MongoDB
- Oracle RAC常見啟動失敗故障分析Oracle
- Oracle Data Guard 快速啟動故障切換指南Oracle
- 【RAC】使用crsctl工具控制Oracle RAC叢集的啟停Oracle
- Oracle叢集件元件Oracle元件
- oracle叢集patchingOracle
- 伺服器叢集的故障轉移方案伺服器
- mongodb叢集節點故障的切換方法MongoDB
- linux叢集故障判斷兩則（一）Linux