Oracle Rac crs無法啟動

OS:
ORACLE LINUX 5.7

DB:
11.2.0.3

RAC：YES

故障：
1、兩節點RAC，節點分別為linuxdb1、linuxdb2，其中節點linuxdb2伺服器出現故障，無法啟動
2、只啟動節點linuxdb1即可，然後供測試部門使用
3、crsctl無法啟動報錯:CRS-1714:Unable to discover any voting files

#crsctl start cluster -n linuxdb1

CRS-2672: Attempting to start 'ora.cssd' on 'linuxdb1'
CRS-2672: Attempting to start 'ora.diskmon' on 'linuxdb1'
CRS-2676: Start of 'ora.diskmon' on 'linuxdb1' succeeded
CRS-2674: Start of 'ora.cssd' on 'linuxdb1' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'linuxdb1'
CRS-2681: Clean of 'ora.cssd' on 'linuxdb1' succeeded
CRS-4000: Command Start failed, or completed with errors.

檢視日誌：
[cssd(1934)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:03.698
[/u01/app/11.2.0/grid/bin/cssdagent(1920)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:52:6} in /u01/app/11.2.0/grid/log/linuxdb1/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2014-02-12 07:57:03.699
[cssd(1934)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:03.699
[cssd(1934)]CRS-1603:CSSD on node linuxdb1 shutdown by user.
2014-02-12 07:57:05.492
[cssd(1934)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:09.753
[ohasd(6518)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'linuxdb1'.
2014-02-12 07:57:21.551
[cssd(2113)]CRS-1713:CSSD daemon is started in clustered mode
2014-02-12 07:57:21.676
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:36.694
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:57:51.713
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log
2014-02-12 07:58:06.732
[cssd(2113)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/linuxdb1/cssd/ocssd.log

檢視多路徑和asmlib狀態：

/etc/init.d/oracleasm listdisks一切正常

 

檢視磁碟資訊：
#fdisk -l

Disk /dev/sdb: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      267350  2147483647+  ee  EFI GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 2.2 TB (2199023255552 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Disk /dev/sdc: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1      267350  2147483647+  ee  EFI GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 2.2 TB (2199023255552 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Disk /dev/sdd: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1      267350  2147483647+  ee  EFI GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 2.2 TB (2199023255552 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Disk /dev/sde: 2199.0 GB, 2199023255552 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1      267350  2147483647+  ee  EFI GPT

 

檢視oracleasm資訊：

# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver.  It is generated
# By running /etc/init.d/oracleasm configure.  Please use that method
# to modify this file
#

# ORACLEASM_ENABELED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=asmadmin

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="multipath sd"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE="sdf sdg sdh sdi"

* fdisk -l 的資訊與oracleasm的磁碟資訊不符

fdisk -l的磁碟為sdb sdc sdd sde

將ORACLEASM_SCANEXCLUDE="sdf sdg sdh sdi" -> ORACLEASM_SCANEXCLUDE="sdb sdc sdd sde"


重啟crs，OK

檢查狀態：

$ srvctl status asm

$ srvctl status instance -d devdb -i devdb1

$ srvctl status database -d devdb

節點linuxdb1連線資料庫一切正常。