RAC因為localhost磁碟空間不夠導致has程式掛起

 

[oracle@his1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Wed Jan 15 13:12:42 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to an idle instance.
下面可以啟動資料庫找不到spfile
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+ORADATA/crds3db/spfilecrds3db.ora'
ORA-17503: ksfdopn:2 Failed to open file +ORADATA/crds3db/spfilecrds3db.ora
ORA-29701: unable to connect to Cluster Synchronization Service
下面程式下面進群主要代理程式沒有起來
CRS-4133: Oracle High Availability Services has been stopped.
[root@his2 bin]# ./crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services
[root@his2 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

從下面叢集日誌分析出因為No space left on device
2014-01-13 10:46:06.004
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A6182402

2014-01-13 10:46:48.097
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:46:47.883
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A4002844

2014-01-13 10:47:40.153
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A5325456

2014-01-13 10:47:46.072
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No 2014-01-13 10:47:52.121
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:47:48.186
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A8256687

2014-01-13 10:48:37.307
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A0629414

2014-01-13 10:49:04.149
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:49:36.161
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:49:37.729
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A3396298

2014-01-13 10:50:12.990
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A2786508

2014-01-13 10:50:13.039
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A7075711

2014-01-13 10:50:40.191
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A9099265

2014-01-13 10:51:15.827
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A8800243

2014-01-13 10:51:20.208
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:51:48.478
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A4302993

2014-01-13 10:52:24.230
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:52:40.548
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A5271303

2014-01-13 10:52:54.826
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No 2014-01-13 10:53:36.254
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:54:08.267
[ctssd(5813)]CRS-2408:The clock on host his1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-01-13 10:54:24.904
[cssd(5742)]CRS-10000:CLSU-00100: Operating System function: mkdir failed with error data: 28
CLSU-00101: Operating System error message: No space left on device
CLSU-00103: error location: authprep6
CLSU-00104: additional error information: failed to make dir /app/grid/product/11.2.0/grid/auth/css/his1/A0800143

[root@his1 ~]# /etc/init.d/init.ohasd run
[grid@his1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[grid@his1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       his1                     Started            
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       his1                                        
ora.crf
      1        ONLINE  ONLINE       his1                                        
ora.crsd
      1        ONLINE  ONLINE       his1                                        
ora.cssd
      1        ONLINE  ONLINE       his1                                        
ora.cssdmonitor
      1        ONLINE  ONLINE       his1                                        
ora.ctssd
      1        ONLINE  ONLINE       his1                     ACTIVE:0           
ora.diskmon
      1        ONLINE  ONLINE       his1                                        
ora.drivers.acfs
      1        ONLINE  ONLINE       his1                                        
ora.evmd
      1        ONLINE  ONLINE       his1                                        
ora.gipcd
      1        ONLINE  ONLINE       his1                                        
ora.gpnpd
      1        ONLINE  ONLINE       his1                                        
ora.mdnsd
      1        ONLINE  ONLINE       his1
轉載時候提示原作者或者以路徑連線方式        請您尊重筆記
技術群：132304250