Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment_948456.1
Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment (Doc ID 948456.1)
In this Document
Applies to:Oracle Database - Enterprise Edition - Version 10.1.0.2 to 11.1.0.8 [Release 10.1 to 11.1]Information in this document applies to any platform. Purpose
The note lists most known issues in 11gR2 Grid Infrastructure (in short GI) + pre-11gR2 database environment.
Refer to note 1568834.1 for pre-12.1 database issues in 12c Grid Infrastructure environment Details
1. Error creating or starting pre-11.2 database:
ORA-01501: CREATE DATABASE failed
ORA-00200: control file could not be created ORA-00202: control file: '+DG_DATA/racdb/control01.ctl' ORA-17502: ksfdcre:4 Failed to create file +DG_DATA/racdb/control01.ctl ORA-15001: diskgroup "DG_DATA" does not exist or is not mounted ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/prod/spfileprod.ora' ORA-17503: ksfdopn:2 Failed to open file +DATA/prod/spfileprod.ora ORA-15077: could not locate ASM instance serving a required diskgroup ORA-29701: unable to connect to Cluster Manager
ORA-27504: IPC error creating OSD context
Solution: To start a pre11gR2 database in 11gR2 Grid Infrastructure environment, node(s) must be pinned. To pin node(s), as root execute:
$GRID_HOME/bin/crsctl pin css -n
To find out whether node(s) is pinned or not:
$GRID_HOME/bin/olsnodes -t -n
"DBCA could not startup the ASM instance configured on this node. To processd with database creation using ASM you need the ASM instance to be up and running. Do you want to recreate the ASM instance on this node?"
oracle.sysman.assistants.util.CommonUtils.getListenerProperties(CommonUtils.java:421)
java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:151)
3. SRVCTL fails to start instance if OCR is located in an ASM diskgroup or with different permission/ownership. The racgimon log (located in $RDBMS_HOME/log/$HOST/racg/imon_${DBNAME}.log) shows the following message:
2009-10-17 11:20:22.093: [ OCROSD][7866809875]utopen:6':failed in stat OCR file/disk +DATA, errno=2, os err string=No such file or directory
2009-10-17 11:20:22.093: [ OCROSD][7866809875]utopen:7:failed to open OCR file/disk +DATA , errno=2, os err string=No such file or directory 2009-10-17 11:20:22.093: [ OCRRAW][7866809875]proprinit: Could not open raw device 2009-10-17 11:20:22.093: [ default][7866809875]a_init:7!: Backend init unsuccessful : [26] .. 2009-10-17 11:20:22.094: [ CSSCLNT][7866809875]clsssinit: Unable to access OCR device in OCR init.PROC-26: Error while accessing the physical storage OperatingSystem error [No such file or directory] [2] 2009-10-17 11:20:22.094: [ RACG][7866809875] [23974][7866809875][ora.default]: racgimon exiting clsz init failed
2009-11-05 14:31:19.922: [ A**][1342593344] Agent received the message: RESOURCE_START[ora.db10.db102.inst 1 1] ID 4098:632
.. 2009-11-05 14:31:19.924: [ A**][1275476288] Executing command: start for resource: ora.db10.db102.inst 1 1 2009-11-05 14:31:19.924: [ora.db10.db102.inst][1275476288] [start] START action called. 2009-11-05 14:31:19.924: [ora.db10.db102.inst][1275476288] [start] Executing action script: /home/app/oracle/product/10.2/db/bin/racgwrap[start] .. 2009-11-05 14:31:22.781: [ora.db10.db102.inst][1275476288] [start] Enter user-name: Connected to an idle instance. 2009-11-05 14:31:22.781: [ora.db10.db102.inst][1275476288] [start] 2009-11-05 14:31:22.782: [ora.db10.db102.inst][1275476288] [start] SQL> ORA-01565: error in identifying file '+DATA/db10/spfiledb10.ora' 2009-11-05 14:31:22.782: [ora.db10.db102.inst][1275476288] [start] ORA-17503: ksfdopn:2 Failed to open file +DATA/db10/spfiledb10.ora 2009-11-05 14:31:22.782: [ora.db10.db102.inst][1275476288] [start] ORA-15077: could not locate ASM instance serving a required diskgroup 2009-11-05 14:31:22.782: [ora.db10.db102.inst][1275476288] [start] 2009-11-05 14:31:22.782: [ora.db10.db102.inst][1275476288] [start] ORA-01078: failure in processing system parameters
$GRID_HOME/bin/crsctl stat res -t
.. ora.DATA.dg OFFLINE OFFLINE racnode1 OFFLINE OFFLINE racnode2 ..
$RDBMS_HOME/bin/srvctl start service -d b1 -s sb1
$GRID_HOME/bin/crsctl stat res .. NAME=ora.b1.sb1.cs TYPE=application TARGET=ONLINE STATE=ONLINE on eyrac1f NAME=ora.b1.sb1.b11.srv TYPE=application TARGET=OFFLINE STATE=OFFLINE NAME=ora.b1.sb1.b12.srv TYPE=application TARGET=OFFLINE STATE=OFFLINE NAME=ora.eons TYPE=ora.eons.type TARGET=ONLINE , ONLINE STATE=ONLINE on eyrac1f, ONLINE on eyrac2f
A. Create a dummy pre11.2 resource entry alphabetically after service entry. In the example above, creating a dummy resource ora.b2.db should workaround the problem
$RDBMS_HOME/bin/srvctl add database -d b2 -o $RDBMS_HOME B. Try to start all services for the database: $RDBMS_HOME/bin/srvctl start service -d b1
Note that fix for bug 8373758 does not include fix for bug 8262786.
ORA-15025: could not open disk '/dev/rdsk/disk1'
ORA-27041: unable to open file SVR4 Error: 13: Permission denied
$GRID_HOME/bin/setasmgidwrap o=/home/oracle/10.2/bin/oracle
KFSG-00312: not an Oracle binary: '/home/oracle/10.2/bin/oracle'
CRS-5809: Failed to execute 'ACTION_SCRIPT' value of '/ocw/crs10/bin/racgwrap' for 'ora.db10.db'. Error information 'cmd /ocw/crs10/bin/racgwrap not found'
CRS-2680: Clean of 'ora.db10.db' on 'node1' failed
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'node1' has failed
CRS-2675: Stop of 'ora.crsd' on 'node1' failed CRS-4000: Command Stop failed, or completed with errors.
A. As pre-11.2 database owner, execute the following command for each pre-11.2 database: crsctl modify res ora. Or B. As pre-11.2 database owner, recreate database resource in OCR with note 1069369.1
$DBHOME/bin/srvctl config service -d racstr
rac_u PREF: racstr1 racstr2 AVAIL: rac_s PREF: racstr1 AVAIL: racstr2 $DBHOME/bin/srvctl status service -d racstr Service rac_u is running on instance(s) racstr1, racstr2 Service rac_s is running on instance(s) racstr1 $GRID_HOME/bin/crsctl status res ora.strdt01.vip NAME=ora.strdt01.vip TYPE=ora.cluster_vip_net1.type TARGET=ONLINE STATE=ONLINE on strdt01 Disable public network on node where instance racstr1 is running, VIP failover to another node: $GRID_HOME/bin/crsctl status res ora.strdt01.vip NAME=ora.racha602.vip TYPE=ora.cluster_vip_net1.type TARGET=ONLINE STATE=INTERMEDIATE on racha603 <== Vip failover to other node $DBHOME/bin/srvctl status service -d racstr Service rac_u is running on instance(s) racstr1, racstr2 <== Service still running on racstr1 Service rac_s is running on instance(s) racstr1 <== Service did not failover to racstr2
PRKO-2010 : Error in adding instance to node: node1
PRKR-1008 : adding of instance dba21 on node node1 to cluster database dba2 failed. CRS-2518: Invalid directory path '/home/oracle/product/11.1/db/bin/racgwrap' CRS-0241: Invalid directory path
$ oifcfg getif
eth3 120.0.0.0 global public eth1 10.1.0.0 global cluster_interconnect Instance alert.log: ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:skgxnqtsz failed with ! status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: SKGXN not av clsssinit ret = 21 interconnect information is not available from OCR WARNING: No cluster interconnect has been specified. Depending on the communication driver configured Oracle cluster traffic may be directed to the public interface of this machine. Oracle recommends that RAC clustered databases be configured with a private interconnect for enhanced security and performance.
For 11gR2 GI + pre-10.2.0.5 database environment, it's not recommended to place OCR on ASM. Other workaround is to set init.ora parameter cluster_interconnects and ignore the warning.
$ srvctl
PRKA-2019 : Error executing command "/ocw/b201/bin/crs_stat". File is missing. $ ocrdump -stdout -keyname SYSTEM.ORA_CRS_HOME [SYSTEM.ORA_CRS_HOME] ORATEXT : /ocw/b201
# ${11.2.0.2GI_HOME}/bin/clscfg -upgrade -lastnode -g
Note:
2011-03-14 23:22:52.228: [ RACG][1108842816] [14693][1108842816][ora.SD302.SD3021.inst]: CLSR-0521: Event ora.ha.racdb.racdb1.inst.down is rejected by EVM daemon
2011-03-14 23:22:52.228: [ RACG][1108842816] [14693][1108842816][ora.SD302.SD3021.inst]: clsrcepevm: clsrcepostevt status = 17 2011-03-14 23:22:52.228: [ RACG][1108842816] [14693][1108842816][ora.SD302.SD3021.inst]: clsrcep:evm post return 1 2011-03-14 23:22:54.458: [ RACG][1108842816] [14693][1108842816][ora.SD302.SD3021.inst]: CLSR-0521: Event sys.ora.clu.crs.app.trigger is rejected by EVM daemon 2011-03-14 23:23:06.495: [ RACG][1108842816] [14693][1108842816][ora.SD302.SD3021.inst]: clsrcexecut: env _USR_ORA_PFILE=/ocw/grid/racg/tmp/ora.racdb.racdb1.inst.ora 2011-03-14 23:23:06.495: [ RACG][1108842816] [14693][1108842816][ora.SD302.SD3021.inst]: clsrcexecut: cmd = /database/db205/bin/racgeut -e _USR_ORA_DEBUG=0 -e ORACLE_SID=racdb1 540 /database/db205/bin/racgimon stop ora.racdb.racdb1.inst
2011-02-16 06:13:55.965: [ EVMAPP][4163668704] EVMD Started
.. 2011-02-16 06:13:55.980: [ EVMD][4163668704] Could not open /ocw/grid/evm/admin/conf/evmdaemon.conf Reconfiguration aborted.
ls -l $GRID_HOME/evm/admin/conf
-rw------- 1 root root 3032 Feb 19 14:42 evm.auth -rw------- 1 root root 2318 Feb 19 14:42 evmdaemon.conf -rw------- 1 root root 4871 Feb 19 14:42 evmlogger.conf
ls -l $GRID_HOME/evm/admin/conf
-rw-r--r-- 1 root root 3032 Feb 19 14:42 evm.auth -rw-r--r-- 1 root root 2318 Feb 19 14:42 evmdaemon.conf -rw-r--r-- 1 root root 4871 Feb 19 14:42 evmlogger.conf
INFO: Read: PRCR-1079 : Failed to start resource ora.asm
INFO: Read: CRS-2672: Attempting to start 'ora.asm' on 'racnode2' INFO: Read: CRS-5017: The resource action "ora.asm start" encountered the following error: INFO: Read: ORA-29702: error occurred in Cluster Group Service operation INFO: Read: . For details refer to "(:CLSN00107:)" in "/ocw/grid/log/racnode2/agent/ohasd/oraagent_oracle/oraagent_oracle.log". INFO: Read: INFO: Read: CRS-2674: Start of 'ora.asm' on 'racnode2' failed
2011-02-25 15:27:55.803: [ A**][1173416256] {0:8:64} A** Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:214148
2011-02-25 15:27:55.804: [ A**][1173416256] {0:8:64} Received state change for ora.db111.db1111.inst 1 1 [old state = ONLINE, new state = UNKNOWN] 2011-02-25 15:27:55.804: [ A**][1173416256] {0:8:64} Received state LABEL change for ora.db111.db1111.inst 1 1 [old label = , new label = CHECK TIMED OUT]
2011-02-25 15:17:55.791: [ora.db111.db1111.inst][1111284032] {3:47926:62288} [check] Check action called..
2011-02-25 15:17:55.791: [ora.db111.db1111.inst][1111284032] {3:47926:62288} [check] CHECK action called. 2011-02-25 15:17:55.791: [ora.db111.db1111.inst][1111284032] {3:47926:62288} [check] Executing action script: /db/pre11.2/bin/racgwrap[check] .. 2011-02-25 15:27:55.800: [ A**][1093445952] {3:47926:62288} Created alert : (:CRSAGF00113:) : Aborting the command: check for resource: ora.db111.db1111.inst 1 1 2011-02-25 15:27:55.800: [ora.db111.db1111.inst][1093445952] {3:47926:62288} [check] Abort operation called for command: CHECK 2011-02-25 15:27:55.800: [ora.db111.db1111.inst][1093445952] {3:47926:62288} [check] Killing action script: check 2011-02-25 15:27:55.800: [ A**][1093445952] {3:47926:62288} Command: check for resource: ora.db111.db1111.inst 1 1 completed with status: TIMEDOUT 2011-02-25 15:27:55.801: [ A**][1115486528] {3:47926:62288} ora.db111.db1111.inst 1 1 state changed from: ONLINE to: UNKNOWN 2011-02-25 15:27:55.801: [ A**][1115486528] {3:47926:62288} ora.db111.db1111.inst 1 1 would be continued to monitored! 2011-02-25 15:27:55.801: [ A**][1115486528] {0:8:64} Generating new Tint for unplanned state change. Original Tint: {3:47926:62288} 2011-02-25 15:27:55.801: [ A**][1115486528] {0:8:64} Agent sending message to PE: RESOURCE_STATUS[Proxy] ID 20481:214148 2011-02-25 15:27:55.822: [ora.db111.db1111.inst][1111284032] {3:47926:62288} [stop] Stop action called.. 2011-02-25 15:27:55.822: [ora.db111.db1111.inst][1111284032] {3:47926:62288} [stop] STOP action called. 2011-02-25 15:27:55.822: [ora.db111.db1111.inst][1111284032] {3:47926:62288} [stop] Executing action script: /db/pre11.2/bin/racgwrap[stop]
2011-02-25 15:39:17.866: [ RACG][1141213504] [27191][1141213504][ora.db111.db1111.inst]: timeout: killed the spawned process
2011-02-25 15:39:17.867: [ RACG][1141213504] [27191][1141213504][ora.db111.db1111.inst]: clsrcexecut: cmd = /db/pre11.2/bin/racgeut -e _USR_ORA_DEBUG=0 -e ORACLE_SID=db1111 540 /db/pre11.2/bin/racgimon stop ora.db111.db1111.inst
..
Shutting down instance: further logons disabled .. Shutting down instance (immediate) License high water mark = nnnn
Pre-11.2 database doesn't start after clusterware/ASM upgraded to 11.2:
srvctl start database -d racdb1 -o mount
PRCR-1079 : Failed to start resource ora.racdb1.db CRS-5017: The resource action "ora.racdb1.db start" encountered the following error: ORA-01078: failure in processing system parameters ORA-01565: error in identifying file '+DATA/racdb1/spfileracdb1.ora' ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb1/spfileracdb1.ora ORA-15077: could not locate ASM instance serving a required diskgroup ORA-29701: unable to connect to Cluster Manager
ALTER DATABASE MOUNT
Solution: As per the following documentation, it's unsupported to have OS hostname longer than 8 characters on hp-ux Itanium: http://download.oracle.com/docs/cd/B28359_01/relnotes.111/b32071/toc.htm#CHDEHEIB The solution is to deconfig and deinstall clusterware, make hostname shorter, install, config clusterware and register databases. Refer to note 942166.1 for details to deconfig, deinstall steps.
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/racdb/spfileracdb.ora' ORA-17503: ksfdopn:2 Failed to open file +DATA/racdb/spfileracdb.ora ORA-15055: unable to connect to ASM instance
Solution:
This is due to bug 12660458 which has been fixed in 11.2.0.3 and above, interim patch 12660458 exists for certain platform/version
$
24. Pre-11.2 database fails to start if ORA_NLS10 is set wrongly: If it happens while starting existing database, sqlplus startup fails:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA1/RACDB/PARAMETERFILE/spfile.722.724675221' ORA-17503: ksfdopn:2 Failed to open file +DATA1/RACDB/PARAMETERFILE/spfile.722.724675221 ORA-15056: additional error message ORA-17503: ksfdopn:2 Failed to open file +DATA1/RACDB/parameterfile/spfile.722.724675221 ORA-03113: end-of-file on communication channel Process ID: Session ID: 4393 Serial number: 1 ORA-06512: at line 4 Solution: The cause is environment variable ORA_NLS10 has invalid setting, once it's unset, database is able to come up
The answer is no, refer to note 1484800.1 for details 26. Pre-11.2 DB creation fails with ORA-15045, ORA-17502, ORA-15081 when storage is on ASM Refer to note 1384180.1 for solution
27. Pre-11.2 service does not failover upon instance crash? This is due to bug 13879428, refer to note 1489751.1 for details.
28. Pre-11.2 DB is slow after Single Instance ASM is upgraded to 11.2.0.2 or 11.2.0.3
This is due to bug 13868167, when this happens, many css*.log is created in "$ORACLE_HOME/log/
2012-01-02 03:04:05.678: [ CSSCLNT][1]clsssinit: error(22 PROC-22: The OCR back end has an invalid format) in OCR initialization
Bug 13868167 is fixed in 11.2.0.4, refer to note 1444483.1 for workaround.
29. Pre-11.2 service may not failover in 11gR2 GI
$ srvctl relocate service -d TEST -s service_name -i test2 -t test1 This is due to misconfiguration, refer to note 1539624.1 for more details.
ReferencesNOTE:1064804.1 - Apply Grid Infrastructure/CRS Patch in Mixed Version RAC Database EnvironmentNOTE:810663.1 - 11.1.0.X CRS Bundle Patch Information NOTE:1069369.1 - How to Delete From or Add Resource to OCR in Oracle Clusterware NOTE:405820.1 - 10.2.0.X CRS Bundle Patch Information NOTE:1539624.1 - Srvctl Relocate (10.2) Service in 11.2 Grid Infrastructure Fails With Reported error message PRKP-1033 : Cannot relocate service |
|
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/17252115/viewspace-1409678/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- Database Creation on 11.2 Grid Infrastructure with Role SeparationDatabaseASTStruct
- How to Troubleshoot Grid Infrastructure Startup IssuesASTStruct
- 【MOS】Top 5 Grid Infrastructure Startup Issues (文件 ID 1368382.1)ASTStruct
- How to Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]ASTStruct
- Troubleshoot Grid Infrastructure Startup Issues (Doc ID 1050908.1)ASTStruct
- 重新配置 11gR2 Grid InfrastructureASTStruct
- 【RAC】How to Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]ASTStruct
- 【MOS】How to backup or restore OLR in 11.2/12c Grid InfrastructureRESTASTStruct
- oracle linux 11.2 rac grid infrastructure add scan ipOracleLinuxASTStruct
- redhat linux 11.2 rac grid infrastructure add scan ipRedhatLinuxASTStruct
- backup or restore OLR in 11.2 Grid Infrastructure (Doc ID 1193643.1)RESTASTStruct
- 【轉】How to recover from root.sh on 11.2 Grid Infrastructure FailedASTStructAI
- 為11gR2 Grid Infrastructure增加新的public網路ASTStruct
- 重新配置與解除安裝11gR2 Grid InfrastructureASTStruct
- zt_Oracle 11gR2 grid infrastructure安裝配置系列OracleASTStruct
- 重新配置與解除安裝 11gR2 Grid InfrastructureASTStruct
- 11g oracle database installation with oracle grid infrastructure on linux(文件)OracleDatabaseASTStructLinux
- 11gR2中安裝Grid Infrastructure出現的IPMI簡介ASTStruct
- Oracle Grid Infrastructure for a Standalone ServerOracleASTStructServer
- clone grid INfrastructure Home and clusterwareASTStruct
- Oracle Clusterware and Oracle Grid InfrastructureOracleASTStruct
- Oracle grid infrastructure 解除安裝OracleASTStruct
- 常見的 11gR2 Grid Infrastructure 升級問題 (文件 ID 1602048.1)ASTStruct
- DNS and DHCP Setup Example for Grid Infrastructure GNSDNSASTStruct
- 記錄下 patch Grid Infrastructure for StandaloneASTStruct
- 安裝 11gR2 Grid Infrastructure(CRS)失敗的處理過程 (文件 ID 1946678.1)ASTStruct
- Oracle Grid Infrastructure Patch Set Update 11.2.0.4.3OracleASTStruct
- Troubleshooting Database Control Startup IssuesDatabase
- Linux/Unix 平臺上升級到 11gR2 Grid Infrastructure(GI) 出錯的處理方法LinuxASTStruct
- oracle 11gR2 新增deinstall命令來解除安裝grid和databaseOracleDatabase
- rhel 5.5 下安裝11gR2 Rac 的 Grid infrastructure 時關於ntp的一點強調ASTStruct
- Troubleshooting Database Hang Issues (Doc ID 1378583.1)Database
- Top 5 Database and/or Instance Performance Issues in RAC EnvironmentDatabaseORM
- Apply PSU for Grid Infrastructure Standalone and DB with Oracle RestartAPPASTStructOracleREST
- 升級Grid Infrastructure到10.2.0.2 遭遇bug 9413827ASTStruct
- Database and/or Instance Performance Issues in RAC Environment_1373500.1DatabaseORM
- How to Collect Diagnostics for Database Hanging Issues_452358.1Database
- 【GRID】Grid Infrastructure 啟動的五大問題 (Doc ID 1526147.1)ASTStruct