How to Restore CRS after accidentally run localconfig on RAC system_747415.1

rongshiyuan發表於2014-05-08

How to Restore CRS after accidentally run localconfig on RAC system (Doc ID 747415.1)

In this Document

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.1 to 11.1.0.7 [Release 10.2 to 11.1]
Information in this document applies to any platform.
This could affect any system on 10g

Goal

Localconfig command is used for single instance RDBMS to config cssd which support ASM.
On RAC system, cssd is part of CRS, ASM will be supported once CRS is installed and running.
There is absolutely no requirement to run localconfig on RAC system and it should never be used on RAC system as it will destroy CRS configuration.

In this note, we will discuss a few scenarios related with localconfig command being run on RAC system, include "localconfig delete" and "localconfig reset" and how to restore CRS configuration if localconfig script accidentally run on RAC system for any reason.

Solution

Case I. "$ORACLE_HOME/bin/localconfig delete" has been done on some node(s) but not all of RAC system

Symptoms:
no /etc/oracle/ocr.loc (or /var/opt/oracle/ocr.loc) exist on affected node(s)

The OCR has been removed in case it stands on a clustered file system. The OCRMIRROR still exist in case it has been defined.

Scenario A. If both OCR and OCRMIRROR are on cluster file system

In this scenario, the OCRMIRROR can be used to recreate the OCR. As root user:

1. find the OCR location

grep "CRS_OCR_LOCATIONS=" /install/rootconfig |head -1

2. touch and set correct ownership and permission, this can be done on either node, for example:

touch /cfs/ocrdisk1
chown root:oinstall /cfs/ocrdisk1
chmod 640 /cfs/ocrdisk1

3. on one unaffected node, replace the ocr device:
ocrconfig -replace ocr

ocrconfig -replace ocr /cfs/ocrdisk1

4. on affected node(s) where "localconfig delete" has been run, as root user, rerun root.sh:

/root.sh

CRS will startup automatically after this.

Scenario B. If only OCR is used, no OCRMIRROR has been defined and it is on cluster file system

As the result of localconfig delete, crsd.bin will go down on the good node(s) as well, crsd.log reports:

[ OCRAPI][1225398592]procr_ctx_set_invalid_no_abort: ctx set to invalid
[ OCRAPI][1225398592]procr_ctx_set_invalid: aborting...

The OCR need to be recreated from backup.

1. find OCR location

grep "CRS_OCR_LOCATIONS=" /install/rootconfig |head -1

2. touch and set correct ownership and permission

touch /cfs/ocrdisk1
chown root:oinstall /cfs/ocrdisk1
chmod 640 /cfs/ocrdisk1

3. look for latest OCR backup generated automatically

ocrconfig -showbackup

4. restore an OCR backup, eg:

ocrconfig -restore /u01/oracle/product/10.2.0/crs/cdata/crs/backup00.ocr

Once OCR restored, crsd.bin should start automatically on the good node(s).
5. on bad node(s), rerun /root.sh as root user.

/root.sh

CRS will startup automatically after this.

Scenario C. If OCR or OCRMIRROR is on raw device

localconfig delete will only remove the OCR device name (eg: /dev/raw/raw1), it will not remove the content of raw device.
1. restore the device name
on Linux: as root, run:

service rawdevice restart

Please check with other OS vendor for restore the device name, or a reboot will restore the name too.
2. on bad node(s), rerun /root.sh as root user

/root.sh

CRS will startup automatically after this.

Case II. "$ORACLE_HOME/bin/localconfig reset" has been done on 1 or all nodes of RAC system

Symptom:
On the node(s) where "localconfig reset" has been run:
1. /etc/oracle/ocr.loc is like:
ocrconfig_loc=/cdata/localhost/local.ocr
local_only=TRUE

2. in /etc/inittab, following lines exist:
h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 h1:35:respawn:/etc/init.d/init.cssd run >/dev/null 2>&1 CRS can not start on the affected node. Reboot will not help.

Solution: on all node(s) where "localconfig reset" has been done:

Caution! proceed with followings only when /etc/oracle/ocr.loc has not been manually modified to point back to real OCR. Eg, its content should still be:

ocrconfig_loc=/cdata/localhost/local.ocr
local_only=TRUE

If you have modified ocr.loc to point to real OCR device, please restore the content back to above before continue with the followings. Otherwise the following commands will zero out the real OCR device.

as root user:
1. run /install/rootdelete.sh, it will remove the init* scripts and place back the blank inittab

/install/rootdelete.sh

2. run /install/rootdeinstall.sh, it will blank out the $ORACLE_HOME/cdata/localhost/local.ocr and remove the ocr.loc

/install/rootdeinstall.sh

3. run /root.sh, CRS should start automatically after this.

/root.sh

For all other cases, either restore the OCR from backup or follow Note 399482.1 How to recreate OCR/Voting disk accidentally deleted to reinitialize OCR and voting disks.