9I RAC驅逐案例分析－ORA-29740

In this Document

Applies to:

Purpose

Purpose: Understanding and Troubleshooting Instance Evictions.

Symptoms of an instance eviction: Instance terminates with ORA-29740, Instance Abort, Instance Kill

Troubleshooting Steps

Background

What is an instance eviction?

A RAC database has several instances.
In an instance eviction, one or more instances are suddenly aborted ("evicted").
The decision to evict these instance(s) is by mutual consensus of all the instances.

Why do instances get evicted?

Prevent problems from occuring that would affect the entire clustered database.
Evict an unresponsive instance instead of allowing cluster-wide hang to occur
Evict an instance which can't communicate with other instances to avoid a "split brain" situation - in other words, to preserve cluster consistency.

How can I tell that I have had an instance eviction?

The instance will be shut down abruptly.

In most cases, the alert log will contain this message:

In a few cases, the ORA-29740 message will not be present, and this message will show instead:

What is the most common cause of instance evictions?

The most common reason is a communications failure.
The Oracle background processes communicate with each other across the private interconnect.
If other instances cannot communicate with one instance, an instance is evicted
This is known as a Communications reconfiguration.
Chief causes of the communications failure: Network issues, OS Load issues

Key files for troubleshooting instance evictions

1. Alert log from each instance.
2. LMON trace file from each instance.

Steps to diagnose instance evictions

1. Look in the alert logs from all instances for eviction message.
2. For ora-29740, check the lmon traces for eviction reason.
3. Review alert logs for additional information.
4. Checks to carry out based on the findings of steps 1, 2, 3.

Step 1. Look in the alert logs from all instances for eviction message.

Look for the following messages in the alert log:

a) Look for ora-29740 in the alert log of the instance that got restarted.

Example:

b) If no ora-29740, look for the following messages:

In the evicted instance:

In one of the surviving instances:

This means that the instance was evicted by another instance, but its LMON did not terminate it with ora-29740.

Step 2. For ora-29740, check the lmon traces for eviction reason.

Check the lmon traces for all instances for a line with "kjxgrrcfgchk: Initiating reconfig".
This will give a reason code such as "kjxgrrcfgchk: Initiating reconfig, reason 3".

Note: make sure that the timestamp of this line is shortly before the time of the ORA-29740.
There is a reconfiguration every time an instance joins or leaves the cluster; reason 1 or 2.
So make sure that you have found the right reconfiguration in the LMON trace.

The reconfiguration reasons are:

Reason 0 = No reconfiguration
Reason 1 = The Node Monitor generated the reconfiguration.
Reason 2 = An instance death was detected.
Reason 3 = Communications Failure
Reason 4 = Reconfiguration after suspend

For ora-29740, by far the most common reconfiguration reason is Reason 3 = Communications Failure
This means that the background processes of one or more instances have registered a communication problem with each other.

In 11.2, the alert log may also print the following for a Reason 3 reconfiguration:

If you find a different reconfiguration reason, double check to make sure that you have got the right reconfiguration, ie. the last "kjxgrrcfgchk" message before the ora-29740 occurred. See Document 219361.1 for more information on the other reconfiguration reasons.

Step 3. Review alert logs for additional information.

Look for any of the following messages in the alert log of any instance, shortly before the eviction:

Example: Instance 1's alert log shows:
IPC Send timeout detected.Sender: ospid 1519
Receiver: inst 8 binc 997466802 ospid 23309

This means that Instance 1's process with OS pid 1519 was trying to send a message to Instance 8's process with OS pid 23309. Instance 1 ospid 1519 timed out while waiting for acknowledgement from Instance 8 ospid 23309

To find out which background process corresponds to each ospid, look BACKWARDS in the corresponding alert log to the PRIOR startup. The ospid's of all background processes are listed at instance startup.

Example:
Thu Apr 25 16:35:41 2013
LMON started with pid=11, OS id=15510
Thu Apr 25 16:35:41 2013
LMD0 started with pid=12, OS id=15512
Thu Apr 25 16:35:41 2013
LMS0 started with pid=13, OS id=15514 at elevated priority

Broadly speaking, there are 2 kinds of reason to see IPC send timeout messages in the alert log:
(1) Network problem with communication over the interconnect, so the IPC message does not get through.
(2) The sender or receiver process is not progressing. This could be caused by OS load or scheduling problem, or by database/process hang or blocked at DB wait level.

These messages are sometimes seen in a communications reconfiguration.

Example 1:

Example 2:

Either of these messages indicates a split-brain situation. This indicates a sustained and severe problem with communication between instances over the interconnect.

See the following note to understand split-brain further:
Document 1425586.1 - What is Split Brain in Oracle Clusterware and Real Application Cluster