How to Analyze Core Files for Oracle Goldengate Processes on Unix/Linux (Doc ID 1360352.1)

In this Document

Applies to:

Purpose

When troubleshooting Oracle GoldenGate (OGG) issues, especially for abend/hang problem, core file information may be collected. This Note answers the questions about how to analyze core files and what to collect from them.

Questions and Answers

How to generate a core file ?

1. If extract and/or replicat abend

A. make sure ulimit settings allow the core generation at operating system level:
"ulimit -c" will show the current setting.
"ulimit -c unlimited" will allow the core file to be generated to its maximum size.
after changing the ulimit settings, GoldenGate manager and GGSCI must be restarted to make the change effective.

B. add parameter to extract or replicat
ABORTONFATALERROR
This parameter will not affect performance under normal conditions. It will generate core dump when the extract/replicat abends abnormally.

2. When the OGG process is still running
For example, when the OGG extract is running but hanging, core file (as well as other information) may help to troubleshoot the problem.
The exact command is different for different OS and versions. Please check with the OS vendor documentation or your System Administrator for details. As well see KM note 1812.1 for more information.

A. gcore
  (e.g., in Sun Solaris, Linux, HP-UX)

B. gencore
  (e.g., in IBM AIX)

C. debug utilities (e.g., dbx, gdb)
example1: extract is running with pid 1234
cd
gdb ./extract
gdb> generate-core-file
gdb> detach
gdb> quit

example2: extract is running with pid 1234
gdb
gdb> attach 1234
gdb> dumpcore
gdb> quit

Where can the core file be found ?

The core file may be searched in following directories:

1. GoldenGate installation home

2. Directories specified by OS configuration
e.g.,
hp-ux: /etc/rc.config.d/savecrash
solaris: /etc/coreadm.conf
linux: /proc/sys/kernel/core_pattern
OS vendor and your system administrator may be contacted for further information.

3. Oracle 11g: OCI Automatic Diagnostic Repository (ADR) location
Fault diagnosability was introduced into OCI in Oracle Database 11g.
The location of the ADR base is determined by OCI in the following order:

(for more detail of ADR, see http://download.oracle.com/docs/cd/E11882_01/appdev.112/e10646/oci10new.htm#LNOCI1020)

A. OCI first looks in the file sqlnet.ora (if it exists) for a statement such as (Linux or UNIX):
ADR_BASE=/foo/adr

adr (the name of a directory) must exist and be writable by all operating system users who execute OCI applications and want to share the same ADR base. foo stands for a path name. The location of sqlnet.ora is given in the directory $TNS_ADMIN (%TNS_ADMIN% on Windows). If there is no $TNS_ADMIN then the current directory is used. If ADR_BASE is set and one sqlnet.ora is shared by all users, then OCI stops searching when directory adr does not exist or is not writable by the user. If ADR_BASE is not set, then OCI continues the search, testing for the existence of certain directories.

For example, if sqlnet.ora contains the entry ADR_BASE=/home/chuck/test then the ADR base is /home/chuck/test/oradiag_chuck and the ADR home could be something like /home/chuck/test/oradiag_chuck/diag/clients/user_chuck/host_4144260688_11.

B. $ORACLE_BASE (%ORACLE_BASE% on Windows) exists. In this case, the client subdirectory exists because it was created during installation of the database using Oracle Universal Installer.

For example, if $ORACLE_BASE is /home/chuck/obase then the ADR base is /home/chuck/obase and the ADR home could be similar to /home/chuck/obase/diag/clients/user_chuck/host_4144260688_11.

C. $ORACLE_HOME (%ORACLE_BASE% on Windows) exists. In this case, the client subdirectory exists because it was created during installation of the database using Oracle Universal Installer.

For example, if $ORACLE_HOME is /ade/chuck_l1/oracle then the ADR base is /ade/chuck_l1/oracle/log and the ADR home could be similar to /ade/chuck_l1/oracle/log/diag/clients/user_chuck/host_4144260688_11.


D. Operating system home directory: $HOME on Linux or UNIX, or %USERPROFILE% on Windows. On Linux or UNIX the location could be something like this for user chuck: /home/chuck/oradiag_chuck. On Windows, a folder named Oracle is created under C:\Documents and Settings\chuck.

For example, in an Instant Client, if $HOME is /home/chuck then the ADR base is /home/chuck/oradiag_chuck and the ADR home could be /home/chuck/oradiag_chuck/diag/clients/user_chuck/host_4144260688_11.


E. Temporary directory in the Linux or UNIX operating system: /var/tmp.
For example, in an Instant Client, if $HOME is not writable, then the ADR base is /var/tmp/oradiag_chuck and the ADR home could be /var/tmp/oradiag_chuck/diag/clients/user_chuck/host_4144260688_11.

What to do with a core file?

The core file itself is hard to read, unless the related information can be provided (from which executable is the core file, which environment is it from, ...). It should be analyzed in the same OS/setup, where it is generated.
In most cases it is not necessary to provide the core file itself, but to create the stack trace out of it.
First find out, from which executable the core is generated. This can be done e.g. with 'file', sometimes 'what' is helpful or other tools - see note 1812.1:
file core

Then, depended on the debug tools available, create a stack trace and upload it it support for analysis, if requested.

References