In this Document

Applies to:

Symptoms

Database performance continues to get slower and slower the longer the database is left running.  You may also notice a continuing increase in the amount of paging space usage the longer the database is left running.  However, database performance returns to normal after rebooting the system, or shutting down and restarting the database.

The slowdown can appear as increased latch contention for system critical latches such as the "library cache" and "shared pool" latches.

Cause

It is likely that you have not tuned the AIX Virtual Memory Manager (VMM).  The default values for the AIX VMM are generally not appropriate for use with relational databases.  The default values for the AIX VMM will gradually allow up to 80% of physical memory to be used to buffer file I/O.  Since Oracle is already buffering file I/O in the SGA, the same data is unnecessarily being buffered twice, and leaves only 20% of physical memory to run the Oracle database(s) and all other programs.  This causes the majority of the Oracle database to be pushed out of physical memory to paging space;  thus greatly impacting database performance.

The information in this article does not apply or the impact will be much less if you are using one or more of the following storage types for the database datafiles because AIX does not buffer file I/O for these types:

Raw logical volumes, filesystems using the Concurrent I/O (CIO) or Direct I/O (DIO) option

If the database is in archive mode, and the archives are written to a filesystem that does not use CIO or DIO, then this article still applies, even if the datafiles are located on a raw logical volume or a filesystem using CIO or DIO.

If your RMAN backups are being written to a filesystem that does not use CIO or DIO, then this article still applies, even if the datafiles are located on a raw logical volume or a filesystem using CIO or DIO.

There is no "built-in" support for CIO or DIO in Oracle Database 9iR2 (9.2.0) or lower, though you can force the use of CIO (JFS2) or DIO (JFS) with filesystem mount options.

Solution

To check whether your system is using the untuned default values for the AIX VMM, run the command:

 If you do not have the /usr/sbin/vmo file, you will need to have your AIX systems administrator load the AIX fileset "bos.perf.tune".  The vmo command will list out all of the VMM parameters and their current values.  The parameters you want to examine are the following:

 Here is an example of the vmo report:

The untuned default settings are MINPERM%=20%, MAXPERM%=80%, and MAXCLIENT%=80%.  There is no "correct" value for these parameters and only extensive testing will reveal the optimal values.  The optimal value may be different for different databases on the same system, so keep this in mind when tuning the VMM and choose values which work best for all of the databases you will be running on the system.  Use the following values as a starting point:

To get a snapshot of how much physical memory is being used by AIX to buffer file I/O, run the command:

The svmon command is part of the same AIX fileset "bos.perf.tune" that vmo belongs to.  The last line of the svmon output should be "in use".  Add the values for "in use / pers" and "in use / clnt".  Now divide the sum by the value for "memory / size".  For best database performance, this value should generally not be higher than 30% (0.30).

Here is an example of the svmon output:

In this example, (in use / pers) 76126 plus (in use / clnt) 6244 equals 82370.  82370 divided by (memory / size) 131072 equals 0.628 or approximately 63% of physical memory being used by AIX to buffer file I/O.  This indicates the AIX VMM needs to be tuned to allow more physical memory to be used by Oracle and other processes and less physical memory to be used to buffer file I/O.

Remember that although AIX associates this memory with the Oracle processes (because Oracle requested the file I/O), all of the memory used to buffer file I/O is completely allocated and controlled by AIX, not Oracle.  If you need help checking, setting, or tuning the AIX VMM, contact your AIX systems administrator and/or IBM Support.  You may also want to review the AIX "Performance Management Guide" by IBM linked in the References section below.

---

UPDATE: After this article was originally written, IBM has introduced a new VMM parameter which is also very helpful with this issue. The parameter is...

The default value is "1", but it is recommended to set this to "0". This setting hints to the VMM to only steal file pages (from the AIX file buffer cache) and leave the computational pages (from the SGA) alone.

This new lru_file_repage parameter is only available on AIX 5.2 ML04 or higher and AIX 5.3 ML01 or higher.

As documented in the IBM "VMM Tuning Tip" referenced below, if you are running on AIX 5.2 ML04 or higher, or AIX 5.3 ML01 or higher, IBM is now recommending the following VMM settings for use with programs which need to protect computational memory (like Oracle):

(IBM does not recommend decreasing the maxpin% value.  See the "Support for pinned memory" link in the References section below.)

Where "percent_of_real_memory" = ( (size of SGA / size of physical memory) *100) + 3
and
Set Oracle database parameter LOCK_SGA to TRUE in the pfile/spfile
and
Sum of all SGAs on the system do not exceed approximately 60% of physical memory

------------------------------------------------------------------------
Accessibility of Links to External Web Sites

This documentation may contain links to Web sites of other companies
or organizations that Oracle Corporation does not own or control.
Oracle Corporation neither evaluates nor makes any representations
regarding the accessibility of these Web sites.
------------------------------------------------------------------------