Database performance gets slower the longer the database is running

Applies to:
Oracle Server - Enterprise Edition - Version:
AIX5L Based Systems (64-bit)
AIX Based Systems (32-bit)
Bull Escala RL AIX (64-bit)
AIX 4.3 Based Systems (64-bit)
Oracle databases running on AIX based systems.

Symptoms
----------------
Database performance continues to get slower and slower the longer the database is left running.  You may also notice a continuing increase in the amount of paging space usage the longer the database is left running.  However, database performance returns to normal after rebooting the system, or shutting down and restarting the database.

Cause
----------
It is likely that you have not tuned the AIX Virtual Memory Manager (VMM). The default values for the AIX VMM are generally not appropriate for use with relational databases. The default values for the AIX VMM will gradually allow up to 80% of physical memory to be used to buffer file I/O. Since Oracle is already buffering file I/O in the SGA, the same data is unnecessarily being buffered twice, and leaves only 20% of physical memory to run the Oracle database(s) and all other programs.  This causes the majority of the Oracle database to be pushed out of physical memory to paging space, thus greatly impacting database performance.

The information in this article does not apply, or the impact will be much less, if you are using one or more of the following storage types for the database datafiles, because AIX does not buffer file I/O for these types:

Raw logical volumes, filesystems using the Concurrent I/O (CIO) option, filesystems using the Direct I/O (DIO) option

Note that there is no "built-in" support for CIO or DIO in Oracle Database 9iR2 (9.2.0) or lower, though you can force the use of CIO (JFS2) or DIO (JFS) with filesystem mount options.

Also note that tuning the AIX VMM is outside the scope of Oracle Support.  If you need help with checking, setting, or tuning the AIX VMM beyond what is covered in this article, you must contact your AIX systems administrator and/or IBM Support for further assistance.

Solution
--------------
To check whether your system is using the untuned default values for the AIX VMM, run the command:

/usr/sbin/vmo -a

If you do not have the /usr/sbin/vmo file you will need to have your AIX systems administrator load the AIX fileset "bos.perf.tune".  The vmo command will list out all of the VMM parameters and their current values.  The parameters you want to examine are:

MINPERM%, MAXPERM%, and MAXCLIENT%

Here is an example of the vmo report:

# vmo –a
memory_frames = 1572864
pinnable_frames = 1431781
maxfree = 1088
minfree = 960
minperm% = 20
minperm = 294356
maxperm% = 80
maxperm = 1177427
strict_maxperm = 0
maxpin% = 80
maxpin = 1258292
maxclient% = 80
lrubucket = 131072
…
The untuned default settings are MINPERM%=20%, MAXPERM%=80%, and MAXCLIENT%=80%.  There is no "correct" value for these parameters and only extensive testing will reveal the optimal values.  The optimal value may be different for different databases on the same system, so keep this in mind when tuning the VMM and choose values which work best for all of the databases you will be running on the system.  Use the following values as a starting point:

MINPERM% = 10-15%, MAXPERM% = 20-30%, MAXCLIENT% = MAXPERM%

To get a snapshot of how much physical memory is being used by AIX to buffer file I/O, run the command:

/usr/bin/svmon -G

The svmon command is part of the same AIX fileset "bos.perf.tune" that vmo belongs to.  The last line of the svmon output should be "in use".  Add the values for "in use / pers" and "in use / clnt".  Now divide the sum by the value for "memory / size".  For best database performance, this value should generally not be higher than 30% (0.30).

Here is an example of the svmon output:

# svmon -G

       size       inuse     free        pin     virtual
memory  131072   129432   1640   11704   50091
pg space 262144   100913

            work       pers      clnt  lpage
pin      11704           0         0     0
in use  47062   76126   6244     0

In this example, (in use / pers) 76126 plus (in use / clnt) 6244 equals 82370.  82370 divided by (memory / size) 131072 equals 0.628 or approximately 63% of physical memory being used by AIX to buffer file I/O.  This indicates the AIX VMM needs to be tuned to allow more physical memory to be used by Oracle and other processes, and less physical memory to be used to buffer file I/O.

Remember that although AIX associates this memory with the Oracle processes (because Oracle requested the file I/O), all of the memory used to buffer file I/O is completely allocated and controlled by AIX, not Oracle.  If you need help checking, setting, or tuning the AIX VMM, contact your AIX systems administrator and/or IBM Support.  You may also want to review the AIX "Performance Management Guide" by IBM linked in the References section below.

---

UPDATE: After this article was originally written, IBM has introduced a new VMM parameter which is also very helpful with this issue. The parameter is...

*****************
lru_file_repage
*****************
The default value is "1", but it is recommended to set this to "0". This setting hints to the VMM to only steal file pages (from the AIX file buffer cache) and leave the computational pages (from the SGA) alone.

This new lru_file_repage parameter is only available on AIX 5.2 ML04+ and AIX 5.3 ML01+

---

UPDATE2: As documented in the IBM "VMM Tuning Tip" referenced below, IBM is now recommending the following VMM settings for use with programs which need to protect computational memory (like Oracle):

maxperm%=maxclient%=(A high value, typically 90%)
minperm%=(A low value, typically 10%)
strict_maxperm=0 (default)
strict_maxclient=1 (default)
lru_file_repage=0.
------------------------------------------------------------------------
Accessibility of Links to External Web Sites

This documentation may contain links to Web sites of other companies
or organizations that Oracle Corporation does not own or control.
Oracle Corporation neither evaluates nor makes any representations
regarding the accessibility of these Web sites.
------------------------------------------------------------------------

References
-----------------
Note 123754.1 - AIX: Determining Oracle memory usage on AIX
Cover page for the AIX Performance Management Guide by IBM
Index to AIX VMM Page Replacement Tuning from AIX Performance Management Guide
Index to AIX Memory Usage from AIX Performance Management Guide
VMM Tuning Tip: Protecting Computational Memory

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/12361284/viewspace-132876/，如需轉載，請註明出處，否則將追究法律責任。