How to enable Large Page Feature on AIX-Based Systems [ID 372157.1]

In this Document
  Goal
  Solution
  References

Applies to:

Goal

How to enable Large Page Feature on AIX-Based Systems ?

See:

Oracle Database Administrator's Reference
10g Release 2 (10.2) for UNIX-Based Operating Systems
Part Number B15658-05

-or-

Oracle Database Administrator's Reference
11g Release 1 (11.1) for Linux and UNIX-Based Operating Systems
Part Number B32009-05

-or-

Oracle Database Administrator's Reference
11g Release 2 (11.2) for Linux and UNIX-Based Operating Systems
Part Number E10839-12

Solution

Large page memory

Starting with the AIX V5.1 operating system when running on IBM POWER4 or POWER5 processors, pSeries systems support two virtual page sizes: 4 kilobytes (standard page) and 16 megabytes (large page). When using large pages to map virtual memory, the translation look aside buffer (TLB) is able to map more virtual memory with a given number of entries, resulting in a lower TLB miss rate for applications that use a large amount of virtual memory. Additionally, when using large pages, there are fewer page boundaries, which improve the performance of prefetching. 

Both online transaction processing (OLTP) and data warehouse environments can benefit from using large pages. 

For applications such as Oracle, which typically uses a large amount of virtual memory, using large page memory will generally result in improved performance. There are three types of memory with which Oracle can use large pages: shared memory (SGA), process data (.data), and the instruction text (.text). In most Oracle applications, the SGA dominates the virtual memory usage and, consequently, most of the benefit of using large pages is achieved by using them for the SGA. In some special Oracle applications, using large pages for .data and .text can provide some additional performance benefit. However, because of the granularity of allocation with large pages, each Oracle process might have a larger memory footprint. Large page text and data is supported on AIX  and only for 64-bit processes and the 64-bit AIX kernel. For most applications, the recommendation is to use large pages only for the SGA, and not for .text or .data. 

Starting with Oracle 9i, when the Oracle initialization parameter LOCK_SGA is set to TRUE, Oracle will request large pages when allocating shared memory (shmget() call has SHM_LGPAGE flag set). For the AIX operating system to use large pages when allocating shared memory, the Oracle user ID must have CAP_BYPASS_RAC_VMM and CAP_PROPAGATE capabilities. Also, the AIX large page pool must be configured (as shown in the example below). When using large pages on an Oracle Real Application Cluster (RAC) database, where the srvctl command is used to start and stop the RAC database instances, it is also necessary to set the CAP_BYPASS_RAC_VMM and CAP_PROPAGATE capabilities for the root user ID. 

Large pages are always pinned, and large page memory cannot be used for standard memory. If a pool of large page memory is configured, then this memory will be unusable for allocation of standard memory even if no other application is currently using large pages. 

Example configuration for using large pages for the Oracle SGA 
============================================================== 

Give the Oracle user ID the CAP_BYPASS_RAC_VMM and CAP_PROPAGATE capabilities by following these steps: 

1. First check the current capabilities: # lsuser -a capabilities oracle 

Note: only the root user can display the capabilities attribute. 

2. Add the CAP_BYPASS_RAC_VMM and CAP_PROPAGATE capabilities to the list of capabilities already assigned to this user ID, if any: 
#chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle 

3. Configure the AIX large page pool by calculating the number of large pages required for the SGA: 
num_of_large_pages = INT((total_SGA_size-1)/16MB)+1 

#vmo -o lru_file_repage=0

Note: If a reboot is required on the system, because of changing the large page configuration, this will be 
indicated in the output of the vmo command. 


Example configuration for using large pages for .text and .data 
=============================================================== 

These configuration steps are in addition to the steps required for using large page SGA: 

1. Calculate the additional large pages required for .text and .data. 

2. Start the Oracle instance and execute the following command as root: 

#svmona-P
for example:
#svmon -P 552974
------------------------------------------------------------------------------ 
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage 
552974 oracle 95447 4012 0 86224 Y N N 
Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual 
181958 70000000 work default shmat/mmap - 58243 0 0 58243 
1e197e 70000001 work default shmat/mmap - 12663 0 0 12663 
1a17da 10 clnt text data BSS heap - 9212 0 - - 
0 0 work kernel segment - 7821 3990 0 7821 
d08ad 90000000 work loader segment - 4776 0 0 4776 
1f1a7f 11 work text data BSS heap - 1924 0 0 1924 
121272 90020014 work shared library text - 375 0 0 375 
21a82 8001000a work private load - 138 0 0 138 
1a40 9001000a work shared library text - 113 0 0 113 
d1a6d 80020014 work private load - 107 0 0 107 
: 
: 

The row with an Esid of 10 corresponds to the .text segment, and the row with an Esid of 11 corresponds to the .data segment. When checking the .text segment for all the Oracle processes, notice that they all have the same Vsid. This is because they share this read-only segment. The .data section is private to each Oracle process so that the total large page requirement is dependent on the number of foreground and background Oracle processes when using large page data. The column labeled Inuse contains the number of four kilobyte pages in that segment. 

Use the following formula to calculate the number of additional pages for large page .text and .data: 

num_of_additional_pages = INT((.text_pages+4095)/4096) + N*(INT(.data_pages+4095)/4096) 

Using the svmon example output and assuming 25 foreground and background processes (N=25), then: 
num_of_additional_pages = INT((9212+4095)/4096) + 25*(INT((1924+4095)/4096)) = 28 

Add this number to the large pages needed for the SGA and execute vmo: 
#vmo -p -o lgpg_regions=num_of_large_pages+num_of_additional_pages -o lgpg_size=16777216 

Edit the XCOFF file header in the oracle binary to enable it to use large page data: 
# ldedit -b lpdata $ORACLE_HOME/bin/oracle

Prior to starting the Oracle instance and listener, export LDR_CNTRL in the Oracle user ID used to start the instance and listener: 
#export LDR_CNTRL=LARGE_PAGE_TEXT=Y@LARGE_PAGE_DATA=M 


Note: This must be set in addition to editing the XCOFF file header to allow both large page text and large page data. Setting the LARGE_PAGE_DATA=M option allocates only enough large pages for the data segment up to the brk value. The @ character is used to separate the definition of multiple values to the LDR_CNTRL variable. 


Tools to monitor memory usage 
============================= 

This section discusses tools to monitor memory usage, such as t and svmon commands. 

vmstat 

The vmstat command also aids with monitoring memory usage. 
Use the vmstat -l command the and free large pages. 

$ vmstat -l 
System configuration: lcpu=2 mem=7424MB 
kthr memory page faults cpu large-page 
----- ----------- ------------------------ ------------ ----------- ----------- 
r b avm fre re pi po fr sr cy in sy cs us sy id wa alp flp 
1 1 433591 338295 0 0 0 0 0 0 85 382 340 0 0 99 0 0 0 

In this vmstat output, the alp column indicates the number of allocated large pages, and the flp column is the number of free large pages. 

svmon 

The svmon command is useful for looking at the memory used by a process or a group of processes. The -P option used in the Large page memory section shows the memory usage for a specific process. The -C oracle option shows the memory usage for all of the Oracle processes running on the system. Only root authority can use the svmon command. 

IMPORTANT NOTES:

1) Patch 7226548 is available to resolve the issue on 10.2.0.4 with large pages not being used, 
and therefore the workaround is no longer needed. 

2) This note applies to AIX5L and AIX 6.1.

3) This is an AIX OS specific configuration and because large pages may be automatically configured in newer versions of AIX, you should always refer to AIX specific documentation or contact IBM for detailed information.

References