Applies to: 

Goal

Interprocess communication (IPC) is the sharing of data and synchronization of events among processes. Shared memory provides efficient data sharing among multiple processes since data are not required to be moved across multiple process address spaces. Shared memory allows sharing of physical memory pages by multiple processes. That means multiple processes can attach or have mappings to the same physical memory segment. Access to shared memory can be performed by simple pointer dereference in the code. Oracle uses shared memory to cache frequently used data blocks (buffer cache) and to facilitate communication (shared pool) among oracle processes (pmon, smon, dbwr, lgwr, and oracle shadow proceses). Semaphores are used to control concurrency between processes when data in shared memory needs to be modified. See detail.

Oracle Database uses shared memory extensively, Misconfiguration of shared memory during Oracle Database setup can result in lower than expected performance. The goal of this document is to identify those symptoms that lead to slower Oracle Database performance and present solutions to overcome these issues.

Solution

Type of Shared Memory

Solaris supports three types of Shared Memory:

• Pageable Shared Memory: This is the same as any other anonymous memory (heap, stack, copy-on-write). Shared memory segment is allocated via shmget(2). Attaching to the shared memory segment via shmat() system call without passing  SHM_SHARED_MMU or SHM_PAGEABLE flags results in pageable mapping to the shared memory segment. Page fault into the shared memory segment results in virtual to physical memory mapping. All pages residing in the shared memory segment have a backing store, to which the pages within the segment can be migrated during memory shortages.  Similar to stack, heap, COW (copy-on-write), shared memory pages are anonymous (Anon) pages and thus they are backed up by virtual swap. These pages can be swapped out during memory shortages which can have a negative effect on application performance. Also, the kernel needs to acquire locks on pages when syncing data cached in shared memory segment. This can induce high lock contention on large SMP systems.

• Intimate Shared Memory (ISM): ISM is an optimized shared memory that deals with the shortcoming of pageable shared memory. Shared memory segment is allocated via shmget(2).  Attaching to the shared memory via shmat() system call with SHM_SHARE_MMU flags results in a ISM segment. Unlike non-ISM memory, ISM memory is automatically locked by the kernel and does not require disk swap reservation. This ensures that ISM pages are not paged out during memory shortages. Also, IO performed on pages in ISM segment uses a fast locking mechanism that reduces kernel lock contention and in turn CPU kernel overhead. ISM offers several additional features such as: Large Page support (page sizes of 4M-2G are supported depending on the platform, see pagesize(1)) to improve TLB reach and cache hits; Low memory footprints due to shared virtual-to-physical translation tables (or mapping structures) for sharing memory pages amongst multiple processes; and a reduction in the number of memory pointers needed by applications due to the larger pages. This reduction in complexity translates into noticeable performance improvements, especially on systems with large amounts of memory. [It is important to note that shmget(2) requires swap reservation when it allocates shared memroy segment. It is only after shmat() is called with SHM_SHARED_MMU that the swap reservation against disk is released and switched to swap reservation against memory when pages are locked in memory]
  

• Dynamic Intimate Shared Memory (DISM): DISM is similar to ISM, but it can be dynamically resized depending on the application demand. Shared memory segment is allocated via shmget(2). Attaching to the shared memory via shmat() system call with SHM_PAGEABLE flags results in a DISM segment. With ISM, it is not possible to change the size of the segment once it has been created. For example, Oracle Database needs to be restarted if the buffer cache size needs to be increased. DISM was introduced as a RAS feature that allows Oracle Database to deal with dynamic reconfiguration events such as adding or removing memory from the system. With this feature, a large DISM segment can be created when the database is started (see doc: Dynamic SGA). The database can selectively lock and unlock sections of this segment as memory requirements change. Unlike ISM, where the kernel locks the shared memory segment, responsibility for locking and unlocking (mlock(3C)) is placed on the database or application.  This provides flexibility to adjust memory requirements dynamically. Once DISM is locked properly by the database, it behaves in the same way as ISM, both in functionality and performance. That means availability benefits of DISM can be realized without compromising performance.

Oracle SGA and PGA Memory

SGA:The SGA (System Global Area) is a group of shared memory structures, known as SGA components, that contain data and control information for one Oracle Database instance (an instance is the database program and its memory). The SGA is shared by all server and background processes. The SGA holds information for all oracle processes. The SGA stores data blocks and control information needed by the database. The SGA is divided into several memory structures such as: buffer cache, shared pool, redo log buffers etc..

PGA: A PGA (Program Global Area) is a memory region that contains data and control information for a server process. It is non-shared memory created by the Oracle Database when a server process is started. Access to the PGA is exclusive to the server process. There is one PGA for each server process. Background processes also allocate their own PGAs. The total PGA memory allocated for all background and server processes attached to an Oracle Database instance is referred to as the total instance PGA memory and the collection of all individual PGAs is referred to as the total instance PGA or just instance PGA. It contains global variables and data structures and control information for a server process. An example of such information is the runtime area of a cursor. Each time a cursor is executed, a new runtime area is created for that cursor in the PGA memory region of the server process in which it is executing. The performance of complex long running queries, typical of a DSS environment, depend to a large extent on the memory available in the PGA which is also called work area.

Automatic Memory Managment (Oracle 11g): Automatic Memory Management (AMM) was introduced in Oralce 11g release. AMM manages both SGA and PGA memory, allowing it to shift memory from SGA to PGA and vice versa. Configuring AMM will cause the Solaris to use Dynamic Intimate Shared Memory (DISM). This allows for shared memory segments to be dynamically resizable. Oracle uses DISM for its dynamic System Global Area (SGA) capability when AMM is configured. Regardless of whether Oracle Database uses ISM or DISM, it can always exchange the memory between dynamically sizable components such as the buffer cache, the shared pool, and the large pool after it starts an instance. Oracle Database can relinquish memory from one dynamic SGA component and allocate it to another component

How to enable ISM and DISM in Oracle

Oracle Database uses DISM if it is available on the system, and if the value of the SGA_MAX_SIZE initialization parameter is larger than the size required for all SGA components combined: the shared pool, the redo buffers, the large pool, the Java pool, the fixed size during Oracle setup.  This enables Oracle Database to lock only the amount of physical memory that is used. Oracle Database also uses Optimized Shared Memory if it is available on the system, else uses DISM, when MEMORY_TARGET or MEMORY_MAX_TARGET is set by the user. By default, SGA_MAX_SIZE is not set during database installation and hence it takes the same value as SGA_TARGET.  The use of SGA_TARGET is not a necessity to use DISM. Oracle will use DISM instead of ISM if SGA_MAX_SIZE is set larger than all database buffers, see doc.  DISM is enabled by default. To enable ISM, set SGA_TARGET less than 50% of SGA_MAX_SIZE or MEMORY_MAX_TARGET. See article

Oracle can dynamically increase SGA_TARGET size until it reaches SGA_MAX_SIZE.  Solaris does not allocate or reserve memory of size SGA_MAX_SIZE at instance startup, instead it allocates memory only when needed. Once DISM segment has been successfully allocated, Oracle automatically locks an amount of memory determined by the total of all database buffers described above. DBA can also alter the size of database buffers (db_cache_size) and the shared pool (shared_pool_size). Oracle then locks additional memory up to the maximum of SGA_MAX_SIZE value. Oracle can also release memory for use elsewhere by the operating system, as part of Oracle Automatic Memory Management (AMM) and Automatic Shared Memory Management (ASMM).

The use of DISM requires that the ora_dism process starts. Since DISM requires the application to lock memory and since memory locking can only be carried out by applications with superuser privileges, Oracle implemented a daemon that runs with root privileges. Since Oracle does not normally run with superuser privileges, Oracle simply makes the $ORACLE_HOME/bin/oradism binary setuid root to accomplish the task. When running Oracle in a non-global zone (container), the proc_lock_memory privilege should be assigned to the ora_dism process. This privilege is now available by default in non-global zones. See article.

Example of configuring Oracle to use DISM:

SQL> alter system set sga_max_size=1000M scope=spfile;
SQL> alter system set sga_target=600M scope=spfile;
SQL> shutdown immediate
SQL> startup
$ ps -ef|grep ism
root 28976 5865 0 10:54:00 ? 0:17 ora_dism_mydb

The ora_dism process will not start unless SGA_MAX_SIZE is greater than sum of all SGA components. In that case, Oracle will use ISM. Thus the Oracle Database uses ISM if the entire shared memory segment is in use at startup or if the value of the SGA_MAX_SIZE parameter is equal to or smaller than the size required for all SGA components combined.

MEMORY_TARGET was introduced in Oracle Database 11g to allow automatic memory tuning.  By default, it is set to 40% of physical memory configured. Furthermore, 60% of the memory addressed by MEMORY_TARGET is allocated to the SGA and the remaining 40% to the PGA. The SGA/PGA mix can be changed dynamically. Consequently, SGA_TARGET (SGA component) which is expected to be locked can be only 60% of MEMORY_TARGET and that represents 24% of physical memory ( 60% of 40% of physical memory), just below the 25% that can be allocated without changing project.max-shm-memory parameter (or zone.max-shm-memory if running in an Oracle Solaris Container). If memory is dynamically diverted from the PGA to the SGA, the 25% threshold is likely to be exceeded. It is very important to set the project.max-shm-memory parameter (or zone.maxshm-memory) high enough to ensure that all SGA memory can be locked under all circumstances.

 Note: To use DISM, you must have ISM enabled (which is the default case on Oracle 10g/11g). You will need always to check parameter: *._use_ism= TRUE

Checking if ISM or DISM is in use by Oracle

• When DISM is in effect, depending on the demand, Oracle locks additional memory, subject to the upper limit imposed by sga_max_size, or releases memory for use elsewhere by the operating system. The results of alter system actions are shown in the alert log file, typically located in $ORACLE_HOME/rdbms/log/alert_$ORACLE_SID.log. An example is shown below:

CKPT: Current size = 37904 MB, Target size = 3760 MB
Fri Jun 21 21:29:50 2002
Completed checkpoint up to RBA [0x7e.2.10], SCN: 0x0000.25c3f387
CKPT: Resize completed for buffer pool DEFAULT for blocksize 2048
Fri Jun 21 21:29:54 2002
ALTER SYSTEM SET db_cache_size='3932160000' SCOPE=MEMORCKPT: Begin
resize of buffer pool 3 (DEFAULT for block size 2048)Y;

• % pmap -xs ;  This will mark the memory segment as: shmid=n (Pageable SHM), ism shmid=n (ISM), dism shmid=n (DISM)

$ pmap -xs 15492
     15492:  ./maps
      Address  Kbytes     RSS    Anon  Locked Pgsz Mode   Mapped File
     00010000       8       8       -       -   8K r-x--  maps
     00020000       8       8       8       -   8K rwx--  maps
     00022000    3960    3960    3960       -   8K rwx--    [ heap ]
     00400000    8192    8192    8192       -   4M rwx--    [ heap ]
     00C00000    4096       -       -       -    - rwx--    [ heap ]
     01000000    4096    4096    4096       -   4M rwx--    [ heap ]
     03000000    1024    1024       -       -   8K rw-s-  dev:0,2 ino:4628487
     04000000     512     512     512       -   8K rw---  dev:0,2 ino:4628487
     04080000     512     512       -       -    - rw---  dev:0,2 ino:4628487
     05000000     512     512     512       -   8K rw--R  dev:0,2 ino:4628487
     05080000     512     512       -       -    - rw--R  dev:0,2 ino:4628487
     06000000    1024    1024    1024       -   8K rw---    [ anon ]
     07000000     512     512     512       -   8K rw--R    [ anon ]
     08000000    8192    8192       -    8192    - rwxs-    [ dism shmid=0x5 ]
     09000000    4096    4096       -       -   8K rwxs-    [ dism shmid=0x4 ]
     0A000000    4096       -       -       -    - rwxs-    [ dism shmid=0x2 ]
     0B000000    8192    8192       -    8192   4M rwxsR    [ ism shmid=0x3 ]
     FF280000     136     136       -       -   8K r-x--  libc.so.1

....

• % ipcs -m; prints information about active shared memory segments. The ISMATTACH column in the output shows the number of processes attached to the shared memory segment as ISM. The output does not differentiate between ISM or DISM.

Identifying ISM and DISM issues in Oracle

• Examine the alert log for each Oracle instance. The following entry indicates that Oracle was unable to start the oradism process with superuser privileges, which is responsible for locking the SGA when DISM is used. Performance will be significantly degraded for this instance, since SGA memory is not locked.
  
  Wed Jul 16 14:03:39 2003
  WARNING: -------------------------------
  WARNING: oradism not set up correctly.
  Dynamic ISM can not be locked. Please
  setup oradism, or unset sga_max_size.
  [diagnostic 0, 5, 0]
  ----------------------------------------

• In Oracle10g, a new table (x$ksmge ) can be examined to determine the state of locks for each granule of SGA memory (each granule is typically 16Mbytes in size). Oracle tracks SGA memory use in internal numbers of granules for each SGA component. The memory for dynamic components in the SGA is allocated in the unit of granules. Granule size is determined by total SGA size.  If the total SGA size is equal or less than 1G, then granule size is 4M. Otherwise, it will be 16M. Example: We can view the number of SGA components allocated in the per instance configuration. The queries below can be better utilized in finding issues with respect to components in the SGA and their allocation.

 1: SQL> select sum(gransize)/1024/1024 Size_MB,grantype

 2: 2 from x$ksmge

 3: 3 where granstate ='ALLOC'

 4: 4 group by grantype;

 5:  

 6: SIZE_MB   GRANTYPE

 7: ---------- ----------

 8: 336          1

 9: 8          2

 10: 16          4

 11: 360          7

 12: 16          3

 13:  

 14: SQL> select sum(a.gransize)/1024/1024 Size_MB,a.grantype,b.component,count(*) Number_of_Granules

 15: 2 from x$ksmge a, x$kmgsct b

 16: 3 where a.grantype = b.grantype

 17: 4 and a.granstate ='ALLOC'

 18: 5 group by a.grantype,b.component

 19: 6  ;

 20:  

 21: SIZE_MB   GRANTYPE COMPONENT                 NUMBER_OF_GRANULES

 22: ---------- ---------- ------------------------- ------------------

 23: 8          2 large pool                                 1

 24: 16          3 java pool                                  2

 25: 336          1 shared pool                               42

 26: 16          4 streams pool                               2

 27: 360          7 DEFAULT buffer cache                      45

 28:  

 29: SQL> select sum(gransize)/1024/1024

 30: 2 from x$ksmge;

 31:  

 32: SUM(GRANSIZE)/1024/1024

 33: -----------------------

 34: 736

Idenfitying ISM and DISM issues in Solaris

• oradism program should have appropriate permissions. Make sure oradism program has setuid root permission. It can be set by running:

% chown root $ORACLE_HOME/bin/oradism
% chmod 4755 $ORACLE_HOME/bin/oradism
% ls -l $ORACLE_HOME/bin/oradism
-rwsr-xr-x 1 root dba 9280 Apr 9 2002 /export/home/oracle/bin/oradism*

• If the oradism process dies in Oracle10g, no action is necessary; Oracle is able restart it. In Oracle9i, though, the only practical solution is to shutdown and restart the instance. To check if oradism process is running, type:

       % ps -aef | grep dism

       root 747 1 0 13:57:26 ? 19:42 ora_dism_custdb
       oradba1 18456 18391 0 23:37:08 pts/6 0:00 grep dism

•  Systems with active use of unlocked SGA memory can be identified using lockstat(1M). The lockstat utility shows mutex contention. As root, look for mutex contention in the segspt_softunlock or spt_anon_getpages functions. Two examples are included below showing the first few lines of a lockstat profile from a system running an active Oracle instance with unlocked SGA memory. The following command can be used to generate these profiles.
  
  # /usr/sbin/lockstat -A -n 200000 sleep 10
  
  Here are the two sample profiles:
  
  Adaptive mutex spin: 22908 events in 10.006 seconds (2290 events/sec)
  
  Count indv cuml rcnt spin Lock Caller
  ---------------------------------------------------------------------------
  4896 21% 21% 1.00 27532 0x3001314ec88
  spt_anon_getpages+0x54
  81 0% 22% 1.00 11 0x300001a1e98 sc_flush+0x1c4
  78 0% 22% 1.00 2 pse_mutex+0xb0 page_unlock+0x1c
  77 0% 22% 1.00 2 pse_mutex+0x368 page_unlock+0x1c
  
  Adaptive mutex spin: 561050 events in 10.017 seconds (56011 events/sec)
  Count indv cuml rcnt spin Lock Caller
  ---------------------------------------------------------------------------
  549729 98% 98% 1.00 146 0x300085a2580
  segspt_softunlock+0xc4
  834 0% 98% 1.00 10685 0x300085a2578
  spt_anon_getpages+0x54
  715 0% 98% 1.00 12 0x300085a2580
  spt_anon_getpages+0x64
  
  The presence of such mutexes indicates that SGA memory is not locked. This will result in slow Oracle Database performance. 

•  guds, performance data collection script captures kernel stacks of all processes/threads in the system in file "threadlist.out.*" using: "echo "::threadlist -v"|mdb -k >threadlist.out". Search for stacks with string segspt_dismfault. If found, it is a sign that DISM is not locked. 

unix:page_lookup(0x300120ca840, 0xc0318fc000) - frame recycled
genunix:swap_getapage+0x6c(0x300120ca840, 0xc0318fc000, 0x2000, 0x0, 0x2a112dd2e48, 0x2000, 0x300a9a02c78, , 0xf)
genunix:swap_getconpage+0x78(0x300120ca840, , 0x2000, 0x0, 0x2a112dd2e48, 0x2000, 0x0, 0x2a112dd3054, , , 0x294e74000)
genunix:anon_map_getpages+0x3b4(0x300ca9286b8, 0x140000, 0x6, 0x300a9a02c78, 0x280000000, 0xf, , , , 0x0, , 0x0, , 0x0)
genunix:spt_anon_getpages_task+0x2e4(0x0, 0x1, 0x2a112dd3308, 0x2a112dd3138, 0x80000000, 0x40000)
genunix:vmtask_task+0x7c(0x2a112dd31f0)
genunix:vmtask_run_job+0xdc(0x1, 0x1, 0x11f53a4, 0x2a112dd3308, 0x0)
genunix:spt_anon_getpages+0x98(0x300a9a02c78, 0x280000000, 0x80000000, 0x600c6800000, 0x0, 0x0)
genunix:segspt_dismfault+0x364(0x3009788f200, 0x300b5300a50, 0x6eb874000, 0x2000, 0x2, 0x1)
genunix:as_fault+0x3f0(0x3009788f200, 0x3006b4b2488, 0x6eb874000, 0x2000, 0x2, 0x1)
genunix:as_pagelock+0x11c(0x3006b4b2488, 0x2a112dd3838, 0x6eb874000, 0x2000, 0x1, 0x0)
genunix:_uio_pagelock+0x140(, 0x1, 0x2a112dd3a88, 0x2a112dd3810)
zfs:zfs_write+0x544(0x300d3932d00, 0x2a112dd3a88, 0x40, 0x3006e5cc3a8, 0x0)
genunix:fop_write+0x84(0x300d3932d00, 0x2a112dd3a88, 0x40, 0x3006e5cc3a8, 0x0)
genunix:pwrite+0x208(, , 0x2000?)
unix:syscall_trap+0xac()

------

unix:sfmmu_pagearray_setup+0x64(0x30000000, 0x3009cfd2000, 0x2a1081aa520, 0x1)
unix:sfmmu_tteload_addentry+0x394(0x30009b382c0, , 0x2a1081aa520, 0x30000000, 0x3009cfd2000, 0x41, 0x3ff)
unix:sfmmu_tteload_array+0x5c(0x30009b382c0, 0x2a1081aa520, 0x30000000, 0x3009cfd2000, 0x41, 0x3ff)
unix:hat_do_memload_array+0x128(0x30009b382c0, 0x30000000, 0x10000000, 0x3009cfd2000, 0xf, 0x41, 0x3ff)
unix:hat_memload_array+0x20(0x30009b382c0, 0x30000000, 0x10000000, 0x3ff, 0xf, 0x41)
genunix:segspt_dismfault+0x220(0x3000ca27140, 0x300729ba380, 0x57eeac000, 0x2000, 0x2, 0x1)
genunix:as_fault+0x4c8(0x3000ca27140, 0x60064b99c28, 0x57eeac000, 0x2000, 0x2, 0x1)
genunix:as_pagelock+0x118(0x60064b99c28, 0x2a1081aaa88, 0x57eeac000, 0x2000?, 0x1)
vxfs:vx_bp_lock_pages+0x44(0x300874ce880, 0x300874ce7f8, 0x100, 0x60064b99c28, 0x57eeac000, 0x2000, 0x2a1081aaa88)
vxfs:vx_dio_physio+0x294(0x300322aa000, 0x300874ce840, 0x799c000, 0x12665c000, 0x65c000, 0x2a1081aacb0, , , 0x300874ce7f8, , , , 0x300874ce778, 0x0)
vxfs:vx_dio_rdwri+0x410(0x300322aa000, 0x300874ce840, 0x100, 0x12665c000, 0x12665e000, 0x300874ce7f8, 0x2a1081ab058, , , 0x0, 0x0)
fdd:fdd_write_end+0x508(, 0x300874ce840, 0x300874ce7f8, 0x2a1081ab058, 0x2a1081ab05c, 0x300874ce778)
fdd:fdd_rw+0x704(0x300322aa000, 0x300874ce840, 0x300874ce7f8, 0x300874ce778, 0x2a1081ab05c, 0x2a1081ab058, 0x100, 0x30032a0ef48)
fdd:fdd_odm_rw+0x294(0x3008d35b140, 0x300874ce778, 0x100, 0x30032a0ef48, 0x12665c000, 0x100000000)
odm:odm_vx_aio+0xb8(0x3008d35b140, 0x300874ce738, 0x100, 0x30032a0ef48, 0x300874ce6c0, 0x7a7d4214)
odm:odm_vx_io+0x1c(0x3008d35b140, 0x300874ce738, 0x100, , 0x300874ce6c0, 0x7a7d4214)
odm:odm_io_issue+0xf8(0x30086b123c0, 0x2a1081ab42c, , , , 0x30086b123d0)
odm:odm_io_start+0x188(0x2a1081ab42c)
odm:odm_io_req+0xb20(0x3008a689198, 0x30074f11000, 0x55, 0x0, 0x4b32a0, 0x2a1081ab5ec)
odm:odm_request_io+0x118(0x3008a689198, 0x3007dbb79a0, 0x0, 0x30074f11000, 0x1000)
odm:odm_io+0x264(0x3007dbb79a0, , 0x2a1081ab78c)
odm:odm_io_stat+0x154(0x3007dbb79a0, 0x0)
odm:odmioctl+0x2b4(0x300898eed80, 0x56584f1c, 0xffffffff7fff5d88, 0x202003, 0x30032a0ef48)
genunix:fop_ioctl+0x20(0x300898eed80, 0x56584f1c, 0xffffffff7fff5d88, 0x202003, , 0x2a1081abadc)
genunix:ioctl+0x184()
unix:syscall_trap+0xac()

• statit utility is a widely-used, although unsupported utility. In statit, look for faults due to s/w lcking request. A non-zero result on a system with an active Oracle instance indicates that some or all SGA memory is not locked. The following example illustrates how to check for this behavior:
  
  % statit sleep 30 | grep lcking
  31.62 faults due to s/w lcking req 0.00 kernel as as_flt()s
  
  The example above shows softlocks, indicating a problem with SGA memory locking.

• ENOMEM errno can be generated due to a failure to allocate memory when DISM is not locked and can result in crashing the Oracle instance. This is due to Solaris Bug ID: 6559612 - multiple softlocks on a DISM segment should decrement availrmem just once. The fix is integrated into Solaris 10 Update 8 (Patchid: 141444-06). The workaround is to always have DISM segments locked. When Oracle performs I/O to DISM pages that are not locked, instead of using an optimized code path the kernel uses softlocks, a slower locking mechanism, which has bad performance implications and also leads to exhaustion of availrmem.  Due to bug 6559612, availrmem is decremented for each page that is locked for IO. If you are doing I/O to a 4MB page, available swap space will go down by 4MB. The problem is that no count is kept of the number of processes holding a softlock on a particular page, which means that availrmem could be decremented multiple times unnecessarily. If the DISM memory is locked, the problem doesn't occur because softlocks are not used.

• ECC Memory Errors: Due to Correctable/Uncorrectable (CE/UE) memory errors in ISM/DISM segment pages, the page retirement thread in Solaris tries to unlock any page that is scheduled for retirement. This may result in soft locking of pages in the SGA that inadvertently cause slow Database performance. This issue is addressed in Solaris BUG: 6997864 - Oracle database degradation when page in ISM segment is scheduled for retirement. The fix is integrated into Solaris 10 Update 11 (Patchid: 147440-20). The workaround is to restart the Oracle Database or reboot the system after replacing affected memory DIMMs. One can identify if page retirement is in progress by running:
  • % kstat -p -n page_retire; This will show pages pending retirement due to CE/UE memory errors. Example:

unix:0:page_retire:pages_deferred       558
unix:0:page_retire:pages_deferred_kernel        0
unix:0:page_retire:pages_fma    2954
unix:0:page_retire:pages_limit  65949
unix:0:page_retire:pages_limit_exceeded 0
unix:0:page_retire:pages_multiple_ce    0
unix:0:page_retire:pages_notdequeued    0
unix:0:page_retire:pages_notenqueued    0
unix:0:page_retire:pages_pending        13 <<<
unix:0:page_retire:pages_pending_kas    12
unix:0:page_retire:pages_retire_request 0
unix:0:page_retire:pages_retire_request_free    0
unix:0:page_retire:pages_retired        2954
unix:0:page_retire:pages_ue     0
unix:0:page_retire:pages_ue_cleared_freed       0
unix:0:page_retire:pages_ue_cleared_retired     0
unix:0:page_retire:pages_ue_persistent  0
unix:0:page_retire:pages_unretired      0

• • Kernel profiling data collected using "lockstat -kIW -i 977 -D 20 -s 40 sleep 10" will show high CPU usage due to soft locking on pages in ISM segment.

Profiling interrupt: 244798 events in 2.097 seconds (116759 events/sec)

-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Hottest CPU+PIL        Caller
105120  43%  43% 0.00      887 cpu[74]                mutex_delay_default

     nsec ------ Time Distribution ------ count     Stack
      128 |                               2056      current_thread
      256 |@@@                            12328     plat_lock_delay
      512 |@@                             9051      mutex_vector_enter
     1024 |@@@@@@@@@@@@                   44247     segspt_shmpagelock
     2048 |@@@@@@@@@                      33228     as_pagelock
     4096 |@                              4150      vx_bp_lock_pages
     8192 |                               60        vx_dio_physio
                                                    vx_dio_rdwri
                                                    fdd_rw
                                                    fdd_odm_rw
                                                    odm_vx_io
                                                    odm_io_sync
                                                    odm_io
                                                    odm_io_stat
                                                    odmioctl
                                                    fop_ioctl
                                                    ioctl
                                                    syscall_trap
-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Hottest CPU+PIL        Caller
95419  39%  82% 0.00      913 cpu[31]                mutex_delay_default

     nsec ------ Time Distribution ------ count     Stack
      128 |                               706       current_thread
      256 |@                              5016      plat_lock_delay
      512 |@@@                            9948      mutex_vector_enter
     1024 |@@@@@@@@@@@@@@                 46507     segspt_shmpagelock
     2048 |@@@@@@@@@                      29907     as_pagelock
     4096 |@                              3232      vx_bp_lock_pages
     8192 |                               72        vx_dio_physio
                                                    vx_dio_rdwri
                                                    fdd_write_end
                                                    fdd_rw
                                                    fdd_odm_rw
                                                    odm_vx_aio
                                                    odm_vx_io
                                                    odm_io_issue
                                                    odm_io_start
                                                    odm_io_req
                                                    odm_request_io

• "echo ::threadlist |mdb -k" will report kernel stacks with active softlocking due to unlocked ISM segment

   mutex_vector_enter+0x428()
   segspt_shmpagelock+0x154()
   as_pagelock+0xd4()
   vx_bp_lock_pages+0x40()
   vx_dio_physio+0x2ec()
   vx_dio_rdwri+0x414()
   fdd_write_end+0x4c0()
   fdd_rw+0x744()
   fdd_odm_rw+0x308()
   odm_vx_aio+0xdc()
   odm_vx_io+0x20()
   odm_io_issue+0x110()
   odm_io_start+0x1b8()
   odm_io_reaper+0xd4()
   thread_start+4()

• VAC conflict" or "Virtual Address Cache conflict": It is a situation that can happen when a page of memory is explicitly mapped by a program to a virtual address that conflicts with the virtual addresses of other mappings to the same
  page of memory. The situation causes the Solaris[TM] operating system to have to work very hard to avoid data corruption. Visible symptoms are many CPU xcalls, mutex contention on an anon_array mutex and slow memory accesses
   to that page. This issue can happen when a mixture of ISM and non ISM attachments are performed by different processes on the same shared memory segment. One can identify the issue by running:
  • 
    

    /usr/sbin/lockstat -C -p -s30 sleep 1; Look for routines: anon_array_lock, annon_array_enter etc..

    
    
  • 
    

    /usr/bin/kstat -p unix:0:sfmmu_global_stat:sf_unload_conflict 1; This will dump conflicts/second in sfmmu global stats structure

    
    

Best Practices for Configuring ISM and DISM

• Always use the Oracle Database installer to create new installations of the Oracle Database
• If instance migration is carried out, ensure that it is done with superuser privileges
• If using NFS to mount Oracle Database binaries, ensure that NFS mounts allow root access
• Configure plenty of disk backed swap when using DISM. DISM segments require double virtual swap reservation. Disk backed swap space is needed to back the full DISM segment size since memory is not automatically locked.
• Install the latest Solaris Kernel Update to avoid running into the bugs mentioned above.
• It is recommended to set following NUMA tunable in OracleDB  init.ora (hidden parameter):
  OracleDB 8.1.7->11.1.0 _enable_NUMA_optimization=TRUE
  OracleDB 11.2.x _enable_NUMA_support=TRUE
  
  This is to avoid NUMA related latencies when DISM memory is allocated in non-local NUMA node. You should run oracle with NUMA tunable enabled. If you want to enable NUMA while under "ISM" there is a minimum S10 patch-level KU 147440-07 or newer required. Without this Patch enabling NUMA with ISM would result in additonal overhead without much benefits. If interested in understanding and monitoring NUMA behavior and latencies consider using new and enhanced NUMA tools provided in Sol11 to get better view and control on how application memory is allocated and used. These tools are available as a separate package for Solaris 10. Discussion is provided below:
  
           o lgrpinfo -l : Latency of memory access between different lgroups
           o pmap -L :  What lgroup is used for memory allocation.  Memory allocation in remote lgroup can result in higher memory latencies
           o pmadmvice: To control memory placement.
           o plgrp -vG :  Associated lgroup of all application threads.
           o plgrp - S :  To move all application threads to home lgroup

References

Related …

Products …

• Sun Microsystems > Operating Systems > Solaris Operating System > Solaris SPARC Operating System > Performance > Slow System, GUDS, DTrace, Kernel Resources

Keywords …

BAD PERFORMANCE;DATABASE PERFORMANCE;DATABASE SETUP;DYNAMIC INTIMATE SHARED MEMORY;DYNAMIC SGA;HIGH CPU USAGE;INTIMATE SHARED MEMORY;MEMORY ERRORS;MEMORY_MAX_TARGET;MEMORY_TARGET;PAGE RETIREMENT;PERFORMANCE;SLOW DATABASE