[20191202]關於hugepages相關問題.txt

[20191202]關於hugepages相關問題.txt

--//上午在測試hugepages相關問題時遇到一個問題,自己以前疏忽了,就是在use_large_pages=true的情況下.
--//如果在/etc/sysctl.conf引數檔案中vm.nr_hugepages定義不夠大,即使定義vm.nr_overcommit_hugepages足夠.
--//也不會向vm.nr_overcommit_hugepages借用記憶體,導致進入混合模式.
--//透過例子說明:

1.環境:
SCOTT@book> @ ver1
PORT_STRING                    VERSION        BANNER
------------------------------ -------------- --------------------------------------------------------------------------------
x86_64/Linux 2.4.xx            11.2.0.4.0     Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

SCOTT@book> alter system set use_large_pages=true scope=spfile;
System altered.

# grep "^vm.nr" /etc/sysctl.conf
vm.nr_hugepages = 104
vm.nr_overcommit_hugepages = 1000

# sysctl -p

2.測試:
--//重啟資料庫看看,檢查alert檔案發現:
Adjusting the default value of parameter parallel_max_servers
from 480 to 170 due to the value of parameter processes (200)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 51 GB

Total Shared Global Region in Large Pages = 208 MB (33%)

Large Pages used by this instance: 104 (208 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 104 (208 MB)
Large Page size = 2048 KB

RECOMMENDATION:
  Total System Global Area size is 618 MB. For optimal performance,
  prior to the next instance restart:
  1. Increase the number of unused large pages by
 at least 205 (page size 2048 KB, total size 410 MB) system wide to
  get 100% of the System Global Area allocated with large pages
********************************************************************
--//可以發現僅僅使用定義vm.nr_hugepages = 104.

$ grep -i page /proc/meminfo
AnonPages:        165700 kB
PageTables:        18904 kB
AnonHugePages:         0 kB
HugePages_Total:     104
HugePages_Free:       49
HugePages_Rsvd:       49
HugePages_Surp:        0
Hugepagesize:       2048 kB

$ ps -ef | grep pmo[n] | awk '{print $2}' | xargs -IQ grep SYSV /proc/Q/numa_maps
60000000 default file=/SYSV00000000\040(deleted) huge dirty=1 mapmax=25 N0=1
60c00000 default file=/SYSV00000000\040(deleted) huge
6a400000 default file=/SYSV00000000\040(deleted) huge
6c400000 default file=/SYSV00000000\040(deleted) huge
6cc00000 default file=/SYSV00000000\040(deleted) huge
--//僅僅5個共享記憶體段使用hugepages.
6d000000 interleave:0-1 file=/SYSV00000000\040(deleted) dirty=785 mapmax=25 active=245 N0=400 N1=385
86800000 interleave:0-1 file=/SYSVe8a8ec10\040(deleted) dirty=1 mapmax=25 N1=1

$ sysresv
IPC Resources for ORACLE_SID "book" :
Shared Memory:
ID              KEY
413040641       0x00000000
413073410       0x00000000
413106179       0x00000000
413138949       0x00000000
413171718       0x00000000
413204487       0x00000000
413237256       0xe8a8ec10
Semaphores:
ID              KEY
335151104       0x6aa88594
Oracle Instance alive for sid "book"
--//也就是在use_large_pages=true的情況下不會向vm.nr_overcommit_hugepages引數借用記憶體.

3.測試:
SYS@book> alter system set use_large_pages=only scope=spfile;
System altered.

# grep "^vm.nr" /etc/sysctl.conf
vm.nr_hugepages = 104
vm.nr_overcommit_hugepages = 204

# sysctl -p
..

SYS@book> startup
ORA-27137: unable to allocate large pages to create a shared memory segment
Linux-x86_64 Error: 12: Cannot allocate memory
Additional information: 2097152

--//104+204 = 308,還不夠大.
# grep "^vm.nr" /etc/sysctl.conf
vm.nr_hugepages = 104
vm.nr_overcommit_hugepages = 1000

# sysctl -p
..

SYS@book> startup
ORACLE instance started.
Total System Global Area  643084288 bytes
Fixed Size                  2255872 bytes
Variable Size             205521920 bytes
Database Buffers          427819008 bytes
Redo Buffers                7487488 bytes
Database mounted.
Database opened.

--//alert輸出如下:
Adjusting the default value of parameter parallel_max_servers
from 480 to 170 due to the value of parameter processes (200)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = ONLY
Per process system memlock (soft) limit = 51 GB

Large Pages unused system wide = 308 (616 MB)
Large Pages configured system wide = 308 (616 MB)
Large Page size = 2048 KB

ERROR:
  Failed to allocate shared global region with large pages, unix errno = 12.
  Aborting Instance startup.
  ORA-27137: unable to allocate Large Pages to create a shared memory segment

ACTION:
  Total System Global Area size is 624 MB. Prior to next instance restart:
 1. Increase the number of unused large pages (page size 2048 KB)
 to at least 312 (624 MB) to allocate 100% System Global Area
 with large pages.
********************************************************************
--//以上是報ORA-27137的情況.

Mon Dec 02 15:59:56 2019
Adjusting the default value of parameter parallel_max_servers
from 480 to 170 due to the value of parameter processes (200)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = ONLY
Per process system memlock (soft) limit = 51 GB

Total Shared Global Region in Large Pages = 618 MB (100%)

Large Pages used by this instance: 309 (618 MB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 309 (618 MB)
Large Page size = 2048 KB
********************************************************************

$ ps -ef | grep pmo[n] | awk '{print $2}' | xargs -IQ grep SYSV /proc/Q/numa_maps
60000000 default file=/SYSV00000000\040(deleted) huge dirty=1 mapmax=25 N0=1
60c00000 default file=/SYSV00000000\040(deleted) huge dirty=37 mapmax=25 N0=30 N1=7
86800000 interleave:0-1 file=/SYSVe8a8ec10\040(deleted) huge dirty=1 mapmax=25 N0=1

$ sysresv

IPC Resources for ORACLE_SID "book" :
Shared Memory:
ID              KEY
413401089       0x00000000
413433858       0x00000000
413466627       0xe8a8ec10
Semaphores:
ID              KEY
335314944       0x6aa88594
Oracle Instance alive for sid "book"

3.總結:
--//設定核心引數vm.nr_overcommit_hugepages ,僅僅在use_large_pages=only的情況下才會有效.