Linux 下配置 HugePages

HugePages是透過使用大頁記憶體來取代傳統的4kb記憶體頁面，使得管理虛擬地址數變少，加快了從虛擬地址到實體地址的對映以及透過摒棄記憶體頁面的換入換出以提高記憶體的整體效能。尤其是對於8GB以上的記憶體以及較大的Oracle SGA size，建議配值並使用HugePage特性。本文基於x86_64 Linux下來描述如何配值 HugePages。
    有關HugePages的特性請參考：Linux HugePage 特性

  
1、為什麼需要配值HugePages ？
a、Larger Page Size and Less # of Pages: 
    Default page size is 4K whereas the HugeTLB size is 2048K. That means the system would need to handle 512 times less pages.

b、No Page Table Lookups: 
    Since the HugePages are not subject to replacement (despite regular pages), page table lookups are not required.

c、Better Overall Memory Performance: 
    On virtual memory systems (any modern OS) each memory operation is actually two abstract memory operations. With HugePages, since there are less number of pages to work on, the possible bottleneck on page table access is clearly avoided.

d、No Swapping: 
    We must avoid swapping to happen on Linux OS at all Document 1295478.1. HugePages are not swappable (whereas regular pages are). Therefore there is no page replacement mechanism overhead. HugePages are universally regarded as pinned.

e、No 'kswapd' Operations:
     kswapd will get very busy if there is a very large area to be paged (i.e. 13 million page table entries for 50GB memory) and will use an incredible amount of CPU resource. When HugePages are used, kswapd is not involved in managing them. See also Document 361670.1

 

2、配值HugePages
  下面列出了配值HugePages的所有步驟

a、檢視當前系統是否配值HugePages
  下面的查詢中HugePages相關的幾個值都為0，表明當前未配值HugePages，其次可以看到Hugepagesize為2MB。
  $ grep Huge /proc/meminfo
  HugePages_Total:   0
  HugePages_Free:    0
  HugePages_Rsvd:    0
  Hugepagesize:     2048 kB
    
b、修改使用者的memlock限制
  透過修改/etc/security/limits.conf 配值檔案來實現
  該引數的值通常配值位略小於當前的已安裝系統記憶體，如當前你的系統記憶體為64GB，可以做如下設定
  *   soft   memlock    60397977
  *   hard   memlock    60397977

  上述的設定單位為kb，不會降低系統效能。至少也要配值為略大於系統上所有SGA的總和。
  使用ulimit -l 來校驗該設定

 

c、禁用AMM(Oracle 11g)
  如果當前的Oracle 版本為10g，可以跳過此步驟。
  如果當前的Oracle 版本為11g，由於AMM(Automatic Memory Management)特性與Hugepages不相容，需要禁用AMM。
    ALTER SYSTEM RESET memory_target SCOPE=SPFILE;

    ALTER SYSTEM RESET memory_max_target SCOPE=SPFILE;
    ALTER SYSTEM SET sga_target=g SCOPE=SPFILE;
    ALTER SYSTEM SET pga_aggregate_target=g SCOPE=SPFILE;
    SHUTDOWN IMMEDIATE; 

    STARTUP;
    
d、計算vm.nr_hugepages 的值    
  使用Oracle 提供的指令碼hugepages_settings.sh的指令碼來計算vm.nr_hugepages的值
  在執行指令碼之前確保所有的Oracle 例項已啟動以及ASM也啟動(存在的情形下)
  $ ./hugepages_settings.sh
  ...
  Recommended setting: vm.nr_hugepages = 1496

 

e、 編輯/etc/sysctl.conf 來設定vm.nr_hugepages引數
  $ sysctl -w vm.nr_hugepages = 1496  
  $ sysctl -p
  
  -- Author : Robinson
  -- Blog   : http://blog.csdn.net/robinson_0612
  
f、停止所有的Instance並重啟server
  上述的所有步驟已經實現了動態修改，但對於HugePages的分配需要重新啟動server才能生效。

 

h、驗證配值
  HugePages相關引數的值會隨著當前伺服器上的例項的停止與啟動而動態發生變化
  通常情況下，HugePages_Free的值應當小於HugePages_Total的值，在HugePages被使用時HugePages_Rsvd值應當為非零值。
  $ grep Huge /proc/meminfo
  HugePages_Total:   131
  HugePages_Free:     20
  HugePages_Rsvd:     20
  Hugepagesize:     2048 kB 
  
  如下面的情形，當伺服器上僅有的一個例項被關閉後，HugePages_Rsvd的值為零。且HugePages_Free等於HugePages_Total
  $ grep Huge /proc/meminfo
  HugePages_Total:   131
  HugePages_Free:    131
  HugePages_Rsvd:      0
  Hugepagesize:     2048 kB   

 

3、使用HugePages的注意事項
  下面的三種情形應當重新配置HugePages
    a、實體記憶體的增減或減少
    b、在當前伺服器上新增或移出Instance
    c、Instance的SGA大小增加或減少   
  如果未能調整HugePages，可能會引發下面的問題
    a、資料庫效能地下
    b、出現記憶體不足或者過度使用交換空間
    c、資料庫例項不能被啟動
    d、關鍵性系統服務故障
   
4、HugePages特性的常見故障處理

Symptom A：
    System is running out of memory or swapping 
Possible Cause： 
    Not enough HugePages to cover the SGA(s) and therefore the area reserved for HugePages are wasted where SGAs are allocated through regular pages. 
Troubleshooting Action：
    Review your HugePages configuration to make sure that all SGA(s) are covered.

 

Symptom B：
    Databases fail to start 
Possible Cause：
    memlock limits are not set properly 
Troubleshooting Action：
    Make sure the settings in limits.conf apply to database owner account.

 

Symptom C：
    One of the database fail to start while another is up 
Possible Cause：
    The SGA of the specific database could not find available HugePages and remaining RAM is not enough. 
Troubleshooting Action：
    Make sure that the RAM and HugePages are enough to cover all your database SGAs

 

Symptom D：
    Cluster Ready Services (CRS) fail to start 
Possible Cause：
    HugePages configured too large (maybe larger than installed RAM)
Troubleshooting Action： 
    Make sure the total SGA is less than the installed RAM and re-calculate HugePages.

 

Symptom E：
    HugePages_Total = HugePages_Free
Possible Cause： 
    HugePages are not used at all. No database instances are up or using AMM. 
Troubleshooting Action：
   Disable AMM and make sure that the database instances are up.

 

Symptom F：
    Database started successfully and the performance is slow 
Possible Cause：
    The SGA of the specific database could not find available HugePages and therefore the SGA is handled by regular pages, which leads to slow performance 
Troubleshooting Action：
    Make sure that the HugePages are many enough to cover all your database SGAs

Reference: [ID 361468.1]

 

5、計算vm.nr_hugepages 值的指令碼