記一次 redis 事件註冊不當導致的記憶體洩露

successgo發表於2024-07-18

線上的程式跑著跑著記憶體越來越大,並且沒有下降的趨勢,重啟一下程式也只能短暫恢復。透過 htop 命令再按一下 M 鍵按記憶體佔用大小排個序,程式會佔好幾個G。那好,讓我們來分析一下。

收集dump

透過 tophtop 程序管理器,或 ps 命令查詢到目標程序 id,然後使用如下命令生成 dump:

createdump --full <PID>

dump 會自動儲存在 /tmp 目錄,檔名格式一般是 coredump.PID

小試牛刀

拿到 dump 後,透過 windbg 開啟。首先使用命令 !address -summary 檢視一下。

注意:有的朋友可能開啟 dump 檔案後,發現 sos 擴充套件載入失敗,需要手動載入一下,輸入命令 .load sos 即可。

0:000> !address -summary

                                     
Mapping file section regions...
Mapping module regions...
Mapping heap regions...
*** WARNING: Unable to verify timestamp for libc-2.17.so

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
<unknown>                               869 ffffffff`f409b600 (  16.000 EB) 100.00%  100.00%
Image                                  1117        0`0b565a00 ( 181.397 MB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
                                        808 ffffffff`018de000 (  16.000 EB)          100.00%
MEM_PRIVATE                            1178        0`fdd23000 (   3.966 GB)   0.00%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
                                        808 ffffffff`018de000 (  16.000 EB) 100.00%  100.00%
MEM_COMMIT                             1178        0`fdd23000 (   3.966 GB)   0.00%    0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          520        0`f6bf0000 (   3.855 GB)   0.00%    0.00%
PAGE_EXECUTE_READ                       160        0`05141000 (  81.254 MB)   0.00%    0.00%
PAGE_READONLY                           466        0`0147c000 (  20.484 MB)   0.00%    0.00%
PAGE_EXECUTE_WRITECOPY                   32        0`00b76000 (  11.461 MB)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unknown>                              7ffd`7f5ad000 ffff8002`80053000 (  16.000 EB)
Image                                  7f44`7390c000        0`013d2000 (  19.820 MB)

從輸出結果中 MEM_COMMIT 一欄可以看出,記憶體佔了3.9G。

下一步要看是託管記憶體還是非託管記憶體吃掉。

託管記憶體

使用 eeheap 命令檢視:

0:000> !sos eeheap 
Loader Heap:
----------------------------------------
System Domain:        7f447979a8c0
LoaderAllocator:      7f447979a8c0
LowFrequencyHeap:     7f4406ce0000(10000:c000) 7f4406cc0000(10000:10000) 7f4406ca0000(10000:10000) 7f4406c80000(10000:10000) 7f4406c50000(10000:10000) 7f4406c20000(10000:10000) 7f4406c00000(10000:10000) 7f4406be0000(10000:10000) 7f4406bc0000(10000:10000) 7f4406b60000(10000:10000) 7f4406b30000(10000:10000) 7f4406b00000(10000:10000) 7f4406ad0000(10000:10000) 7f4406a90000(20000:20000) 7f4406a70000(10000:a000) 7f4406a20000(10000:f000) 7f4406a00000(10000:10000) 7f44069e0000(10000:10000) 7f44069d0000(10000:10000) 7f4406990000(40000:40000) 7f4406950000(30000:29000) 7f4406940000(10000:10000) 7f4406720000(10000:f000) 7f4406700000(10000:10000) 7f44066e0000(10000:10000) 7f44066d0000(10000:10000) 7f44066b0000(10000:10000) 7f4406690000(10000:10000) 7f4406670000(10000:10000) 7f4406540000(10000:10000) 7f4406530000(10000:10000) 7f4406390000(10000:b000) 7f4406330000(10000:10000) 7f4406300000(10000:10000) 7f44062e0000(10000:10000) 7f4406270000(10000:10000) 7f4406240000(10000:10000) 7f4406210000(10000:10000) 7f4406200000(10000:10000) 7f44061d0000(30000:2a000) 7f44061b0000(10000:4000) 7f4406190000(10000:10000) 7f4406170000(10000:10000) 7f4406160000(10000:f000) 7f4406140000(10000:c000) 7f4406110000(10000:10000) 7f44060e0000(10000:10000) 7f4405ed0000(10000:10000) 7f4405eb0000(10000:10000) 7f4405e90000(10000:10000) 7f4405e70000(10000:10000) 7f4405e50000(10000:f000) 7f4405d70000(10000:10000) 7f4405d50000(10000:10000) 7f4405d20000(10000:10000) 7f4405d00000(10000:10000) 7f4405ce0000(10000:10000) 7f4405cd0000(10000:10000) 7f4405cb0000(10000:f000) 7f4405c90000(10000:f000) 7f4405bd0000(10000:10000) 7f4405b40000(10000:10000) 7f4405b20000(10000:f000) 7f4405af0000(10000:10000) 7f4405ad0000(10000:10000) 7f4405aa0000(30000:2f000) 7f4405a90000(10000:e000) 7f4405a70000(10000:10000) 7f4405a30000(10000:10000) 7f4405a10000(10000:10000) 7f44059a0000(10000:10000) 7f4405980000(10000:10000) 7f4405970000(10000:10000) 7f4405950000(10000:10000) 7f4405730000(10000:10000) 7f4405720000(10000:10000) 7f4405700000(10000:10000) 7f44056e0000(10000:10000) 7f44056b0000(10000:10000) 7f44056a0000(10000:10000) 7f4405670000(10000:10000) 7f4405650000(10000:10000) 7f4405630000(10000:f000) 7f44055e0000(10000:10000) 7f44055d0000(10000:10000) 7f44055c0000(10000:10000) 7f44055a0000(10000:10000) 7f4405590000(10000:10000) 7f4405580000(10000:10000) 7f4405560000(10000:10000) 7f4405540000(10000:10000) 7f4405520000(10000:10000) 7f4405510000(10000:10000) 7f44054f0000(10000:10000) 7f44054e0000(10000:10000) 7f44054c0000(10000:10000) 7f44054a0000(10000:10000) 7f4405490000(10000:10000) 7f4405280000(10000:d000) 7f4405260000(10000:10000) 7f4405240000(10000:10000) 7f4405230000(10000:10000) 7f4405210000(10000:10000) 7f4405200000(10000:10000) 7f44051f0000(10000:10000) 7f44051a0000(10000:10000) 7f4405180000(10000:10000) 7f4405150000(10000:10000) 7f4405120000(10000:10000) 7f44050e0000(10000:10000) 7f4405050000(10000:10000) 7f4405020000(10000:10000) 7f4404ff0000(10000:10000) 7f4404fc0000(10000:10000) 7f4404fb0000(10000:10000) 7f4404f90000(10000:f000) 7f4404f50000(10000:10000) 7f4404f40000(10000:10000) 7f4404f30000(10000:10000) 7f4404f10000(10000:7000) 7f4404e60000(10000:10000) 7f4404e50000(10000:e000) 7f4404e30000(10000:d000) 7f4404d20000(10000:10000) 7f4404d10000(10000:10000) 7f4404cf0000(10000:7000) 7f4404c70000(10000:10000) 7f4404c50000(10000:10000) 7f4404a40000(10000:b000) 7f4404a20000(10000:e000) 7f4404a00000(10000:10000) 7f4404970000(10000:10000) 7f4404950000(10000:10000) 7f4404940000(10000:10000) 7f4404930000(10000:7000) 7f4404920000(10000:f000) 7f4404900000(10000:10000) 7f44048e0000(10000:10000) 7f4404830000(10000:10000) 7f4404810000(10000:10000) 7f44047f0000(10000:c000) 7f4404710000(10000:10000) 7f44046f0000(10000:10000) 7f44046d0000(10000:10000) 7f44046a0000(10000:10000) 7f4404550000(10000:10000) 7f44044a0000(10000:10000) 7f44043d0000(10000:10000) 7f44043c0000(10000:10000) 7f44040c0000(10000:b000) 7f4404000000(10000:e000) 7f4403ed0000(10000:10000) 7f4403e30000(10000:10000) 7f4403d10000(10000:10000) 7f4403d00000(10000:f000) 7f4403c90000(10000:a000) 7f4403bc0000(10000:10000) 7f4403bb0000(10000:f000) 7f44037a0000(10000:a000) 7f4403770000(10000:10000) 7f4403730000(10000:10000) 7f44036d0000(10000:10000) 7f4403660000(10000:10000) 7f4403510000(10000:10000) 7f44033f0000(10000:c000) 7f44032a0000(10000:10000) 7f4403280000(10000:10000) 7f4403270000(10000:f000) 7f4403220000(40000:3f000) 7f4402980000(20000:1a000) 7f4402650000(10000:d000) 7f4402620000(10000:d000) 7f4402580000(10000:10000) 7f4402360000(10000:f000) 7f4402210000(10000:10000) 7f4402140000(10000:10000) 7f4402050000(10000:10000) 7f4402030000(10000:10000) 7f4401ee0000(10000:10000) 7f4401d70000(10000:10000) 7f4401cb0000(10000:10000) 7f4401ad0000(10000:10000) 7f44018b0000(10000:f000) 7f44016d0000(10000:10000) 7f4401520000(10000:10000) 7f44013b0000(10000:10000) 7f4401210000(20000:1f000) 7f4400f90000(10000:3000) 7f4400de0000(10000:10000) 7f4400d20000(10000:10000) 7f4400b00000(10000:f000) 7f4400a20000(10000:10000) 7f4400870000(10000:10000) 7f4400730000(20000:20000) 7f44004d0000(10000:2000) 7f4400280000(10000:10000) 7f4400200000(10000:e000) 7f44001f0000(10000:9000) 7f4400100000(10000:10000) 7f44000f0000(10000:10000) 7f44000a0000(10000:10000) 7f4400000000(10000:10000) 7f43fffb0000(50000:50000) 7f43ff4e0000(3000:1000) Size: 0xd3b000 (13873152) bytes total, 0xb4000 (737280) bytes wasted.
HighFrequencyHeap:    7f4406cd0000(10000:f000) 7f4406cb0000(10000:10000) 7f4406c70000(10000:10000) 7f4406c60000(10000:10000) 7f4406c40000(10000:10000) 7f4406c30000(10000:10000) 7f4406c10000(10000:10000) 7f4406bf0000(10000:10000) 7f4406bd0000(10000:10000) 7f4406b70000(10000:10000) 7f4406b50000(10000:10000) 7f4406b40000(10000:10000) 7f4406b10000(10000:10000) 7f4406af0000(10000:10000) 7f4406ae0000(10000:10000) 7f4406ac0000(10000:10000) 7f4406ab0000(10000:10000) 7f4406a80000(10000:10000) 7f4406a60000(10000:10000) 7f4406a50000(10000:10000) 7f4406a40000(10000:10000) 7f4406a30000(10000:10000) 7f4406a10000(10000:10000) 7f44069f0000(10000:10000) 7f4406980000(10000:10000) 7f4406730000(10000:10000) 7f4406710000(10000:10000) 7f44066f0000(10000:10000) 7f44066c0000(10000:10000) 7f44066a0000(10000:10000) 7f4406680000(10000:10000) 7f4406620000(10000:10000) 7f4406550000(10000:10000) 7f44063a0000(10000:10000) 7f4406380000(10000:f000) 7f4406320000(10000:10000) 7f4406310000(10000:10000) 7f44062f0000(10000:f000) 7f44062d0000(10000:10000) 7f4406280000(10000:10000) 7f4406230000(10000:f000) 7f44061c0000(10000:10000) 7f44061a0000(10000:10000) 7f4406180000(10000:10000) 7f4406150000(10000:10000) 7f4406130000(10000:10000) 7f4406100000(10000:10000) 7f44060f0000(10000:10000) 7f4405ec0000(10000:10000) 7f4405ea0000(10000:10000) 7f4405e80000(10000:10000) 7f4405e60000(10000:10000) 7f4405e30000(10000:10000) 7f4405d60000(10000:10000) 7f4405d40000(10000:10000) 7f4405d10000(10000:10000) 7f4405cf0000(10000:10000) 7f4405cc0000(10000:10000) 7f4405ca0000(10000:10000) 7f4405c80000(10000:10000) 7f4405b50000(10000:10000) 7f4405b30000(10000:10000) 7f4405b10000(10000:10000) 7f4405b00000(10000:10000) 7f4405ae0000(10000:10000) 7f4405a80000(10000:f000) 7f4405a60000(10000:10000) 7f4405a20000(10000:10000) 7f4405990000(10000:10000) 7f4405960000(10000:10000) 7f4405740000(10000:10000) 7f4405710000(10000:10000) 7f44056f0000(10000:10000) 7f44056d0000(10000:10000) 7f44056c0000(10000:10000) 7f4405680000(10000:10000) 7f4405660000(10000:10000) 7f4405640000(10000:10000) 7f4405610000(10000:10000) 7f4405600000(10000:10000) 7f44055f0000(10000:10000) 7f44055b0000(10000:10000) 7f4405570000(10000:10000) 7f4405530000(10000:10000) 7f4405500000(10000:10000) 7f44054d0000(10000:10000) 7f44054b0000(10000:10000) 7f4405270000(10000:10000) 7f4405250000(10000:10000) 7f4405220000(10000:10000) 7f44051e0000(10000:10000) 7f4405190000(10000:10000) 7f4405170000(10000:10000) 7f4405160000(10000:10000) 7f4405140000(10000:10000) 7f4405130000(10000:10000) 7f4405110000(10000:10000) 7f4405100000(10000:10000) 7f44050f0000(10000:10000) 7f4405060000(10000:10000) 7f4405040000(10000:10000) 7f4405030000(10000:10000) 7f4405010000(10000:10000) 7f4405000000(10000:10000) 7f4404fe0000(10000:10000) 7f4404fd0000(10000:10000) 7f4404fa0000(10000:10000) 7f4404f80000(10000:10000) 7f4404f60000(10000:10000) 7f4404f20000(10000:10000) 7f4404f00000(10000:10000) 7f4404e40000(10000:10000) 7f4404d00000(10000:10000) 7f4404cd0000(10000:10000) 7f4404c60000(10000:10000) 7f4404a30000(10000:10000) 7f4404a10000(10000:10000) 7f4404980000(10000:10000) 7f4404960000(10000:10000) 7f4404910000(10000:10000) 7f44048f0000(10000:10000) 7f44048d0000(10000:10000) 7f44048c0000(10000:10000) 7f4404820000(10000:10000) 7f4404800000(10000:10000) 7f44047e0000(10000:10000) 7f4404700000(10000:10000) 7f44046e0000(10000:10000) 7f44046c0000(10000:10000) 7f44046b0000(10000:10000) 7f4404690000(10000:10000) 7f44045b0000(10000:10000) 7f4404540000(10000:10000) 7f44044b0000(10000:10000) 7f4404490000(10000:10000) 7f4404480000(10000:10000) 7f44043e0000(10000:10000) 7f4404020000(10000:10000) 7f4404010000(10000:10000) 7f4403fa0000(10000:10000) 7f4403ee0000(10000:10000) 7f4403e40000(10000:10000) 7f4403d30000(10000:10000) 7f4403d20000(10000:10000) 7f4403ca0000(10000:10000) 7f4403c70000(10000:10000) 7f4403c60000(10000:10000) 7f4403bd0000(10000:10000) 7f44037b0000(10000:10000) 7f4403790000(10000:10000) 7f4403780000(10000:10000) 7f4403760000(10000:10000) 7f4403750000(10000:10000) 7f4403740000(10000:10000) 7f4403720000(10000:10000) 7f4403710000(10000:10000) 7f44036e0000(10000:10000) 7f44036c0000(10000:10000) 7f44036b0000(10000:10000) 7f4403420000(10000:10000) 7f4403410000(10000:10000) 7f4403400000(10000:10000) 7f44033a0000(10000:10000) 7f4403330000(10000:10000) 7f4403290000(10000:10000) 7f4403260000(10000:10000) 7f4402660000(10000:10000) 7f4402640000(10000:10000) 7f4402630000(10000:10000) 7f4402590000(10000:10000) 7f4402480000(10000:10000) 7f4402370000(10000:10000) 7f44022e0000(10000:10000) 7f4402230000(10000:10000) 7f4402220000(10000:10000) 7f4402130000(10000:10000) 7f4402120000(10000:10000) 7f4402070000(10000:10000) 7f4402060000(10000:10000) 7f4402040000(10000:10000) 7f4402020000(10000:10000) 7f4401df0000(10000:10000) 7f4401d80000(10000:10000) 7f4401d60000(10000:10000) 7f4401d00000(10000:10000) 7f44019c0000(10000:10000) 7f4401850000(10000:10000) 7f44016e0000(10000:10000) 7f4401600000(10000:10000) 7f4401510000(10000:10000) 7f44012e0000(10000:10000) 7f4400f20000(10000:10000) 7f4400d40000(10000:10000) 7f4400d30000(10000:10000) 7f4400b70000(10000:10000) 7f4400a30000(10000:10000) 7f44009c0000(10000:10000) 7f44004e0000(10000:10000) 7f4400110000(10000:10000) 7f44000e0000(10000:10000) 7f44000d0000(10000:10000) 7f44000c0000(10000:10000) 7f44000b0000(10000:10000) 7f4400010000(10000:10000) 7f43ff4e4000(9000:5000) Size: 0xcc0000 (13369344) bytes total, 0x8000 (32768) bytes wasted.
StubHeap:             7f4405070000(10000:5000) 7f43ff4ed000(3000:3000) Size: 0x8000 (32768) bytes total.
IndirectionCellHeap:  7f4405690000(10000:5000) 7f43ff4f0000(6000:6000) Size: 0xb000 (45056) bytes total.
LookupHeap:           7f4404f70000(10000:6000) 7f43ff4ff000(4000:4000) Size: 0xa000 (40960) bytes total.
ResolveHeap:          7f43ff534000(57000:20000) Size: 0x20000 (131072) bytes total.
DispatchHeap:         7f43ff503000(31000:e000) Size: 0xe000 (57344) bytes total.
CacheEntryHeap:       7f4406220000(10000:3000) 7f43ff4f6000(9000:9000) Size: 0xc000 (49152) bytes total.
Total size:           Size: 0x1a52000 (27598848) bytes total, 0xbc000 (770048) bytes wasted.
----------------------------------------
Domain 1:             5558fbe901d0
LoaderAllocator:      5558fbe901d0
No unique loader heaps found.
----------------------------------------
JIT Manager:          5558fbe936c0
LoaderCodeHeap:       7f4400020000(80000:36000) Size: 0x36000 (221184) bytes total.
LoaderCodeHeap:       7f4401940000(80000:39000) Size: 0x39000 (233472) bytes total.
LoaderCodeHeap:       7f44032b0000(80000:42000) Size: 0x42000 (270336) bytes total.
LoaderCodeHeap:       7f4403be0000(80000:49000) Size: 0x49000 (299008) bytes total.
LoaderCodeHeap:       7f44044c0000(80000:5a000) Size: 0x5a000 (368640) bytes total.
LoaderCodeHeap:       7f4404840000(80000:5f000) Size: 0x5f000 (389120) bytes total.
LoaderCodeHeap:       7f4404a50000(200000:18f000) Size: 0x18f000 (1634304) bytes total.
LoaderCodeHeap:       7f4405290000(200000:1c4000) Size: 0x1c4000 (1851392) bytes total.
LoaderCodeHeap:       7f4405750000(200000:1b1000) Size: 0x1b1000 (1773568) bytes total.
LoaderCodeHeap:       7f4405ee0000(200000:1b6000) Size: 0x1b6000 (1794048) bytes total.
LoaderCodeHeap:       7f4406740000(200000:15f000) Size: 0x15f000 (1437696) bytes total.
HostCodeHeap:         7f4403c80000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4404ce0000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4405a50000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4405d80000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4406120000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4406260000(10000:10000) Size: 0x10000 (65536) bytes total.
HostCodeHeap:         7f4406290000(40000:40000) Size: 0x40000 (262144) bytes total.
HostCodeHeap:         7f4406340000(40000:40000) Size: 0x40000 (262144) bytes total.
HostCodeHeap:         7f4406b80000(40000:40000) Size: 0x40000 (262144) bytes total.
Total size:           Size: 0xaec000 (11452416) bytes total.
----------------------------------------

========================================
Number of GC Heaps: 1
----------------------------------------
generation 0 starts at 7f423d1f52c8
generation 1 starts at 7f423cc8b198
generation 2 starts at 7f43c7fff000
ephemeral segment allocation context: none
Small object heap
         segment            begin        allocated        committed allocated size         committed size        
    7f419d080000     7f419d081000     7f41acef9e78     7f41acf1a000 0xfe78e78 (266833528)  0xfe9a000 (266969088) 
    7f41bbfe0000     7f41bbfe1000     7f41cbb9b950     7f41cbbbc000 0xfbba950 (263956816)  0xfbdc000 (264093696) 
    7f4203ff0000     7f4203ff1000     7f4213feffb0     7f4213ff0000 0xfffefb0 (268431280)  0x10000000 (268435456)
    7f422bffa000     7f422bffb000     7f423a3a02f0     7f423a3c1000 0xe3a52f0 (238703344)  0xe3c7000 (238841856) 
    7f428bff0000     7f428bff1000     7f429bfeffe0     7f429bff0000 0xfffefe0 (268431328)  0x10000000 (268435456)
    7f429bff2000     7f429bff3000     7f42abff2000     7f42abff2000 0xffff000 (268431360)  0x10000000 (268435456)
    7f42abffe000     7f42abfff000     7f42bbffdfd0     7f42bbffe000 0xfffefd0 (268431312)  0x10000000 (268435456)
    7f42d3ffe000     7f42d3fff000     7f42e3ffe000     7f42e3ffe000 0xffff000 (268431360)  0x10000000 (268435456)
    7f42fb030000     7f42fb031000     7f430b030000     7f430b030000 0xffff000 (268431360)  0x10000000 (268435456)
    7f4317ffe000     7f4317fff000     7f4327ffe000     7f4327ffe000 0xffff000 (268431360)  0x10000000 (268435456)
    7f43c7ffe000     7f43c7fff000     7f43d7ffdfe8     7f43d7ffe000 0xfffefe8 (268431336)  0x10000000 (268435456)
    7f423bffe000     7f423bfff000     7f423d33f780     7f423e51b000 0x1340780 (20187008)   0x251d000 (38916096)  
Large object heap starts at 7f43d7fff000
         segment            begin        allocated        committed allocated size         committed size        
    7f40ed054000     7f40ed055000     7f40f1055038     7f40f1056000 0x4000038 (67108920)   0x4002000 (67117056)  
    7f41d3fe4000     7f41d3fe5000     7f41d7fe5038     7f41d8006000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41dbfe6000     7f41dbfe7000     7f41dffe7038     7f41e0008000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41ebfea000     7f41ebfeb000     7f41effeb038     7f41f000c000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41f3fec000     7f41f3fed000     7f41f7fed038     7f41f800e000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f41fbfee000     7f41fbfef000     7f41fffef038     7f4200010000 0x4000038 (67108920)   0x4022000 (67248128)  
    7f432fffe000     7f432ffff000     7f4337063918     7f4337084000 0x7064918 (117852440)  0x7086000 (117989376) 
    7f43d7ffe000     7f43d7fff000     7f43de5eb030     7f43de5ec000 0x65ec030 (106872880)  0x65ee000 (106881024) 
Pinned object heap starts at 7f43dffff000
         segment            begin        allocated        committed allocated size         committed size        
    7f43dfffe000     7f43dffff000     7f43e02bc4f8     7f43e02c0000 0x2bd4f8 (2872568)     0x2c2000 (2891776)    
------------------------------
GC Allocated Heap Size:    Size: 0xd4a1f110 (3567382800) bytes.
GC Committed Heap Size:    Size: 0xd5d3c000 (3587424256) bytes.

Total bytes consumed by CLR: 0xd827a000 (3626475520)

可以看到,CLR 吃掉了3.6G。Ok,到這兒基本可以確定是託管記憶體問題。

相信大家都注意到小物件堆和大物件堆有大量的記憶體段,吃掉了主要的託管記憶體。

接下來使用 dumpheap -stat 來統計下託管堆上記憶體所有物件:

0:000> !sos dumpheap -stat
Statistics:
          MT      Count   TotalSize Class Name
7f4406bf5ef0          1          24 NPOI.OpenXmlFormats.ExtendedPropertiesDocument
7f4406bf60a0          1          24 NPOI.OpenXmlFormats.CustomPropertiesDocument
7f4406bf67e8          1          24 NPOI.OpenXmlFormats.CT_CustomProperties
7f4406bf74e0          1          24 NPOI.OpenXmlFormats.CT_Property[]
7f4406bf8540          1          24 NPOI.HSSF.Util.HSSFColor+Black
7f4406bf8678          1          24 NPOI.HSSF.Util.HSSFColor+White
...
...
7f440213aa08      3,270     313,920 System.Reflection.Emit.DynamicMethod
7f44000c8080      4,135     358,828 System.Int32[]
7f4401519498     10,309     365,192 System.RuntimeType[]
7f440480b990          2     458,800 StackExchange.Redis.RawResult[]
7f440213faa0      3,270     470,880 System.Reflection.Emit.DynamicILGenerator
7f440185dcf8      5,133     492,768 System.Reflection.RuntimeParameterInfo
7f44000e7e60      3,216     551,160 System.String[]
7f440185e0a8      6,986     558,880 System.Signature
7f44012e8f10     10,812   1,124,448 System.Reflection.RuntimeMethodInfo
7f44000c7bc8     15,112   1,305,648 System.SByte[]
7f44000cd2e0     30,854   2,294,568 System.String
7f4400110eb0        224   5,064,502 System.Char[]
7f4400b784f0     18,748  21,734,523 System.Byte[]
7f44046efe70  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.EndPointEventArgs>
7f44047001b0  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.HashSlotMovedEventArgs>
7f4404700488  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.InternalErrorEventArgs>
7f44046efc10  6,541,452 418,652,928 System.EventHandler<StackExchange.Redis.RedisErrorEventArgs>
5558fbe656a0    293,562 474,504,032 Free
7f440001b0f8      5,222 538,033,592 System.Object[]
7f44046ef988 13,082,904 837,305,856 System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs>
Total 39,789,084 objects, 3,567,106,882 bytes

這個命令會按照 mt 彙總統計每個物件的數量及所佔記憶體大小,並按記憶體大小排序,我們看到詢問有幾個物件格外搶眼,其中最後一個有 1000w 個 EvevtHandler 物件,佔 837 M,那就繼續往下看。

罪魁禍首

我們需要看一下這 1000w 物件到底什麼來頭,因為上述是按 mt 彙總的,所以現在用 dumpheap -mt 展開看一下:

0:000> !sos dumpheap -mt 7f44046ef988
         Address               MT           Size
    7f419d0810c0     7f44046ef988             64 
    7f419d081100     7f44046ef988             64 
    7f419d081240     7f44046ef988             64  
...
    7f419d2f0e50     7f44046ef988             64 
    7f419d2f0f90     7f44046ef988             64 
    7f419d2f0fd0     7f44046ef988             64 

Statistics:
          MT  Count TotalSize Class Name
7f44046ef988 13,160   842,240 System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs>
Total 13,160 objects, 842,240 bytes

物件太多,不可能展示完,所以這裡做了一個省略。好的,接下來抽查幾個物件使用命令 dumpobj 看一下:

0:000> !sos dumpobj /d 7f419d2f0fd0
Name:        System.EventHandler`1[[StackExchange.Redis.ConnectionFailedEventArgs, StackExchange.Redis]]
MethodTable: 00007f44046ef988
EEClass:     00007f44046f8b58
Tracked Type: false
Size:        64(0x40) bytes
File:        /usr/share/dotnet/shared/Microsoft.NETCore.App/6.0.6/System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007f4400015290  40001bd        8        System.Object  0 instance 00007f419d2f0fd0 _target
00007f4400015290  40001be       10        System.Object  0 instance 0000000000000000 _methodBase
00007f44000c3eb8  40001bf       18        System.IntPtr  1 instance 00007F43FF4ED1E0 _methodPtr
00007f44000c3eb8  40001c0       20        System.IntPtr  1 instance 00007F4403BEF508 _methodPtrAux
00007f4400015290  4000243       28        System.Object  0 instance 0000000000000000 _invocationList
00007f44000c3eb8  4000244       30        System.IntPtr  1 instance 0000000000000000 _invocationCount

抽查幾個後發現,都是一樣。那既然這些物件一直不被GC回收,肯定有什麼原因,使用 gcroot 檢視一下都被誰引用:

0:000> !sos gcroot 7f419d2f0fd0
Caching GC roots, this may take a while.
Subsequent runs of this command will be faster.

HandleTable:
    00007f447a911310 (strong handle)
          -> 7f43e0008c28     System.Object[] 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer (static variable: System.Object._locker)
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

Thread 1f4f:
    7f44724de9b0 7f440488ac93 StackExchange.Redis.PhysicalBridge+<ProcessBacklogAsync>d__97.MoveNext()
        rbp-78: 00007f44724de9c8
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

        rbp-80: 00007f44724de9c0
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724dea50 7f44048888f9
        rbp-8: 00007f44724dea78
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724dea90 7f44048882fb
        rbp-10: 00007f44724dea90
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724deab0 7f44048882ac StackExchange.Redis.PhysicalBridge.ProcessBacklogAsync()
        rbp-30: 00007f44724deab0
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

        rbp-8: 00007f44724dead8
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

    7f44724deaf0 7f4404887f3f StackExchange.Redis.PhysicalBridge+<>c.<StartBacklogProcessor>b__93_0(System.Object)
        rbp-10: 00007f44724deaf0
          -> 7f43c81b7e30     StackExchange.Redis.PhysicalBridge 
          -> 7f43c81a4aa0     StackExchange.Redis.ConnectionMultiplexer 
          -> 7f423d33c828     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 
          -> 7f41f3fed020     System.Object[] 
          -> 7f419d2f0fd0     System.EventHandler<StackExchange.Redis.ConnectionFailedEventArgs> 

Found 8 unique roots.

注意:因為物件太多,遍歷查詢引用鏈需要一點兒時間,請耐心等待!

我們發現,它們都被 StackExchange.Redis.ConnectionMultiplexer 引用,這是一個 redis 連線物件。再經過大膽猜測和簡單分析,其餘幾個巨量物件列表都是這個 redis 物件的事件。說明程式中引入了 StackExchange 的 redis 擴充套件包,並在專案中某處使用了 redis 操作,這個物件到底在何處,可能出問題的程式碼又在哪兒?

定位程式碼

有經驗的朋友可能會知道,redis 這個物件大機率是一個靜態變數。現在我們來驗證一下,試圖查詢這個物件對應的靜態變數所在位置,使用 s-q 命令查詢是哪個記憶體地址(B)上儲存此這個 redis 物件地址(A)。

首先縮小記憶體查詢範圍:

0:000> !sos dumpobj /d 7f43e0008c28
Name:        System.Object[]
MethodTable: 00007f440001b0f8
EEClass:     00007f440001b070
Tracked Type: false
Size:        32664(0x7f98) bytes
Array:       Rank 1, Number of elements 4080, Type CLASS (Print Array)
Fields:
None

我們需要用到這個 Object 物件陣列的起始位置及數量用作如下命令的偏移:

0:000> s-q 7f43e0008c28 L?0x7f98 7f43c81a4aa0
00007f43`e000aaa0  00007f43`c81a4aa0 00007f43`ca4922a0

這樣,我們就找到了靜態變數的位置(B):

00007f43`e000aaa0

讓我們繼續查詢是哪些程式碼在使用這個靜態變數:

0:000> s-b 0 L?0xffffffffffffffff a0 aa 00 e0 43 7f 00 00
00007f44`044f9f81  a0 aa 00 e0 43 7f 00 00-33 ff 48 89 38 48 b8 6c  ....C...3.H.8H.l
00007f44`044fb96a  a0 aa 00 e0 43 7f 00 00-48 83 3e 00 40 0f 94 c6  ....C...H.>.@...
00007f44`044fb9b3  a0 aa 00 e0 43 7f 00 00-48 83 3f 00 74 38 48 bf  ....C...H.?.t8H.
00007f44`044fb9c3  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 b8 28 b0 6e  ....C...H.?H.(.n
00007f44`044fba71  a0 aa 00 e0 43 7f 00 00-48 8b b5 50 ff ff ff e8  ....C...H..P....
00007f44`044fbaa9  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d b8 48  ....C...H.?H.}.H
00007f44`044fbb03  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d a8 48  ....C...H.?H.}.H
00007f44`044fbb5d  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d 98 48  ....C...H.?H.}.H
00007f44`044fbbb7  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 7d 88 48  ....C...H.?H.}.H
00007f44`044fbc11  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 bd 78 ff  ....C...H.?H..x.
00007f44`044fbc74  a0 aa 00 e0 43 7f 00 00-48 8b 3f 48 89 bd 68 ff  ....C...H.?H..h.
00007f44`044fbcc7  a0 aa 00 e0 43 7f 00 00-48 8b 00 48 89 45 d0 90  ....C...H..H.E..

OK 我們找到不少,抽幾個看一下反彙編程式碼:

0:000> !sos u 00007f44`044fb96a
Normal JIT generated code
Wisder.xRiver.BaseCore.Util.xRedis.get_Instance()
ilAddr is 00007F447142F990 pImport is 00000143049FEE00
Begin 00007F44044FB890, size 485
00007f44`044fb890 55              push    rbp
00007f44`044fb891 4881ecd0000000  sub     rsp,0D0h
00007f44`044fb898 488dac24d0000000 lea     rbp,[rsp+0D0h]
00007f44`044fb8a0 33c0            xor     eax,eax
00007f44`044fb8a2 48898538ffffff  mov     qword ptr [rbp-0C8h],rax
00007f44`044fb8a9 c4413857c0      vxorps  xmm8,xmm8,xmm8
00007f44`044fb8ae 48b840ffffffffffffff mov rax,0FFFFFFFFFFFFFF40h
00007f44`044fb8b8 c5797f0428      vmovdqa xmmword ptr [rax+rbp],xmm8
00007f44`044fb8bd c5797f440510    vmovdqa xmmword ptr [rbp+rax+10h],xmm8
00007f44`044fb8c3 c5797f440520    vmovdqa xmmword ptr [rbp+rax+20h],xmm8
00007f44`044fb8c9 4883c030        add     rax,30h
00007f44`044fb8cd 75e9            jne     00007f44`044fb8b8
00007f44`044fb8cf 4889a530ffffff  mov     qword ptr [rbp-0D0h],rsp
00007f44`044fb8d6 48b870e26402447f0000 mov rax,7F440264E270h
00007f44`044fb8e0 833800          cmp     dword ptr [rax],0
00007f44`044fb8e3 7405            je      00007f44`044fb8ea
00007f44`044fb8e5 e88659dd74      call    libcoreclr!JIT_DbgIsJustMyCode (00007f44`792d1270)
00007f44`044fb8ea 90              nop
00007f44`044fb8eb 48bf90aa00e0437f0000 mov rdi,7F43E000AA90h
00007f44`044fb8f5 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fb8f8 393f            cmp     dword ptr [rdi],edi
00007f44`044fb8fa e829ccb2fb      call    00007f44`00028528 (System.String.get_Length(), mdToken: 0000000006000764)
00007f44`044fb8ff 8945cc          mov     dword ptr [rbp-34h],eax
00007f44`044fb902 837dcc00        cmp     dword ptr [rbp-34h],0
00007f44`044fb906 400f94c6        sete    sil
00007f44`044fb90a 400fb6f6        movzx   esi,sil
00007f44`044fb90e 8975fc          mov     dword ptr [rbp-4],esi
00007f44`044fb911 837dfc00        cmp     dword ptr [rbp-4],0
00007f44`044fb915 7451            je      00007f44`044fb968
00007f44`044fb917 90              nop
00007f44`044fb918 48bf90ec0c00447f0000 mov rdi,7F44000CEC90h (MT: System.Exception)
00007f44`044fb922 e809c8dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fb927 48898540ffffff  mov     qword ptr [rbp-0C0h],rax
00007f44`044fb92e bf49220000      mov     edi,2249h
00007f44`044fb933 48be00de6402447f0000 mov rsi,7F440264DE00h
00007f44`044fb93d e87ecedc74      call    libcoreclr!JIT_StrCns (00007f44`792c87c0)
00007f44`044fb942 48898538ffffff  mov     qword ptr [rbp-0C8h],rax
00007f44`044fb949 488bb538ffffff  mov     rsi,qword ptr [rbp-0C8h]
00007f44`044fb950 488bbd40ffffff  mov     rdi,qword ptr [rbp-0C0h]
00007f44`044fb957 e89cd1b2fb      call    00007f44`00028af8 (System.Exception..ctor(System.String), mdToken: 00000000060003CB)
00007f44`044fb95c 488bbd40ffffff  mov     rdi,qword ptr [rbp-0C0h]
00007f44`044fb963 e8f832dd74      call    libcoreclr!IL_Throw (00007f44`792cec60)
00007f44`044fb968 48bea0aa00e0437f0000 mov rsi,7F43E000AAA0h
00007f44`044fb972 48833e00        cmp     qword ptr [rsi],0
00007f44`044fb976 400f94c6        sete    sil
00007f44`044fb97a 400fb6f6        movzx   esi,sil
00007f44`044fb97e 8975f8          mov     dword ptr [rbp-8],esi
00007f44`044fb981 837df800        cmp     dword ptr [rbp-8],0
00007f44`044fb985 0f8409010000    je      00007f44`044fba94
00007f44`044fb98b 90              nop
00007f44`044fb98c 48be98aa00e0437f0000 mov rsi,7F43E000AA98h
00007f44`044fb996 488b36          mov     rsi,qword ptr [rsi]
00007f44`044fb999 488975f0        mov     qword ptr [rbp-10h],rsi
00007f44`044fb99d 33f6            xor     esi,esi
00007f44`044fb99f 8975e8          mov     dword ptr [rbp-18h],esi
00007f44`044fb9a2 488d75e8        lea     rsi,[rbp-18h]
00007f44`044fb9a6 488b7df0        mov     rdi,qword ptr [rbp-10h]
00007f44`044fb9aa e811dbb2fb      call    00007f44`000294c0 (System.Threading.Monitor.Enter(System.Object, Boolean ByRef), mdToken: 0000000006002BA0)
00007f44`044fb9af 90              nop
00007f44`044fb9b0 90              nop
00007f44`044fb9b1 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fb9bb 48833f00        cmp     qword ptr [rdi],0
00007f44`044fb9bf 7438            je      00007f44`044fb9f9
00007f44`044fb9c1 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fb9cb 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fb9ce 48b828b06e04447f0000 mov rax,7F44046EB028h
00007f44`044fb9d8 393f            cmp     dword ptr [rdi],edi
00007f44`044fb9da ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.get_IsConnected(), mdToken: 00000000060001A2)
00007f44`044fb9dc 89854cffffff    mov     dword ptr [rbp-0B4h],eax
00007f44`044fb9e2 83bd4cffffff00  cmp     dword ptr [rbp-0B4h],0
00007f44`044fb9e9 400f94c7        sete    dil
00007f44`044fb9ed 400fb6ff        movzx   edi,dil
00007f44`044fb9f1 89bd64ffffff    mov     dword ptr [rbp-9Ch],edi
00007f44`044fb9f7 eb0a            jmp     00007f44`044fba03
00007f44`044fb9f9 c78564ffffff01000000 mov dword ptr [rbp-9Ch],1
00007f44`044fba03 8bbd64ffffff    mov     edi,dword ptr [rbp-9Ch]
00007f44`044fba09 400fb6ff        movzx   edi,dil
00007f44`044fba0d 897de4          mov     dword ptr [rbp-1Ch],edi
00007f44`044fba10 837de400        cmp     dword ptr [rbp-1Ch],0
00007f44`044fba14 7470            je      00007f44`044fba86
00007f44`044fba16 90              nop
00007f44`044fba17 48bf90aa00e0437f0000 mov rdi,7F43E000AA90h
00007f44`044fba21 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fba24 e86ff6ffff      call    00007f44`044fb098 (StackExchange.Redis.ConfigurationOptions.Parse(System.String), mdToken: 0000000006000156)
00007f44`044fba29 48898558ffffff  mov     qword ptr [rbp-0A8h],rax
00007f44`044fba30 488bbd58ffffff  mov     rdi,qword ptr [rbp-0A8h]
00007f44`044fba37 48897dd8        mov     qword ptr [rbp-28h],rdi
00007f44`044fba3b 488b7dd8        mov     rdi,qword ptr [rbp-28h]
00007f44`044fba3f be10270000      mov     esi,2710h
00007f44`044fba44 393f            cmp     dword ptr [rdi],edi
00007f44`044fba46 e8edf7ffff      call    00007f44`044fb238 (StackExchange.Redis.ConfigurationOptions.set_ConnectTimeout(Int32), mdToken: 0000000006000121)
00007f44`044fba4b 90              nop
00007f44`044fba4c 488b7dd8        mov     rdi,qword ptr [rbp-28h]
00007f44`044fba50 be983a0000      mov     esi,3A98h
00007f44`044fba55 393f            cmp     dword ptr [rdi],edi
00007f44`044fba57 e82cf9ffff      call    00007f44`044fb388 (StackExchange.Redis.ConfigurationOptions.set_SyncTimeout(Int32), mdToken: 000000000600014B)
00007f44`044fba5c 90              nop
00007f44`044fba5d 488b7dd8        mov     rdi,qword ptr [rbp-28h]
00007f44`044fba61 33f6            xor     esi,esi
00007f44`044fba63 e838e3ffff      call    00007f44`044f9da0 (StackExchange.Redis.ConnectionMultiplexer.Connect(StackExchange.Redis.ConfigurationOptions, System.IO.TextWriter), mdToken: 00000000060001BC)
00007f44`044fba68 48898550ffffff  mov     qword ptr [rbp-0B0h],rax
00007f44`044fba6f 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fba79 488bb550ffffff  mov     rsi,qword ptr [rbp-0B0h]
00007f44`044fba80 e80cb7f274      call    libcoreclr!JIT_CheckedWriteBarrier (00007f44`79427191)
00007f44`044fba85 90              nop
00007f44`044fba86 90              nop
00007f44`044fba87 90              nop
00007f44`044fba88 eb00            jmp     00007f44`044fba8a
00007f44`044fba8a 488bfc          mov     rdi,rsp
00007f44`044fba8d e854020000      call    00007f44`044fbce6 (Wisder.xRiver.BaseCore.Util.xRedis.get_Instance(), mdToken: 00000000060000B3)
00007f44`044fba92 90              nop
00007f44`044fba93 90              nop
00007f44`044fba94 48bf88f96e04447f0000 mov rdi,7F44046EF988h (MT: System.EventHandler`1[[StackExchange.Redis.ConnectionFailedEventArgs, StackExchange.Redis]])
00007f44`044fba9e e88dc6dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbaa3 488945c0        mov     qword ptr [rbp-40h],rax
00007f44`044fbaa7 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbab1 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbab4 48897db8        mov     qword ptr [rbp-48h],rdi
00007f44`044fbab8 488b7dc0        mov     rdi,qword ptr [rbp-40h]
00007f44`044fbabc 33f6            xor     esi,esi
00007f44`044fbabe 48ba10f5be03447f0000 mov rdx,7F4403BEF510h
00007f44`044fbac8 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbad2 e8b1b6b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbad7 488b7db8        mov     rdi,qword ptr [rbp-48h]
00007f44`044fbadb 488b75c0        mov     rsi,qword ptr [rbp-40h]
00007f44`044fbadf 48b818b16e04447f0000 mov rax,7F44046EB118h
00007f44`044fbae9 393f            cmp     dword ptr [rdi],edi
00007f44`044fbaeb ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ConnectionFailed(System.EventHandler`1<StackExchange.Redis.ConnectionFailedEventArgs>), mdToken: 00000000060001F9)
00007f44`044fbaed 90              nop
00007f44`044fbaee 48bf88f96e04447f0000 mov rdi,7F44046EF988h (MT: System.EventHandler`1[[StackExchange.Redis.ConnectionFailedEventArgs, StackExchange.Redis]])
00007f44`044fbaf8 e833c6dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbafd 488945b0        mov     qword ptr [rbp-50h],rax
00007f44`044fbb01 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbb0b 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbb0e 48897da8        mov     qword ptr [rbp-58h],rdi
00007f44`044fbb12 488b7db0        mov     rdi,qword ptr [rbp-50h]
00007f44`044fbb16 33f6            xor     esi,esi
00007f44`044fbb18 48ba08f5be03447f0000 mov rdx,7F4403BEF508h
00007f44`044fbb22 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbb2c e857b6b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbb31 488b7da8        mov     rdi,qword ptr [rbp-58h]
00007f44`044fbb35 488b75b0        mov     rsi,qword ptr [rbp-50h]
00007f44`044fbb39 48b838b16e04447f0000 mov rax,7F44046EB138h
00007f44`044fbb43 393f            cmp     dword ptr [rdi],edi
00007f44`044fbb45 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ConnectionRestored(System.EventHandler`1<StackExchange.Redis.ConnectionFailedEventArgs>), mdToken: 00000000060001FF)
00007f44`044fbb47 90              nop
00007f44`044fbb48 48bf10fc6e04447f0000 mov rdi,7F44046EFC10h (MT: System.EventHandler`1[[StackExchange.Redis.RedisErrorEventArgs, StackExchange.Redis]])
00007f44`044fbb52 e8d9c5dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbb57 488945a0        mov     qword ptr [rbp-60h],rax
00007f44`044fbb5b 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbb65 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbb68 48897d98        mov     qword ptr [rbp-68h],rdi
00007f44`044fbb6c 488b7da0        mov     rdi,qword ptr [rbp-60h]
00007f44`044fbb70 33f6            xor     esi,esi
00007f44`044fbb72 48ba00f5be03447f0000 mov rdx,7F4403BEF500h
00007f44`044fbb7c 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbb86 e8fdb5b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbb8b 488b7d98        mov     rdi,qword ptr [rbp-68h]
00007f44`044fbb8f 488b75a0        mov     rsi,qword ptr [rbp-60h]
00007f44`044fbb93 48b878b16e04447f0000 mov rax,7F44046EB178h
00007f44`044fbb9d 393f            cmp     dword ptr [rdi],edi
00007f44`044fbb9f ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ErrorMessage(System.EventHandler`1<StackExchange.Redis.RedisErrorEventArgs>), mdToken: 000000000600020F)
00007f44`044fbba1 90              nop
00007f44`044fbba2 48bf70fe6e04447f0000 mov rdi,7F44046EFE70h (MT: System.EventHandler`1[[StackExchange.Redis.EndPointEventArgs, StackExchange.Redis]])
00007f44`044fbbac e87fc5dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbbb1 48894590        mov     qword ptr [rbp-70h],rax
00007f44`044fbbb5 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbbbf 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbbc2 48897d88        mov     qword ptr [rbp-78h],rdi
00007f44`044fbbc6 488b7d90        mov     rdi,qword ptr [rbp-70h]
00007f44`044fbbca 33f6            xor     esi,esi
00007f44`044fbbcc 48baf8f4be03447f0000 mov rdx,7F4403BEF4F8h
00007f44`044fbbd6 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbbe0 e8a3b5b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbbe5 488b7d88        mov     rdi,qword ptr [rbp-78h]
00007f44`044fbbe9 488b7590        mov     rsi,qword ptr [rbp-70h]
00007f44`044fbbed 48b848b16e04447f0000 mov rax,7F44046EB148h
00007f44`044fbbf7 393f            cmp     dword ptr [rdi],edi
00007f44`044fbbf9 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_ConfigurationChanged(System.EventHandler`1<StackExchange.Redis.EndPointEventArgs>), mdToken: 0000000006000202)
00007f44`044fbbfb 90              nop
00007f44`044fbbfc 48bfb0017004447f0000 mov rdi,7F44047001B0h (MT: System.EventHandler`1[[StackExchange.Redis.HashSlotMovedEventArgs, StackExchange.Redis]])
00007f44`044fbc06 e825c5dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbc0b 48894580        mov     qword ptr [rbp-80h],rax
00007f44`044fbc0f 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbc19 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbc1c 4889bd78ffffff  mov     qword ptr [rbp-88h],rdi
00007f44`044fbc23 488b7d80        mov     rdi,qword ptr [rbp-80h]
00007f44`044fbc27 33f6            xor     esi,esi
00007f44`044fbc29 48ba18f5be03447f0000 mov rdx,7F4403BEF518h
00007f44`044fbc33 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbc3d e846b5b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbc42 488bbd78ffffff  mov     rdi,qword ptr [rbp-88h]
00007f44`044fbc49 488b7580        mov     rsi,qword ptr [rbp-80h]
00007f44`044fbc4d 48b868b16e04447f0000 mov rax,7F44046EB168h
00007f44`044fbc57 393f            cmp     dword ptr [rdi],edi
00007f44`044fbc59 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_HashSlotMoved(System.EventHandler`1<StackExchange.Redis.HashSlotMovedEventArgs>), mdToken: 000000000600020C)
00007f44`044fbc5b 90              nop
00007f44`044fbc5c 48bf88047004447f0000 mov rdi,7F4404700488h (MT: System.EventHandler`1[[StackExchange.Redis.InternalErrorEventArgs, StackExchange.Redis]])
00007f44`044fbc66 e8c5c4dc74      call    libcoreclr!JIT_NewS_MP_FastPortable (00007f44`792c8130)
00007f44`044fbc6b 48898570ffffff  mov     qword ptr [rbp-90h],rax
00007f44`044fbc72 48bfa0aa00e0437f0000 mov rdi,7F43E000AAA0h
00007f44`044fbc7c 488b3f          mov     rdi,qword ptr [rdi]
00007f44`044fbc7f 4889bd68ffffff  mov     qword ptr [rbp-98h],rdi
00007f44`044fbc86 488bbd70ffffff  mov     rdi,qword ptr [rbp-90h]
00007f44`044fbc8d 33f6            xor     esi,esi
00007f44`044fbc8f 48ba20f5be03447f0000 mov rdx,7F4403BEF520h
00007f44`044fbc99 48b9e0d14eff437f0000 mov rcx,7F43FF4ED1E0h
00007f44`044fbca3 e8e0b4b2fb      call    00007f44`00027188 (System.MulticastDelegate.CtorOpened(System.Object, IntPtr, IntPtr), mdToken: 00000000060004ED)
00007f44`044fbca8 488bbd68ffffff  mov     rdi,qword ptr [rbp-98h]
00007f44`044fbcaf 488bb570ffffff  mov     rsi,qword ptr [rbp-90h]
00007f44`044fbcb6 48b828b16e04447f0000 mov rax,7F44046EB128h
00007f44`044fbcc0 393f            cmp     dword ptr [rdi],edi
00007f44`044fbcc2 ff10            call    qword ptr [rax] (StackExchange.Redis.ConnectionMultiplexer.add_InternalError(System.EventHandler`1<StackExchange.Redis.InternalErrorEventArgs>), mdToken: 00000000060001FC)
00007f44`044fbcc4 90              nop
00007f44`044fbcc5 48b8a0aa00e0437f0000 mov rax,7F43E000AAA0h
00007f44`044fbccf 488b00          mov     rax,qword ptr [rax]
00007f44`044fbcd2 488945d0        mov     qword ptr [rbp-30h],rax
00007f44`044fbcd6 90              nop
00007f44`044fbcd7 eb00            jmp     00007f44`044fbcd9
00007f44`044fbcd9 488b45d0        mov     rax,qword ptr [rbp-30h]
00007f44`044fbcdd 4881c4d0000000  add     rsp,0D0h
00007f44`044fbce4 5d              pop     rbp
00007f44`044fbce5 c3              ret
00007f44`044fbce6 55              push    rbp
00007f44`044fbce7 4883ec10        sub     rsp,10h
00007f44`044fbceb 488b2f          mov     rbp,qword ptr [rdi]
00007f44`044fbcee 48892c24        mov     qword ptr [rsp],rbp
00007f44`044fbcf2 488dadd0000000  lea     rbp,[rbp+0D0h]
00007f44`044fbcf9 8b7de8          mov     edi,dword ptr [rbp-18h]
00007f44`044fbcfc 400fb6ff        movzx   edi,dil
00007f44`044fbd00 85ff            test    edi,edi
00007f44`044fbd02 740a            je      00007f44`044fbd0e
00007f44`044fbd04 488b7df0        mov     rdi,qword ptr [rbp-10h]
00007f44`044fbd08 e83317dd74      call    libcoreclr!JIT_MonExit_Portable (00007f44`792cd440)
00007f44`044fbd0d 90              nop
00007f44`044fbd0e 90              nop
00007f44`044fbd0f 4883c410        add     rsp,10h
00007f44`044fbd13 5d              pop     rbp
00007f44`044fbd14 c3              ret

重點來了,Wisder.xRiver.BaseCore.Util.xRedis.get_Instance() 這個方法其實是 Instance 屬性的 get 方法。

可以繼續使用 ip2md 命令將指令地址轉換出模組資訊:

0:000> !sos ip2md 00007f44`044fb96a
MethodDesc:   00007f4403caf880
Method Name:          Wisder.xRiver.BaseCore.Util.xRedis.get_Instance()
Class:                00007f4403d1a278
MethodTable:          00007f4403cafe48
mdToken:              00000000060000B3
Module:               00007f440264de00
IsJitted:             yes
Current CodeAddr:     00007f44044fb890
Version History:
  ILCodeVersion:      0000000000000000
  ReJIT ID:           0
  IL Addr:            00007f447142f990
     CodeAddr:           00007f44044fb890  (MinOptJitted)
     NativeCodeVersion:  0000000000000000

找到模組後,檢視一下資訊就可以找到對應的 dll 檔案:

0:000> !sos DumpModule /d 00007f440264de00
Name: /xxx/xRiver.BaseCore.dll
Attributes:              PEFile IsFileLayout 
TransientFlags:          00209011 
Assembly:                00005558fc05f540
BaseAddress:             00007F447142A000
PEAssembly:              00005558FC03ABE0
ModuleId:                00007F440264E990
ModuleIndex:             0000000000000064
LoaderHeap:              0000000000000000
TypeDefToMethodTableMap: 00007F4402650000
TypeRefToMethodTableMap: 00007F4402650560
MethodDefToDescMap:      00007F4402650EE8
FieldDefToDescMap:       00007F4402653550
MemberRefToDescMap:      0000000000000000
FileReferencesMap:       00007F4402655948
AssemblyReferencesMap:   00007F4402655950
MetaData start address:  00007F4471444540 (148124 bytes)

那麼接下來就是找到這個 dll 檔案對應的原始碼專案,重點關注一下里面 get_Instance 方法。

namespace Wisder.xRiver.BaseCore.Util
{
    public static class xRedis
    {
        private static ConnectionMultiplexer _instance = null;

        /// <summary>
        /// 使用一個靜態屬性來返回已連線的例項,如下列中所示。
        /// 這樣,一旦 ConnectionMultiplexer 斷開連線,便可以初始化新的連線例項。
        /// </summary>
        public static ConnectionMultiplexer Instance
        {
            get
            {
                if (Constr.Length == 0)
                {
                    throw new Exception("Redis連線字串未設定!");
                }

                if (_instance == null)
                {
                    lock (_locker)
                    {
                        if (_instance == null || !_instance.IsConnected)
                        {
                            var options = ConfigurationOptions.Parse(Constr);
                            options.ConnectTimeout = 10000; // 設定連線超時時間為10秒
                            options.SyncTimeout = 15000; // 設定同步操作的超時時間為15秒

                            _instance = ConnectionMultiplexer.Connect(options); //Constr
                        }
                    }
                }

                //註冊如下事件
                _instance.ConnectionFailed += MuxerConnectionFailed;
                _instance.ConnectionRestored += MuxerConnectionRestored;
                _instance.ErrorMessage += MuxerErrorMessage;
                _instance.ConfigurationChanged += MuxerConfigurationChanged;
                _instance.HashSlotMoved += MuxerHashSlotMoved;
                _instance.InternalError += MuxerInternalError;
                return _instance;
            }
        }
    }
}

解決思路:把事件註冊挪到最裡面的 if 內部,_instance 賦值的後面。後續修改後可以觀察下情況。

總結分析

一個事件註冊會產生一個 64 Byte 大小的物件,不要小看它,積少成多,照樣會導致記憶體洩露問題。

這讓我不由得想起了 PHP 語言,一個請求結束就會銷燬程序,可有效避免記憶體洩露問題,實在不行再重啟一下 php 服務,也可以解決記憶體洩露問題。回到 .net 這一類平臺上,就可以好好寫程式碼,提高程式碼質量,避免記憶體洩露等問題。

相關文章