利用Windbg分析高記憶體佔用問題

微軟技術棧發表於2021-12-29

大家好,我是本期的微軟 MVP 實驗室研究員——馮輝。本篇文章主要介紹如何利用Windbg分析應用程式中的記憶體問題,從託管堆到非託管堆的探索以及到記憶體的分配,接下來我們一起來探索吧。

image.png

近期有幾位朋友使用我們的Magicodes.IE反饋在匯出過程中記憶體暴漲,接下來我們通過windbg來看一下什麼原因導致的。

我們先通過address -summary來看一下當前應用記憶體佔用量。

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    581     7df8`ef0c9000 ( 125.972 TB)           98.42%
<unknown>                              1678      206`ffb9e000 (   2.027 TB)  99.99%    1.58%
Image                                   950        0`064fd000 ( 100.988 MB)   0.00%    0.00%
Heap                                     58        0`050f6000 (  80.961 MB)   0.00%    0.00%
Stack                                   156        0`04380000 (  67.500 MB)   0.00%    0.00%
Other                                    11        0`019ad000 (  25.676 MB)   0.00%    0.00%
TEB                                      52        0`00068000 ( 416.000 kB)   0.00%    0.00%
PEB                                       1        0`00001000 (   4.000 kB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED                              282      200`038a6000 (   2.000 TB)  98.64%    1.56%
MEM_PRIVATE                            1674        7`07184000 (  28.111 GB)   1.35%    0.02%
MEM_IMAGE                               950        0`064fd000 ( 100.988 MB)   0.00%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                581     7df8`ef0c9000 ( 125.972 TB)           98.42%
MEM_RESERVE                             295      205`f8659000 (   2.023 TB)  99.79%    1.58%
MEM_COMMIT                             2611        1`188ce000 (   4.384 GB)   0.21%    0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         1595        1`0dc6c000 (   4.215 GB)   0.20%    0.00%
PAGE_EXECUTE_READ                       156        0`04d66000 (  77.398 MB)   0.00%    0.00%
PAGE_READONLY                           600        0`03851000 (  56.316 MB)   0.00%    0.00%
PAGE_NOACCESS                            99        0`021f2000 (  33.945 MB)   0.00%    0.00%
PAGE_EXECUTE_READWRITE                   19        0`0027b000 (   2.480 MB)   0.00%    0.00%
PAGE_WRITECOPY                           90        0`001a0000 (   1.625 MB)   0.00%    0.00%
PAGE_READWRITE | PAGE_GUARD              52        0`0009e000 ( 632.000 kB)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                    189`0413c000     7c6b`01ed4000 ( 124.418 TB)
<unknown>                              7dfb`2a153000      1f9`bd2ef000 (   1.976 TB)
Image                                  7ffc`883c1000        0`009ba000 (   9.727 MB)
Heap                                    183`0e9a1000        0`00f01000 (  15.004 MB)
Stack                                    37`62980000        0`0017b000 (   1.480 MB)
Other                                   183`77707000        0`01775000 (  23.457 MB)
TEB                                      37`62600000        0`00002000 (   8.000 kB)
PEB                                      37`627dd000        0`00001000 (   4.000 kB)

MEM_COMMIT佔用了4.384G,接下來我們利用eeheap -gc來檢查託管堆。

0:000> !eeheap -gc
GC Allocated Heap Size:    Size: 0x11ac2568 (296494440) bytes.
GC Committed Heap Size:    Size: 0x120e7000 (302936064) bytes.

根據這些記憶體來看,似乎問題不是這裡,大量的記憶體還是出現在非託管。我們利用Windows NT堆來看一下,其實在Windows中大多數的使用者堆分配器都在ntdll.dll中的NT堆管理器API(RtlAllocateHeap/RtlFreeHeap)上建立,比如說C中的malloc/free和new/delete,另外還有COM框架中的SysAllocString以及在Win32中的LocalAlloc、GlobalAlloc和HeapAlloc,雖然說這些分配器都會建立不同的堆來儲存它們的記憶體,但是他們最終都要呼叫ntdll.dll中的NT堆來實現。

0:000> !heap -s


************************************************************************************************************************
                                              NT HEAP STATS BELOW
************************************************************************************************************************
NtGlobalFlag enables following debugging aids for new heaps:
    stack back traces
LFH Key                   : 0x7cfd4cc2db4ddb4d
Termination on corruption : ENABLED
          Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                            (k)     (k)    (k)     (k) length      blocks cont. heap 
-------------------------------------------------------------------------------------
0000018378fd0000 08000002   65128  15296  64928   1720   177    17    2      c   LFH
    External fragmentation  11 % (177 free blocks)
00000183775c0000 08008000      64      4     64      2     1     1    0      0      
000001837aa90000 08001002    1280    108   1080     26     3     2    0      0   LFH
000001837ad20000 08001002      60      8     60      2     1     1    0      0      
000001837aca0000 08041002      60      8     60      5     1     1    0      0      
000001887bfd0000 08001002      60     20     60      1     2     1    0      0      
000001830cf30000 08001002    3324   1364   3124     19    10     3    0      0   LFH
000001830ce30000 08001002      60      8     60      5     1     1    0      0      
-------------------------------------------------------------------------------------

輸出結果如上所示,NT堆內容好少....什麼原因....好吧根據maoni所說,似乎是驗證出了問題。

在Windows上面所有的user mode allocations最終都是通過VirtualAlloc來獲得記憶體,bitmaps也好,GC heap也好。

不同的是你去直接呼叫VirtualAlloc還是使用其他方式去呼叫它。如果是不託管的記憶體,GC並不管轄它,當然也不知道它們的存在。

GC沒有管轄這些記憶體,所以說還是我們編寫的程式碼有問題,我們返過來再考慮一個事情,“匯出進行時,記憶體會大量增加,匯出完成後記憶體會降低下去”。我們來看一下程式碼,如下所示,其實我們現在明白的是,在我們執行期間肯定是這些記憶體一直“持有”,並沒有被釋放掉。

app.MapGet("/excel", async content =>
{
    string path = Path.Combine(Directory.GetCurrentDirectory(), "test.xlsx");
    List<TestDto> list = new();
    for (int i = 0; i < 400; i++)
    {
        list.Add(new TestDto
        {
            ImageUrl = "https://gimg2.baidu.com/image_search/src=http%3A%2F%2Fup.enterdesk.com%2Fedpic_source%2F53%2F0a%2Fda%2F530adad966630fce548cd408237ff200.jpg&refer=http%3A%2F%2Fup.enterdesk.com&app=2002&size=f9999,10000&q=a80&n=0&g=0n&fmt=jpeg?sec=1641193100&t=417a589da8c9ba3103ed74c33fbd6c70"
        });
    }
    Stopwatch stopwatch = Stopwatch.StartNew();
    ExcelExporter exporter = new ExcelExporter();
    await exporter.Export(path, list);
    stopwatch.Stop();
    await content.Response.WriteAsync(stopwatch.Elapsed.TotalSeconds.ToString());
});

根據記憶體的表現和我們的理論,我們繼續利用Windbg來排查一下,現在其實我們可以發現,這些物件最終還是被GC收回了,帶著理論我們繼續構思,GC是知道哪些物件可以終結的對吧?並且它們在變成不可到達時呼叫它們的終結器,在GC中會利用finalization queue來記錄這些終結物件。所以說我們是不是可以查一下?如下所示,我們來看一下。

0:000> !finalizequeue
----------------------------------
Statistics for all finalizable objects (including all objects ready for finalization):
              MT    Count    TotalSize Class Name
00007ffc2dc23818        1           24 System.Net.Security.SafeCredentialReference
00007ffc2dac4238        1           24 System.WeakReference
00007ffc2d6eb908        1           24 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]]
00007ffc2d6e4120        1           24 System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]]
00007ffc2d572b68        1           24 System.WeakReference`1[[Microsoft.Extensions.DependencyInjection.ServiceProvider, Microsoft.Extensions.DependencyInjection]]
00007ffc2d429258        1           24 System.WeakReference`1[[System.IO.FileSystemWatcher, System.IO.FileSystem.Watcher]]
00007ffc2dd15c20        1           32 Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle
00007ffc2d6de4d8        1           32 Internal.Cryptography.Pal.Native.SafeLocalAllocHandle
00007ffc2d68fa00        1           32 Internal.Cryptography.Pal.Native.SafeCertStoreHandle
00007ffc2d3a5cc0        1           32 System.Net.Quic.Implementations.MsQuic.Internal.SafeMsQuicRegistrationHandle
00007ffc2db390c8        1           40 Interop+WinHttp+SafeWinHttpHandle
00007ffc2d69a420        1           40 Internal.Cryptography.Pal.Native.SafeCertContextHandle
00007ffc2d5bea18        1           40 System.Diagnostics.EventLog
00007ffc2dc29a38        1           48 System.Net.Security.SafeFreeCredential_SECURITY
00007ffc2d963f80        2           48 System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]]
00007ffc2d7a3750        2           48 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]]
00007ffc2d685e10        1           56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Char, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
00007ffc2d44c4d0        1           56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Byte, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
00007ffc2d96be68        1           64 CellStore`1[[System.Uri, System.Private.Uri]]
00007ffc2d96b780        1           64 FlagCellStore
00007ffc2d96af48        1           64 CellStore`1[[System.Object, System.Private.CoreLib]]
00007ffc2d96a5b8        1           64 CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]]
00007ffc2d6ddab8        2           64 Internal.Cryptography.Pal.Native.SafeChainEngineHandle
00007ffc2d69d528        2           64 Internal.Win32.SafeHandles.SafeRegistryHandle
00007ffc2d685bc8        2           64 Microsoft.Win32.SafeHandles.SafeWaitHandle
00007ffc2d685280        3           72 System.Threading.ThreadInt64PersistentCounter+ThreadLocalNodeFinalizationHelper
00007ffc2d5f5f50        3           72 System.Runtime.InteropServices.PosixSignalRegistration
00007ffc2d4299d0        1           72 Microsoft.Win32.SafeHandles.SafeFileHandle
00007ffc2d6e40b8        1           80 System.Runtime.Loader.DefaultAssemblyLoadContext
00007ffc2dac9ed0        2           96 PageIndex
00007ffc2d96d0c8        2           96 ColumnIndex
00007ffc2d464470        3          120 System.Gen2GcCallback
00007ffc2d40a620        1          120 System.IO.FileSystemWatcher
00007ffc2d96bc18        2          128 CellStore`1[[System.Int32, System.Private.CoreLib]]
00007ffc2dac20c8        2          144 System.Reflection.Emit.DynamicResolver
00007ffc2d680f10        3          144 System.Threading.LowLevelLock
00007ffc2d683c48        3          168 System.Threading.ThreadPoolWorkQueueThreadLocals
00007ffc2d681e80        1          176 System.Threading.LowLevelLifoSemaphore
00007ffc2dc25ef0        1          184 System.Collections.Concurrent.CDSCollectionETWBCLProvider
00007ffc2db8e658        1          184 System.Net.NetEventSource
00007ffc2db8c378        1          184 System.Net.NetEventSource
00007ffc2db38f90        1          184 System.Net.NetEventSource
00007ffc2d90c658        1          184 Microsoft.IO.RecyclableMemoryStreamManager+Events
00007ffc2d689b48        1          184 Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource
00007ffc2d66f9f8        1          184 System.Diagnostics.Tracing.FrameworkEventSource
00007ffc2d66b720        1          184 System.Net.NetEventSource
00007ffc2d44d128        1          184 System.Buffers.ArrayPoolEventSource
00007ffc2d2e2ec8        1          184 System.Diagnostics.Tracing.NativeRuntimeEventSource
00007ffc2d694e10        1          192 System.Threading.Tasks.TplEventSource
00007ffc2d572ab0        1          192 Microsoft.Extensions.DependencyInjection.DependencyInjectionEventSource
00007ffc2d505f00        1          200 Microsoft.Extensions.Logging.EventSource.LoggingEventSource
00007ffc2db8ade8        1          224 System.Net.NameResolutionTelemetry
00007ffc2d428b08        7          224 System.Threading.PreAllocatedOverlapped
00007ffc2d563c78        1          232 System.Diagnostics.DiagnosticSourceEventSource
00007ffc2d61fe88        1          240 Microsoft.AspNetCore.Hosting.HostingEventSource
00007ffc2db6b788        8          256 System.Threading.TimerQueue+AppDomainTimerSafeHandle
00007ffc2d690270        1          280 System.Net.Sockets.SocketsTelemetry
00007ffc2db6bc80        1          296 System.Net.Http.HttpTelemetry
00007ffc2d68b998        1          336 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource
00007ffc2dc21998        1          360 System.Net.Security.NetSecurityTelemetry
00007ffc2d2dae28        1          384 System.Diagnostics.Tracing.RuntimeEventSource
00007ffc2d66ad60       10          480 System.Net.Sockets.SafeSocketHandle
00007ffc2d2e0240       21          504 System.WeakReference`1[[System.Diagnostics.Tracing.EventSource, System.Private.CoreLib]]
00007ffc2d2b0538        9          648 System.Threading.Thread
00007ffc2d77a188        2          704 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver
00007ffc2d90cec0        6          960 Microsoft.IO.RecyclableMemoryStream
00007ffc2d5fc658       10         1280 System.Net.Sockets.Socket
00007ffc2d68d898        4         1536 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs
00007ffc2d2dc778       42         4704 System.Diagnostics.Tracing.EventSource+OverrideEventProvider
00007ffc2daec058      356        14240 System.Drawing.Bitmap
Total 553 objects

WOW!!!,看上面356個System.Drawing.Bitmap在等待回收,看起來這是我們的影響因素,我們來查一下程式碼。

try
{
    cell.Value = string.Empty;
    Bitmap bitmap;
    if (url.IsBase64StringValid())
    {
        bitmap = url.Base64StringToBitmap();
    }
    else
    {
        bitmap = Extension.GetBitmapByUrl(url);
    }

    if (bitmap == null)
    {
        cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt;
    }
    else
    {
        ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap);
        AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)),
            colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset);
        CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height;
        pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height);
    }

}
catch (Exception)
{
    cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt;
}

在ExcelPicture物件中去使用Bitmap物件,對於線上圖片源來說,我們會讀取並儲存到Bitmap中,但是我們發現並沒有對該物件進行釋放操作,所以導致大量的Bitmap一直沒有釋放,我們通過using來處理一下。

using (ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap))
{
    AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)),
        colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset);
    CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height;
    pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height);
}

一個帶有終結器的新物件是必須要被新增進finalization queue中的,這個行為也被稱為“終結註冊(registering for finalization)”。當然我也建議你選擇使用SOSEX擴充套件外掛,它提供了finalization類似的內容,似乎看起來更直觀一些,如下所示。

下載地址:http://www.stevestechspot.com...


:000> .load D:\sosex_64\sosex.dll
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000> !finq -stat
Generation 0:
       Count      Total Size   Type
---------------------------------------------------------
          54            2160   System.Drawing.Bitmap

54 objects, 2,160 bytes

Generation 1:
       Count      Total Size   Type
---------------------------------------------------------
           1             184   Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource
           1             336   Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource
           4            1536   System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs
           1              32   Internal.Cryptography.Pal.Native.SafeCertStoreHandle
           1             280   System.Net.Sockets.SocketsTelemetry
           1             192   System.Threading.Tasks.TplEventSource
           1              40   Internal.Cryptography.Pal.Native.SafeCertContextHandle
           2              64   Internal.Win32.SafeHandles.SafeRegistryHandle
           2              64   Internal.Cryptography.Pal.Native.SafeChainEngineHandle
           1              32   Internal.Cryptography.Pal.Native.SafeLocalAllocHandle
           1              80   System.Runtime.Loader.DefaultAssemblyLoadContext
           1              24   System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]]
           1              24   System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]]
           2             704   Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver
           2              48   System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]]
           1             184   Microsoft.IO.RecyclableMemoryStreamManager+Events
           6             960   Microsoft.IO.RecyclableMemoryStream
           2              48   System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]]
           1              64   CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]]
           1              64   CellStore`1[[System.Object, System.Private.CoreLib]]
           1              64   FlagCellStore
           2             128   CellStore`1[[System.Int32, System.Private.CoreLib]]
           1              64   CellStore`1[[System.Uri, System.Private.Uri]]
           2              96   ColumnIndex
           2             144   System.Reflection.Emit.DynamicResolver
           1              24   System.WeakReference
           2              96   PageIndex
         302           12080   System.Drawing.Bitmap
           1             184   System.Net.NetEventSource
           1              40   Interop+WinHttp+SafeWinHttpHandle
           8             256   System.Threading.TimerQueue+AppDomainTimerSafeHandle
           1             296   System.Net.Http.HttpTelemetry
           1             224   System.Net.NameResolutionTelemetry
           1             184   System.Net.NetEventSource
           1             184   System.Net.NetEventSource
           1             360   System.Net.Security.NetSecurityTelemetry
           1              24   System.Net.Security.SafeCredentialReference
           1             184   System.Collections.Concurrent.CDSCollectionETWBCLProvider
           1              48   System.Net.Security.SafeFreeCredential_SECURITY
           1              32   Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle

499 objects, 30,736 bytes

Generation 2:
0 objects, 0 bytes

TOTAL: 553 objects, 32,896 bytes

可能大家都會像我一開始有個疑問,你這個圖片我看了...沒有那麼大,並且在Windbg中也沒有表現大小呀。首先我們先來看一下這個圖片的質量。圖片的畫素為2560x1440,位深為24目前已知這些資訊,我們計算一下未壓縮的圖片大小。

2560x1440x24/8

10M左右一張圖,已知圖片數x10M=3G,其實對於這個問題來說,這並不屬於記憶體洩漏。

總結

這篇文章主要介紹如何利用Windbg分析應用程式中的記憶體問題,從託管堆到非託管堆的探索以及到記憶體的分配,最終根據記憶體的表現和理論確認記憶體的問題,當然對於記憶體分析建議大家不一定非要鍾情一個工具,當然可以結合著PerfView一起做也許效果更佳。

微軟最有價值專家(MVP)

bc93fde364ea9dd3d9106b58e805b770.png

微軟最有價值專家是微軟公司授予第三方技術專業人士的一個全球獎項。28年來,世界各地的技術社群領導者,因其線上上和線下的技術社群中分享專業知識和經驗而獲得此獎項。

MVP是經過嚴格挑選的專家團隊,他們代表著技術最精湛且最具智慧的人,是對社群投入極大的熱情並樂於助人的專家。MVP致力於通過演講、論壇問答、建立網站、撰寫部落格、分享視訊、開源專案、組織會議等方式來幫助他人,並最大程度地幫助微軟技術社群使用者使用Microsoft技術。
更多詳情請登入官方網站:
https://mvp.microsoft.com/zh-cn


歡迎關注微軟中國MSDN訂閱號,獲取更多最新發布!
image.png

相關文章