關於記一次 Go 服務記憶體洩漏問題調查
記錄下這個問題是因為這不同於傳統典型的 Go 服務記憶體洩漏。
8月15日前後,釘釘監控告警群不定時出現服務機器記憶體佔用超 80%
告警,一開始以為是 Prometheus exporter 堆積造成的,準備下掉 Prometheus
exporter 的整合程式碼,後來覺 得這個堆積速度太快了點,不到一天就吃掉
6GB,十分誇張,所以有空的時候用 gops 快照了 一下,記憶體情況如下圖
https://github.com/users/mchinee/projects/1
https://github.com/users/mchinee/projects/2
https://github.com/users/mchinee/projects/3
https://github.com/users/mchinee/projects/4
https://github.com/users/mchinee/projects/5
https://github.com/users/mchinee/projects/6
https://github.com/users/mchinee/projects/7
https://github.com/users/mchinee/projects/8
https://github.com/users/mchinee/projects/9
https://github.com/users/mchinee/projects/10
https://github.com/users/mchinee/projects/11
https://github.com/users/mchinee/projects/12
https://github.com/users/mchinee/projects/13
https://github.com/users/mchinee/projects/14
https://github.com/users/mchinee/projects/15
https://github.com/users/mchinee/projects/16
https://github.com/users/mchinee/projects/17
https://github.com/users/mchinee/projects/18
https://github.com/users/mchinee/projects/19
https://github.com/users/mchinee/projects/20
https://github.com/users/mchinee/projects/21
https://github.com/users/mchinee/projects/22
https://github.com/users/mchinee/projects/23
https://github.com/users/mchinee/projects/24
https://github.com/users/mchinee/projects/25
https://github.com/users/mchinee/projects/26
https://github.com/users/mchinee/projects/27
https://github.com/users/mchinee/projects/28
https://github.com/users/mchinee/projects/29
https://github.com/users/mchinee/projects/30
https://github.com/users/mchinee/projects/31
https://github.com/users/mchinee/projects/32
https://github.com/users/mchinee/projects/33
https://github.com/users/mchinee/projects/34
https://github.com/users/mchinee/projects/35
https://github.com/users/mchinee/projects/36
https://github.com/users/mchinee/projects/37
https://github.com/users/mchinee/projects/38
https://github.com/users/mchinee/projects/39
https://github.com/users/mchinee/projects/40
https://github.com/users/mchinee/projects/41
https://github.com/users/mchinee/projects/42
https://github.com/users/mchinee/projects/43
https://github.com/users/mchinee/projects/44
https://github.com/users/mchinee/projects/45
https://github.com/users/mchinee/projects/46
https://github.com/users/mchinee/projects/47
https://github.com/users/mchinee/projects/48
https://github.com/users/mchinee/projects/49
https://github.com/users/mchinee/projects/50
https://github.com/users/mchinee/projects/51
https://github.com/users/mchinee/projects/52
https://github.com/users/mchinee/projects/53
https://github.com/users/mchinee/projects/54
https://github.com/users/mchinee/projects/55
https://github.com/users/mchinee/projects/56
https://github.com/users/mchinee/projects/57
https://github.com/users/mchinee/projects/58
https://github.com/users/mchinee/projects/59
https://github.com/users/mchinee/projects/60
https://github.com/users/mchinee/projects/61
https://github.com/users/mchinee/projects/62
https://github.com/users/mchinee/projects/63
https://github.com/users/mchinee/projects/64
https://github.com/users/mchinee/projects/65
https://github.com/users/mchinee/projects/66
https://github.com/users/mchinee/projects/67
https://github.com/users/mchinee/projects/68
https://github.com/users/mchinee/projects/69
https://github.com/users/mchinee/projects/70
https://github.com/users/mchinee/projects/71
https://github.com/users/mchinee/projects/72
https://github.com/users/mchinee/projects/73
https://github.com/users/mchinee/projects/74
https://github.com/users/mchinee/projects/75
https://github.com/users/mchinee/projects/76
https://github.com/users/mchinee/projects/77
https://github.com/users/mchinee/projects/78
https://github.com/users/mchinee/projects/79
https://github.com/users/mchinee/projects/80
https://github.com/users/mchinee/projects/81
https://github.com/users/mchinee/projects/82
https://github.com/users/mchinee/projects/83
https://github.com/users/mchinee/projects/84
https://github.com/users/mchinee/projects/85
https://github.com/users/mchinee/projects/86
https://github.com/users/mchinee/projects/87
https://github.com/users/mchinee/projects/88
https://github.com/users/mchinee/projects/89
https://github.com/users/mchinee/projects/90
https://github.com/users/mchinee/projects/91
https://github.com/users/mchinee/projects/92
https://github.com/users/mchinee/projects/93
https://github.com/users/mchinee/projects/94
https://github.com/users/mchinee/projects/95
https://github.com/users/mchinee/projects/96
https://github.com/users/mchinee/projects/97
https://github.com/users/mchinee/projects/98
https://github.com/users/mchinee/projects/99
https://github.com/users/mchinee/projects/100
https://github.com/users/mchinee/projects/101
https://github.com/users/mchinee/projects/102
https://github.com/users/mchinee/projects/103
https://github.com/users/mchinee/projects/104
https://github.com/users/mchinee/projects/105
https://github.com/users/mchinee/projects/106
https://github.com/users/mchinee/projects/107
https://github.com/users/mchinee/projects/108
https://github.com/users/mchinee/projects/109
https://github.com/users/mchinee/projects/110
https://github.com/users/mchinee/projects/111
https://github.com/users/mchinee/projects/112
https://github.com/users/mchinee/projects/113
https://github.com/users/mchinee/projects/114
https://github.com/users/mchinee/projects/115
https://github.com/users/mchinee/projects/116
https://github.com/users/mchinee/projects/117
https://github.com/users/mchinee/projects/118
https://github.com/users/mchinee/projects/119
https://github.com/users/mchinee/projects/120
https://github.com/users/mchinee/projects/121
https://github.com/users/mchinee/projects/122
https://github.com/users/mchinee/projects/123
https://github.com/users/mchinee/projects/124
https://github.com/users/mchinee/projects/125
https://github.com/users/mchinee/projects/126
https://github.com/users/mchinee/projects/127
https://github.com/users/mchinee/projects/128
https://github.com/users/mchinee/projects/129
https://github.com/users/mchinee/projects/130
https://github.com/users/mchinee/projects/131
https://github.com/users/mchinee/projects/132
https://github.com/users/mchinee/projects/133
https://github.com/users/mchinee/projects/134
https://github.com/users/mchinee/projects/135
https://github.com/users/mchinee/projects/136
https://github.com/users/mchinee/projects/137
https://github.com/users/mchinee/projects/138
https://github.com/users/mchinee/projects/139
https://github.com/users/mchinee/projects/140
https://github.com/users/mchinee/projects/141
https://github.com/users/mchinee/projects/142
https://github.com/users/mchinee/projects/143
https://github.com/users/mchinee/projects/144
https://github.com/users/mchinee/projects/145
https://github.com/users/mchinee/projects/146
https://github.com/users/mchinee/projects/147
https://github.com/users/mchinee/projects/148
https://github.com/users/mchinee/projects/149
https://github.com/users/mchinee/projects/150
https://github.com/users/mchinee/projects/151
https://github.com/users/mchinee/projects/152
https://github.com/users/mchinee/projects/153
https://github.com/users/mchinee/projects/154
https://github.com/users/mchinee/projects/155
https://github.com/users/mchinee/projects/156
https://github.com/users/mchinee/projects/157
https://github.com/users/mchinee/projects/158
https://github.com/users/mchinee/projects/159
https://github.com/users/mchinee/projects/160
https://github.com/users/mchinee/projects/161
https://github.com/users/mchinee/projects/162
https://github.com/users/mchinee/projects/163
https://github.com/users/mchinee/projects/164
https://github.com/users/mchinee/projects/165
https://github.com/users/mchinee/projects/166
https://github.com/users/mchinee/projects/167
https://github.com/users/mchinee/projects/168
https://github.com/users/mchinee/projects/169
https://github.com/users/mchinee/projects/170
https://github.com/users/mchinee/projects/171
https://github.com/users/mchinee/projects/172
https://github.com/users/mchinee/projects/173
https://github.com/users/mchinee/projects/174
https://github.com/users/mchinee/projects/175
https://github.com/users/mchinee/projects/176
https://github.com/users/mchinee/projects/177
https://github.com/users/mchinee/projects/178
https://github.com/users/mchinee/projects/179
https://github.com/users/mchinee/projects/180
https://github.com/users/mchinee/projects/181
https://github.com/users/mchinee/projects/182
https://github.com/users/mchinee/projects/183
https://github.com/users/mchinee/projects/184
https://github.com/users/mchinee/projects/185
https://github.com/users/mchinee/projects/186
https://github.com/users/mchinee/projects/187
https://github.com/users/mchinee/projects/188
https://github.com/users/mchinee/projects/189
https://github.com/users/mchinee/projects/190
https://github.com/users/mchinee/projects/191
https://github.com/users/mchinee/projects/192
https://github.com/users/mchinee/projects/193
https://github.com/users/mchinee/projects/194
https://github.com/users/mchinee/projects/195
https://github.com/users/mchinee/projects/196
https://github.com/users/mchinee/projects/197
https://github.com/users/mchinee/projects/198
https://github.com/users/mchinee/projects/199
https://github.com/users/mchinee/projects/200
https://github.com/users/mchinee/projects/201
https://github.com/users/mchinee/projects/202
https://github.com/users/mchinee/projects/203
https://github.com/users/mchinee/projects/204
https://github.com/users/mchinee/projects/205
https://github.com/users/mchinee/projects/206
https://github.com/users/mchinee/projects/207
https://github.com/users/mchinee/projects/208
https://github.com/users/mchinee/projects/209
https://github.com/users/mchinee/projects/210
https://github.com/users/mchinee/projects/211
https://github.com/users/mchinee/projects/212
https://github.com/users/mchinee/projects/213
https://github.com/users/mchinee/projects/214
https://github.com/users/mchinee/projects/215
https://github.com/users/mchinee/projects/216
https://github.com/users/mchinee/projects/217
https://github.com/users/mchinee/projects/218
https://github.com/users/mchinee/projects/219
https://github.com/users/mchinee/projects/220
https://github.com/users/mchinee/projects/221
https://github.com/users/mchinee/projects/222
https://github.com/users/mchinee/projects/223
https://github.com/users/mchinee/projects/224
https://github.com/users/mchinee/projects/225
https://github.com/users/mchinee/projects/226
https://github.com/users/mchinee/projects/227
https://github.com/users/mchinee/projects/228
https://github.com/users/mchinee/projects/229
https://github.com/users/mchinee/projects/230
https://github.com/users/mchinee/projects/231
https://github.com/users/mchinee/projects/232
https://github.com/users/mchinee/projects/233
https://github.com/users/mchinee/projects/234
https://github.com/users/mchinee/projects/235
https://github.com/users/mchinee/projects/236
https://github.com/users/mchinee/projects/237
https://github.com/users/mchinee/projects/238
https://github.com/users/mchinee/projects/239
https://github.com/users/mchinee/projects/240
https://github.com/users/mchinee/projects/241
https://github.com/users/mchinee/projects/242
https://github.com/users/mchinee/projects/243
https://github.com/users/mchinee/projects/244
https://github.com/users/mchinee/projects/245
https://github.com/users/mchinee/projects/246
https://github.com/users/mchinee/projects/247
https://github.com/users/mchinee/projects/248
https://github.com/users/mchinee/projects/249
https://github.com/users/mchinee/projects/250
https://github.com/users/mchinee/projects/251
https://github.com/users/mchinee/projects/252
https://github.com/users/mchinee/projects/253
https://github.com/users/mchinee/projects/254
https://github.com/users/mchinee/projects/255
https://github.com/users/mchinee/projects/256
https://github.com/users/mchinee/projects/257
https://github.com/users/mchinee/projects/258
https://github.com/users/mchinee/projects/259
https://github.com/users/mchinee/projects/260
https://github.com/users/mchinee/projects/261
https://github.com/users/mchinee/projects/262
https://github.com/users/mchinee/projects/263
https://github.com/users/mchinee/projects/264
https://github.com/users/mchinee/projects/265
https://github.com/users/mchinee/projects/266
https://github.com/users/mchinee/projects/267
https://github.com/users/mchinee/projects/268
https://github.com/users/mchinee/projects/269
https://github.com/users/mchinee/projects/270
https://github.com/users/mchinee/projects/271
https://github.com/users/mchinee/projects/272
https://github.com/users/mchinee/projects/273
https://github.com/users/mchinee/projects/274
https://github.com/users/mchinee/projects/275
https://github.com/users/mchinee/projects/276
https://github.com/users/mchinee/projects/277
https://github.com/users/mchinee/projects/278
https://github.com/users/mchinee/projects/279
https://github.com/users/mchinee/projects/280
https://github.com/users/mchinee/projects/281
https://github.com/users/mchinee/projects/282
https://github.com/users/mchinee/projects/283
https://github.com/users/mchinee/projects/284
https://github.com/users/mchinee/projects/285
https://unsplash.com/collections/69915378/
https://unsplash.com/collections/74812508/
https://unsplash.com/collections/60915643/
https://unsplash.com/collections/98939806/
https://unsplash.com/collections/18698100/
https://unsplash.com/collections/23930917/
https://unsplash.com/collections/27236459/
https://unsplash.com/collections/27989740/
https://unsplash.com/collections/39809727/
https://unsplash.com/collections/18792076/
https://unsplash.com/collections/15669474/
https://unsplash.com/collections/49306324/
https://unsplash.com/collections/35866117/
https://unsplash.com/collections/82394632/
https://unsplash.com/collections/25347124/
https://unsplash.com/collections/47459128/
https://unsplash.com/collections/63196118/
https://unsplash.com/collections/61964753/
https://unsplash.com/collections/58125121/
https://unsplash.com/collections/73265572/
https://unsplash.com/collections/43950857/
https://unsplash.com/collections/79152325/
https://unsplash.com/collections/54861433/
https://unsplash.com/collections/28150607/
https://unsplash.com/collections/90400295/
https://unsplash.com/collections/15974965/
https://unsplash.com/collections/44655825/
https://unsplash.com/collections/91730201/
https://unsplash.com/collections/80009231/
https://unsplash.com/collections/26813616/
https://unsplash.com/collections/16600248/
https://unsplash.com/collections/67127874/
https://unsplash.com/collections/51375357/
https://unsplash.com/collections/46641606/
https://unsplash.com/collections/38602667/
https://unsplash.com/collections/87796444/
https://unsplash.com/collections/37100261/
https://unsplash.com/collections/31564297/
https://unsplash.com/collections/90736570/
https://unsplash.com/collections/98089042/
https://unsplash.com/collections/33251296/
https://unsplash.com/collections/76055195/
https://unsplash.com/collections/50363024/
https://unsplash.com/collections/79347635/
https://unsplash.com/collections/77432865/
https://unsplash.com/collections/13413575/
https://unsplash.com/collections/13618475/
https://unsplash.com/collections/38936155/
https://unsplash.com/collections/44822118/
https://unsplash.com/collections/77361405/
https://unsplash.com/collections/55127386/
https://unsplash.com/collections/82143618/
https://unsplash.com/collections/79853743/
https://unsplash.com/collections/55022907/
https://unsplash.com/collections/49112072/
https://unsplash.com/collections/76646867/
https://unsplash.com/collections/66057426/
https://unsplash.com/collections/70924286/
https://unsplash.com/collections/75203762/
https://unsplash.com/collections/26252762/
https://unsplash.com/collections/25951818/
https://unsplash.com/collections/68949147/
https://unsplash.com/collections/62807819/
https://unsplash.com/collections/19468731/
https://unsplash.com/collections/50013326/
https://unsplash.com/collections/98906283/
https://unsplash.com/collections/16442168/
https://unsplash.com/collections/26961265/
https://unsplash.com/collections/89458228/
https://unsplash.com/collections/23011360/
https://unsplash.com/collections/71825411/
https://unsplash.com/collections/44887384/
https://unsplash.com/collections/95284929/
https://unsplash.com/collections/73102671/
https://unsplash.com/collections/26654692/
https://unsplash.com/collections/37636046/
https://unsplash.com/collections/32139074/
https://unsplash.com/collections/23852498/
https://unsplash.com/collections/34057470/
https://unsplash.com/collections/78067557/
https://unsplash.com/collections/44479344/
https://unsplash.com/collections/48081167/
https://unsplash.com/collections/79962025/
https://unsplash.com/collections/36540228/
https://unsplash.com/collections/48220710/
https://unsplash.com/collections/62476640/
https://unsplash.com/collections/34299916/
https://unsplash.com/collections/64903603/
https://unsplash.com/collections/36790470/
https://unsplash.com/collections/85282109/
https://unsplash.com/collections/78052373/
https://unsplash.com/collections/34763343/
https://unsplash.com/collections/51908810/
https://unsplash.com/collections/21271257/
https://unsplash.com/collections/84481581/
https://unsplash.com/collections/16209898/
https://unsplash.com/collections/85627471/
https://unsplash.com/collections/98583923/
https://unsplash.com/collections/51047129/
https://unsplash.com/collections/59193225/
https://unsplash.com/collections/72181781/
https://unsplash.com/collections/53812290/
https://unsplash.com/collections/66127690/
https://unsplash.com/collections/68896005/
https://unsplash.com/collections/60372449/
https://unsplash.com/collections/76814601/
https://unsplash.com/collections/41789691/
https://unsplash.com/collections/58450836/
https://unsplash.com/collections/65533836/
https://unsplash.com/collections/97693974/
https://unsplash.com/collections/15793161/
https://unsplash.com/collections/56173699/
https://unsplash.com/collections/98575510/
https://unsplash.com/collections/84840906/
https://unsplash.com/collections/41828606/
https://unsplash.com/collections/49444633/
https://unsplash.com/collections/21793403/
https://unsplash.com/collections/40437640/
https://unsplash.com/collections/80322090/
https://unsplash.com/collections/82741953/
https://unsplash.com/collections/65286188/
https://unsplash.com/collections/21494152/
https://unsplash.com/collections/54236115/
https://unsplash.com/collections/14966184/
https://unsplash.com/collections/85808238/
https://unsplash.com/collections/73690029/
https://unsplash.com/collections/15048015/
https://unsplash.com/collections/30414688/
https://unsplash.com/collections/86072403/
https://unsplash.com/collections/78156573/
https://unsplash.com/collections/82450630/
https://unsplash.com/collections/72297495/
https://unsplash.com/collections/92188946/
https://unsplash.com/collections/50249469/
https://unsplash.com/collections/43478184/
https://unsplash.com/collections/61391977/
https://unsplash.com/collections/29123712/
https://unsplash.com/collections/79566308/
https://unsplash.com/collections/65485829/
https://unsplash.com/collections/98092883/
https://unsplash.com/collections/53027796/
https://unsplash.com/collections/63900024/
https://unsplash.com/collections/81367802/
https://unsplash.com/collections/71050312/
https://unsplash.com/collections/20040599/
https://unsplash.com/collections/79603271/
https://unsplash.com/collections/76941003/
https://unsplash.com/collections/76436850/
https://unsplash.com/collections/56185907/
https://unsplash.com/collections/46601702/
https://unsplash.com/collections/47698511/
https://unsplash.com/collections/73860897/
https://unsplash.com/collections/99229623/
https://unsplash.com/collections/41289333/
https://unsplash.com/collections/38604673/
https://unsplash.com/collections/69716884/
https://unsplash.com/collections/21929738/
https://unsplash.com/collections/52249503/
https://unsplash.com/collections/53609607/
https://unsplash.com/collections/98940255/
https://unsplash.com/collections/18402614/
https://unsplash.com/collections/89589929/
https://unsplash.com/collections/81910718/
https://unsplash.com/collections/63508945/
https://unsplash.com/collections/60137934/
https://unsplash.com/collections/95917788/
https://unsplash.com/collections/90387067/
https://unsplash.com/collections/27995808/
https://unsplash.com/collections/90210668/
https://unsplash.com/collections/99338977/
https://unsplash.com/collections/51894937/
https://unsplash.com/collections/94518612/
https://unsplash.com/collections/74848977/
https://unsplash.com/collections/81534686/
https://unsplash.com/collections/68265519/
https://unsplash.com/collections/33457265/
https://unsplash.com/collections/13036211/
https://unsplash.com/collections/95031313/
https://unsplash.com/collections/94376809/
https://unsplash.com/collections/69665825/
https://unsplash.com/collections/68338903/
https://unsplash.com/collections/74294061/
https://unsplash.com/collections/34022385/
https://unsplash.com/collections/45336547/
https://unsplash.com/collections/25203159/
https://unsplash.com/collections/27366545/
https://unsplash.com/collections/84057977/
https://unsplash.com/collections/96471673/
https://unsplash.com/collections/14734056/
https://unsplash.com/collections/84258887/
https://unsplash.com/collections/69616705/
https://unsplash.com/collections/31898609/
https://unsplash.com/collections/47674071/
https://unsplash.com/collections/72815484/
https://unsplash.com/collections/88026855/
https://unsplash.com/collections/23639565/
https://unsplash.com/collections/30402072/
https://unsplash.com/collections/66947933/
https://unsplash.com/collections/67940672/
https://unsplash.com/collections/84728948/
https://unsplash.com/collections/74623985/
https://unsplash.com/collections/87560170/
https://unsplash.com/collections/51394298/
https://unsplash.com/collections/29491173/
https://unsplash.com/collections/77392125/
https://unsplash.com/collections/45009534/
https://unsplash.com/collections/40905492/
https://unsplash.com/collections/29656963/
https://unsplash.com/collections/23389046/
https://unsplash.com/collections/92549322/
https://unsplash.com/collections/63532454/
https://unsplash.com/collections/23888333/
https://unsplash.com/collections/69802601/
https://unsplash.com/collections/35139941/
https://unsplash.com/collections/24248460/
https://unsplash.com/collections/17671281/
https://unsplash.com/collections/91641013/
https://unsplash.com/collections/72496131/
https://unsplash.com/collections/50990271/
https://unsplash.com/collections/22039470/
https://unsplash.com/collections/51696384/
https://unsplash.com/collections/99414095/
https://unsplash.com/collections/14969096/
https://unsplash.com/collections/65584214/
https://unsplash.com/collections/99297598/
https://unsplash.com/collections/41433526/
https://unsplash.com/collections/43613069/
https://unsplash.com/collections/29417051/
https://unsplash.com/collections/34642227/
https://unsplash.com/collections/72834693/
https://unsplash.com/collections/51546417/
https://unsplash.com/collections/41719448/
https://unsplash.com/collections/99286402/
https://unsplash.com/collections/81518084/
https://unsplash.com/collections/37686402/
https://unsplash.com/collections/35865935/
https://unsplash.com/collections/65162038/
https://unsplash.com/collections/55410890/
https://unsplash.com/collections/88408830/
https://unsplash.com/collections/56252690/
https://unsplash.com/collections/52532743/
https://unsplash.com/collections/53255096/
https://unsplash.com/collections/59381135/
https://unsplash.com/collections/92858633/
https://unsplash.com/collections/25898058/
https://unsplash.com/collections/77006027/
https://unsplash.com/collections/47035038/
https://unsplash.com/collections/98011228/
https://unsplash.com/collections/72092951/
https://unsplash.com/collections/68733298/
https://unsplash.com/collections/41331897/
https://unsplash.com/collections/26944291/
https://unsplash.com/collections/55485592/
https://unsplash.com/collections/64742951/
https://unsplash.com/collections/58262291/
https://unsplash.com/collections/54388864/
https://unsplash.com/collections/39925616/
https://unsplash.com/collections/15993017/
https://unsplash.com/collections/73298484/
https://unsplash.com/collections/84686632/
https://unsplash.com/collections/51053045/
https://unsplash.com/collections/46083192/
https://unsplash.com/collections/19900952/
https://unsplash.com/collections/14560960/
https://unsplash.com/collections/36385100/
https://unsplash.com/collections/87553876/
https://unsplash.com/collections/37419878/
https://unsplash.com/collections/68521852/
https://unsplash.com/collections/61548622/
https://unsplash.com/collections/64390884/
https://unsplash.com/collections/99005236/
https://unsplash.com/collections/67866278/
https://unsplash.com/collections/40218206/
https://unsplash.com/collections/91598400/
https://unsplash.com/collections/77768178/
https://unsplash.com/collections/82704264/
https://unsplash.com/collections/45817850/
https://unsplash.com/collections/62800790/
https://unsplash.com/collections/54767717/
https://unsplash.com/collections/33868771/
https://unsplash.com/collections/21239553/
https://unsplash.com/collections/70331628/
https://unsplash.com/collections/71797427/
https://unsplash.com/collections/70005190/
https://unsplash.com/collections/98341822/
https://unsplash.com/collections/56834394/
https://unsplash.com/collections/68858930/
https://unsplash.com/collections/61931384/
https://unsplash.com/collections/17817256/
https://unsplash.com/collections/16453559/
(pprof) /app/bin # gops memstats 1
alloc: 54.66MB (57317968 bytes)
total-alloc: 186.21GB (199938346560 bytes)
sys: 6.72GB (7215977904 bytes)
lookups: 0
mallocs: 2906600570
frees: 2906521966
heap-alloc: 54.66MB (57317968 bytes)
heap-sys: 6.44GB (6909984768 bytes)
heap-idle: 6.38GB (6846406656 bytes)
heap-in-use: 60.63MB (63578112 bytes)
heap-released: 6.33GB (6794371072 bytes)
heap-objects: 78604
stack-in-use: 2.12MB (2228224 bytes)
stack-sys: 2.12MB (2228224 bytes)
stack-mspan-inuse: 266.02KB (272408 bytes)
stack-mspan-sys: 31.88MB (33423360 bytes)
stack-mcache-inuse: 6.78KB (6944 bytes)
stack-mcache-sys: 16.00KB (16384 bytes)
other-sys: 4.58MB (4802459 bytes)
gc-sys: 251.55MB (263773240 bytes)
next-gc: when heap-alloc >= 100.19MB (105060944 bytes)
last-gc: 2020-08-14 20:54:01.063567774 +0800 CST
gc-pause-total: 982.706218ms
gc-pause: 17439
num-gc: 3484
enable-gc: true
debug-gc: false
分析從上面的指標可以得出以下結論
- 當前正在使用的堆記憶體大致是 60 多M
- HeapReleased = 6.33 GB 返還給作業系統的實體記憶體的位元組數是 6.33GB (
HeapReleased 統計了從idle
span中返還給作業系統,沒有被重新獲取的記憶體大小)
那問題就抽象為:\為什麼 HeapReleased 上升,RSS 沒有下降?\
搜尋了一下有人踩過這個坑了,參考:https://zhuanlan.zhihu.com/p/...
這是因為 Go 底層用 mmap 申請的記憶體,會用 madvise 釋放記憶體。具體見
go/src/runtime/mem~linux~.go 的程式碼。madvise 將某段記憶體標記為不再使用時,有兩種方式 MADV~DONTNEED~ 和
MADV~FREE~(通過標誌引數傳入):
- MADV~DONTNEED標記過的記憶體如果再次使用~,會觸發缺頁中斷
- MADV~FREE標記過的記憶體~,核心會等到記憶體緊張時才會釋放。在釋放之前,這塊記憶體依然
可以複用。這個特性從linux 4.5版本核心開始支援顯然,MADV~FREE是一種用空間換時間的優化~。
- 在Go 1.12之前,linux 平臺下 Go runtime
中的sysUnsed使用madvise(MADV~DONTNEED~)- 在Go 1.12之後,在MADV~FREE可用時會優先使用MADVFREE~
具體見 https://github.com/golang/go/...
Go
1.12之後,提供了一種方式強制回退使用MADV~DONTNEED的方式~,在執行程式前新增
GODEBUG=madvdontneed=1。具體見
https://github.com/golang/go/...
另外還有一個疑點:現在知道問題出現的條件是 Go \> 1.12 && Linux Kernel \>
4.5,但是 為什麼之前阿里雲沒有出現,系統8月10號整體遷移到 aws 出現了?
調查發現
阿里雲機器核心 4.4.0-105-generic,而 aws 機器核心
4.15.0-1054-aws,剛好符合預期,可以解釋。
==
補充:8月24日
使用 GODEBUG=madvdontneed=1 強制回退使用
MADV~DONTNEED~,沒有再出現記憶體洩漏問題。記憶體穩定在 200MB 左右
相關文章
- 關於PHP記憶體洩漏的問題PHP記憶體
- ThreadLocal記憶體洩漏問題thread記憶體
- AFN的記憶體洩漏問題記憶體
- redisson記憶體洩漏問題排查Redis記憶體
- 線上問題排查例項分析|關於 Redis 記憶體洩漏Redis記憶體
- 線上問題排查例項分析|關於Redis記憶體洩漏Redis記憶體
- 關於java中的記憶體洩漏Java記憶體
- 如何查詢記憶體洩漏記憶體
- iOS八種記憶體洩漏問題iOS記憶體
- iOS 11 textField記憶體洩漏問題iOS記憶體
- bea記憶體洩漏問題診斷記憶體
- HibernateDaoSupport 記憶體洩漏的問題!記憶體
- 記一次 .NET 某HIS系統後端服務 記憶體洩漏分析後端記憶體
- 記憶體洩漏記憶體
- 記一次堆外記憶體洩漏分析記憶體
- 對於記憶體洩漏問題的簡單認知記憶體
- BufferedImage記憶體洩漏和溢位問題記憶體
- 記一次 .NET 某消防物聯網 後臺服務 記憶體洩漏分析記憶體
- 記憶體洩漏問題分析之非託管資源洩漏記憶體
- 記一次使用 laravel-s 記憶體洩漏Laravel記憶體
- js記憶體洩漏JS記憶體
- Java記憶體洩漏Java記憶體
- webView 記憶體洩漏WebView記憶體
- Javascript記憶體洩漏JavaScript記憶體
- jvm 記憶體洩漏JVM記憶體
- 一次 Java 記憶體洩漏的排查Java記憶體
- 解決記憶體洩漏(1)-ApacheKylin InternalThreadLocalMap洩漏問題分析記憶體Apachethread
- 分析記憶體洩漏和goroutine洩漏記憶體Go
- 如何解決JVM OutOfMemoryError記憶體洩漏問題?JVMError記憶體
- 【Java面試題】之記憶體洩漏Java面試題記憶體
- 記一次記憶體佔用問題的調查過程記憶體
- 記憶體分析與記憶體洩漏定位記憶體
- 記憶體洩漏和記憶體溢位記憶體溢位
- Go 記憶體洩漏?不是那麼簡單!Go記憶體
- 請教 關於記憶體洩漏的檢測方法記憶體
- 前端面試查漏補缺--(十三) 記憶體洩漏前端面試記憶體
- 記一次Go websocket 專案記憶體洩露排查 + 使用Go pprof定位記憶體洩露GoWeb記憶體洩露
- 記憶體洩漏的有關知識記憶體