tomcat服務突然無響應了，匯出記憶體堆疊和執行緒堆疊，分析後發現是同步鎖使用不合理導致的。
[root@prd-dtb-web-01 ~]# pgrep java
10472
[root@prd-dtb-web-01 ~]# jmap -heap 10472
Attaching to process ID 10472, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.111-b14
using thread-local object allocation.
Parallel GC with 2 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 1983905792 (1892.0MB)
NewSize = 41943040 (40.0MB)
MaxNewSize = 661127168 (630.5MB)
OldSize = 83886080 (80.0MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 487063552 (464.5MB)
used = 26314992 (25.095932006835938MB)
free = 460748560 (439.40406799316406MB)
5.402784070362958% used
From Space:
capacity = 72351744 (69.0MB)
used = 71945680 (68.61274719238281MB)
free = 406064 (0.3872528076171875MB)
99.43876404693161% used
To Space:
capacity = 84934656 (81.0MB)
used = 0 (0.0MB)
free = 84934656 (81.0MB)
0.0% used
PS Old Generation
capacity = 254279680 (242.5MB)
used = 136744120 (130.40935516357422MB)
free = 117535560 (112.09064483642578MB)
53.77705367570071% used
36326 interned Strings occupying 4333960 bytes.
[root@prd-dtb-web-01 ~]# jmap -dump:file=dump_dtb 10472
Dumping heap to /root/dump_dtb ...
Heap dump file created
[root@prd-dtb-web-01 ~]# jstack 10472 > thread_dtb
使用Eclipse MemoryAnalyzer對記憶體堆疊的分析，發現執行緒已經佔滿了。
透過對執行緒堆疊檔案內容的分析，發現大量執行緒都處於waiting to lock狀態，進一步發現，對應程式碼使用了synchronized同步鎖，一個執行緒內部訪問資料庫發生了超時，長時間佔用了該鎖，導致其它執行緒都處於等待狀態。
...
"http-nio-8002-exec-26" #52 daemon prio=5 os_prio=0 tid=0x00007f951c01b000 nid=0x291e waiting for monitor entry [0x00007f9530dc9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at cn.friendsure.tdtb.services.WeixinPayService.payed(WeixinPayService.java:273)
- waiting to lock <0x000000008a9103b0> (a cn.friendsure.tdtb.services.WeixinPayService)
at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:446)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:434)
...
at cn.friendsure.tdtb.logics.OrderLogic.transferOrder(OrderLogic.java:531)
- locked <0x000000008a9104e0> (a cn.friendsure.tdtb.logics.OrderLogic)
at cn.friendsure.tdtb.services.WeixinPayService.transferOrder(WeixinPayService.java:478)
at cn.friendsure.tdtb.services.WeixinPayService.payed(WeixinPayService.java:399)
- locked <0x000000008a9103b0> (a cn.friendsure.tdtb.services.WeixinPayService)
at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
...
【解決方案】
去掉不必要的同步鎖。
【總結】
涉及IO的方法，儘量不要使用synchronized關鍵字，如果一定要用，要確保程式邏輯中有明確的超時控制機制，並且超時時間不要太長。