1. 問題
spring-cloud-gateway 作為統一的請求入口，負責轉發請求到相應的微服務中去。
採用的 Spring Cloud 的版本為 Finchley SR2。
測試一個介面的效能，發現 tps 只有 1000 req/s 左右就上不去了。
[root@hystrix-dashboard wrk]# wrk -t 10 -c 200 -d 30s --latency -s post-test.lua 'http://10.201.0.28:8888/api/v1/json'
Running 30s test @ http://10.201.0.28:8888/api/v1/json
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 188.34ms 110.13ms 2.00s 78.43%
Req/Sec 106.95 37.19 333.00 77.38%
Latency Distribution
50% 165.43ms
75% 243.48ms
90% 319.47ms
99% 472.64ms
30717 requests in 30.04s, 7.00MB read
Socket errors: connect 0, read 0, write 0, timeout 75
Requests/sec: 1022.62
Transfer/sec: 238.68KB
複製程式碼其中 post-test.lua 內容如下：
request = function()
local headers = {}
headers["Content-Type"] = "application/json"
local body = [[{
"biz_code": "1109000001",
"channel": "7",
"param": {
"custom_id": "ABCD",
"type": "test",
"animals": ["cat", "dog", "lion"],
"retcode": "0"
}
}]]
return wrk.format('POST', nil, headers, body)
end
複製程式碼閘道器的邏輯是讀取請求中 body 的值，根據 biz_code 欄位去記憶體中匹配路由，然後轉發請求到對應的微服務中去。
2. 排查
- 測試介面本身的效能：
[root@hystrix-dashboard wrk]# wrk -t 10 -c 200 -d 30s --latency -s post-test.lua 'http://10.201.0.32:8776/eeams-service/api/v1/json'
Running 30s test @ http://10.201.0.32:8776/eeams-service/api/v1/json
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 26.72ms 8.59ms 260.23ms 89.66%
Req/Sec 752.18 101.46 0.94k 78.67%
Latency Distribution
50% 23.52ms
75% 28.02ms
90% 35.58ms
99% 58.25ms
224693 requests in 30.02s, 50.83MB read
Requests/sec: 7483.88
Transfer/sec: 1.69MB
複製程式碼發現介面的 tps 可以達到 7000+。
- 通過 spring-boot-admin 檢視閘道器的 cpu、記憶體等佔用情況，發現都沒有用滿；檢視執行緒狀況，發現 reactor-http-nio 執行緒組存在阻塞情況。對於響應式程式設計來說，reactor-http-nio 執行緒出現阻塞結果是災難性的。
- 通過 jstack 命令分析執行緒狀態，定位阻塞的程式碼（第 19 行）：
"reactor-http-nio-4" #19 daemon prio=5 os_prio=0 tid=0x00007fb784d7f240 nid=0x80b waiting for monitor entry [0x00007fb71befc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
- waiting to lock <0x000000008b0cec30> (a java.lang.Object)
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:282)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:753)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:624)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:608)
at org.springframework.http.codec.json.Jackson2JsonEncoder.<init>(Jackson2JsonEncoder.java:54)
at org.springframework.http.codec.support.AbstractCodecConfigurer$AbstractDefaultCodecs.getJackson2JsonEncoder(AbstractCodecConfigurer.java:177)
at org.springframework.http.codec.support.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getSseEncoder(DefaultServerCodecConfigurer.java:99)
at org.springframework.http.codec.support.DefaultServerCodecConfigurer$ServerDefaultCodecsImpl.getObjectWriters(DefaultServerCodecConfigurer.java:90)
at org.springframework.http.codec.support.AbstractCodecConfigurer.getWriters(AbstractCodecConfigurer.java:121)
at org.springframework.http.codec.support.DefaultServerCodecConfigurer.getWriters(DefaultServerCodecConfigurer.java:39)
at org.springframework.web.reactive.function.server.DefaultHandlerStrategiesBuilder.build(DefaultHandlerStrategiesBuilder.java:103)
at org.springframework.web.reactive.function.server.HandlerStrategies.withDefaults(HandlerStrategies.java:90)
at org.springframework.cloud.gateway.support.DefaultServerRequest.<init>(DefaultServerRequest.java:81)
at com.glsc.imf.dbg.route.RouteForJsonFilter.filter(RouteForJsonFilter.java:34)
at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.lambda$filter$0(FilteringWebHandler.java:115)
at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain$$Lambda$800/1871561393.get(Unknown Source)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
複製程式碼最終定位到問題程式碼為：
DefaultServerRequest req = new DefaultServerRequest(exchange); // 這行程式碼存在效能問題
return req.bodyToMono(JSONObject.class).flatMap(body -> {
...
});
複製程式碼這裡的邏輯是我需要讀取請求中 body 的值，並轉化為 json，之後根據其中的特定欄位去匹配路由，然後進行轉發。這裡選擇了先把 exchange 轉化為 DefaultServerRequest，目的是為了使用該類的 bodyToMono 方法，可以方便的進行轉換。
3. 解決
改寫程式碼以實現同樣的功能：
return exchange.getRequest().getBody().collectList()
.map(dataBuffers -> {
ByteBuf byteBuf = Unpooled.buffer();
dataBuffers.forEach(buffer -> {
try {
byteBuf.writeBytes(IOUtils.toByteArray(buffer.asInputStream()));
} catch (IOException e) {
e.printStackTrace();
}
});
return JSON.parseObject(new String(byteBuf.array()));
})
.flatMap(body -> {
...
});
複製程式碼之後進行測試，
[root@hystrix-dashboard wrk]# wrk -t 10 -c 200 -d 30s --latency -s post-test.lua 'http://10.201.0.28:8888/api/v1/json'
Running 30s test @ http://10.201.0.28:8888/api/v1/json
10 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 48.47ms 45.85ms 325.87ms 88.55%
Req/Sec 548.13 202.55 760.00 80.01%
Latency Distribution
50% 31.18ms
75% 39.44ms
90% 112.18ms
99% 227.19ms
157593 requests in 30.02s, 35.94MB read
Requests/sec: 5249.27
Transfer/sec: 1.20MB
複製程式碼發現 tps 從 1000 提升到了 5000+，問題解決。