Java訪問Elasticsearch報錯Request cannot be executed; I/O reactor status: STOPPED

Naylor發表於2022-01-04

簡介

使用ES過程中遇到一個Request cannot be executed; I/O reactor status: STOPPED 的異常，大概意思是和server端的連線異常終止了。開始以為是引用的版本不對，或者自己使用問題，後來發現就是因為OOM導致程式當機，進而引發連線終止。

環境

功能

SpringBoot 的程式通過 SpringDataElasticsearch 訪問ES-server 獲取資料。

ES-SERVER

版本：7.15.2

ES-CLIENT

ES-CLIENT 就是 SpringBoot 程式，核心pom依賴：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <version>2.5.3</version>
</dependency>

問題

部署到測試環境之後，使用者量大了之後介面報錯，報錯資訊為：

Request cannot be executed; I/O reactor status: STOPPED

後端程式處於假死狀態，訪問其他介面一直轉圈，無法響應內容
重啟程式之後一切正常，使用者量大了之後又有此問題
詳細報錯日誌為：

2021-12-31/19:26:31.748||||^_^|[pool-4-thread-1] ERROR o.a.http.impl.nio.client.InternalHttpAsyncClient 66 - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: Java heap space
2021-12-31/19:26:32.783||||^_^|[http-nio-8092-exec-3] ERROR com.nghsmart.datacenter.handler.ExceptionHandler 67 - handleRuntimeException:
java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:888)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1624)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1594)
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1110)
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.lambda$search$11(ElasticsearchRestTemplate.java:296)
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:383)
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.search(ElasticsearchRestTemplate.java:296)
        at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.search(AbstractElasticsearchTemplate.java:513)
        at com.nghsmart.datacenter.domain.area.service.impl.AreaDashboradServiceImpl.getRotationStatisticsInfo(AreaDashboradServiceImpl.java:166)
        at com.nghsmart.datacenter.domain.area.controller.AreaDashboradController.getRotationStatisticsInfo(AreaDashboradController.java:76)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at com.nghsmart.commonauthentication.filter.JwtAuthenticationTokenFilter.doFilterInternal(JwtAuthenticationTokenFilter.java:52)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
        at org.apache.http.util.Asserts.check(Asserts.java:46)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:279)
... 69 common frames omitted

測試環境使用者量一大就報，本地IDEA中一次也沒有報這個問題
使用JMETER設定50個執行緒併發訪問可以穩定復現此問題

原因

程式介面中將一塊很大的資料存進JAVA集合中引發了oom，oom異常導致程式當機，處於假死狀態，進而導致 ES-CLIENT 和 ES-SERVER 端的 http 連線異常終止，然後org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning 方法報異常。

ensureRunning:

protected void ensureRunning() {
    final Status currentStatus = this.status.get();
    Asserts.check(currentStatus == Status.ACTIVE, "Request cannot be executed; " +
            "I/O reactor status: %s", currentStatus);
}

check:

public static void check(final boolean expression, final String message, final Object arg) {
    if (!expression) {
        throw new IllegalStateException(String.format(message, arg));
    }
}

SpringDataElasticsearch 和ES-SERVER 是長連結，只要報了OOM，當前和 ES-SERVER 的連線執行緒都將報異常，也就是說，雖然OOM只報了一次，但是可能有多個執行緒都在 Asserts.check 方法中報異常
其實不僅僅是和ES-SERVER 的連線異常關閉，觀察大量日誌後發現和nacos 的連線也有問題：

2022-01-04 15:59:07.366 [com.alibaba.nacos.client.remote.worker]  INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1]Server healthy check fail,currentConnection=1641282430796_192.168.1.90_43108
2022-01-04 15:59:07.367 [com.alibaba.nacos.client.remote.worker]  INFO com.alibaba.nacos.common.remote.client.printIfInfoEnabled:60 - [117be10e-119f-4253-a348-71c95a8978a1] try to re connect to a new server ,server is  not appointed,will choose a random server.

測試環境執行SpringBoot記憶體配置為：-Xms256m -Xmx512m，使用 java -jar 命令啟動。測試環境能復現，本地idea無法復現的原因是記憶體配置不同導致的，本地idea記憶體給的2g。編輯 run/debug configuration ，設定 VM options 和測試環境相同，使用jemter壓測本地idea中跑的程式，也能復現此問題

解決辦法

優化介面，解決OOM問題
增大測試環境記憶體配置

解決過程

HttpClient問題

這個思路並沒有解決我的問題，但是解決問題的過程中花了大量時間研究。其實應該仔細閱讀報錯log的上下文和介面程式碼上下文，一開始沒有留意到oom的異常

大概意思是說低版本 httpClient 有這個問題，高版本就沒有這個問題，是apache-httpclient 導致的。具體是因為捕獲到執行緒異常之後http連線就主動終止了，解決思路是升級版本或者是重寫處理器，遇到異常之後不終止連線。

升級httpclient相關pom依賴：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <!-- 排除 httpclient、httpcore、 httpcore-nio這三個依賴，並手動引入最新版本，解決es客戶端報Request cannot be executed; I/O reactor status: STOPPED問題 -->
    <exclusions>
<!--                <exclusion>-->
<!--                    <groupId>org.apache.httpcomponents</groupId>-->
<!--                    <artifactId>httpclient</artifactId>-->
<!--                </exclusion>-->
<!--                <exclusion>-->
<!--                    <groupId>org.apache.httpcomponents</groupId>-->
<!--                    <artifactId>httpcore</artifactId>-->
<!--                </exclusion>-->
<!--                <exclusion>-->
<!--                    <groupId>org.apache.httpcomponents</groupId>-->
<!--                    <artifactId>httpcore-nio</artifactId>-->
<!--                </exclusion>-->
        <exclusion>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
        </exclusion>
    </exclusions>
</dependency>


<!-- start -->
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-client</artifactId>
    <version>7.15.2</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore-nio</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<!--  end  -->
<!-- start：排除 httpclient、httpcore、 httpcore-nio這三個依賴，並手動引入最新版本，解決es客戶端報Request cannot be executed; I/O reactor status: STOPPED問題 -->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.13</version>
</dependency>

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore-nio</artifactId>
    <version>4.4.14</version>
</dependency>

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.4.14</version>
</dependency>

<!--  end   -->

重寫ElasticsearchRestTemplate：


import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager;
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.message.BasicHeader;
import org.apache.http.nio.reactor.IOReactorException;
import org.apache.http.nio.reactor.IOReactorExceptionHandler;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@Slf4j
@Configuration
public class EsClientConfig {
   
    @Value("${es.host}")
    private String host;
    @Value("${es.port}")
    private Integer port;
   
    @Bean
    public ElasticsearchRestTemplate initElasticsearchRestTemplate() {
        RestClientBuilder builder = RestClient.builder(new HttpHost(host, port))
                .setRequestConfigCallback(
                        config -> config.setConnectTimeout(180000)
                                .setConnectionRequestTimeout(180000)
                                .setSocketTimeout(180000))
                .setHttpClientConfigCallback(
                        httpClientBuilder -> {
                            httpClientBuilder.setMaxConnTotal(100);
                            httpClientBuilder.setMaxConnPerRoute(50);
                            List<Header> headers = new ArrayList<>(2);
                            headers.add(new BasicHeader("Connection", "keep-alive"));
                            headers.add(new BasicHeader("Keep-Alive", "720"));
                            httpClientBuilder.setDefaultHeaders(headers);
                            httpClientBuilder.setKeepAliveStrategy(DefaultConnectionKeepAliveStrategy.INSTANCE);
                            try {
                                DefaultConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
                                ioReactor.setExceptionHandler(new IOReactorExceptionHandler() {
                                    @Override
                                    public boolean handle(IOException e) {
                                        log.debug("System may be unstable: IOReactor encountered a checked exception : " + e.getMessage());
                                        log.debug("start setHttpClientConfigCallback handle IOException e printStackTrace");
                                        e.printStackTrace();
                                        log.debug("end setHttpClientConfigCallback handle IOException e printStackTrace");
                                        // Return true to note this exception as handled, it will not be re-thrown
                                        return true;
                                    }
                                    @Override
                                    public boolean handle(RuntimeException e) {
                                        log.debug("System may be unstable: IOReactor encountered a runtime exception : " + e.getMessage() + ",e is {}", e);
                                        log.debug("start setHttpClientConfigCallback handle RuntimeException e printStackTrace ");
                                        e.printStackTrace();
                                        log.debug("end setHttpClientConfigCallback handle RuntimeException e printStackTrace ");
                                        // Return true to note this exception as handled, it will not be re-thrown
                                        return true;
                                    }
                                });
                                httpClientBuilder.setConnectionManager(new PoolingNHttpClientConnectionManager(ioReactor));
                            } catch (IOReactorException e) {
                                throw new RuntimeException(e);
                            }
                            return httpClientBuilder;
                        }
                );
        ElasticsearchRestTemplate elasticsearchRestTemplate = new ElasticsearchRestTemplate(new RestHighLevelClient(builder));
        return elasticsearchRestTemplate;
    }
}

關於此問題相關參考：

ElasticSearch：Request cannot be executed； I/O reactor status: STOPPED
2020-11-02
ElasticsearchReact
IRP(I/O Request Package)詳解
2018-09-04
Package
Java I/O
2024-07-07
Java
【java】I/O流
2018-04-11
Java
Java（8）I/O
2020-11-26
Java
SAP RETAIL 執行事務程式碼WRP1，報錯-Transaction WRP1 cannot be executed-
2021-11-08
AI
監控報I/O問題，怎麼辦？
2018-07-19
JAVA I/O系統
2018-11-13
Java
Java 非同步 I/O
2022-01-19
Java非同步
HTTP Status 400 – Bad Request
2024-06-07
HTTP
Java™ 教程（命令列I/O）
2019-01-19
Java命令列
K3S 系列文章-5G IoT 閘道器裝置 POD 訪問報錯 DNS 'i/o timeout'分析與解決
2023-02-18
DNS
深入理解Java I/O模型
2019-05-07
Java模型
"萬字" Java I/O 詳解
2023-02-11
Java
解決String cannot be cast to java.util.List報錯
2020-12-23
ASTJava
allowedOrigins cannot contain the gateway 報錯
2021-07-10
AIGateway
SpringCloud問題解決：spring-cloud-eureka啟動出錯Cannot execute request on any known server
2018-04-02
SpringGCCloudServer
yarn add @fortawesome/Font Awesome-free 之後報錯 EIO: i/o error
2020-02-26
YarnError
Java I/O模型及其底層原理
2020-06-10
Java模型
《Java 高階篇》六：I/O 流
2024-10-02
Java
解決使用mitmprox抓包可以訪問網頁，但是使用python request 呼叫該網站介面報錯問題
2023-10-28
MIT網頁Python網站
計算機I/O與I/O模型
2019-05-10
計算機模型
chorme訪問https的一個報錯
2021-11-01
ORMHTTP
執行SQL發生錯誤！錯誤：disk I/O error
2024-10-01
SQLError
gateway 報錯 allowedOrigins cannot contain the special
2021-09-01
GatewayAI
Java NIO學習系列五：I/O模型
2019-07-22
Java模型
ORACLE OMS啟動失敗之BEA-300040 I/O error while writing node manager status
2020-03-20
OracleErrorWhile
I/O流
2018-07-26
Elasticsearch 開放外網訪問
2020-01-07
Elasticsearch
關閉tomcat報錯Cannot allocate memory
2018-08-24
Tomcat
前端報錯：cannot read property length of undefined
2024-04-03
前端Undefined
【雜談】Java I/O的底層實現
2018-11-08
Java
Java非阻塞I/O模型之NIO說明
2021-07-04
Java模型
Netty權威指南：Java的I/O演進
2021-04-06
NettyJava
Python教程：精簡概述I/O模型與I/O操作
2020-05-25
Python模型
LightDB Canopy 常見報錯問題分析（二）DETAIL: Distributed relations cannot have
2023-03-02
AI
關於I/O
2018-05-09
c++ I/O
2024-09-01
C++