Java程式碼解決ElasticSearch的Result window is too large問題

Aruze發表於2018-11-23
JavaElasticsearch

呼叫ElasticSearch做分頁查詢時報錯：

QueryPhaseExecutionException[Result window is too large, from + size must be less than or equal to: [10000] but was [666000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.]; }

提示用from+size方式有1萬條資料查詢的限制，需要更改index.max_result_window引數的值。

翻了下elasticsearch官網的文件：

index.max_result_window
The maximum value of from + size for searches to this index.Defaults to 10000. 
Search requests take heap memory and time proportional to from + size and this limits that memory. 
See Scroll or Search After for a more efficient alternative to raising this.

說是用傳統方式（from + size）查詢佔用記憶體空間且比較消耗時間，所以做了限制。

問題是用scroll方式做後臺分頁根本行不通。

不說用scroll方式只能一頁頁的翻這種不人性化的操作。頁碼一多，scrollId也很難管理啊。

所以繼續鼓搗傳統方式的分頁。

上網查了下設定max_result_window的方法，全都是用crul或者http方式改的。

後來無意間看到了一篇文件： https://blog.csdn.net/tzconn/article/details/83309516

結合之前逛elastic中文社群的時候知道這個引數是索引級別的。於是小試了一下，結果竟然可以了。

java程式碼如下：

public SearchResponse search(String logIndex, String logType, QueryBuilder query, 
                 List<AggregationBuilder> agg, int page, int size) {
        page = page > 0 ? page - 1 : page;
        TransportClient client = getClient();
        SearchRequestBuilder searchRequestBuilder = client.prepareSearch(logIndex.split(","))
                .setTypes(logType.split(","))
                .setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
                .addSort("createTime", SortOrder.DESC);

        if (agg != null && !agg.isEmpty()) {
            for (int i = 0; i < agg.size(); i++) {
                searchRequestBuilder.addAggregation(agg.get(i));
            }
        }
        updateIndexs(client, logIndex, page, size);

        SearchResponse searchResponse = searchRequestBuilder
                .setQuery(query)
                .setFrom(page * size)
                .setSize(size)
                .get();
        return searchResponse;
    }

    //更新索引的max_result_window引數
    private boolean updateIndexs(TransportClient client, String indices, int from, int size) {
        int records = from * size + size;
        if (records <= 10000) return true;
        UpdateSettingsResponse indexResponse = client.admin().indices()
                .prepareUpdateSettings(indices)
                .setSettings(Settings.builder()
                        .put("index.max_result_window", records)
                        .build()
                ).get();
        return indexResponse.isAcknowledged();
    }

搞定。

當然這段程式碼不好的地方在於：

每次查詢超過10000萬條記錄的時候，都會去更新一次index。

這對原本就偏慢的from+size查詢來說，更是雪上加霜了。

 

相關文章