基於Grafana和Prometheus的監視系統(3):java客戶端使用

基於Grafana和Prometheus的監視系統(3):java客戶端使用

0.基本說明

• 如何在程式碼中進行指標埋點: prometheus java client的使用
• 如何生成jvm 指標資料: hotspot expoter的使用
• 在spring boot 中進行指標採集
• 非指標資料的採集: Elasticsearch的使用

1. java client的使用

[https://github.com/prometheus/client_java]
使用方式和log4j的使用很相似

 <!-- The client -->
        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient</artifactId>
            <version>0.4.0</version>
        </dependency>
        <!-- Hotspot JVM metrics-->
        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_hotspot</artifactId>
            <version>0.4.0</version>
        </dependency>
        <!-- Exposition HTTPServer-->
        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_httpserver</artifactId>
            <version>0.4.0</version>
        </dependency>
        <!-- Pushgateway exposition-->
        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_pushgateway</artifactId>
            <version>0.4.0</version>
        </dependency>

class A {
 static CollectorRegistry registry = new CollectorRegistry();
    // 請求總數統計
    static final  Counter counterRequest = Counter.build().name("TsHistoryData_RequestTotal").help("xx").
            labelNames("request").register(registry);
    // 請求時間
    static final Gauge costTime = Gauge.build().name("TsHistoryData_CostTime").help("xx").
            labelNames("id").register(registry);
    // 直方圖, 統計在某個bucket時間的請求的個數.(linearBuckets | exponentialBuckets)
    static final Histogram requestLatency_hgm = Histogram.build()
            .name("TsHistoryData_RequestLatency_hgm").exponentialBuckets(0.5,2,10).help("Request latency in seconds.").register(registry);
    static final Summary requestLatency_suy = Summary.build()
            .name("TsHistoryData_RequestLatency_suy").
                    quantile(0.1, 0.01).
                    quantile(0.3, 0.01).
                    quantile(0.5,0.01).
                    quantile(0.7, 0.01).
                    quantile(0.9, 0.01).
                    quantile(0.95, 0.01).help("Request latency in seconds.").register(registry);
 void function1() {
        String requestId = System.currentTimeMillis() + "";
        counterRequest.labels("request").inc();
        Gauge.Timer costTimeTimer = costTime.labels(requestId).startTimer();
        Histogram.Timer requestTimer = requestLatency_hgm.startTimer();
        Summary.Timer requestTimer_suy = requestLatency_suy.startTimer();
        ......
        costTimeTimer.setDuration();
        requestTimer.observeDuration();
        requestTimer_suy.observeDuration();
        Monitor.pushMetricToPushGateway(registry, "TS_HISTORY_DATA");
  }
}

2. hotspot expoter

對於程式執行時的jvm指標進行監控

public static void jvmExport() throws Exception {
        DefaultExports.initialize();
        InetSocketAddress address = new InetSocketAddress("192.168.1.222", 9201);
        Server server = new Server(address);
        ServletContextHandler context = new ServletContextHandler();
        context.setContextPath("/");
        server.setHandler(context);
        context.addServlet(new ServletHolder(new MetricsServlet()), "/metrics");
        server.start();
        // server.join();
    }

3.在spring boot中的使用

1.新增Maven的依賴

<!-- Hotspot JVM metrics-->
        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_hotspot</artifactId>
            <version>0.4.0</version>
        </dependency>
        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_spring_boot</artifactId>
            <version>0.4.0</version>
        </dependency>
        <!-- Exposition servlet -->
        <dependency>
            <groupId>io.prometheus</groupId>
            <artifactId>simpleclient_servlet</artifactId>
            <version>0.4.0</version>
        </dependency>

2. 啟用Prometheus Metrics Endpoint
   新增註解@EnablePrometheusEndpoint啟用Prometheus Endpoint,這裡同時使用了simpleclient_hotspot中提供的DefaultExporter,該Exporter展示當前應用JVM的相關資訊

@SpringBootApplication
@EnablePrometheusEndpoint
public class CoreApplication extends WebMvcConfigurerAdapter implements CommandLineRunner {

    public static void main(String[] args) {
        SpringApplication springApplication = new SpringApplication(CoreApplication.class);
        springApplication.run(args);
    }

    @Override
    public void run(String... strings) throws Exception {
        DefaultExports.initialize();
    }
}

向外暴露指標的介面

@Configuration
public class MonitoringConfig {

    @Bean
    ServletRegistrationBean servletRegistrationBean() {

        return new ServletRegistrationBean(new MetricsServlet(), "/metrics");
    }
}

這樣訪問 localhost:8080/metrics 可以看到jvm相關的指標.

3.新增攔截器，為監控埋點做準備
除了獲取應用JVM相關的狀態以外，我們還可能需要新增一些自定義的監控Metrics實現對系統效能，以及業務狀態進行採集，以提供日後優化的相關支撐資料。首先我們使用攔截器處理對應用的所有請求。

繼承WebMvcConfigurerAdapter類，複寫addInterceptors方法，對所有請求/**新增攔截器

@SpringBootApplication
@EnablePrometheusEndpoint
public class CoreApplication extends WebMvcConfigurerAdapter implements CommandLineRunner {

    @Override
    public void addInterceptors(InterceptorRegistry registry) {
        registry.addInterceptor(new PrometheusMetricsInterceptor()).addPathPatterns("/**");
    }
}

PrometheusMetricsInterceptor整合HandlerInterceptorAdapter，通過複寫父方法，實現對請求處理前/處理完成的處理。

@Component
public class PrometheusMetricsInterceptor extends HandlerInterceptorAdapter {
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
    return super.preHandle(request, response, handler);
}

@Override
public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
    super.afterCompletion(request, response, handler, ex);
}
}

4.自定義指標

@Component
public class PrometheusMetricsInterceptor extends HandlerInterceptorAdapter {

    static final Counter requestCounter = Counter.build()
            .name("module_core_http_requests_total").labelNames("path", "method", "code")
            .help("Total requests.").register();

    static final Gauge inprogressRequests = Gauge.build()
            .name("module_core_http_inprogress_requests").labelNames("path", "method", "code")
            .help("Inprogress requests.").register();

    static final Gauge requestTime = Gauge.build()
            .name("module_core_http_requests_costTime").labelNames("path", "method", "code")
            .help("requests cost time.").register();

    static final Histogram requestLatencyHistogram = Histogram.build().labelNames("path", "method", "code")
            .name("module_core_http_requests_latency_seconds_histogram").help("Request latency in seconds.")
            .register();

    static final Summary requestLatency = Summary.build()
            .name("module_core_http_requests_latency_seconds_summary")
            .quantile(0.5, 0.05)
            .quantile(0.9, 0.01)
            .labelNames("path", "method", "code")
            .help("Request latency in seconds.").register();
    private Histogram.Timer histogramRequestTimer;

    private Summary.Timer summaryTimer;

    private Gauge.Timer gaugeTimer;

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        String requestURI = request.getRequestURI();
        String method = request.getMethod();
        int status = response.getStatus();
        inprogressRequests.labels(requestURI, method, String.valueOf(status)).inc();
        histogramRequestTimer = requestLatencyHistogram.labels(requestURI, method, String.valueOf(status)).startTimer();
        summaryTimer = requestLatency.labels(requestURI, method, String.valueOf(status)).startTimer();
        gaugeTimer = requestTime.labels(requestURI, method, String.valueOf(status)).startTimer();
        return super.preHandle(request, response, handler);
    }

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {

        String requestURI = request.getRequestURI();
        String method = request.getMethod();
        int status = response.getStatus();

        requestCounter.labels(requestURI, method, String.valueOf(status)).inc();
        inprogressRequests.labels(requestURI, method, String.valueOf(status)).dec();
        histogramRequestTimer.observeDuration();
        summaryTimer.observeDuration();
        gaugeTimer.setDuration();
        super.afterCompletion(request, response, handler, ex);
    }
}

5.使用Collector暴露業務指標
除了在攔截器中使用Prometheus提供的Counter,Summary,Gauage等構造監控指標以外，我們還可以通過自定義的Collector實現對相關業務指標的暴露

@SpringBootApplication
@EnablePrometheusEndpoint
public class CoreApplication extends WebMvcConfigurerAdapter implements CommandLineRunner {
@Autowired
private CustomExporter customExporter;

...省略的程式碼

@Override
public void run(String... args) throws Exception {
    ...省略的程式碼
    customExporter.register();
}
}

CustomExporter整合自io.prometheus.client.Collector，在呼叫Collector的register()方法後，當訪問/metrics時，則會自動從Collector的collection()方法中獲取採集到的監控指標。

由於這裡CustomExporter存在於Spring的IOC容器當中，這裡可以直接訪問業務程式碼，返回需要的業務相關的指標。

@Component
public class CustomExporter extends Collector {
    @Override
    public List<MetricFamilySamples> collect() {
        List<MetricFamilySamples> mfs = new ArrayList<>();

        GaugeMetricFamily labeledGauge =
                new GaugeMetricFamily("module_core_custom_metrics", "custom metrics", Collections.singletonList("labelname"));


        labeledGauge.addMetric(Collections.singletonList("labelvalue"), 1);

        mfs.add(labeledGauge);
        return mfs;
    }
}

4. ES的使用

docker compose [https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html]
java client

<dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>6.0.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch -->
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>6.0.0</version>
        </dependency>

public class EsConfig {

    private static RestHighLevelClient client;
    public  static RestHighLevelClient init() throws Exception{
        client = new RestHighLevelClient(
                RestClient.builder(
                        new HttpHost("192.168.1.223", 9200, "http")
                        ));
        return client;
    }
    public static void close() throws Exception{
        client.close();
    }

}

 try {
            client = EsConfig.init();

            Map<String, Object> jsonMap = new HashMap<>();
            jsonMap.put("Id", requestId);
            jsonMap.put("Date", new Date());
            jsonMap.put("Value", tt);
            IndexRequest indexRequest = new IndexRequest("ts_data_history", "doc", requestId)
                    .source(jsonMap);
            IndexResponse indexResponse = client.index(indexRequest);
            client.close();
        } catch (Exception e) {
            e.printStackTrace();
        }