如何設定Samza的metrics

devos發表於2014-04-28

參考這個裡邊對API的呼叫

參考這裡列出的可用配置
在正常的配置之外，需要額外新增的配置。新增這些配置以後，就可以在StreamTask裡使用metrics的API來提供metrics資訊
 
  1. 需要哪些配置？
    1. 設定用哪種reporter factory，Samza自帶有Kafka和JMX兩種
    2. 設定reporter使用的stream,包括設定這個stream的serde
    3. 註冊這個reporter
#設定用來做為輸出流的Stream
#設定samza-metrics這個stream的system
streams.samza-metrics.system= kafka 
#設定samza-metrics這個stream的名字,即在kafka裡對應的topic的名字
streams.samza-metrics.stream= samza-metrics
#配置reporter
#設定reporter factory這裡使用的MetricsSnapshotReporterFactory,把JSON格式的metrics發給Kafka。這個class要是Factory的名字，samza當前的document裡的配置是錯的
metrics.reporter.samza-metrics.class= org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory
 
#samza-metrics這個reporter10s傳送一次metrics
metrics.reporter.samza-metrics.window.ms= 10000
 
#定義samza-metrics這個reporter對應的stream。此reporter使用kafka這個system下的samza-metrics這個stream。注意一定要使用
#system.stream這種寫法，即寫成kafka.samza-metrics
#這個配置是必須的，但是Samza的config文件裡沒有列出
metrics.reporter.samza-metrics.stream= kafka.samza-metrics
 
#註冊samza-metrics這個reporter
metrics.reporters= samza-metrics
#配置serde
# Serializers 定義可以使用的Serde factory的名字
serializers.registry.json.class= org.apache.samza.serializers.JsonSerdeFactory
serializers.registry.metrics.class= org.apache.samza.serializers.MetricsSnapshotSerdeFactory
 
#定義samza-metrics這個stream，用來做訊息serde的Serde factory為metrics。由於這個stream輸出的是metrics的資訊，因此必須使用這個serde factory
systems.kafka.streams.samza-metrics.samza.msg.serde= metrics
 
 
最後會有三種metrics。StramTask的、SystemProducer的、SystemConsumer的

{"metrics":{"org.apache.samza.container.TaskInstanceMetrics":{"process-calls":28575767,"messages-sent":0,"commit-calls":12,"window-skipped":30204500,"kafka-pizza-offset":"36950462","commit-skipped":30204488,"send-skipped":30204500,"window-calls":0,"send-calls":0},"class hs.samza.simple.SimpleKafkaTask":{"messageCount":28500000}},"header":{"reset-time":1398672217312,"job-id":"1","time":1398672937854,"host":"hadoop-node-1","container-name":"samza-container-2","source":"Partition-2","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}


{"metrics":{"org.apache.samza.system.kafka.KafkaSystemProducerMetrics":{"kafka-producer-sends":38,"kafka-partition-2-producer-buffer-size":0,"kafka-flushes":38,"kafka-metricssnapshotreporterfactory-producer-buffer-size":0,"kafka-samza-container-2-producer-buffer-size":0,"kafka-producer-reconnects":0,"kafka-flush-sizes":38}},"header":{"reset-time":1398672217312,"job-id":"1","time":1398672937855,"host":"hadoop-node-1","container-name":"samza-container-2","source":"MetricsSnapshotReporterFactory","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}


{"metrics":{"org.apache.samza.system.SystemConsumersMetrics":{"blocking-poll-timeout":10,"kafka-messages-per-poll":15667419,"chose-object":29071417,"kafka-ssp-fetches-per-poll":1501156,"max-buffered-messages-per-stream-partition":1000,"ssps-needed-by-chooser":1,"kafka-pizza-messages-chosen":29071417,"unprocessed-messages":0,"chose-null":1483650,"kafka-polls":15667420,"poll-timeout":10},"org.apache.samza.metrics.JvmMetrics":{"threads-runnable":5,"mem-heap-committed-mb":313.5625,"threads-new":0,"mem-non-heap-committed-mb":24.75,"mem-heap-used-mb":93.080475,"mem-non-heap-used-mb":24.63375,"threads-terminated":0,"ps marksweep-gc-time-millis":41,"ps scavenge-gc-count":380,"ps scavenge-gc-time-millis":16289,"gc-time-millis":16330,"threads-blocked":0,"threads-timed-waiting":6,"ps marksweep-gc-count":1,"threads-waiting":4,"gc-count":381},"org.apache.samza.container.SamzaContainerMetrics":{"process-null-envelopes":1483649,"process-envelopes":29071417,"process-calls":30555067,"commit-calls":30555066,"window-calls":30555066,"send-calls":30555066},"org.apache.samza.system.chooser.RoundRobinChooserMetrics":{"buffered-messages":0},"org.apache.samza.system.kafka.KafkaSystemConsumerMetrics":{"kafka-pizza-3-offset-change":37437828,"poll-count":15667420,"kafka-10.5.132.122-9092-topic-partitions":1,"no-more-messages-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":false,"kafka-10.5.132.122-9092-messages-read":36263,"blocking-poll-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":0,"kafka-pizza-3-bytes-read":1512471142,"kafka-pizza-3-messages-read":29071417,"kafka-10.5.132.122-9092-skipped-fetch-requests":8,"blocking-poll-timeout-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":728147,"kafka-pizza-3-messages-behind-high-watermark":0,"buffered-message-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":0,"kafka-10.5.132.122-9092-bytes-read":1512471142,"kafka-10.5.132.122-9092-reconnects":0},"org.apache.samza.system.kafka.KafkaSystemProducerMetrics":{"kafka-producer-sends":0,"kafka-flushes":14,"kafka-partition-3-producer-buffer-size":0,"kafka-producer-reconnects":0,"kafka-flush-sizes":0},"org.apache.samza.system.SystemProducersMetrics":{"partition-3-sends":0,"partition-3-flushes":14,"flushes":14,"sends":0}},"header":{"reset-time":1398672219649,"job-id":"1","time":1398673001854,"host":"hd-e.cdh","container-name":"samza-container-3","source":"samza-container-3","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}

 

看著頭都大了。應該還是JMX那種reporter看著舒服點。以後再試吧……

相關文章