ganglia 是監控的hadoop 叢集的最佳系統.

安裝步驟就不寫了. 比較簡單.

主要說下安裝過程, hadoop 的metrics 的配置問題.

hadoop-metrics2.properties

如果採用預設配置, 一般情況下, 你會苦不堪言.

主要問題是預設的把所有的metrics 監控項全部監控出來了.

那麼就帶來了一個問題.

每個任務, 會派生出很多container ,然後每個container 又有很多指標.

因為一直忙著救火, 就沒有截圖保留案底了.

我統計一下 , 一個hadoo 節點下面關於container 的 rrd 檔案就有11700+

單個hadoop 節點在一天內的監控資料大於 7.5GB .

到什麼情況呢, 在ganglia 裡面已經無法開啟單個節點的監控頁面了. 直接死了.

同樣的hbase 的 hadoop-metrics2-hbase.properties 中也會涉及到

regionserver 的問題. 每個節點上有很多regions ,然後每個region 都涉及到N多的metrics 監控項.

然後導致rrd 檔案暴漲. 效果跟前面一樣. 煩死人.

怎麼解決呢? 查文件.

在官方文件裡找到了一點答案. 然後就開始各種演繹.

Class	Description
	A glob pattern filter for metrics.
	A regex pattern filter for metrics

正則表達.

Fields inherited from class org.apache.hadoop.metrics2.filter.AbstractPatternFilter

EXCLUDE_KEY, EXCLUDE_TAGS_KEY, INCLUDE_KEY, INCLUDE_TAGS_KEY

透過正則表達可以限制某些metrics 監控項的採集等等.

到這裡問題基本就解決了.

我列出來我的配置的監控項做個示例:

more hadoop-metrics2-hbase.properties

hbase.sink.file-all.class=org.apache.hadoop.metrics2.sink.FileSink

hbase.sink.file-all.filename=all.metrics

############################################################

*.period=10

*.sink.ganglia.period=10

*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31

*.source.filter.class=org.apache.hadoop.metrics2.filter.RegexFilter

*.record.filter.class=${*.source.filter.class}

*.metric.filter.class=${*.source.filter.class}

hbase.sink.ganglia.metric.filter.exclude=.*_(max|min|mean|median|percentile)

hbase.sink.ganglia.record.filter.exclude=Regions

hbase.sink.ganglia.source.filter.exclude=.*Regions.*

hbase.sink.ganglia.period=10

hbase.sink.ganglia.servers=gangliaserver:8649

more hadoop-metrics2.properties

################################################

*.period=10

*.sink.ganglia.period=10

*.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31

*.source.filter.class=org.apache.hadoop.metrics2.filter.RegexFilter

*.record.filter.class=${*.source.filter.class}

*.metric.filter.class=${*.source.filter.class}

resourcemanager.sink.ganglia.metric.filter.exclude=.*_(max|min|mean|median|percentile)

resourcemanager.sink.ganglia.source.filter.exclude=.*_(container)*

resourcemanager.sink.ganglia.record.filter.exclude=container*

resourcemanager.sink.ganglia.source.filter.exclude=*.ontainerResource.*

nodemanager.sink.ganglia.metric.filter.exclude=.*_(max|min|mean|median|percentile)

nodemanager.sink.ganglia.source.filter.exclude=.*_(container)*

#nodeemanager.sink.ganglia.record.filter.exclude=*container.*

nodeemanager.sink.ganglia.record.filter.exclude=*container*

nodemanager.sink.ganglia.source.filter.exclude=*ontainerResource*

nodemanager.sink.ganglia.source.filter.exclude=*Containers*

mrappmaster.sink.ganglia.metric.filter.exclude=.*_(max|min|mean|median|percentile)

mrappmaster.sink.ganglia.source.filter.exclude=.*_(container)*

mrappmaster.sink.ganglia.record.filter.exclude=*container.*

mrappmaster.sink.ganglia.record.filter.exclude=*container*

mrappmaster.sink.ganglia.source.filter.exclude=*ontainerResource*

jobhistoryserver.sink.ganglia.metric.filter.exclude=.*_(max|min|mean|median|percentile)

jobhistoryserver.sink.ganglia.record.filter.exclude=*container.*

jobhistoryserver.sink.ganglia.record.filter.exclude=*container*

jobhistoryserver.sink.ganglia.source.filter.exclude=*ontainerResource*

##############################################################

............. 後面的關於server 的配置部分省略.....

ganglia 監控hadoop 叢集的配置檔案問題的一個說明.

Fields inherited from class org.apache.hadoop.metrics2.filter.AbstractPatternFilter

相關文章