一個MapReduce 程式示例 細節決定成敗(三) ：Combiner

self_control發表於2016-05-27
上一篇中，我們寫了一個統計輸入檔案中 a~z 每個字元個數的mr 程式。通過檢視執行日誌的Counter 計數可以看到map 到 reduce 之間的網路傳輸是多少。
而本篇我們將介紹的Combiner 是一個非常重要的元件，主要可以用來減少網路傳輸。原理是在每個map 任務執行節點上，先把map的輸出進行彙總，然後再傳輸到
reducer任務，也可以稱此為一個map 端的reduce操作。
先上程式碼。因為我們的combiner 邏輯與 reducer邏輯相同，所以我們簡單的使用MyWordCountReducer來設定成任務的combiner 。

點選(此處)摺疊或開啟

  1. job.setCombinerClass(MyWordCountJob.MyWordCountReducer.class);

重點看執行日誌：

點選(此處)摺疊或開啟

  1. File System Counters
  2.                 FILE: Number of bytes read=422
  3.                 FILE: Number of bytes written=338601
  4.                 FILE: Number of read operations=0
  5.                 FILE: Number of large read operations=0
  6.                 FILE: Number of write operations=0
  7.                 HDFS: Number of bytes read=556
  8.                 HDFS: Number of bytes written=103
  9.                 HDFS: Number of read operations=12
  10.                 HDFS: Number of large read operations=0
  11.                 HDFS: Number of write operations=2
  12.         Job Counters
  13.                 Launched map tasks=3
  14.                 Launched reduce tasks=1
  15.                 Data-local map tasks=3
  16.                 Total time spent by all maps in occupied slots (ms)=355336
  17.                 Total time spent by all reduces in occupied slots (ms)=66400
  18.         Map-Reduce Framework
  19.                 Map input records=8
  20.                 Map output records=137
  21.                 Map output bytes=822
  22.                 Map output materialized bytes=434
  23.                 Input split bytes=399
  24.                 Combine input records=137
  25.                 Combine output records=52
  26.                 Reduce input groups=25
  27.                 Reduce shuffle bytes=434
  28.                 Reduce input records=52
  29.                 Reduce output records=25
  30.                 Spilled Records=104
  31.                 Shuffled Maps =3
  32.                 Failed Shuffles=0
  33.                 Merged Map outputs=3
  34.                 GC time elapsed (ms)=274
  35.                 CPU time spent (ms)=3430
  36.                 Physical memory (bytes) snapshot=1078874112
  37.                 Virtual memory (bytes) snapshot=3947868160
  38.                 Total committed heap usage (bytes)=884539392
  39.         Shuffle Errors
  40.                 BAD_ID=0
  41.                 CONNECTION=0
  42.                 IO_ERROR=0
  43.                 WRONG_LENGTH=0
  44.                 WRONG_MAP=0
  45.                 WRONG_REDUCE=0
  46.         File Input Format Counters
  47.                 Bytes Read=157
  48.         File Output Format Counters
  49.                 Bytes Written=103
通過日誌可以看出， map 的輸入輸出記錄數為8和137 這個與沒combiner沒影響。但我們發現reduce的輸入由137變成了52。也就是說通過網路傳輸的資料量減少了。
同時也可以發現這個變化是combine 影響的。 可以看到 map 輸出的137條記錄實際是進行combine中處理了，combine 輸入為map輸入的137條記錄，combine  輸出的52條記錄進行到reduce中繼續彙總。
注意：combine 是在map 端的彙總，沒有網路傳輸，它彙總的也只是combiner所在節點的map的輸入，不是全部節點的資料。
reduce 中處理的才是所有節點的資料。
下一篇，我們介紹另一種類似combiner功能的實現方式。in-map 聚合。

最後還是把全部程式碼貼到後面吧


點選(此處)摺疊或開啟

  1. package wordcount;

  3. import java.io.IOException;

  5. import org.apache.commons.lang.StringUtils;
  6. import org.apache.hadoop.conf.Configuration;
  7. import org.apache.hadoop.conf.Configured;
  8. import org.apache.hadoop.fs.Path;
  9. import org.apache.hadoop.io.IntWritable;
  10. import org.apache.hadoop.io.LongWritable;
  11. import org.apache.hadoop.io.Text;
  12. import org.apache.hadoop.mapreduce.Job;
  13. import org.apache.hadoop.mapreduce.Mapper;
  14. import org.apache.hadoop.mapreduce.Reducer;
  15. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  16. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  17. import org.apache.hadoop.util.Tool;
  18. import org.apache.hadoop.util.ToolRunner;
  19. import org.apache.log4j.Logger;

  21. public class MyWordCountJob extends Configured implements Tool {
  22.         Logger log = Logger.getLogger(MyWordCountJob.class);

  24.         public static class MyWordCountMapper extends
  25.                         Mapper<LongWritable, Text, Text, IntWritable> {
  26.                 Logger log = Logger.getLogger(MyWordCountJob.class);

  28.                 Text mapKey = new Text();
  29.                 IntWritable mapValue = new IntWritable(1);
  30.                 @Override
  31.                 protected void map(LongWritable key, Text value, Context context)
  32.                                 throws IOException, InterruptedException {
  33.                         for(char c :value.toString().toLowerCase().toCharArray()){
  34.                                 if(c>='a' && c <='z'){
  35.                                         mapKey.set(String.valueOf(c));
  36.                                         context.write(mapKey, mapValue);
  37.                                 }
  38.                         }
  39.                 }

  41.         }


  44.         public static class MyWordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
  45.                 Text rkey = new Text();
  46.                 IntWritable rvalue = new IntWritable(1);
  47.                 @Override
  48.                 protected void reduce(Text key, Iterable<IntWritable> values,Context context)
  49.                                 throws IOException, InterruptedException {
  50.                         int n=0;
  51.                         for(IntWritable value :values){
  52.                                 n+= value.get();
  53.                         }
  54.                         rvalue.set(n);
  55.                         context.write(key, rvalue);
  56.                 }
  57.         }

  59.         @Override
  60.         public int run(String[] args) throws Exception {
  61.                 //valid the parameters
  62.                 if(args.length !=2){
  63.                         return -1;
  64.                 }

  66.                 Job job = Job.getInstance(getConf(), "MyWordCountJob");
  67.                 job.setJarByClass(MyWordCountJob.class);

  69.                 Path inPath = new Path(args[0]);
  70.                 Path outPath = new Path(args[1]);

  72.                 outPath.getFileSystem(getConf()).delete(outPath,true);
  73.                 TextInputFormat.setInputPaths(job, inPath);
  74.                 TextOutputFormat.setOutputPath(job, outPath);


  77.             job.setMapperClass(MyWordCountJob.MyWordCountMapper.class);
  78.                 job.setReducerClass(MyWordCountJob.MyWordCountReducer.class);
  79.                 job.setCombinerClass(MyWordCountJob.MyWordCountReducer.class);
  80.             job.setInputFormatClass(TextInputFormat.class);
  81.             job.setOutputFormatClass(TextOutputFormat.class);

  83.                 job.setMapOutputKeyClass(Text.class);
  84.                 job.setMapOutputValueClass(IntWritable.class);
  85.                 job.setOutputKeyClass(Text.class);
  86.                 job.setOutputValueClass(IntWritable.class);

  88.                 return job.waitForCompletion(true)?0:1;
  89.         }
  90.         public static void main(String [] args){
  91.                 int result = 0;
  92.                 try {
  93.                         result = ToolRunner.run(new Configuration(), new MyWordCountJob(), args);
  94.                 } catch (Exception e) {
  95.                         e.printStackTrace();
  96.                 }
  97.                 System.exit(result);
  98.         }

  100. }
下一篇中，我們使用in-map Aggregation來優化我們的任務。在某種情況下，可以比combiner更有效率。 
一個MapReduce 程式示例 細節決定成敗(四) ：In-Map Aggregation 

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/30066956/viewspace-2107885/，如需轉載，請註明出處，否則將追究法律責任。

相關文章