一個MapReduce 程式示例 細節決定成敗(六) ：CombineFileInputFormat

self_control發表於2016-05-30
ORM

hadoop的mr 任務設計上是針對大檔案的，但實踐中難免會遇到大量小檔案的情況，就像我們這個字元數量統計的mr。 

輸入是三個小檔案。所以每個檔案至少都會產生一下split，每個split 又會產生一個map 任務。對於很少的資料量，啟動一個jvm的代價顯得有點過大了。
所以這時就需要我們使用CombineFileInputFormat來對輸入的小檔案進行合併。


本次實驗，我們使用自定義的InputFormat，用來減少mapper 任務資料。

既然要寫一個自定義的InputFormat 那我們先來了解一下什麼是InputFormat、以及相關的概念inputsplit、 recordreader.


InputFormat 描述了一個MR Job的輸入格式。那有以下幾個作用：

  1. 校驗輸入檔案是否有效。
  2. 把輸入檔案劃分為邏輯的InputSplit ,每一個InputSplit 例項會送往不同的Mapper(重點，這是我們本篇的重點，也是優化的地方，意味著過多的inputsplit 會造成過多的map 任務)。
  3. 提供一下RecordReader,用來從inputsplit例項中讀取key value 來供maper 使用。

基於檔案輸入的InputFormat預設是按照輸入檔案的大小來劃分inputsplit。一般情況下，inputsplit的最大值為分散式檔案系統的塊大小（預設為128m、之前版本為64m）,inputsplit的最小值可以通過mapreduce.input.fileinputformat.split.minsize.進行設定。

InputSplit

InputSplit 描述了哪兒些資料裝被送往不同的maper。一般情況下，InputSplit 包含的是基於位元組的資料檢視、這些位元組的解析則是由後面要說到的RecordReader.

RecordReader

RecordReader 負責從InputSplit中 解析出供map 函式使用的對。


一下InputFormat中最主要的兩個功能就是getInputSplits 和 createRecordReader 。 而一般 getInputSplit 在FileInputFormat中已經完成，FileInputFormat一般 
是所有基於檔案的InputFormat要繼承的類。
所以重點就放在了RecordReader.
下面直接上程式碼吧。
關注以下幾個地方：

  1. MyCombinedFilesInputFormat  的父類是CombineFileInputFormat
  2. createRecordReader 返回的是一個CombineFileInputFormat，其建構函式資料一個自定義的RecordReader.
  3. MyRecordReader 的父類是RecordReader. 
  4. 關注initialize 方法中split的處理。
MyCombinedFilesInputFormat


點選(此處)摺疊或開啟

  1. package wordcount;

  3. import java.io.IOException;

  5. import org.apache.hadoop.io.LongWritable;
  6. import org.apache.hadoop.io.Text;
  7. import org.apache.hadoop.mapreduce.InputSplit;
  8. import org.apache.hadoop.mapreduce.RecordReader;
  9. import org.apache.hadoop.mapreduce.TaskAttemptContext;
  10. import org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat;
  11. import org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader;
  12. import org.apache.hadoop.mapreduce.lib.input.CombineFileSplit;
  13. import org.apache.hadoop.mapreduce.lib.input.FileSplit;
  14. import org.apache.hadoop.mapreduce.lib.input.LineRecordReader;


  17. public class MyCombinedFilesInputFormat extends CombineFileInputFormat<LongWritable, Text> {


  20.         @Override
  21.         public RecordReader<LongWritable, Text> createRecordReader(InputSplit split, TaskAttemptContext context) throws IOException {
  22.                 return new CombineFileRecordReader<LongWritable, Text>((CombineFileSplit) split,context,MyRecordReader.class);
  23.         }
  24.         public static class MyRecordReader extends RecordReader<LongWritable, Text> {
  25.                 private Integer index;
  26.                 private LineRecordReader reader;

  28.                 public MyRecordReader(CombineFileSplit split, TaskAttemptContext context, Integer index) {
  29.                         this.index = index;
  30.                         reader = new LineRecordReader();
  31.                 }

  33.                 @Override
  34.                 public void initialize(InputSplit split, TaskAttemptContext context)
  35.                                 throws IOException, InterruptedException {
  36.                         CombineFileSplit cfsplit = (CombineFileSplit) split;
  37.                         FileSplit fileSplit = new FileSplit(cfsplit.getPath(index),
  38.                                                                                                 cfsplit.getOffset(index),
  39.                                                                                                 cfsplit.getLength(index),
  40.                                                                                                 cfsplit.getLocations()
  41.                                         );
  42.                         reader.initialize(fileSplit, context);
  43.                 }

  45.                 @Override
  46.                 public boolean nextKeyValue() throws IOException, InterruptedException {
  47.                         return reader.nextKeyValue();
  48.                 }

  50.                 @Override
  51.                 public LongWritable getCurrentKey() throws IOException, InterruptedException {
  52.                         return reader.getCurrentKey();
  53.                 }

  55.                 @Override
  56.                 public Text getCurrentValue() throws IOException,
  57.                                 InterruptedException {
  58.                         return reader.getCurrentValue();
  59.                 }

  61.                 @Override
  62.                 public float getProgress() throws IOException, InterruptedException {
  63.                         return reader.getProgress();
  64.                 }

  66.                 @Override
  67.                 public void close() throws IOException {
  68.                         reader.close();
  69.                 }

  71.         }
  72. }

下面看一下job的配置
注意以下兩個地方：

  1. job.setInputFormatClass(MyCombinedFilesInputFormat.class); 
  2. MyCombinedFilesInputFormat.setMaxInputSplitSize(job, 1024*1024*64);


點選(此處)摺疊或開啟

  1. 點選(此處)摺疊或開啟
  2.         @Override
  3.         public int run(String[] args) throws Exception {
  4.                 //valid the parameters
  5.                 if(args.length !=2){
  6.                         return -1;
  7.                 }

  9.                 Job job = Job.getInstance(getConf(), "MyWordCountJob");
  10.                 job.setJarByClass(MyWordCountJob.class);

  12.                 Path inPath = new Path(args[0]);
  13.                 Path outPath = new Path(args[1]);

  15.                 outPath.getFileSystem(getConf()).delete(outPath,true);
  16.                 TextInputFormat.setInputPaths(job, inPath);
  17.                 TextOutputFormat.setOutputPath(job, outPath);


  20.                 job.setMapperClass(MyWordCountJob.MyWordCountMapper.class);
  21.                 job.setReducerClass(MyWordCountJob.MyWordCountReducer.class);

  23.                 job.setInputFormatClass(MyCombinedFilesInputFormat.class);
  24.                 MyCombinedFilesInputFormat.setMaxInputSplitSize(job, 1024*1024*64);
  25.                 job.setOutputFormatClass(TextOutputFormat.class);
  26.                 job.setMapOutputKeyClass(Text.class);
  27.                 job.setMapOutputValueClass(IntWritable.class);
  28.                 job.setOutputKeyClass(Text.class);
  29.                 job.setOutputValueClass(IntWritable.class);


  32.                 return job.waitForCompletion(true)?0:1;
  33.         }
執行一下驗證是否影響了map的任務數。
可以看到，輸入是三個小檔案、但只起了一個map task。

點選(此處)摺疊或開啟

  1. [train@sandbox MyWordCount]$ hadoop jar mywordcount.jar mrdemo/ output
  2. 16/05/12 11:12:48 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.252.131:8050
  3. 16/05/12 11:12:49 INFO input.FileInputFormat: Total input paths to process : 3
  4. 16/05/12 11:12:49 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 1, size left: 157
  5. 16/05/12 11:12:49 INFO mapreduce.JobSubmitter: number of splits:1
  6. 16/05/12 11:12:49 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
  7. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  8. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
  9. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
  10. 16/05/12 11:12:49 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
  11. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  12. 16/05/12 11:12:49 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
  13. 16/05/12 11:12:49 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
  14. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  15. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
  16. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
  17. 16/05/12 11:12:49 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
  18. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  19. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
  20. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
  21. 16/05/12 11:12:49 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
  22. 16/05/12 11:12:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462517728035_0104
  23. 16/05/12 11:12:50 INFO impl.YarnClientImpl: Submitted application application_1462517728035_0104 to ResourceManager at sandbox.hortonworks.com/192.168.252.131:8050
  24. 16/05/12 11:12:50 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1462517728035_0104/
  25. 16/05/12 11:12:50 INFO mapreduce.Job: Running job: job_1462517728035_0104
  26. 16/05/12 11:12:58 INFO mapreduce.Job: Job job_1462517728035_0104 running in uber mode : false
  27. 16/05/12 11:12:58 INFO mapreduce.Job: map 0% reduce 0%
  28. 16/05/12 11:13:10 INFO mapreduce.Job: map 100% reduce 0%
  29. 16/05/12 11:13:17 INFO mapreduce.Job: map 100% reduce 100%
  30. 16/05/12 11:13:18 INFO mapreduce.Job: Job job_1462517728035_0104 completed successfully
  31. 16/05/12 11:13:18 INFO mapreduce.Job: Counters: 43
  32.         File System Counters
  33.                 FILE: Number of bytes read=1198
  34.                 FILE: Number of bytes written=170905
  35.                 FILE: Number of read operations=0
  36.                 FILE: Number of large read operations=0
  37.                 FILE: Number of write operations=0
  38.                 HDFS: Number of bytes read=487
  39.                 HDFS: Number of bytes written=108
  40.                 HDFS: Number of read operations=8
  41.                 HDFS: Number of large read operations=0
  42.                 HDFS: Number of write operations=2
  43.         Job Counters
  44.                 Launched map tasks=1
  45.                 Launched reduce tasks=1
  46.                 Other local map tasks=1
  47.                 Total time spent by all maps in occupied slots (ms)=71600
  48.                 Total time spent by all reduces in occupied slots (ms)=33720
  49.         Map-Reduce Framework
  50.                 Map input records=8
  51.                 Map output records=149
  52.                 Map output bytes=894
  53.                 Map output materialized bytes=1198
  54.                 Input split bytes=330
  55.                 Combine input records=0
  56.                 Combine output records=0
  57.                 Reduce input groups=26
  58.                 Reduce shuffle bytes=1198
  59.                 Reduce input records=149
  60.                 Reduce output records=26
  61.                 Spilled Records=298
  62.                 Shuffled Maps =1
  63.                 Failed Shuffles=0
  64.                 Merged Map outputs=1
  65.                 GC time elapsed (ms)=54
  66.                 CPU time spent (ms)=1850
  67.                 Physical memory (bytes) snapshot=445575168
  68.                 Virtual memory (bytes) snapshot=1995010048
  69.                 Total committed heap usage (bytes)=345636864
  70.         Shuffle Errors
  71.                 BAD_ID=0
  72.                 CONNECTION=0
  73.                 IO_ERROR=0
  74.                 WRONG_LENGTH=0
  75.                 WRONG_MAP=0
  76.                 WRONG_REDUCE=0
  77.         File Input Format Counters
  78.                 Bytes Read=0
  79.         File Output Format Counters
  80.                 Bytes Written=108


本篇中簡單處理了MyRecordReader,在下篇中演示如何使用一下MyRecordReader 實現讀取自定義的Key Value對輸入到map函式中。

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/30066956/viewspace-2109215/，如需轉載，請註明出處，否則將追究法律責任。

相關文章