細節決定成敗 MapReduce任務實戰 倒排索引

self_control發表於2016-06-15
索引
今天在偶然看到一個博文，裡面講述如何使用mapreduce進行倒排索引處理。那就拿這個任務當成本篇實戰任務吧。

一、任務描述


hdfs 上有三個檔案，內容下上面左面框中所示。右框中為處理完成後的結果檔案。
倒排索引（Inverted index），也常被稱為反向索引、置入檔案或反向檔案，是一種索引方法，被用來儲存在全文搜尋下某個單詞在一個文件或者一組文件中的儲存位置的對映。它是文件檢索系統中最常用的資料結構。通過倒排索引，可以根據單詞快速獲取包含這個單詞的文件列表。倒排索引主要由兩個部分組成：“單詞詞典”和“倒排檔案”
這個任務與傳統的倒排索引任務不同的地方是加上了每個檔案中的頻數。

二、實現思路

  • 首先關注結果中有檔名稱，這個我們有兩種方式處理：1、自定義InputFormat，在其中的自定義RecordReader中，直接通過InputSplit得到Path，繼而得到FileName;2、在Mapper中，通過上下文可以取到Split，也可以得到fileName。這個任務中我們使用第二種方式，得到filename.
  • 在mapper中，得到filename 及 word，封裝到一個自定義keu中。value 使用IntWritable。在map  中直接輸出值為1的IntWritable物件。
  • 對進入reduce函式中的key進行分組控制，要求按word相同的進入同一次reduce呼叫。所以需要自定義GroupingComparator。

三、實現程式碼

自定義Key， WordKey程式碼。 注意這裡面有個故意設定的坑。

點選(此處)摺疊或開啟

  1. package indexinverted;

  3. import java.io.DataInput;
  4. import java.io.DataOutput;
  5. import java.io.IOException;

  7. import org.apache.hadoop.io.WritableComparable;


  10. public class WordKey implements WritableComparable <WordKey> {

  12.         private String fileName;
  13.         private String word;

  15.         @Override
  16.         public void write(DataOutput out) throws IOException {
  17.                 out.writeUTF(fileName);
  18.                 out.writeUTF(word);
  19.         }

  21.         @Override
  22.         public void readFields(DataInput in) throws IOException {
  23.                 this.fileName = in.readUTF();
  24.                 this.word = in.readUTF();
  25.         }

  27.         @Override
  28.         public int compareTo(WordKey key) {
  29.                 int r = fileName.compareTo(key.fileName);
  30.                 if(r==0)
  31.                         r = word.compareTo(key.word);
  32.                 return r;
  33.         }

  35.         public String getFileName() {
  36.                 return fileName;
  37.         }

  39.         public void setFileName(String fileName) {
  40.                 this.fileName = fileName;
  41.         }

  43.         public String getWord() {
  44.                 return word;
  45.         }

  47.         public void setWord(String word) {
  48.                 this.word = word;
  49.         }
  50. }
Mapper、 Reducer、 IndexInvertedGroupingComparator （我喜歡把一些小的類當成內部類，放到Job類中，這樣程式碼比較簡單）
Reduce函式中處理輸出結果有點繁瑣，可以不用太關注。

點選(此處)摺疊或開啟

  1. package indexinverted;
  2. import java.io.IOException;
  3. import java.util.HashMap;
  4. import java.util.LinkedHashMap;

  6. import org.apache.hadoop.conf.Configuration;
  7. import org.apache.hadoop.conf.Configured;
  8. import org.apache.hadoop.fs.FileSystem;
  9. import org.apache.hadoop.fs.Path;
  10. import org.apache.hadoop.io.IntWritable;
  11. import org.apache.hadoop.io.LongWritable;
  12. import org.apache.hadoop.io.Text;
  13. import org.apache.hadoop.io.WritableComparable;
  14. import org.apache.hadoop.io.WritableComparator;
  15. import org.apache.hadoop.mapreduce.Job;
  16. import org.apache.hadoop.mapreduce.Mapper;
  17. import org.apache.hadoop.mapreduce.Reducer;
  18. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  19. import org.apache.hadoop.mapreduce.lib.input.FileSplit;
  20. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  21. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  22. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  23. import org.apache.hadoop.util.Tool;
  24. import org.apache.hadoop.util.ToolRunner;
  25. import org.apache.log4j.Logger;


  28. public class MyIndexInvertedJob extends Configured implements Tool{

  30.         public static class IndexInvertedMapper extends Mapper<LongWritable,Text,WordKey,IntWritable>{
  31.                 private WordKey newKey = new WordKey();
  32.                 private IntWritable ONE = new IntWritable(1);
  33.                 private String fileName ;
  34.                
  35.                 @Override
  36.                 protected void map(LongWritable key, Text value, Context context)
  37.                                 throws IOException, InterruptedException {
  38.                         newKey.setFileName(fileName);
  39.                         String words [] = value.toString().split(" ");
  40.                         for(String w:words){
  41.                                 newKey.setWord(w);
  42.                                 context.write(newKey, ONE);
  43.                         }
  44.                 }
  45.                 @Override
  46.                 protected void setup(Context context) throws IOException,
  47.                                 InterruptedException {
  48.                         FileSplit inputSplit = (FileSplit) context.getInputSplit();
  49.                         fileName = inputSplit.getPath().getName();
  50.                 }

  52.         }
  53.         public static class IndexInvertedReducer extends Reducer<WordKey,IntWritable,Text,Text>{
  54.                 private Text outputKey = new Text();

  56.                 @Override
  57.                 protected void reduce(WordKey key, Iterable<IntWritable> values,Context context)
  58.                                 throws IOException, InterruptedException {
  59.                         outputKey.set(key.getWord());
  60.                         LinkedHashMap <String,Integer> map = new LinkedHashMap<String,Integer>();
  61.                         for(IntWritable v :values){
  62.                                 if(map.containsKey(key.getFileName())){
  63.                                         map.put(key.getFileName(), map.get(key.getFileName())+ v.get());
  64.                                 }
  65.                                 else{
  66.                                         map.put(key.getFileName(), v.get());
  67.                                 }
  68.                         }
  69.                         StringBuilder sb = new StringBuilder();
  70.                         sb.append("{");
  71.                         for(String k: map.keySet()){
  72.                                 sb.append("(").append(k).append(",").append(map.get(k)).append(")").append(",");
  73.                         }
  74.                         sb.deleteCharAt(sb.length()-1).append("}");
  75.                         context.write(outputKey, new Text(sb.toString()));
  76.                 }

  78.         }
  79.         public static class IndexInvertedGroupingComparator extends WritableComparator{
  80.                 Logger log = Logger.getLogger(getClass());
  81.                 public IndexInvertedGroupingComparator(){
  82.                         super(WordKey.class,true);
  83.                 }

  85.                 @Override
  86.                 public int compare(WritableComparable a, WritableComparable b) {
  87.                         WordKey key1 = (WordKey) a;
  88.                         WordKey key2 = (WordKey) b;
  89.                         log.info("==============key1.getWord().compareTo(key2.getWord()):"+key1.getWord().compareTo(key2.getWord()));
  90.                         return key1.getWord().compareTo(key2.getWord());
  91.                 }

  93.         }
  94.         @Override
  95.         public int run(String[] args) throws Exception {
  96.                 Job job = Job.getInstance(getConf(), "IndexInvertedJob");
  97.                 job.setJarByClass(getClass());
  98.                 Configuration conf = job.getConfiguration();

  100.                 Path in = new Path("myinvertedindex/");
  101.                 Path out = new Path("myinvertedindex/output");
  102.                 FileSystem.get(conf).delete(out,true);
  103.                 FileInputFormat.setInputPaths(job, in);
  104.                 FileOutputFormat.setOutputPath(job, out);

  106.                 job.setInputFormatClass(TextInputFormat.class);
  107.                 job.setOutputFormatClass(TextOutputFormat.class);

  109.                 job.setMapperClass(IndexInvertedMapper.class);
  110.                 job.setMapOutputKeyClass(WordKey.class);
  111.                 job.setMapOutputValueClass(IntWritable.class);

  113.                 job.setReducerClass(IndexInvertedReducer.class);
  114.                 job.setOutputKeyClass(Text.class);
  115.                 job.setOutputValueClass(Text.class);

  117.                 job.setGroupingComparatorClass(IndexInvertedGroupingComparator.class);

  119.                 return job.waitForCompletion(true)?0:1;
  120.         }

  122.         public static void main(String [] args){
  123.                 int r = 0 ;
  124.                 try{
  125.                         r = ToolRunner.run(new Configuration(), new MyIndexInvertedJob(), args);
  126.                 }catch(Exception e){
  127.                         e.printStackTrace();
  128.                 }
  129.                 System.exit(r);
  130.         }

  132. }

四、檢視結果

hadoop dfs -cat myinvertedindex/output/part-r-00000 

點選(此處)摺疊或開啟

  1. MapReduce {(1.txt,1)}
  2. is {(1.txt,1)}
  3. simple {(1.txt,1)}
  4. MapReduce {(2.txt,1)}
  5. is {(2.txt,2)}
  6. powerful {(2.txt,1)}
  7. simple {(2.txt,1)}
  8. Hello {(3.txt,1)}
  9. MapReduce {(3.txt,2)}
  10. bye {(3.txt,1)}
檢視結果發現問題：單詞並沒有合併到一起，這會是什麼原因？

五、深坑回填

檢視結果發現問題：單詞並沒有合併到一起，這是什麼原因？
GroupingComparator 是在什麼樣的基礎上起作用的？是配置了按word相同輸入到同一次reduce呼叫就一定會相同的word都進入同一個reduce呼叫嗎？
NO！ Groupingcomparator中只是比較了相臨的兩個key是否相等。所有要結果正確，就要儲存key的排序與GroupingComparator的排序是相協調。
問題出在了WordKey的排序是先按檔案再按單詞進行排，這樣相臨的key並不是單詞相同的，而是檔案相同的。
所以要改下WordKey的comparTo方法 。修復後程式碼如下：

點選(此處)摺疊或開啟

  1. package indexinverted;

  3. import java.io.DataInput;
  4. import java.io.DataOutput;
  5. import java.io.IOException;

  7. import org.apache.hadoop.io.WritableComparable;


  10. public class WordKey implements WritableComparable <WordKey> {

  12.         private String fileName;
  13.         private String word;

  15.         @Override
  16.         public void write(DataOutput out) throws IOException {
  17.                 out.writeUTF(fileName);
  18.                 out.writeUTF(word);
  19.         }

  21.         @Override
  22.         public void readFields(DataInput in) throws IOException {
  23.                 this.fileName = in.readUTF();
  24.                 this.word = in.readUTF();
  25.         }

  27.         @Override
  28.         public int compareTo(WordKey key) {
  29.                 int r = word.compareTo(key.word);
  30.                 if(r==0)
  31.                         r = fileName.compareTo(key.fileName);
  32.                 return r;
  33.         }

  35.         public String getFileName() {
  36.                 return fileName;
  37.         }

  39.         public void setFileName(String fileName) {
  40.                 this.fileName = fileName;
  41.         }

  43.         public String getWord() {
  44.                 return word;
  45.         }

  47.         public void setWord(String word) {
  48.                 this.word = word;
  49.         }
  50. }


執行結果

點選(此處)摺疊或開啟

  1. Hello {(3.txt,1)}
  2. MapReduce {(1.txt,1),(2.txt,1),(3.txt,2)}
  3. bye {(3.txt,1)}
  4. is {(1.txt,1),(2.txt,2)}
  5. powerful {(2.txt,1)}
  6. simple {(1.txt,1),(2.txt,1)}








來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/30066956/viewspace-2120238/，如需轉載，請註明出處，否則將追究法律責任。

相關文章