1. 設計思路
在MapReduce過程中自帶有排序,可以使用這個預設的排序達到我們的目的。 MapReduce 是按照key值進行排序的,我們在Map過程中將讀入的資料轉化成IntWritable型別,然後作為Map的key值輸出。 Reduce 階段拿到的就是按照key值排序好的<key,value list>,將key值輸出,並根據value list 中元素的個數決定key的輸出次數。
2. 實現
2.1 程式程式碼
package sort; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class SimpleSort { public static class Map extends Mapper<LongWritable, Text, IntWritable, IntWritable> { private IntWritable data; protected void map(LongWritable key, Text value, Context context) throws java.io.IOException, InterruptedException { data = new IntWritable(); String line = value.toString(); data.set(Integer.parseInt(line)); context.write(data, new IntWritable(1)); }; } public static class Reduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable> { private static IntWritable num = new IntWritable(1); protected void reduce(IntWritable key, java.lang.Iterable<IntWritable> values, Context output) throws java.io.IOException, InterruptedException { for ( IntWritable val : values){ output.write(num, key); num = new IntWritable(num.get() + 1); } }; } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { Configuration conf = new Configuration(); Job job = new Job(conf,"simple sort"); job.setJarByClass(SimpleSort.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setOutputKeyClass(IntWritable.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path("/user/hadoop_admin/sortin")); FileOutputFormat.setOutputPath(job, new Path("/user/hadoop_admin/sortout")); System.exit((job.waitForCompletion(true) ? 0 : 1)); } }
2.2 測試結果
測試用例
file1
2 3 1 89 34 21 67 35
file2
38 29 1 23 49 16
執行資訊
16/04/11 10:09:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/04/11 10:09:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. ****hdfs://master:9000/user/hadoop_admin/sortin 16/04/11 10:09:00 INFO input.FileInputFormat: Total input paths to process : 2 16/04/11 10:09:00 WARN snappy.LoadSnappy: Snappy native library not loaded 16/04/11 10:09:00 INFO mapred.JobClient: Running job: job_local_0001 16/04/11 10:09:00 INFO mapred.Task: Using ResourceCalculatorPlugin : null 16/04/11 10:09:00 INFO mapred.MapTask: io.sort.mb = 100 16/04/11 10:09:00 INFO mapred.MapTask: data buffer = 79691776/99614720 16/04/11 10:09:00 INFO mapred.MapTask: record buffer = 262144/327680 16/04/11 10:09:00 INFO mapred.MapTask: Starting flush of map output 16/04/11 10:09:00 INFO mapred.MapTask: Finished spill 0 16/04/11 10:09:00 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 16/04/11 10:09:01 INFO mapred.JobClient: map 0% reduce 0% 16/04/11 10:09:03 INFO mapred.LocalJobRunner: 16/04/11 10:09:03 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done. 16/04/11 10:09:03 INFO mapred.Task: Using ResourceCalculatorPlugin : null 16/04/11 10:09:03 INFO mapred.MapTask: io.sort.mb = 100 16/04/11 10:09:03 INFO mapred.MapTask: data buffer = 79691776/99614720 16/04/11 10:09:03 INFO mapred.MapTask: record buffer = 262144/327680 16/04/11 10:09:03 INFO mapred.MapTask: Starting flush of map output 16/04/11 10:09:03 INFO mapred.MapTask: Finished spill 0 16/04/11 10:09:03 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting 16/04/11 10:09:04 INFO mapred.JobClient: map 100% reduce 0% 16/04/11 10:09:06 INFO mapred.LocalJobRunner: 16/04/11 10:09:06 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done. 16/04/11 10:09:06 INFO mapred.Task: Using ResourceCalculatorPlugin : null 16/04/11 10:09:06 INFO mapred.LocalJobRunner: 16/04/11 10:09:06 INFO mapred.Merger: Merging 2 sorted segments 16/04/11 10:09:06 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 144 bytes 16/04/11 10:09:06 INFO mapred.LocalJobRunner: 16/04/11 10:09:06 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting 16/04/11 10:09:06 INFO mapred.LocalJobRunner: 16/04/11 10:09:06 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now 16/04/11 10:09:06 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to /user/hadoop_admin/sortout 16/04/11 10:09:09 INFO mapred.LocalJobRunner: reduce > reduce 16/04/11 10:09:09 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done. 16/04/11 10:09:10 INFO mapred.JobClient: map 100% reduce 100% 16/04/11 10:09:10 INFO mapred.JobClient: Job complete: job_local_0001 16/04/11 10:09:10 INFO mapred.JobClient: Counters: 19 16/04/11 10:09:10 INFO mapred.JobClient: File Output Format Counters 16/04/11 10:09:10 INFO mapred.JobClient: Bytes Written=71 16/04/11 10:09:10 INFO mapred.JobClient: FileSystemCounters 16/04/11 10:09:10 INFO mapred.JobClient: FILE_BYTES_READ=85835 16/04/11 10:09:10 INFO mapred.JobClient: HDFS_BYTES_READ=97 16/04/11 10:09:10 INFO mapred.JobClient: FILE_BYTES_WRITTEN=239842 16/04/11 10:09:10 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=71 16/04/11 10:09:10 INFO mapred.JobClient: File Input Format Counters 16/04/11 10:09:10 INFO mapred.JobClient: Bytes Read=38 16/04/11 10:09:10 INFO mapred.JobClient: Map-Reduce Framework 16/04/11 10:09:10 INFO mapred.JobClient: Reduce input groups=13 16/04/11 10:09:10 INFO mapred.JobClient: Map output materialized bytes=152 16/04/11 10:09:10 INFO mapred.JobClient: Combine output records=0 16/04/11 10:09:10 INFO mapred.JobClient: Map input records=14 16/04/11 10:09:10 INFO mapred.JobClient: Reduce shuffle bytes=0 16/04/11 10:09:10 INFO mapred.JobClient: Reduce output records=14 16/04/11 10:09:10 INFO mapred.JobClient: Spilled Records=28 16/04/11 10:09:10 INFO mapred.JobClient: Map output bytes=112 16/04/11 10:09:10 INFO mapred.JobClient: Total committed heap usage (bytes)=877854720 16/04/11 10:09:10 INFO mapred.JobClient: Combine input records=0 16/04/11 10:09:10 INFO mapred.JobClient: Map output records=14 16/04/11 10:09:10 INFO mapred.JobClient: SPLIT_RAW_BYTES=230 16/04/11 10:09:10 INFO mapred.JobClient: Reduce input records=14
結果
1 1 2 1 3 2 4 3 5 16 6 21 7 23 8 29 9 34 10 35 11 38 12 49 13 67 14 89