外部排序

AH20發表於2024-10-06
排序
外部排序
如果需要排序的檔案超過了記憶體的大小，就需要分塊排序，按塊寫入檔案，最後再歸併已經有序的分塊檔案。

第一階段 切分排序
1.將原始未排序的大檔案，分為有序的小檔案
    1.1確定大檔案切分的位置
    1.2主執行緒按塊讀取原始檔案,然後交給排序執行緒
    1.3排序執行緒,對塊進行排序
    1.4寫入執行緒將有序的資料，寫入檔案


第二階段 歸併
    將臨時檔案歸併到目標的結果檔案

作為JAVA程式，需要注意的是，第一階段是IO密集型,儘量分配更多的記憶體給Old區
而第二階段是CPU密集型,需要注意Young區,避免過多的GC佔用CPU資源。

需要提前瞭解的技術細節。
1.JAVA切分大檔案
http://blog.itpub.net/29254281/viewspace-1161173/

2.JAVA記憶體對映檔案
http://blog.itpub.net/29254281/viewspace-1162157/

3.JAVA柵欄
http://blog.itpub.net/29254281/viewspace-1164727/

4.觀察者模式、生產者/消費者模式    

5.JVM監控和GC

實驗環境:
雙核CPU,使用1G的JAVA堆記憶體,對2億long型隨機數排序,一般情況下,2億long型隨機數檔案4G左右大小。




總體設計的思路
Main主執行緒負責切分原始檔案,blocking函式根據檔案分片的設定，返回原始檔案的分塊。
這樣主要為了分塊包含整行，而不會切斷資料。
讀取出來的原始未排序資料封裝為Sorter物件,提交給排序執行緒池。
排序之後的資料,封裝為Writer物件,再交給寫入執行緒池,將排序好的分塊寫入臨時檔案。
在每個分塊寫入之後，執行緒在CyclicBarrier柵欄處等待,待所有的分塊寫入完畢,啟用歸併。
需要特別注意的是,在柵欄等待之前,一定要釋放所有的資源,以便JVM GC回收記憶體。
就是Writer物件write方法的下列程式碼

歸併非常消耗CPU資源。
在歸併的過程中，因為每個分塊本身都是有序的,所以只需要一個執行緒計算各個分塊中最小的數字，將其寫入BlockingQueue。
而另一個執行緒,不斷的將佇列的資料順序寫入目標檔案。
這就是Merge物件的作用。這個過程使用了觀察者模式和生產者消費者模式。

實現如下:
  1. import java.io.BufferedInputStream;
  2. import java.io.BufferedOutputStream;
  3. import java.io.File;
  4. import java.io.FileInputStream;
  5. import java.io.FileOutputStream;
  6. import java.io.IOException;
  7. import java.io.RandomAccessFile;
  8. import java.nio.MappedByteBuffer;
  9. import java.nio.channels.FileChannel;
  10. import java.nio.channels.FileChannel.MapMode;
  11. import java.util.ArrayList;
  12. import java.util.Arrays;
  13. import java.util.Collections;
  14. import java.util.List;
  15. import java.util.Queue;
  16. import java.util.concurrent.BlockingQueue;
  17. import java.util.concurrent.BrokenBarrierException;
  18. import java.util.concurrent.ConcurrentLinkedQueue;
  19. import java.util.concurrent.CyclicBarrier;
  20. import java.util.concurrent.ExecutorService;
  21. import java.util.concurrent.Executors;
  22. import java.util.concurrent.LinkedBlockingQueue;

  24. public class Controller {
  25.     public static void main(String[] args) throws IOException {
  26.         Controller c = new Controller(new File("/home/lihuilin/桌面/t.txt"), 15, "/home/lihuilin/桌面/");
  27.     }

  29.     // 排序執行緒池
  30.     private final ExecutorService sortThread;
  31.     // 寫入執行緒池，將排序之後的分片寫入檔案
  32.     private final ExecutorService writerThread;
  33.     // 柵欄,等待所有分片寫入檔案之後，啟動合併
  34.     private final CyclicBarrier barrier;
  35.     // 原始沒有順序的大檔案
  36.     private final File file;
  37.     // 分片大小
  38.     private final int pieces;
  39.     // 輸出目錄
  40.     private final String outDir;
  41.     // 記錄分片寫入臨時檔案的位置
  42.     private final List<File> outFileList = new ArrayList<File>();

  44.     public Controller(File file, int pieces, final String outDir) throws IOException {
  45.         final long start = System.currentTimeMillis();
  46.         sortThread = Executors.newFixedThreadPool(1);
  47.         // 寫入執行緒池的執行緒數一定不能小於分片的大小。否則CyclicBarrier
  48.         // await之後，後續的分片將沒有執行緒可用。await不會釋放執行緒資源。
  49.         writerThread = Executors.newFixedThreadPool(pieces + 1);
  50.         this.file = file;
  51.         this.pieces = pieces;
  52.         this.outDir = outDir;
  53.         this.barrier = new CyclicBarrier(pieces, new Runnable() {

  55.             @Override
  56.             public void run() {
  57.                 long end = System.currentTimeMillis();
  58.                 System.out.println("合併之前總用時:" + (end - start) / 1000 + "s");
  59.                 // 合併有序的分片臨時檔案
  60.                 Merger merger = new Merger(outFileList, outDir);
  61.                 writerThread.submit(merger);
  62.                 try {
  63.                     merger.merge();
  64.                 } catch (IOException e) {
  65.                     // TODO Auto-generated catch block
  66.                     e.printStackTrace();
  67.                 } catch (InterruptedException e) {
  68.                     // TODO Auto-generated catch block
  69.                     e.printStackTrace();
  70.                 }

  72.                 writerThread.shutdown();
  73.                 sortThread.shutdown();
  74.                 end = System.currentTimeMillis();
  75.                 System.out.println("外部排序總用時:" + (end - start) / 1000 + "s");
  76.             }
  77.         });
  78.         action();
  79.     }

  81.     private void action() throws IOException {
  82.         List<Point> list = blocking(file, pieces);
  83.         for (Point p : list) {
  84.             Spilter spilter = new MappedByteBufferSpilter(file, p);
  85.             long[] data = null;
  86.             data = spilter.spilt();
  87.             Sorter s = new Sorter(data, p, writerThread, barrier, outFileList);
  88.             sortThread.submit(s);
  89.         }

  91.     }

  93.     private List<Point> blocking(File file, int piece) throws IOException {
  94.         List<Point> result = new ArrayList<Point>();
  95.         List<Long> list = new ArrayList<Long>();
  96.         list.add(-1L);
  97.         long length = file.length();
  98.         long step = length / piece;
  99.         long index = 0;
  100.         for (int i = 0; i < piece; i++) {
  101.             BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
  102.             if (index + step < length) {
  103.                 index = index + step;
  104.                 in.skip(index);

  106.                 while (in.read() != 10) {
  107.                     index = index + 1;
  108.                 }
  109.                 list.add(index);
  110.                 index++;
  111.             }
  112.             in.close();
  113.         }
  114.         list.add(length - 1);
  115.         for (int i = 0; i < list.size() - 1; i++) {
  116.             long skipSize = list.get(i) + 1;
  117.             long l = list.get(i + 1) - list.get(i);
  118.             result.add(new Point(skipSize, l, outDir));

  120.         }
  121.         return result;
  122.     }

  124. }

  126. class Merger implements Runnable {
  127.     private final List<Worker> workerList = new ArrayList<Worker>();
  128.     private String outDir = null;
  129.     private BlockingQueue<Long> queue = new LinkedBlockingQueue<Long>(1000);
  130.     private volatile boolean finished = false;

  132.     public Merger(List<File> outFileList, String outDir) {
  133.         for (File file : outFileList) {
  134.             Worker worker = new Worker(file, workerList);
  135.             workerList.add(worker);
  136.         }
  137.         this.outDir = outDir;
  138.     }

  140.     @Override
  141.     public void run() {
  142.         try {
  143.             System.out.println("讀取佇列,寫入目標檔案");
  144.             BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(outDir + "result.txt"), 50 * 1024 * 1024);
  145.             while (finished != true || !queue.isEmpty()) {
  146.                 Long l = queue.take();
  147.                 bos.write((l + "\n").getBytes());
  148.             }
  149.             bos.flush();
  150.             bos.close();
  151.         } catch (Exception ex) {
  152.             ex.printStackTrace();
  153.         }
  154.     }

  156.     public void merge() throws IOException, InterruptedException {
  157.         while (workerList.size() != 0) {
  158.             Collections.sort(workerList);
  159.             Worker worker = workerList.get(0);
  160.             Long data = worker.poll();
  161.             if (data == null) {
  162.                 workerList.remove(worker);
  163.             } else {
  164.                 queue.put(data);
  165.             }
  166.         }
  167.         finished = true;
  168.     }

  170.     private class Worker implements Comparable<Worker> {
  171.         private long data;
  172.         private MappedByteBuffer buffer = null;
  173.         private List<Worker> workerList = null;
  174.         private boolean eof = false;

  176.         Worker(File file, List<Worker> workerList) {
  177.             try {
  178.                 RandomAccessFile rFile = new RandomAccessFile(file, "r");
  179.                 FileChannel channel = rFile.getChannel();
  180.                 buffer = channel.map(MapMode.READ_ONLY, 0, channel.size());
  181.                 channel.close();
  182.                 rFile.close();
  183.                 this.workerList = workerList;
  184.                 data = buffer.getLong();
  185.             } catch (IOException e) {
  186.                 // TODO Auto-generated catch block
  187.                 e.printStackTrace();
  188.             }
  189.         }

  191.         public long peek() {
  192.             return data;
  193.         }

  195.         public Long poll() {
  196.             long result = data;
  197.             if (buffer.position() != buffer.limit()) {
  198.                 data = buffer.getLong();
  199.             } else {
  200.                 if (eof == false) {
  201.                     eof = true;
  202.                 } else {
  203.                     return null;
  204.                 }
  205.             }

  207.             return result;
  208.         }

  210.         @Override
  211.         public int compareTo(Worker o) {
  212.             if (this.peek() > o.peek()) {
  213.                 return 1;
  214.             } else if (this.peek() < o.peek()) {
  215.                 return -1;
  216.             } else {
  217.                 return 0;
  218.             }
  219.         }
  220.     }

  222. }

  224. interface Spilter {
  225.     public long[] spilt();
  226. }

  228. class Sorter implements Runnable {
  229.     long[] data;
  230.     Point p;
  231.     ExecutorService writerThread;
  232.     List<File> outFileList;
  233.     CyclicBarrier barrier;

  235.     public Sorter(long[] data, Point p, ExecutorService writerThread, CyclicBarrier barrier, List<File> outFileList) {
  236.         this.data = data;
  237.         this.p = p;
  238.         this.outFileList = outFileList;
  239.         this.barrier = barrier;
  240.         this.writerThread = writerThread;
  241.     }

  243.     public long[] sort() {
  244.         System.out.println("\t開始排序:" + p);
  245.         long start = System.currentTimeMillis();
  246.         Arrays.sort(this.data);
  247.         long end = System.currentTimeMillis();
  248.         System.out.println("\t結束排序:" + p + ",用時:" + (end - start) / 1000);
  249.         return this.data;
  250.     }

  252.     @Override
  253.     public void run() {
  254.         Writer writer = new MappedByteBufferWriter(sort(), p, barrier, outFileList);
  255.         writerThread.submit(writer);
  256.     }
  257. }

  259. interface Writer extends Runnable {
  260.     public void write();
  261. }

  263. class MappedByteBufferWriter implements Writer {
  264.     private static int FLAG = 1;
  265.     private CyclicBarrier barrier = null;
  266.     private File outfile = null;
  267.     private Point point = null;

  269.     private long[] data = null;
  270.     private List<File> outFileList = null;

  272.     public MappedByteBufferWriter(long[] data, Point point, CyclicBarrier barrier, List<File> outFileList) {
  273.         this.data = data;
  274.         this.point = point;
  275.         this.outfile = new File(point.getOutDir() + FLAG + ".txt");
  276.         this.barrier = barrier;
  277.         this.outFileList = outFileList;
  278.         FLAG++;
  279.     }

  281.     @Override
  282.     public void write() {
  283.         try {
  284.             System.out.println("\t\t開始寫入:" + point);
  285.             long start = System.currentTimeMillis();
  286.             FileChannel channel = new RandomAccessFile(this.outfile, "rw").getChannel();
  287.             MappedByteBuffer buffer = channel.map(MapMode.READ_WRITE, 0, this.data.length * 8);
  288.             for (int i = 0; i < data.length; i++) {
  289.                 buffer.putLong(data[i]);
  290.             }
  291.             buffer.force();
  292.             long end = System.currentTimeMillis();
  293.             System.out.println("\t\t結束寫入:" + point + ",用時:" + (end - start) / 1000);
  294.             synchronized (outFileList) {
  295.                 outFileList.add(outfile);
  296.             }
  297.             this.data = null;
  298.             channel.close();
  299.             buffer = null;
  300.             barrier.await();
  301.         } catch (IOException ex) {
  302.             ex.printStackTrace();
  303.         } catch (InterruptedException e) {
  304.             // TODO Auto-generated catch block
  305.             e.printStackTrace();
  306.         } catch (BrokenBarrierException e) {
  307.             // TODO Auto-generated catch block
  308.             e.printStackTrace();
  309.         }
  310.     }

  312.     @Override
  313.     public void run() {
  314.         this.write();
  315.     }

  317. }

  319. class MappedByteBufferSpilter implements Spilter {
  320.     private File file;
  321.     private Point point;

  323.     public MappedByteBufferSpilter(File file, Point p) {
  324.         this.file = file;
  325.         this.point = p;
  326.     }

  328.     @Override
  329.     public long[] spilt() {
  330.         System.out.println("開始讀入:" + point);
  331.         long start = System.currentTimeMillis();
  332.         long[] result = null;
  333.         try {

  335.             FileChannel in = new RandomAccessFile(file, "r").getChannel();
  336.             MappedByteBuffer inBuffer = in.map(MapMode.READ_ONLY, point.getSkipSize(), point.getLength());

  338.             byte[] data = new byte[inBuffer.limit()];
  339.             inBuffer.get(data);
  340.             result = new long[getObjectSize(data)];
  341.             int resultIndex = 0;
  342.             int index = 0;
  343.             int first = 0;
  344.             while (index < data.length) {
  345.                 if (data[index] == 10) {
  346.                     byte[] tmpData = Arrays.copyOfRange(data, first, index);
  347.                     String str = new String(tmpData);
  348.                     result[resultIndex] = Long.valueOf(str);
  349.                     resultIndex++;
  350.                     first = index + 1;
  351.                 }

  353.                 index++;
  354.             }
  355.             in.close();

  357.         } catch (IOException ex) {
  358.             ex.printStackTrace();
  359.         }
  360.         long end = System.currentTimeMillis();
  361.         System.out.println("結束讀入:" + point + ",用時:" + (end - start) / 1000);
  362.         return result;
  363.     }

  365.     private int getObjectSize(byte[] data) {
  366.         int size = 0;
  367.         for (byte b : data) {
  368.             if (b == 10) {
  369.                 size++;
  370.             }
  371.         }
  372.         return size;
  373.     }

  375. }

  377. class Point {
  378.     public Point(long skipSize, long length, String outDir) {
  379.         if (length > Integer.MAX_VALUE) {
  380.             throw new RuntimeException("長度溢位");
  381.         }
  382.         this.skipSize = skipSize;
  383.         this.length = (int) length;
  384.         this.outDir = outDir;
  385.     }

  387.     @Override
  388.     public String toString() {
  389.         return "Point [skipSize=" + skipSize + ", length=" + length + "]";
  390.     }

  392.     private long skipSize;
  393.     private int length;
  394.     private String outDir;

  396.     public String getOutDir() {
  397.         return outDir;
  398.     }

  400.     public long getSkipSize() {
  401.         return skipSize;
  402.     }

  404.     public int getLength() {
  405.         return length;
  406.     }
  407. }
執行:
  1. [lihuilin@lihuilin 桌面]$ java Controller
  2. 開始讀入:Point [skipSize=0, length=271726519]
  3. 結束讀入:Point [skipSize=0, length=271726519],用時:8
  4. 開始讀入:Point [skipSize=271726519, length=271726515]
  5. 開始排序:Point [skipSize=0, length=271726519]
  6. 結束排序:Point [skipSize=0, length=271726519],用時:3
  7. 開始寫入:Point [skipSize=0, length=271726519]
  8. 結束寫入:Point [skipSize=0, length=271726519],用時:2
  9. 結束讀入:Point [skipSize=271726519, length=271726515],用時:9
  10. 開始排序:Point [skipSize=271726519, length=271726515]
  11. 開始讀入:Point [skipSize=543453034, length=271726511]
  12. 結束排序:Point [skipSize=271726519, length=271726515],用時:4
  13. 開始寫入:Point [skipSize=271726519, length=271726515]
  14. 結束寫入:Point [skipSize=271726519, length=271726515],用時:3
  15. 結束讀入:Point [skipSize=543453034, length=271726511],用時:9
  16. 開始讀入:Point [skipSize=815179545, length=271726515]
  17. 開始排序:Point [skipSize=543453034, length=271726511]
  18. 結束排序:Point [skipSize=543453034, length=271726511],用時:3
  19. 開始寫入:Point [skipSize=543453034, length=271726511]
  20. 結束寫入:Point [skipSize=543453034, length=271726511],用時:5
  21. 結束讀入:Point [skipSize=815179545, length=271726515],用時:13
  22. 開始讀入:Point [skipSize=1086906060, length=271726524]
  23. 開始排序:Point [skipSize=815179545, length=271726515]
  24. 結束排序:Point [skipSize=815179545, length=271726515],用時:3
  25. 開始寫入:Point [skipSize=815179545, length=271726515]
  26. 結束寫入:Point [skipSize=815179545, length=271726515],用時:5
  27. 結束讀入:Point [skipSize=1086906060, length=271726524],用時:13
  28. 開始讀入:Point [skipSize=1358632584, length=271726507]
  29. 開始排序:Point [skipSize=1086906060, length=271726524]
  30. 結束排序:Point [skipSize=1086906060, length=271726524],用時:3
  31. 開始寫入:Point [skipSize=1086906060, length=271726524]
  32. 結束寫入:Point [skipSize=1086906060, length=271726524],用時:5
  33. 結束讀入:Point [skipSize=1358632584, length=271726507],用時:12
  34. 開始讀入:Point [skipSize=1630359091, length=271726523]
  35. 開始排序:Point [skipSize=1358632584, length=271726507]
  36. 結束排序:Point [skipSize=1358632584, length=271726507],用時:3
  37. 開始寫入:Point [skipSize=1358632584, length=271726507]
  38. 結束寫入:Point [skipSize=1358632584, length=271726507],用時:5
  39. 結束讀入:Point [skipSize=1630359091, length=271726523],用時:13
  40. 開始讀入:Point [skipSize=1902085614, length=271726514]
  41. 開始排序:Point [skipSize=1630359091, length=271726523]
  42. 結束排序:Point [skipSize=1630359091, length=271726523],用時:3
  43. 開始寫入:Point [skipSize=1630359091, length=271726523]
  44. 結束寫入:Point [skipSize=1630359091, length=271726523],用時:5
  45. 結束讀入:Point [skipSize=1902085614, length=271726514],用時:13
  46. 開始讀入:Point [skipSize=2173812128, length=271726519]
  47. 開始排序:Point [skipSize=1902085614, length=271726514]
  48. 結束排序:Point [skipSize=1902085614, length=271726514],用時:3
  49. 開始寫入:Point [skipSize=1902085614, length=271726514]
  50. 結束寫入:Point [skipSize=1902085614, length=271726514],用時:5
  51. 結束讀入:Point [skipSize=2173812128, length=271726519],用時:13
  52. 開始讀入:Point [skipSize=2445538647, length=271726516]
  53. 開始排序:Point [skipSize=2173812128, length=271726519]
  54. 結束排序:Point [skipSize=2173812128, length=271726519],用時:3
  55. 開始寫入:Point [skipSize=2173812128, length=271726519]
  56. 結束寫入:Point [skipSize=2173812128, length=271726519],用時:5
  57. 結束讀入:Point [skipSize=2445538647, length=271726516],用時:12
  58. 開始讀入:Point [skipSize=2717265163, length=271726517]
  59. 開始排序:Point [skipSize=2445538647, length=271726516]
  60. 結束排序:Point [skipSize=2445538647, length=271726516],用時:3
  61. 開始寫入:Point [skipSize=2445538647, length=271726516]
  62. 結束寫入:Point [skipSize=2445538647, length=271726516],用時:5
  63. 結束讀入:Point [skipSize=2717265163, length=271726517],用時:13
  64. 開始讀入:Point [skipSize=2988991680, length=271726517]
  65. 開始排序:Point [skipSize=2717265163, length=271726517]
  66. 結束排序:Point [skipSize=2717265163, length=271726517],用時:3
  67. 開始寫入:Point [skipSize=2717265163, length=271726517]
  68. 結束寫入:Point [skipSize=2717265163, length=271726517],用時:5
  69. 結束讀入:Point [skipSize=2988991680, length=271726517],用時:12
  70. 開始讀入:Point [skipSize=3260718197, length=271726516]
  71. 開始排序:Point [skipSize=2988991680, length=271726517]
  72. 結束排序:Point [skipSize=2988991680, length=271726517],用時:3
  73. 開始寫入:Point [skipSize=2988991680, length=271726517]
  74. 結束寫入:Point [skipSize=2988991680, length=271726517],用時:5
  75. 結束讀入:Point [skipSize=3260718197, length=271726516],用時:12
  76. 開始讀入:Point [skipSize=3532444713, length=271726515]
  77. 開始排序:Point [skipSize=3260718197, length=271726516]
  78. 結束排序:Point [skipSize=3260718197, length=271726516],用時:3
  79. 開始寫入:Point [skipSize=3260718197, length=271726516]
  80. 結束寫入:Point [skipSize=3260718197, length=271726516],用時:5
  81. 結束讀入:Point [skipSize=3532444713, length=271726515],用時:12
  82. 開始讀入:Point [skipSize=3804171228, length=271726376]
  83. 開始排序:Point [skipSize=3532444713, length=271726515]
  84. 結束排序:Point [skipSize=3532444713, length=271726515],用時:3
  85. 開始寫入:Point [skipSize=3532444713, length=271726515]
  86. 結束寫入:Point [skipSize=3532444713, length=271726515],用時:4
  87. 結束讀入:Point [skipSize=3804171228, length=271726376],用時:12
  88. 開始排序:Point [skipSize=3804171228, length=271726376]
  89. 結束排序:Point [skipSize=3804171228, length=271726376],用時:2
  90. 開始寫入:Point [skipSize=3804171228, length=271726376]
  91. 結束寫入:Point [skipSize=3804171228, length=271726376],用時:3
  92. 合併之前總用時:190s
  93. 讀取佇列,寫入目標檔案
  94. 外部排序總用時:398s
JVM監控:


從監控可以看到第一階段是IO密集型,對於記憶體需要很大;
而第二階段,排序各個分塊對於CPU壓力很大,一定要注意不要讓GC執行緒佔用過多CPU資源,就是Young區不能過小。

上圖中FGC 55之後的就是第二階段的過程,Young GC明顯增多。

驗證:
開始使用1-100的實驗資料，可以正確排序。
大檔案排序之後，可以使用Linux Sort命令驗證。


關於效能優化
外部排序的本質就是排序寫入小檔案，再將小檔案歸併為有序的目標檔案。
所以時間大致應該是拷貝這個檔案的時間乘以2.

但是...
想起在15所的時候，吳老師使用大致與我效能相當的配置，居然只用了230S左右..
優化需要注意的兩點
1.記憶體對映檔案
    檔案複製,分塊檔案寫入然後讀出這種場景使用記憶體對映檔案，避免了核心空間和使用者空間的記憶體複製。並且可以使用堆外作業系統記憶體作為快取.
2.避免GC佔用CPU
    讀取檔案和寫入檔案的時候,Byte和Long做轉換的時候，均採用了String型別作為中轉,
    

    
    後續可以考慮直接將byte和long型互轉,避免引入String型別,這樣就可以避免額外的GC

第一點都很容易想到,而第二點吳老師做了,我沒有實現,這可能是導致效能較慢的原因。
以後有時間再補上這個細節吧。
因為使用了記憶體對映檔案,比了避免誤差,每次實驗之前最好清除快取。

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29254281/viewspace-1167988/，如需轉載，請註明出處，否則將追究法律責任。

相關文章