外部排序優化

壹頁書發表於2014-05-29
排序優化
優化外部排序
主要步驟重新拆分如下
第一階段
1.將未排序的大檔案切分(spilterReceiver)
2.一個執行緒池專門接收byte[]，並將其轉化為long[] (prepareReceiver)
3.一個執行緒池專門排序long[] (sortReceiver)
4.一個執行緒池專門將已經有序的long[]寫入檔案 (writerReceiver)
這個過程直接寫long型別的數字到檔案，因為寫入文字佔用IO資源比直接寫入long型數字要多。

第二階段
1.一個執行緒用於歸併排序,將結果寫入緩衝區 (mergerReceiver)
2.如果緩衝區快滿了，就將其交給一個執行緒池用於寫入最終的檔案。 (mergerReceiver)

優化的重點是平衡CPU，記憶體和硬碟的資源使用。

每個電腦的配置不同，都需要進行不同的調整，才能達到最優化的效果。
單位的電腦是i5的CPU，8G記憶體,7200轉硬碟。
我使用的虛擬機器分配了2個核心,4G記憶體。
跑這個程式大致需要200s左右。
因為第二階段的第一個步驟是非常消耗CPU資源的，單位電腦CPU主頻很高，所以不會成為瓶頸。

而家裡的電腦，應該配置更接近吳老師做實驗的配置。主頻比較低,32位系統,4G記憶體。
這個配置明顯CPU成為瓶頸
在第一個階段，切分出來的檔案，由於CPU不能及時排序並輸出，經常導致OOM
只能在切分一個檔案出來之後，讓切分執行緒sleep一段時間，大量測試得到的結果是這個執行緒至少sleep2500毫秒才能不導致OOM。
也就是說,CPU運算的速度跟不上IO的速度。後來想到一個方法，加大分片。原來4G檔案使用15個分片，現在使用30個分片。資料快入快出,這樣可以避免OOM。至於這個分片的數字，恐怕每個機器都需要實際的測試。
另外在第二階段，也出現了CPU拖累IO,導致效能下降。
因為歸併的時候，需要大量運算各個分片中最小的資料，經常看到計算50M資料,CPU需要2s，而寫入一般1s就可以完成。
而這個過程是單執行緒的，CPU的主頻直接決定了外部排序的總體時間。


  1. import java.io.BufferedInputStream;
  2. import java.io.File;
  3. import java.io.FileInputStream;
  4. import java.io.IOException;
  5. import java.io.RandomAccessFile;
  6. import java.nio.ByteBuffer;
  7. import java.nio.MappedByteBuffer;
  8. import java.nio.channels.FileChannel;
  9. import java.nio.channels.FileChannel.MapMode;
  10. import java.util.ArrayList;
  11. import java.util.Arrays;
  12. import java.util.Collections;
  13. import java.util.List;
  14. import java.util.concurrent.BrokenBarrierException;
  15. import java.util.concurrent.CyclicBarrier;
  16. import java.util.concurrent.ExecutorService;
  17. import java.util.concurrent.Executors;

  19. public class Controller {
  20.     public static void main(String[] args) throws IOException {
  21.         new Controller().action(new File("/home/lihuilin/桌面/t.txt"), 30, "/home/lihuilin/桌面/");

  23.     }

  25.     public void action(File file, int pieces, String outDir) throws IOException {
  26.         Invoker invoker = new Invoker(pieces);
  27.         List<Command> commandList = blocking(file, pieces, outDir);
  28.         for (Command command : commandList) {
  29.             command.setInvoker(invoker);
  30.             invoker.executeCommand(command);
  31.         }
  32.     }

  34.     private List<Command> blocking(File file, int pieces, String outDir) throws IOException {
  35.         List<Command> result = new ArrayList<Command>();
  36.         List<Long> list = new ArrayList<Long>();
  37.         list.add(-1L);
  38.         long length = file.length();
  39.         long step = length / pieces;
  40.         long index = 0;
  41.         for (int i = 0; i < pieces; i++) {
  42.             BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));
  43.             if (index + step < length) {
  44.                 index = index + step;
  45.                 in.skip(index);
  46.                 while (in.read() != 10) {
  47.                     index = index + 1;
  48.                 }
  49.                 list.add(index);
  50.                 index++;
  51.             }
  52.             in.close();
  53.         }
  54.         list.add(length - 1);
  55.         for (int i = 0; i < list.size() - 1; i++) {
  56.             long skipSize = list.get(i) + 1;
  57.             long l = list.get(i + 1) - list.get(i);
  58.             result.add(new SpilterCommand(file, skipSize, l, outDir));
  59.         }
  60.         return result;
  61.     }

  63. }

  65. interface Command extends Comparable<Command> {
  66.     public void setInvoker(Invoker invoker);

  68.     public void setReceiver(Receiver receiver);

  70.     public void execute() throws IOException;

  72.     public long getSkipSize();

  74.     public int getLength();

  76.     public String getOutDir();

  78.     public File getSourceFile();

  80.     public Invoker getInvoker();

  82.     public void clear();

  84.     public File getOutFile();

  86.     public File getTargetFile();
  87. }

  89. class Invoker {
  90.     private Receiver sortReceiver;
  91.     private Receiver spilterReceiver;
  92.     private Receiver writerReceiver;
  93.     private Receiver prepareReceiver;
  94.     private Receiver mergerReceiver;

  96.     public Invoker(int pieces) {
  97.         writerReceiver = new WriterReceiver(pieces);
  98.         spilterReceiver = new SpilterReceiver();
  99.         sortReceiver = new SorterReceiver();
  100.         prepareReceiver = new PrepareReceiver();
  101.         mergerReceiver = new MergerReceiver();
  102.     }

  104.     public void executeCommand(Command command) {
  105.         try {
  106.             if (command instanceof SpilterCommand) {
  107.                 spilterReceiver.action(command);
  108.             } else if (command instanceof PrepareCommand) {
  109.                 prepareReceiver.action(command);
  110.             } else if (command instanceof SortCommand) {
  111.                 sortReceiver.action(command);
  112.             } else if (command instanceof WriterCommand) {
  113.                 writerReceiver.action(command);
  114.             } else if (command instanceof MergeCommand) {
  115.                 mergerReceiver.action(command);
  116.             } else if (command instanceof KillCommand) {
  117.                 writerReceiver.shutdown();
  118.                 spilterReceiver.shutdown();
  119.                 sortReceiver.shutdown();
  120.                 prepareReceiver.shutdown();
  121.                 mergerReceiver.shutdown();
  122.             }
  123.         } catch (IOException ex) {
  124.             ex.printStackTrace();
  125.         }
  126.     }
  127. }

  129. class SpilterCommand extends AbstractCommand {

  131.     public SpilterCommand(File sourceFile, long skipSize, long length, String outDir) {
  132.         super(sourceFile, skipSize, length, outDir);
  133.     }

  135.     @Override
  136.     public void clear() {

  138.     }

  140. }

  142. abstract class AbstractCommand implements Command {
  143.     private Invoker invoker = null;
  144.     private Receiver receiver = null;
  145.     private long skipSize = 0L;
  146.     private int length = 0;
  147.     private String outDir = null;
  148.     private File sourceFile;
  149.     private File outFile = null;
  150.     public static final long START = System.currentTimeMillis();
  151.     private static int FLAG = 0;
  152.     private File targetFile = null;

  154.     private synchronized int getFlag() {
  155.         return FLAG++;
  156.     }

  158.     public AbstractCommand(Command command) {
  159.         this(command.getSourceFile(), command.getSkipSize(), command.getLength(), command.getOutDir());
  160.         this.invoker = command.getInvoker();
  161.         if (command.getOutFile() != null) {
  162.             this.outFile = command.getOutFile();
  163.         } else {
  164.             this.outFile = new File(this.outDir + getFlag() + ".temp");
  165.         }
  166.     }

  168.     public AbstractCommand(File sourceFile, long skipSize, long length, String outDir) {
  169.         if (length > Integer.MAX_VALUE) {
  170.             throw new RuntimeException("長度溢位");
  171.         }
  172.         this.skipSize = skipSize;
  173.         this.length = (int) length;
  174.         this.outDir = outDir;
  175.         this.sourceFile = sourceFile;
  176.         this.targetFile = new File(outDir + "result.temp");
  177.     }

  179.     public void setInvoker(Invoker invoker) {
  180.         this.invoker = invoker;
  181.     }

  183.     public void setReceiver(Receiver receiver) {
  184.         this.receiver = receiver;
  185.     }

  187.     public long getSkipSize() {
  188.         return skipSize;
  189.     }

  191.     public int getLength() {
  192.         return length;
  193.     }

  195.     public String getOutDir() {
  196.         return outDir;
  197.     }

  199.     public File getSourceFile() {
  200.         return sourceFile;
  201.     }

  203.     @Override
  204.     public String toString() {
  205.         return "Command [skipSize=" + skipSize + ", length=" + length + "]";
  206.     }

  208.     public void execute() throws IOException {
  209.         this.receiver.action(this);
  210.     }

  212.     public Invoker getInvoker() {
  213.         return invoker;
  214.     }

  216.     public File getOutFile() {
  217.         return this.outFile;
  218.     }

  220.     public File getTargetFile() {
  221.         return this.targetFile;
  222.     }

  224.     @Override
  225.     public int compareTo(Command o) {
  226.         if (this.skipSize < o.getSkipSize()) {
  227.             return -1;
  228.         }
  229.         return 1;
  230.     }
  231. }

  233. interface Receiver {
  234.     public void action(Command command) throws IOException;

  236.     public void shutdown();
  237. }

  239. class PrepareReceiver implements Receiver {
  240.     private ExecutorService prepareThreadPool;

  242.     public PrepareReceiver() {
  243.         prepareThreadPool = Executors.newFixedThreadPool(4);
  244.     }

  246.     @Override
  247.     public void action(final Command command) throws IOException {
  248.         prepareThreadPool.submit(new Runnable() {

  250.             @Override
  251.             public void run() {
  252.                 System.out.println("\t整理資料:" + command);
  253.                 long start = System.currentTimeMillis();

  255.                 byte[] data = ((PrepareCommand) command).getData();
  256.                 long[] result = new long[getObjectSize(data)];
  257.                 int resultIndex = 0;
  258.                 int index = 0;
  259.                 int first = 0;
  260.                 while (index < data.length) {
  261.                     if (data[index] == 10) {
  262.                         byte[] tmpData = Arrays.copyOfRange(data, first, index);
  263.                         String str = new String(tmpData);
  264.                         result[resultIndex] = Long.valueOf(str);
  265.                         resultIndex++;
  266.                         first = index + 1;
  267.                     }

  269.                     index++;
  270.                 }
  271.                 command.getInvoker().executeCommand(new SortCommand(command, result));

  273.                 result = null;
  274.                 data = null;
  275.                 command.clear();

  277.                 long end = System.currentTimeMillis();
  278.                 System.out.println("\t結束整理資料:" + command + ",用時:" + (end - start) / 1000);
  279.             }

  281.         });

  283.     }

  285.     private int getObjectSize(byte[] data) {
  286.         int size = 0;
  287.         for (byte b : data) {
  288.             if (b == 10) {
  289.                 size++;
  290.             }
  291.         }
  292.         return size;
  293.     }

  295.     @Override
  296.     public void shutdown() {
  297.         this.prepareThreadPool.shutdown();
  298.     }

  300. }

  302. class PrepareCommand extends AbstractCommand {

  304.     private byte[] data;

  306.     public PrepareCommand(Command command, byte[] data) {
  307.         super(command);
  308.         this.data = data;
  309.     }

  311.     public byte[] getData() {
  312.         return this.data;
  313.     }

  315.     @Override
  316.     public void clear() {
  317.         this.data = null;
  318.     }
  319. }

  321. class SpilterReceiver implements Receiver {
  322.     @Override
  323.     public void action(final Command command) throws IOException {
  324.         try {
  325.             System.out.println("開始讀入:" + command);
  326.             long start = System.currentTimeMillis();
  327.             FileChannel in = new RandomAccessFile(command.getSourceFile(), "r").getChannel();
  328.             MappedByteBuffer inBuffer = in.map(MapMode.READ_ONLY, command.getSkipSize(), command.getLength());
  329.             byte[] data = new byte[inBuffer.limit()];
  330.             inBuffer.get(data);
  331.             command.getInvoker().executeCommand(new PrepareCommand(command, data));

  333.             data = null;
  334.             in.close();
  335.             command.clear();

  337.             long end = System.currentTimeMillis();
  338.             System.out.println("結束記憶體讀入:" + (end - start) / 1000 + "s");

  340.         } catch (IOException e) {
  341.             e.printStackTrace();
  342.         }
  343.     }

  345.     @Override
  346.     public void shutdown() {
  347.     }
  348. }

  350. class MergeCommand extends AbstractCommand {
  351.     private List<Command> commandList = null;

  353.     public MergeCommand(List<Command> commandList) {
  354.         super(commandList.get(0));
  355.         this.commandList = commandList;
  356.     }

  358.     @Override
  359.     public void clear() {

  361.     }

  363.     public List<Command> getCommandList() {
  364.         return this.commandList;
  365.     }
  366. }

  368. class KillCommand extends AbstractCommand {

  370.     public KillCommand(Command command) {
  371.         super(command);
  372.     }

  374.     @Override
  375.     public void clear() {

  377.     }

  379. }

  381. class MergerReceiver implements Receiver {
  382.     private ExecutorService mergerThreadPool = Executors.newFixedThreadPool(4);
  383.     private List<Worker> workerList = new ArrayList<Worker>();
  384.     private ByteBuffer bb = null;

  386.     @Override
  387.     public void action(final Command command) throws IOException {
  388.         bb = ByteBuffer.allocate(50 * 1024 * 1024);
  389.         List<Command> list = ((MergeCommand) command).getCommandList();
  390.         for (Command historyCommand : list) {
  391.             Worker worker = new Worker(historyCommand.getOutFile(), workerList);
  392.             workerList.add(worker);
  393.         }

  395.         System.out.println("讀取佇列,寫入目標檔案");
  396.         long start = System.currentTimeMillis();
  397.         RandomAccessFile targetFile = new RandomAccessFile(command.getTargetFile(), "rw");
  398.         FileChannel channel = targetFile.getChannel();
  399.         MappedByteBuffer out = null;
  400.         long index = 0L;
  401.         while (workerList.size() != 0) {
  402.             Worker worker = Collections.min(workerList);
  403.             Long data = worker.poll();
  404.             if (data == null) {
  405.                 workerList.remove(worker);
  406.             } else {

  408.                 bb.put((data + "\n").getBytes());

  410.                 if (bb.position() > 49 * 1024 * 1024) {
  411.                     long end = System.currentTimeMillis();
  412.                     System.out.println("歸併用時:" + (end - start)/1000 + "s");
  413.                     start = System.currentTimeMillis();
  414.                     bb.flip();
  415.                     long temp = bb.limit();
  416.                     MutiWriter writer = new MutiWriter(channel, bb, index);
  417.                     mergerThreadPool.submit(writer);
  418.                     index = index + temp;
  419.                     bb = ByteBuffer.allocate(50 * 1024 * 1024);
  420.                 }

  422.             }
  423.         }

  425.         bb.flip();

  427.         out = channel.map(MapMode.READ_WRITE, index, bb.limit());
  428.         int i = bb.limit();
  429.         while (i > .0) {
  430.             out.put(bb.get());
  431.             i--;
  432.         }
  433.         out.force();
  434.         channel.close();
  435.         targetFile.close();

  437.         KillCommand killCommand = new KillCommand(command);
  438.         command.getInvoker().executeCommand(killCommand);

  440.         long end = System.currentTimeMillis();
  441.         System.out.println("外部排序總用時:" + (end - AbstractCommand.START) / 1000);

  443.     }

  445.     private class MutiWriter implements Runnable {
  446.         private ByteBuffer buffer;
  447.         private MappedByteBuffer out = null;

  449.         public MutiWriter(FileChannel channel, ByteBuffer buffer, long index) {
  450.             this.buffer = buffer;
  451.             try {
  452.                 out = channel.map(MapMode.READ_WRITE, index, buffer.limit());
  453.             } catch (IOException e) {
  454.                 e.printStackTrace();
  455.             }
  456.         }

  458.         @Override
  459.         public void run() {
  460.             long a1 = System.currentTimeMillis();
  461.             int i = buffer.limit();
  462.             while (i > 0) {
  463.                 out.put(buffer.get());
  464.                 i--;
  465.             }
  466.             out.force();
  467.             out = null;
  468.             long a2 = System.currentTimeMillis();
  469.             System.out.println("記憶體對映寫入:" + (a2 - a1) / 1000 + "s");
  470.         }

  472.     }

  474.     @Override
  475.     public void shutdown() {
  476.         mergerThreadPool.shutdown();
  477.     }

  479.     private class Worker implements Comparable<Worker> {
  480.         private long data;
  481.         private MappedByteBuffer buffer = null;
  482.         private List<Worker> workerList = null;
  483.         private boolean eof = false;

  485.         Worker(File file, List<Worker> workerList) {
  486.             try {
  487.                 RandomAccessFile rFile = new RandomAccessFile(file, "r");
  488.                 FileChannel channel = rFile.getChannel();
  489.                 buffer = channel.map(MapMode.READ_ONLY, 0, channel.size());

  491.                 this.workerList = workerList;
  492.                 data = buffer.getLong();

  494.                 rFile.close();
  495.                 channel.close();

  497.             } catch (IOException e) {
  498.                 // TODO Auto-generated catch block
  499.                 e.printStackTrace();
  500.             }
  501.         }

  503.         public long peek() {
  504.             return data;
  505.         }

  507.         public Long poll() {
  508.             long result = data;
  509.             if (buffer.position() != buffer.limit()) {
  510.                 data = buffer.getLong();
  511.             } else {
  512.                 if (eof == false) {
  513.                     eof = true;
  514.                 } else {
  515.                     return null;
  516.                 }
  517.             }

  519.             return result;
  520.         }

  522.         @Override
  523.         public int compareTo(Worker o) {
  524.             if (this.peek() > o.peek()) {
  525.                 return 1;
  526.             } else if (this.peek() < o.peek()) {
  527.                 return -1;
  528.             } else {
  529.                 return 0;
  530.             }
  531.         }
  532.     }

  534. }

  536. class WriterReceiver implements Receiver {
  537.     private ExecutorService writeThreadPool = null;;
  538.     private CyclicBarrier barrier = null;
  539.     private List<Command> commandList = new ArrayList<Command>();

  541.     public WriterReceiver(int pieces) {
  542.         writeThreadPool = Executors.newFixedThreadPool(pieces + 2);
  543.         barrier = new CyclicBarrier(pieces, new Runnable() {

  545.             @Override
  546.             public void run() {
  547.                 long end = System.currentTimeMillis();
  548.                 System.out.println("合併之前總用時:" + (end - AbstractCommand.START) / 1000);

  550.                 MergeCommand command = new MergeCommand(WriterReceiver.this.commandList);
  551.                 command.getInvoker().executeCommand(command);

  553.             }
  554.         });

  556.     }

  558.     @Override
  559.     public void action(final Command command) {
  560.         writeThreadPool.submit(new Runnable() {

  562.             @Override
  563.             public void run() {
  564.                 long[] data = ((WriterCommand) command).getData();

  566.                 try {
  567.                     System.out.println("\t\t\t開始寫入:" + command);
  568.                     long start = System.currentTimeMillis();
  569.                     File outfile = command.getOutFile();
  570.                     FileChannel channel = new RandomAccessFile(outfile, "rw").getChannel();
  571.                     MappedByteBuffer buffer = channel.map(MapMode.READ_WRITE, 0, data.length * 8);
  572.                     for (int i = 0; i < data.length; i++) {
  573.                         buffer.putLong(data[i]);
  574.                     }
  575.                     buffer.force();
  576.                     WriterReceiver.this.commandList.add(command);
  577.                     long end = System.currentTimeMillis();
  578.                     System.out.println("\t\t\t結束寫入:" + command + ",用時:" + (end - start) / 1000);

  580.                     command.clear();
  581.                     data = null;
  582.                     channel.close();
  583.                     buffer = null;

  585.                     barrier.await();
  586.                 } catch (IOException ex) {
  587.                     ex.printStackTrace();
  588.                 } catch (InterruptedException e) {
  589.                     e.printStackTrace();
  590.                 } catch (BrokenBarrierException e) {
  591.                     e.printStackTrace();
  592.                 }
  593.             }
  594.         });
  595.     }

  597.     @Override
  598.     public void shutdown() {
  599.         writeThreadPool.shutdown();
  600.     }
  601. }

  603. class SortCommand extends AbstractCommand {
  604.     private long[] data = null;

  606.     public SortCommand(Command command, long[] data) {
  607.         super(command);
  608.         this.data = data;
  609.     }

  611.     public long[] getData() {
  612.         return this.data;
  613.     }

  615.     @Override
  616.     public void clear() {
  617.         this.data = null;
  618.     }

  620. }

  622. class SorterReceiver implements Receiver {
  623.     private ExecutorService sortThreadPool = Executors.newFixedThreadPool(4);

  625.     public void action(final Command command) {
  626.         sortThreadPool.submit(new Runnable() {

  628.             @Override
  629.             public void run() {
  630.                 System.out.println("\t\t開始排序:" + command);
  631.                 long start = System.currentTimeMillis();

  633.                 long[] data = ((SortCommand) command).getData();
  634.                 Arrays.sort(data);

  636.                 command.getInvoker().executeCommand(new WriterCommand(command, data));

  638.                 data = null;
  639.                 command.clear();

  641.                 long end = System.currentTimeMillis();
  642.                 System.out.println("\t\t結束排序:" + command + ",用時:" + (end - start) / 1000);

  644.             }

  646.         });

  648.     }

  650.     @Override
  651.     public void shutdown() {
  652.         sortThreadPool.shutdown();
  653.     }
  654. }

  656. class WriterCommand extends AbstractCommand {
  657.     private long[] data;

  659.     public WriterCommand(Command command, long[] data) {
  660.         super(command);
  661.         this.data = data;
  662.     }

  664.     public long[] getData() {
  665.         return this.data;
  666.     }

  668.     @Override
  669.     public void clear() {
  670.         this.data = null;
  671.     }

  673. }
最近真的很忙,我這一個單執行緒的CPU居然都跑出了雙核的感覺。
我感覺優化就是找到慢的地方，看看是不是有無效的操作,如果都是有效運算,就只能想辦法使用多執行緒，使壓力分散在多個核心上。
之前吳老師說他用的是直接緩衝區+通道 IO的方式。自己寫程式將long直接轉化為byte。
這個優化的方式我試了一下，不能應用在我的場景。
他的本意是減少GC，但是在我的場景，這種方式更加劇了CPU的爭用。
優化的本質應該是平衡資源

程式日誌,相比最初的版本，效能提升了50s
因為將切分檔案，byte陣列轉為long陣列拆分為了兩個操作，並且使用了多執行緒處理。
  1. 開始讀入:Command [skipSize=0, length=135863265]
  2. 結束記憶體讀入:3s
  3. 整理資料:Command [skipSize=0, length=135863265]
  4. 開始讀入:Command [skipSize=135863265, length=135863254]
  5. 結束整理資料:Command [skipSize=0, length=135863265],用時:4
  6. 開始排序:Command [skipSize=0, length=135863265]
  7. 結束記憶體讀入:4s
  8. 開始讀入:Command [skipSize=271726519, length=135863267]
  9. 整理資料:Command [skipSize=135863265, length=135863254]
  10. 結束排序:Command [skipSize=0, length=135863265],用時:2
  11. 開始寫入:Command [skipSize=0, length=135863265]
  12. 結束記憶體讀入:3s
  13. 開始讀入:Command [skipSize=407589786, length=135863268]
  14. 整理資料:Command [skipSize=271726519, length=135863267]
  15. 結束寫入:Command [skipSize=0, length=135863265],用時:2
  16. 結束整理資料:Command [skipSize=135863265, length=135863254],用時:5
  17. 開始排序:Command [skipSize=135863265, length=135863254]
  18. 結束排序:Command [skipSize=135863265, length=135863254],用時:1
  19. 開始寫入:Command [skipSize=135863265, length=135863254]
  20. 結束整理資料:Command [skipSize=271726519, length=135863267],用時:4
  21. 開始排序:Command [skipSize=271726519, length=135863267]
  22. 結束排序:Command [skipSize=271726519, length=135863267],用時:1
  23. 開始寫入:Command [skipSize=271726519, length=135863267]
  24. 結束寫入:Command [skipSize=135863265, length=135863254],用時:6
  25. 結束寫入:Command [skipSize=271726519, length=135863267],用時:4
  26. 結束記憶體讀入:11s
  27. 整理資料:Command [skipSize=407589786, length=135863268]
  28. 開始讀入:Command [skipSize=543453054, length=135863264]
  29. 結束整理資料:Command [skipSize=407589786, length=135863268],用時:3
  30. 開始排序:Command [skipSize=407589786, length=135863268]
  31. 結束記憶體讀入:4s
  32. 開始讀入:Command [skipSize=679316318, length=135863268]
  33. 整理資料:Command [skipSize=543453054, length=135863264]
  34. 結束排序:Command [skipSize=407589786, length=135863268],用時:1
  35. 開始寫入:Command [skipSize=407589786, length=135863268]
  36. 結束寫入:Command [skipSize=407589786, length=135863268],用時:1
  37. 結束整理資料:Command [skipSize=543453054, length=135863264],用時:3
  38. 開始排序:Command [skipSize=543453054, length=135863264]
  39. 結束排序:Command [skipSize=543453054, length=135863264],用時:1
  40. 開始寫入:Command [skipSize=543453054, length=135863264]
  41. 結束寫入:Command [skipSize=543453054, length=135863264],用時:1
  42. 結束記憶體讀入:7s
  43. 整理資料:Command [skipSize=679316318, length=135863268]
  44. 開始讀入:Command [skipSize=815179586, length=135863264]
  45. 結束記憶體讀入:3s
  46. 開始讀入:Command [skipSize=951042850, length=135863270]
  47. 整理資料:Command [skipSize=815179586, length=135863264]
  48. 開始排序:Command [skipSize=679316318, length=135863268]
  49. 結束整理資料:Command [skipSize=679316318, length=135863268],用時:3
  50. 結束排序:Command [skipSize=679316318, length=135863268],用時:2
  51. 開始寫入:Command [skipSize=679316318, length=135863268]
  52. 結束整理資料:Command [skipSize=815179586, length=135863264],用時:4
  53. 開始排序:Command [skipSize=815179586, length=135863264]
  54. 結束寫入:Command [skipSize=679316318, length=135863268],用時:2
  55. 結束記憶體讀入:5s
  56. 整理資料:Command [skipSize=951042850, length=135863270]
  57. 開始讀入:Command [skipSize=1086906120, length=135863262]
  58. 結束排序:Command [skipSize=815179586, length=135863264],用時:1
  59. 開始寫入:Command [skipSize=815179586, length=135863264]
  60. 結束寫入:Command [skipSize=815179586, length=135863264],用時:1
  61. 整理資料:Command [skipSize=1086906120, length=135863262]
  62. 結束記憶體讀入:3s
  63. 開始讀入:Command [skipSize=1222769382, length=135863265]
  64. 開始排序:Command [skipSize=951042850, length=135863270]
  65. 結束整理資料:Command [skipSize=951042850, length=135863270],用時:4
  66. 結束記憶體讀入:3s
  67. 開始讀入:Command [skipSize=1358632647, length=135863256]
  68. 整理資料:Command [skipSize=1222769382, length=135863265]
  69. 結束排序:Command [skipSize=951042850, length=135863270],用時:2
  70. 開始寫入:Command [skipSize=951042850, length=135863270]
  71. 結束寫入:Command [skipSize=951042850, length=135863270],用時:1
  72. 結束整理資料:Command [skipSize=1086906120, length=135863262],用時:5
  73. 開始排序:Command [skipSize=1086906120, length=135863262]
  74. 結束排序:Command [skipSize=1086906120, length=135863262],用時:1
  75. 開始寫入:Command [skipSize=1086906120, length=135863262]
  76. 結束記憶體讀入:4s
  77. 整理資料:Command [skipSize=1358632647, length=135863256]
  78. 開始讀入:Command [skipSize=1494495903, length=135863268]
  79. 結束寫入:Command [skipSize=1086906120, length=135863262],用時:1
  80. 開始排序:Command [skipSize=1222769382, length=135863265]
  81. 結束整理資料:Command [skipSize=1222769382, length=135863265],用時:6
  82. 結束排序:Command [skipSize=1222769382, length=135863265],用時:1
  83. 開始寫入:Command [skipSize=1222769382, length=135863265]
  84. 整理資料:Command [skipSize=1494495903, length=135863268]
  85. 結束記憶體讀入:4s
  86. 開始讀入:Command [skipSize=1630359171, length=135863255]
  87. 開始排序:Command [skipSize=1358632647, length=135863256]
  88. 結束整理資料:Command [skipSize=1358632647, length=135863256],用時:5
  89. 結束寫入:Command [skipSize=1222769382, length=135863265],用時:1
  90. 結束排序:Command [skipSize=1358632647, length=135863256],用時:1
  91. 開始寫入:Command [skipSize=1358632647, length=135863256]
  92. 結束整理資料:Command [skipSize=1494495903, length=135863268],用時:3
  93. 結束寫入:Command [skipSize=1358632647, length=135863256],用時:2
  94. 開始排序:Command [skipSize=1494495903, length=135863268]
  95. 結束記憶體讀入:4s
  96. 整理資料:Command [skipSize=1630359171, length=135863255]
  97. 開始讀入:Command [skipSize=1766222426, length=135863271]
  98. 結束排序:Command [skipSize=1494495903, length=135863268],用時:1
  99. 開始寫入:Command [skipSize=1494495903, length=135863268]
  100. 結束寫入:Command [skipSize=1494495903, length=135863268],用時:1
  101. 結束整理資料:Command [skipSize=1630359171, length=135863255],用時:3
  102. 開始排序:Command [skipSize=1630359171, length=135863255]
  103. 整理資料:Command [skipSize=1766222426, length=135863271]
  104. 結束記憶體讀入:4s
  105. 開始讀入:Command [skipSize=1902085697, length=135863258]
  106. 結束排序:Command [skipSize=1630359171, length=135863255],用時:2
  107. 開始寫入:Command [skipSize=1630359171, length=135863255]
  108. 結束寫入:Command [skipSize=1630359171, length=135863255],用時:1
  109. 結束整理資料:Command [skipSize=1766222426, length=135863271],用時:3
  110. 開始排序:Command [skipSize=1766222426, length=135863271]
  111. 整理資料:Command [skipSize=1902085697, length=135863258]
  112. 結束記憶體讀入:4s
  113. 開始讀入:Command [skipSize=2037948955, length=135863254]
  114. 結束排序:Command [skipSize=1766222426, length=135863271],用時:1
  115. 開始寫入:Command [skipSize=1766222426, length=135863271]
  116. 結束寫入:Command [skipSize=1766222426, length=135863271],用時:1
  117. 結束整理資料:Command [skipSize=1902085697, length=135863258],用時:3
  118. 開始排序:Command [skipSize=1902085697, length=135863258]
  119. 整理資料:Command [skipSize=2037948955, length=135863254]
  120. 結束記憶體讀入:4s
  121. 開始讀入:Command [skipSize=2173812209, length=135863259]
  122. 結束排序:Command [skipSize=1902085697, length=135863258],用時:2
  123. 開始寫入:Command [skipSize=1902085697, length=135863258]
  124. 結束寫入:Command [skipSize=1902085697, length=135863258],用時:1
  125. 結束整理資料:Command [skipSize=2037948955, length=135863254],用時:3
  126. 開始排序:Command [skipSize=2037948955, length=135863254]
  127. 結束記憶體讀入:4s
  128. 整理資料:Command [skipSize=2173812209, length=135863259]
  129. 開始讀入:Command [skipSize=2309675468, length=135863260]
  130. 結束排序:Command [skipSize=2037948955, length=135863254],用時:2
  131. 開始寫入:Command [skipSize=2037948955, length=135863254]
  132. 結束整理資料:Command [skipSize=2173812209, length=135863259],用時:3
  133. 開始排序:Command [skipSize=2173812209, length=135863259]
  134. 結束記憶體讀入:4s
  135. 整理資料:Command [skipSize=2309675468, length=135863260]
  136. 開始讀入:Command [skipSize=2445538728, length=135863266]
  137. 結束寫入:Command [skipSize=2037948955, length=135863254],用時:2
  138. 結束排序:Command [skipSize=2173812209, length=135863259],用時:2
  139. 開始寫入:Command [skipSize=2173812209, length=135863259]
  140. 結束寫入:Command [skipSize=2173812209, length=135863259],用時:1
  141. 結束整理資料:Command [skipSize=2309675468, length=135863260],用時:3
  142. 開始排序:Command [skipSize=2309675468, length=135863260]
  143. 整理資料:Command [skipSize=2445538728, length=135863266]
  144. 結束記憶體讀入:4s
  145. 開始讀入:Command [skipSize=2581401994, length=135863268]
  146. 結束排序:Command [skipSize=2309675468, length=135863260],用時:2
  147. 開始寫入:Command [skipSize=2309675468, length=135863260]
  148. 結束寫入:Command [skipSize=2309675468, length=135863260],用時:1
  149. 結束整理資料:Command [skipSize=2445538728, length=135863266],用時:4
  150. 開始排序:Command [skipSize=2445538728, length=135863266]
  151. 結束記憶體讀入:4s
  152. 整理資料:Command [skipSize=2581401994, length=135863268]
  153. 開始讀入:Command [skipSize=2717265262, length=135863264]
  154. 結束排序:Command [skipSize=2445538728, length=135863266],用時:2
  155. 開始寫入:Command [skipSize=2445538728, length=135863266]
  156. 結束寫入:Command [skipSize=2445538728, length=135863266],用時:1
  157. 結束整理資料:Command [skipSize=2581401994, length=135863268],用時:3
  158. 開始排序:Command [skipSize=2581401994, length=135863268]
  159. 整理資料:Command [skipSize=2717265262, length=135863264]
  160. 結束記憶體讀入:4s
  161. 開始讀入:Command [skipSize=2853128526, length=135863258]
  162. 結束排序:Command [skipSize=2581401994, length=135863268],用時:2
  163. 開始寫入:Command [skipSize=2581401994, length=135863268]
  164. 結束寫入:Command [skipSize=2581401994, length=135863268],用時:1
  165. 結束整理資料:Command [skipSize=2717265262, length=135863264],用時:3
  166. 開始排序:Command [skipSize=2717265262, length=135863264]
  167. 整理資料:Command [skipSize=2853128526, length=135863258]
  168. 結束記憶體讀入:4s
  169. 開始讀入:Command [skipSize=2988991784, length=135863254]
  170. 結束排序:Command [skipSize=2717265262, length=135863264],用時:2
  171. 開始寫入:Command [skipSize=2717265262, length=135863264]
  172. 結束寫入:Command [skipSize=2717265262, length=135863264],用時:1
  173. 結束整理資料:Command [skipSize=2853128526, length=135863258],用時:3
  174. 開始排序:Command [skipSize=2853128526, length=135863258]
  175. 結束記憶體讀入:4s
  176. 整理資料:Command [skipSize=2988991784, length=135863254]
  177. 開始讀入:Command [skipSize=3124855038, length=135863260]
  178. 結束排序:Command [skipSize=2853128526, length=135863258],用時:2
  179. 開始寫入:Command [skipSize=2853128526, length=135863258]
  180. 結束寫入:Command [skipSize=2853128526, length=135863258],用時:1
  181. 結束整理資料:Command [skipSize=2988991784, length=135863254],用時:3
  182. 開始排序:Command [skipSize=2988991784, length=135863254]
  183. 整理資料:Command [skipSize=3124855038, length=135863260]
  184. 結束記憶體讀入:4s
  185. 開始讀入:Command [skipSize=3260718298, length=135863254]
  186. 結束排序:Command [skipSize=2988991784, length=135863254],用時:2
  187. 開始寫入:Command [skipSize=2988991784, length=135863254]
  188. 結束寫入:Command [skipSize=2988991784, length=135863254],用時:2
  189. 結束整理資料:Command [skipSize=3124855038, length=135863260],用時:4
  190. 開始排序:Command [skipSize=3124855038, length=135863260]
  191. 整理資料:Command [skipSize=3260718298, length=135863254]
  192. 結束記憶體讀入:4s
  193. 開始讀入:Command [skipSize=3396581552, length=135863262]
  194. 結束排序:Command [skipSize=3124855038, length=135863260],用時:1
  195. 開始寫入:Command [skipSize=3124855038, length=135863260]
  196. 結束寫入:Command [skipSize=3124855038, length=135863260],用時:2
  197. 結束整理資料:Command [skipSize=3260718298, length=135863254],用時:3
  198. 開始排序:Command [skipSize=3260718298, length=135863254]
  199. 結束記憶體讀入:4s
  200. 整理資料:Command [skipSize=3396581552, length=135863262]
  201. 開始讀入:Command [skipSize=3532444814, length=135863265]
  202. 結束排序:Command [skipSize=3260718298, length=135863254],用時:1
  203. 開始寫入:Command [skipSize=3260718298, length=135863254]
  204. 結束寫入:Command [skipSize=3260718298, length=135863254],用時:2
  205. 結束整理資料:Command [skipSize=3396581552, length=135863262],用時:3
  206. 開始排序:Command [skipSize=3396581552, length=135863262]
  207. 結束記憶體讀入:4s
  208. 整理資料:Command [skipSize=3532444814, length=135863265]
  209. 開始讀入:Command [skipSize=3668308079, length=135863273]
  210. 結束排序:Command [skipSize=3396581552, length=135863262],用時:2
  211. 開始寫入:Command [skipSize=3396581552, length=135863262]
  212. 結束寫入:Command [skipSize=3396581552, length=135863262],用時:2
  213. 結束整理資料:Command [skipSize=3532444814, length=135863265],用時:3
  214. 開始排序:Command [skipSize=3532444814, length=135863265]
  215. 結束記憶體讀入:4s
  216. 整理資料:Command [skipSize=3668308079, length=135863273]
  217. 開始讀入:Command [skipSize=3804171352, length=135863261]
  218. 結束排序:Command [skipSize=3532444814, length=135863265],用時:2
  219. 開始寫入:Command [skipSize=3532444814, length=135863265]
  220. 結束寫入:Command [skipSize=3532444814, length=135863265],用時:1
  221. 結束整理資料:Command [skipSize=3668308079, length=135863273],用時:3
  222. 開始排序:Command [skipSize=3668308079, length=135863273]
  223. 整理資料:Command [skipSize=3804171352, length=135863261]
  224. 結束記憶體讀入:4s
  225. 開始讀入:Command [skipSize=3940034613, length=135862991]
  226. 結束排序:Command [skipSize=3668308079, length=135863273],用時:2
  227. 開始寫入:Command [skipSize=3668308079, length=135863273]
  228. 結束寫入:Command [skipSize=3668308079, length=135863273],用時:2
  229. 結束整理資料:Command [skipSize=3804171352, length=135863261],用時:3
  230. 開始排序:Command [skipSize=3804171352, length=135863261]
  231. 結束記憶體讀入:4s
  232. 整理資料:Command [skipSize=3940034613, length=135862991]
  233. 結束排序:Command [skipSize=3804171352, length=135863261],用時:1
  234. 開始寫入:Command [skipSize=3804171352, length=135863261]
  235. 結束寫入:Command [skipSize=3804171352, length=135863261],用時:1
  236. 結束整理資料:Command [skipSize=3940034613, length=135862991],用時:3
  237. 開始排序:Command [skipSize=3940034613, length=135862991]
  238. 結束排序:Command [skipSize=3940034613, length=135862991],用時:1
  239. 開始寫入:Command [skipSize=3940034613, length=135862991]
  240. 結束寫入:Command [skipSize=3940034613, length=135862991],用時:1
  241. 合併之前總用時:143
  242. 讀取佇列,寫入目標檔案
  243. 歸併用時:2s
  244. 記憶體對映寫入:1s
  245. 歸併用時:2s
  246. 記憶體對映寫入:1s
  247. 歸併用時:2s
  248. 記憶體對映寫入:1s
  249. 歸併用時:2s
  250. 記憶體對映寫入:1s
  251. 歸併用時:2s
  252. 記憶體對映寫入:1s
  253. 歸併用時:2s
  254. 記憶體對映寫入:1s
  255. 歸併用時:2s
  256. 記憶體對映寫入:1s
  257. 歸併用時:2s
  258. 記憶體對映寫入:2s
  259. 歸併用時:2s
  260. 記憶體對映寫入:1s
  261. 歸併用時:2s
  262. 記憶體對映寫入:1s
  263. 歸併用時:2s
  264. 記憶體對映寫入:1s
  265. 歸併用時:2s
  266. 記憶體對映寫入:1s
  267. 歸併用時:2s
  268. 記憶體對映寫入:1s
  269. 歸併用時:2s
  270. 記憶體對映寫入:1s
  271. 歸併用時:2s
  272. 歸併用時:1s
  273. 記憶體對映寫入:3s
  274. 歸併用時:2s
  275. 歸併用時:2s
  276. 記憶體對映寫入:5s
  277. 記憶體對映寫入:3s
  278. 歸併用時:2s
  279. 記憶體對映寫入:2s
  280. 記憶體對映寫入:2s
  281. 歸併用時:2s
  282. 記憶體對映寫入:2s
  283. 歸併用時:2s
  284. 歸併用時:2s
  285. 記憶體對映寫入:2s
  286. 記憶體對映寫入:1s
  287. 歸併用時:2s
  288. 歸併用時:2s
  289. 記憶體對映寫入:2s
  290. 記憶體對映寫入:2s
  291. 歸併用時:2s
  292. 歸併用時:2s
  293. 記憶體對映寫入:2s
  294. 記憶體對映寫入:2s
  295. 歸併用時:2s
  296. 記憶體對映寫入:2s
  297. 歸併用時:2s
  298. 記憶體對映寫入:2s
  299. 歸併用時:2s
  300. 記憶體對映寫入:2s
  301. 歸併用時:2s
  302. 記憶體對映寫入:2s
  303. 歸併用時:2s
  304. 歸併用時:2s
  305. 記憶體對映寫入:2s
  306. 記憶體對映寫入:2s
  307. 歸併用時:2s
  308. 歸併用時:2s
  309. 記憶體對映寫入:2s
  310. 歸併用時:2s
  311. 記憶體對映寫入:2s
  312. 記憶體對映寫入:2s
  313. 歸併用時:2s
  314. 記憶體對映寫入:2s
  315. 歸併用時:2s
  316. 記憶體對映寫入:2s
  317. 歸併用時:2s
  318. 記憶體對映寫入:2s
  319. 歸併用時:2s
  320. 記憶體對映寫入:2s
  321. 歸併用時:2s
  322. 記憶體對映寫入:2s
  323. 歸併用時:2s
  324. 記憶體對映寫入:2s
  325. 歸併用時:2s
  326. 記憶體對映寫入:2s
  327. 歸併用時:2s
  328. 記憶體對映寫入:2s
  329. 歸併用時:2s
  330. 記憶體對映寫入:2s
  331. 歸併用時:2s
  332. 記憶體對映寫入:2s
  333. 歸併用時:2s
  334. 記憶體對映寫入:2s
  335. 歸併用時:2s
  336. 記憶體對映寫入:2s
  337. 歸併用時:2s
  338. 記憶體對映寫入:2s
  339. 歸併用時:2s
  340. 歸併用時:2s
  341. 記憶體對映寫入:4s
  342. 記憶體對映寫入:2s
  343. 歸併用時:3s
  344. 記憶體對映寫入:2s
  345. 歸併用時:2s
  346. 歸併用時:2s
  347. 記憶體對映寫入:2s
  348. 記憶體對映寫入:2s
  349. 歸併用時:2s
  350. 記憶體對映寫入:2s
  351. 歸併用時:2s
  352. 記憶體對映寫入:2s
  353. 歸併用時:2s
  354. 記憶體對映寫入:2s
  355. 歸併用時:2s
  356. 記憶體對映寫入:2s
  357. 歸併用時:2s
  358. 記憶體對映寫入:2s
  359. 歸併用時:2s
  360. 記憶體對映寫入:2s
  361. 歸併用時:2s
  362. 記憶體對映寫入:2s
  363. 歸併用時:2s
  364. 歸併用時:1s
  365. 歸併用時:2s
  366. 歸併用時:4s
  367. 記憶體對映寫入:6s
  368. 記憶體對映寫入:8s
  369. 記憶體對映寫入:4s
  370. 歸併用時:2s
  371. 歸併用時:2s
  372. 記憶體對映寫入:3s
  373. 記憶體對映寫入:5s
  374. 記憶體對映寫入:2s
  375. 歸併用時:3s
  376. 歸併用時:2s
  377. 記憶體對映寫入:2s
  378. 記憶體對映寫入:2s
  379. 歸併用時:2s
  380. 記憶體對映寫入:1s
  381. 歸併用時:3s
  382. 歸併用時:2s
  383. 記憶體對映寫入:2s
  384. 歸併用時:2s
  385. 記憶體對映寫入:2s
  386. 記憶體對映寫入:2s
  387. 歸併用時:2s
  388. 歸併用時:2s
  389. 記憶體對映寫入:2s
  390. 記憶體對映寫入:2s
  391. 歸併用時:2s
  392. 記憶體對映寫入:1s
  393. 歸併用時:2s
  394. 記憶體對映寫入:2s
  395. 歸併用時:2s
  396. 記憶體對映寫入:2s
  397. 歸併用時:2s
  398. 記憶體對映寫入:2s
  399. 歸併用時:2s
  400. 記憶體對映寫入:1s
  401. 外部排序總用時:351

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29254281/viewspace-1172752/，如需轉載，請註明出處，否則將追究法律責任。

相關文章