RocketMQ中Broker的刷盤原始碼分析

上一篇部落格的最後簡單提了下CommitLog的刷盤  【RocketMQ中Broker的訊息儲存原始碼分析】 （這篇部落格和上一篇有很大的聯絡）

Broker的CommitLog刷盤會啟動一個執行緒，不停地將緩衝區的內容寫入磁碟（CommitLog檔案）中，主要分為非同步刷盤和同步刷盤

非同步刷盤又可以分為兩種方式：
①快取到mappedByteBuffer -> 寫入磁碟（包括同步刷盤）
②快取到writeBuffer -> 快取到fileChannel -> 寫入磁碟 （前面說過的開啟記憶體位元組緩衝區情況下）

 

CommitLog的兩種刷盤模式：

public enum FlushDiskType {
    SYNC_FLUSH,
    ASYNC_FLUSH
}

同步和非同步，同步刷盤由GroupCommitService實現，非同步刷盤由FlushRealTimeService實現，預設採用非同步刷盤

在採用非同步刷盤的模式下，若是開啟記憶體位元組緩衝區，那麼會在FlushRealTimeService的基礎上開啟CommitRealTimeService

 

同步刷盤：

啟動GroupCommitService執行緒：

public void run() {
    CommitLog.log.info(this.getServiceName() + " service started");

    while (!this.isStopped()) {
        try {
            this.waitForRunning(10);
            this.doCommit();
        } catch (Exception e) {
            CommitLog.log.warn(this.getServiceName() + " service has exception. ", e);
        }
    }

    // Under normal circumstances shutdown, wait for the arrival of the
    // request, and then flush
    try {
        Thread.sleep(10);
    } catch (InterruptedException e) {
        CommitLog.log.warn("GroupCommitService Exception, ", e);
    }

    synchronized (this) {
        this.swapRequests();
    }

    this.doCommit();

    CommitLog.log.info(this.getServiceName() + " service end");
}

通過迴圈中的doCommit不斷地進行刷盤

doCommit方法：

private void doCommit() {
    synchronized (this.requestsRead) {
        if (!this.requestsRead.isEmpty()) {
            for (GroupCommitRequest req : this.requestsRead) {
                // There may be a message in the next file, so a maximum of
                // two times the flush
                boolean flushOK = false;
                for (int i = 0; i < 2 && !flushOK; i++) {
                    flushOK = CommitLog.this.mappedFileQueue.getFlushedWhere() >= req.getNextOffset();

                    if (!flushOK) {
                        CommitLog.this.mappedFileQueue.flush(0);
                    }
                }

                req.wakeupCustomer(flushOK);
            }

            long storeTimestamp = CommitLog.this.mappedFileQueue.getStoreTimestamp();
            if (storeTimestamp > 0) {
                CommitLog.this.defaultMessageStore.getStoreCheckpoint().setPhysicMsgTimestamp(storeTimestamp);
            }

            this.requestsRead.clear();
        } else {
            // Because of individual messages is set to not sync flush, it
            // will come to this process
            CommitLog.this.mappedFileQueue.flush(0);
        }
    }
}

其中在GroupCommitService中管理著兩張List：

private volatile List<GroupCommitRequest> requestsWrite = new ArrayList<GroupCommitRequest>();
private volatile List<GroupCommitRequest> requestsRead = new ArrayList<GroupCommitRequest>();

GroupCommitRequest中封裝了一個Offset

private final long nextOffset;

 

這裡就需要看到上一篇部落格結尾提到的handleDiskFlush方法：

public void handleDiskFlush(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt) {
    // Synchronization flush
    if (FlushDiskType.SYNC_FLUSH == this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
        final GroupCommitService service = (GroupCommitService) this.flushCommitLogService;
        if (messageExt.isWaitStoreMsgOK()) {
            GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes());
            service.putRequest(request);
            boolean flushOK = request.waitForFlush(this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());
            if (!flushOK) {
                log.error("do groupcommit, wait for flush failed, topic: " + messageExt.getTopic() + " tags: " + messageExt.getTags()
                    + " client address: " + messageExt.getBornHostString());
                putMessageResult.setPutMessageStatus(PutMessageStatus.FLUSH_DISK_TIMEOUT);
            }
        } else {
            service.wakeup();
        }
    }
    // Asynchronous flush
    else {
        if (!this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
            flushCommitLogService.wakeup();
        } else {
            commitLogService.wakeup();
        }
    }
}

這個方法的呼叫發生在Broker接收到來自Producer的訊息，並且完成了向ByteBuffer的寫入

可以看到，在同步刷盤SYNC_FLUSH模式下，會從AppendMessageResult 中取出WroteOffset以及WroteBytes從而計算出nextOffset，把這個nextOffset封裝到GroupCommitRequest中，然後通過GroupCommitService 的putRequest方法，將GroupCommitRequest新增到requestsWrite這個List中
putRequest方法：

public synchronized void putRequest(final GroupCommitRequest request) {
    synchronized (this.requestsWrite) {
        this.requestsWrite.add(request);
    }
    if (hasNotified.compareAndSet(false, true)) {
        waitPoint.countDown(); // notify
    }
}

在完成List的add操作後，會通過CAS操作修改hasNotified這個原子化的Boolean值，同時通過waitPoint的countDown進行喚醒操作，在後面會有用

由於這裡這裡是同步刷盤，所以需要通過GroupCommitRequest的waitForFlush方法，在超時時間內等待該記錄對應的刷盤完成
而非同步刷盤會通過wakeup方法喚醒刷盤任務，並沒有進行等待，這就是二者區別

回到doCommit方法中，這時會發現這裡是對requestsRead這條List進行的操作，而剛才是將記錄存放在requestsWrite這條List中的
這就和在run方法中的waitForRunning方法有關了：

protected void waitForRunning(long interval) {
   if (hasNotified.compareAndSet(true, false)) {
        this.onWaitEnd();
        return;
    }

    //entry to wait
    waitPoint.reset();

    try {
        waitPoint.await(interval, TimeUnit.MILLISECONDS);
    } catch (InterruptedException e) {
        log.error("Interrupted", e);
    } finally {
        hasNotified.set(false);
        this.onWaitEnd();
    }
}

這裡通過CAS操作修改hasNotified值，從而呼叫onWaitEnd方法；如果修改失敗，則因為await進入阻塞，等待上面所說的putRequest方法將其喚醒，也就是說當Producer傳送的訊息被快取成功後，呼叫handleDiskFlush方法後，喚醒刷盤線工作，當然刷盤執行緒在達到超時時間interval後也會喚醒

再來看看onWaitEnd方法：

protected void onWaitEnd() {
    this.swapRequests();
}

private void swapRequests() {
    List<GroupCommitRequest> tmp = this.requestsWrite;
    this.requestsWrite = this.requestsRead;
    this.requestsRead = tmp;
}

可以看到，這裡是將兩個List進行了交換

這是一個非常有趣的做法，如果熟悉JVM的話，有沒有覺得這其實很像新生代的標記-清除演算法！
當刷盤執行緒阻塞的時候，requestsWrite中會填充記錄，當刷盤執行緒被喚醒工作的時候，首先會將requestsWrite和requestsRead進行交換，那麼此時的記錄就是從requestsRead中讀取的了，而同時requestsWrite會變為空的List，訊息記錄就會往這個空的List中填充，如此往復

可以看到doCommit方法中，當requestsRead不為空的時候，在最後會呼叫requestsRead的clear方法，由此證明了我上面的說法

 

仔細來看看是如何進行刷盤的：

for (GroupCommitRequest req : this.requestsRead) {
   // There may be a message in the next file, so a maximum of
    // two times the flush
    boolean flushOK = false;
    for (int i = 0; i < 2 && !flushOK; i++) {
        flushOK = CommitLog.this.mappedFileQueue.getFlushedWhere() >= req.getNextOffset();

        if (!flushOK) {
            CommitLog.this.mappedFileQueue.flush(0);
        }
    }

    req.wakeupCustomer(flushOK);
}

通過遍歷requestsRead，可以到得到GroupCommitRequest封裝的NextOffset

其中flushedWhere是用來記錄上一次刷盤完成後的offset，若是上一次的刷盤位置大於等於NextOffset，就說明從NextOffset位置起始已經被重新整理過了，不需要重新整理，否則呼叫mappedFileQueue的flush方法進行刷盤

MappedFileQueue的flush方法：

public boolean flush(final int flushLeastPages) {
    boolean result = true;
    MappedFile mappedFile = this.findMappedFileByOffset(this.flushedWhere, this.flushedWhere == 0);
    if (mappedFile != null) {
        long tmpTimeStamp = mappedFile.getStoreTimestamp();
        int offset = mappedFile.flush(flushLeastPages);
        long where = mappedFile.getFileFromOffset() + offset;
        result = where == this.flushedWhere;
        this.flushedWhere = where;
        if (0 == flushLeastPages) {
            this.storeTimestamp = tmpTimeStamp;
        }
    }

    return result;
}

這裡首先根據flushedWhere上一次刷盤完成後的offset，通過findMappedFileByOffset方法，找到CommitLog檔案的對映MappedFile
有關MappedFile及其相關操作在我之前的部落格中介紹過很多次，就不再累贅

再找到MappedFile後，呼叫其flush方法：

MappedFile的flush方法：

public int flush(final int flushLeastPages) {
    if (this.isAbleToFlush(flushLeastPages)) {
        if (this.hold()) {
            int value = getReadPosition();

            try {
                //We only append data to fileChannel or mappedByteBuffer, never both.
                if (writeBuffer != null || this.fileChannel.position() != 0) {
                    this.fileChannel.force(false);
                } else {
                    this.mappedByteBuffer.force();
                }
            } catch (Throwable e) {
                log.error("Error occurred when force data to disk.", e);
            }

            this.flushedPosition.set(value);
            this.release();
        } else {
            log.warn("in flush, hold failed, flush offset = " + this.flushedPosition.get());
            this.flushedPosition.set(getReadPosition());
        }
    }
    return this.getFlushedPosition();
}

首先isAbleToFlush方法：

private boolean isAbleToFlush(final int flushLeastPages) {
    int flush = this.flushedPosition.get();
    int write = getReadPosition();

    if (this.isFull()) {
        return true;
    }

    if (flushLeastPages > 0) {
        return ((write / OS_PAGE_SIZE) - (flush / OS_PAGE_SIZE)) >= flushLeastPages;
    }

    return write > flush;
}

其中flush記錄的是上一次完成重新整理後的位置，write記錄的是當前訊息內容寫入後的位置
當flushLeastPages 大於0的時候，通過：

return ((write / OS_PAGE_SIZE) - (flush / OS_PAGE_SIZE)) >= flushLeastPages;

可以計算出是否滿足page的要求，其中OS_PAGE_SIZE是4K，也就是說1個page大小是4k

由於這裡是同步刷盤，flushLeastPages是0，不對page要求，只要有快取有內容就會刷盤；但是在非同步刷盤中，flushLeastPages是4，也就是說，只有當快取的訊息至少是4（page個數）*4K（page大小）= 16K時，非同步刷盤才會將快取寫入檔案

 

回到MappedFile的flush方法，在通過isAbleToFlush檢查完寫入要求後

int value = getReadPosition();
try {
    //We only append data to fileChannel or mappedByteBuffer, never both.
    if (writeBuffer != null || this.fileChannel.position() != 0) {
        this.fileChannel.force(false);
    } else {
        this.mappedByteBuffer.force();
    }
} catch (Throwable e) {
    log.error("Error occurred when force data to disk.", e);
}

this.flushedPosition.set(value);

首先通過getReadPosition獲取當前訊息內容寫入後的位置，由於是同步刷盤，所以這裡呼叫mappedByteBuffer的force方法，通過JDK的NIO操作，將mappedByteBuffer快取中的資料寫入CommitLog檔案中
最後更新flushedPosition的值

再回到MappedFileQueue的flush方法，在完成MappedFile的flush後，還需要更新flushedWhere的值

此時快取中的資料完成了持久化，同步刷盤結束

 

非同步刷盤：

①FlushCommitLogService：

public void run() {
    CommitLog.log.info(this.getServiceName() + " service started");

    while (!this.isStopped()) {
        boolean flushCommitLogTimed = CommitLog.this.defaultMessageStore.getMessageStoreConfig().isFlushCommitLogTimed();

        int interval = CommitLog.this.defaultMessageStore.getMessageStoreConfig().getFlushIntervalCommitLog();
        int flushPhysicQueueLeastPages = CommitLog.this.defaultMessageStore.getMessageStoreConfig().getFlushCommitLogLeastPages();

        int flushPhysicQueueThoroughInterval =
            CommitLog.this.defaultMessageStore.getMessageStoreConfig().getFlushCommitLogThoroughInterval();

        boolean printFlushProgress = false;

        // Print flush progress
        long currentTimeMillis = System.currentTimeMillis();
        if (currentTimeMillis >= (this.lastFlushTimestamp + flushPhysicQueueThoroughInterval)) {
            this.lastFlushTimestamp = currentTimeMillis;
            flushPhysicQueueLeastPages = 0;
            printFlushProgress = (printTimes++ % 10) == 0;
        }

        try {
            if (flushCommitLogTimed) {
                Thread.sleep(interval);
            } else {
                this.waitForRunning(interval);
            }

            if (printFlushProgress) {
                this.printFlushProgress();
            }

            long begin = System.currentTimeMillis();
            CommitLog.this.mappedFileQueue.flush(flushPhysicQueueLeastPages);
            long storeTimestamp = CommitLog.this.mappedFileQueue.getStoreTimestamp();
            if (storeTimestamp > 0) {
                CommitLog.this.defaultMessageStore.getStoreCheckpoint().setPhysicMsgTimestamp(storeTimestamp);
            }
            long past = System.currentTimeMillis() - begin;
            if (past > 500) {
                log.info("Flush data to disk costs {} ms", past);
            }
        } catch (Throwable e) {
            CommitLog.log.warn(this.getServiceName() + " service has exception. ", e);
            this.printFlushProgress();
        }
    }

    // Normal shutdown, to ensure that all the flush before exit
    boolean result = false;
    for (int i = 0; i < RETRY_TIMES_OVER && !result; i++) {
        result = CommitLog.this.mappedFileQueue.flush(0);
        CommitLog.log.info(this.getServiceName() + " service shutdown, retry " + (i + 1) + " times " + (result ? "OK" : "Not OK"));
    }

    this.printFlushProgress();

    CommitLog.log.info(this.getServiceName() + " service end");
}

flushCommitLogTimed：是否使用定時刷盤
interval：刷盤時間間隔，預設500ms
flushPhysicQueueLeastPages：page大小，預設4個
flushPhysicQueueThoroughInterval：徹底刷盤時間間隔，預設10s

首先根據lastFlushTimestamp（上一次刷盤時間）+ flushPhysicQueueThoroughInterval和當前時間比較，判斷是否需要進行一次徹底刷盤，若達到了需要則將flushPhysicQueueLeastPages置為0

接著根據flushCommitLogTimed判斷
當flushCommitLogTimed為true，使用sleep等待500ms
當flushCommitLogTimed為false，呼叫waitForRunning在超時時間為500ms下阻塞，其喚醒條件也就是在handleDiskFlush中的wakeup喚醒

最後，和同步刷盤一樣，呼叫mappedFileQueue的flush方法
只不過，這裡的flushPhysicQueueLeastPages決定了其是進行徹底重新整理，還是按4page（16K）的標準重新整理

②CommitRealTimeService
這種刷盤方式需要和FlushCommitLogService配合

CommitRealTimeService的run方法：

public void run() {
   CommitLog.log.info(this.getServiceName() + " service started");
    while (!this.isStopped()) {
        int interval = CommitLog.this.defaultMessageStore.getMessageStoreConfig().getCommitIntervalCommitLog();

        int commitDataLeastPages = CommitLog.this.defaultMessageStore.getMessageStoreConfig().getCommitCommitLogLeastPages();

        int commitDataThoroughInterval =
            CommitLog.this.defaultMessageStore.getMessageStoreConfig().getCommitCommitLogThoroughInterval();

        long begin = System.currentTimeMillis();
        if (begin >= (this.lastCommitTimestamp + commitDataThoroughInterval)) {
            this.lastCommitTimestamp = begin;
            commitDataLeastPages = 0;
        }

        try {
            boolean result = CommitLog.this.mappedFileQueue.commit(commitDataLeastPages);
            long end = System.currentTimeMillis();
            if (!result) {
                this.lastCommitTimestamp = end; // result = false means some data committed.
                //now wake up flush thread.
                flushCommitLogService.wakeup();
            }

            if (end - begin > 500) {
                log.info("Commit data to file costs {} ms", end - begin);
            }
            this.waitForRunning(interval);
        } catch (Throwable e) {
            CommitLog.log.error(this.getServiceName() + " service has exception. ", e);
        }
    }

    boolean result = false;
    for (int i = 0; i < RETRY_TIMES_OVER && !result; i++) {
        result = CommitLog.this.mappedFileQueue.commit(0);
        CommitLog.log.info(this.getServiceName() + " service shutdown, retry " + (i + 1) + " times " + (result ? "OK" : "Not OK"));
    }
    CommitLog.log.info(this.getServiceName() + " service end");
}

這裡的邏輯和FlushCommitLogService中相似，之不過引數略有不同

interval：提交時間間隔，預設200ms
commitDataLeastPages：page大小，預設4個
commitDataThoroughInterval：提交完成時間間隔，預設200ms

基本和FlushCommitLogService相似，只不過呼叫了mappedFileQueue的commit方法

public boolean commit(final int commitLeastPages) {
    boolean result = true;
    MappedFile mappedFile = this.findMappedFileByOffset(this.committedWhere, this.committedWhere == 0);
    if (mappedFile != null) {
        int offset = mappedFile.commit(commitLeastPages);
        long where = mappedFile.getFileFromOffset() + offset;
        result = where == this.committedWhere;
        this.committedWhere = where;
    }

    return result;
}

這裡和mappedFileQueue的flush方法很相似，通過committedWhere尋找MappedFile

然後呼叫MappedFile的commit方法：

public int commit(final int commitLeastPages) {
    if (writeBuffer == null) {
        //no need to commit data to file channel, so just regard wrotePosition as committedPosition.
        return this.wrotePosition.get();
    }
    if (this.isAbleToCommit(commitLeastPages)) {
        if (this.hold()) {
            commit0(commitLeastPages);
            this.release();
        } else {
            log.warn("in commit, hold failed, commit offset = " + this.committedPosition.get());
        }
    }

    // All dirty data has been committed to FileChannel.
    if (writeBuffer != null && this.transientStorePool != null && this.fileSize == this.committedPosition.get()) {
        this.transientStorePool.returnBuffer(writeBuffer);
        this.writeBuffer = null;
    }

    return this.committedPosition.get();
}

依舊和MappedFile的flush方法很相似，在isAbleToCommit檢查完page後呼叫commit0方法

MappedFile的commit0方法：

protected void commit0(final int commitLeastPages) {
    int writePos = this.wrotePosition.get();
    int lastCommittedPosition = this.committedPosition.get();

    if (writePos - this.committedPosition.get() > 0) {
        try {
            ByteBuffer byteBuffer = writeBuffer.slice();
            byteBuffer.position(lastCommittedPosition);
            byteBuffer.limit(writePos);
            this.fileChannel.position(lastCommittedPosition);
            this.fileChannel.write(byteBuffer);
            this.committedPosition.set(writePos);
        } catch (Throwable e) {
            log.error("Error occurred when commit data to FileChannel.", e);
        }
    }
}

在 【RocketMQ中Broker的訊息儲存原始碼分析】 

中說過，當使用這種方式時，會先將訊息快取在writeBuffer中而不是之前的mappedByteBuffer
這裡就可以清楚地看到將writeBuffer中從lastCommittedPosition（上次提交位置）開始到writePos（快取訊息結束位置）的內容快取到了fileChannel中相同的位置，並沒有寫入磁碟
在快取到fileChannel後，會更新committedPosition值

回到commit方法，在向fileCfihannel快取完畢後，會檢查committedPosition是否達到了fileSize，也就是判斷writeBuffer中的內容是不是去全部提交完畢

若是全部提交，需要通過transientStorePool的returnBuffer方法來回收利用writeBuffer
transientStorePool其實是一個雙向佇列，由CommitLog來管理
TransientStorePool：

public class TransientStorePool {
    private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.STORE_LOGGER_NAME);

    private final int poolSize;
    private final int fileSize;
    private final Deque<ByteBuffer> availableBuffers;
    private final MessageStoreConfig storeConfig;

    public TransientStorePool(final MessageStoreConfig storeConfig) {
        this.storeConfig = storeConfig;
        this.poolSize = storeConfig.getTransientStorePoolSize();
        this.fileSize = storeConfig.getMapedFileSizeCommitLog();
        this.availableBuffers = new ConcurrentLinkedDeque<>();
    }
    ......
}

returnBuffer方法：

public void returnBuffer(ByteBuffer byteBuffer) {
    byteBuffer.position(0);
    byteBuffer.limit(fileSize);
    this.availableBuffers.offerFirst(byteBuffer);
}

這裡就可以清楚地看到byteBuffer確實被回收了

 

回到MappedFileQueue的commit方法：

public boolean commit(final int commitLeastPages) {
    boolean result = true;
    MappedFile mappedFile = this.findMappedFileByOffset(this.committedWhere, this.committedWhere == 0);
    if (mappedFile != null) {
        int offset = mappedFile.commit(commitLeastPages);
        long where = mappedFile.getFileFromOffset() + offset;
        result = where == this.committedWhere;
        this.committedWhere = where;
    }

    return result;
}

在完成mappedFile的commit後，通過where和committedWhere來判斷是否真的向fileCfihannel快取了 ，只有確實快取了result才是false！
之後會更新committedWhere，並返回result

 

那麼回到CommitRealTimeService的run方法，在完成commit之後，會判斷result
只有真的向fileCfihannel快取後，才會呼叫flushCommitLogService的wakeup方法，也就是喚醒了FlushCommitLogService的刷盤執行緒

唯一和之前分析的FlushCommitLogService不同的地方是在MappedFile的flush方法中：

if (writeBuffer != null || this.fileChannel.position() != 0) {
    this.fileChannel.force(false);
} else {
    this.mappedByteBuffer.force();
}

之前在沒有開啟記憶體位元組緩衝區的情況下，是將mappedByteBuffer中的內容寫入磁碟
而這時，終於輪到fileChannel了

可以看到這裡的條件判斷，當writeBuffer不等與null，或者fileChannel的position不等與0
writeBuffer等於null的情況會在TransientStorePool對其回收之後

到這裡就可以明白開啟記憶體位元組緩衝區的情況下，其實是進行了兩次快取才寫入磁碟

 

至此，Broker的訊息持久化以及刷盤的整個過程完畢