上一篇文章，分析了Netty服務端啟動的初始化過程，今天我們來分析一下Netty中的Reactor執行緒模型
在分析原始碼之前，我們先分析，哪些地方用到了EventLoop？
- NioServerSocketChannel的連線監聽註冊
- NioSocketChannel的IO事件註冊
NioServerSocketChannel連線監聽
在AbstractBootstrap類的initAndRegister()方法中，當NioServerSocketChannel初始化完成後，會呼叫case標記位置的程式碼進行註冊。
final ChannelFuture initAndRegister() {
Channel channel = null;
try {
channel = channelFactory.newChannel();
init(channel);
} catch (Throwable t) {
}
//註冊到boss執行緒的selector上。
ChannelFuture regFuture = config().group().register(channel);
if (regFuture.cause() != null) {
if (channel.isRegistered()) {
channel.close();
} else {
channel.unsafe().closeForcibly();
}
}
return regFuture;
}
AbstractNioChannel.doRegister
按照程式碼的執行邏輯，最終會執行到AbstractNioChannel的doRegister()方法中。
@Override
protected void doRegister() throws Exception {
boolean selected = false;
for (;;) {
try {
//呼叫ServerSocketChannel的register方法，把當前服務端物件註冊到boss執行緒的selector上
selectionKey = javaChannel().register(eventLoop().unwrappedSelector(), 0, this);
return;
} catch (CancelledKeyException e) {
if (!selected) {
// Force the Selector to select now as the "canceled" SelectionKey may still be
// cached and not removed because no Select.select(..) operation was called yet.
eventLoop().selectNow();
selected = true;
} else {
// We forced a select operation on the selector before but the SelectionKey is still cached
// for whatever reason. JDK bug ?
throw e;
}
}
}
}
NioEventLoop的啟動過程
NioEventLoop是一個執行緒，它的啟動過程如下。
在AbstractBootstrap的doBind0方法中，獲取了NioServerSocketChannel中的NioEventLoop，然後使用它來執行繫結埠的任務。
private static void doBind0(
final ChannelFuture regFuture, final Channel channel,
final SocketAddress localAddress, final ChannelPromise promise) {
//啟動
channel.eventLoop().execute(new Runnable() {
@Override
public void run() {
if (regFuture.isSuccess()) {
channel.bind(localAddress, promise).addListener(ChannelFutureListener.CLOSE_ON_FAILURE);
} else {
promise.setFailure(regFuture.cause());
}
}
});
}
SingleThreadEventExecutor.execute
然後一路執行到SingleThreadEventExecutor.execute方法中，呼叫startThread()方法啟動執行緒。
private void execute(Runnable task, boolean immediate) {
boolean inEventLoop = inEventLoop();
addTask(task);
if (!inEventLoop) {
startThread(); //啟動執行緒
if (isShutdown()) {
boolean reject = false;
try {
if (removeTask(task)) {
reject = true;
}
} catch (UnsupportedOperationException e) {
// The task queue does not support removal so the best thing we can do is to just move on and
// hope we will be able to pick-up the task before its completely terminated.
// In worst case we will log on termination.
}
if (reject) {
reject();
}
}
}
if (!addTaskWakesUp && immediate) {
wakeup(inEventLoop);
}
}
startThread
private void startThread() {
if (state == ST_NOT_STARTED) {
if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
boolean success = false;
try {
doStartThread(); //執行啟動過程
success = true;
} finally {
if (!success) {
STATE_UPDATER.compareAndSet(this, ST_STARTED, ST_NOT_STARTED);
}
}
}
}
}
接著呼叫doStartThread()方法，通過executor.execute執行一個任務，在該任務中啟動了NioEventLoop執行緒
private void doStartThread() {
assert thread == null;
executor.execute(new Runnable() { //通過執行緒池執行一個任務
@Override
public void run() {
thread = Thread.currentThread();
if (interrupted) {
thread.interrupt();
}
boolean success = false;
updateLastExecutionTime();
try {
SingleThreadEventExecutor.this.run(); //呼叫boss的NioEventLoop的run方法，開啟輪詢
}
//省略....
}
});
}
NioEventLoop的輪詢過程
當NioEventLoop執行緒被啟動後，就直接進入到NioEventLoop的run方法中。
protected void run() {
int selectCnt = 0;
for (;;) {
try {
int strategy;
try {
strategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());
switch (strategy) {
case SelectStrategy.CONTINUE:
continue;
case SelectStrategy.BUSY_WAIT:
case SelectStrategy.SELECT:
long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
if (curDeadlineNanos == -1L) {
curDeadlineNanos = NONE; // nothing on the calendar
}
nextWakeupNanos.set(curDeadlineNanos);
try {
if (!hasTasks()) {
strategy = select(curDeadlineNanos);
}
} finally {
// This update is just to help block unnecessary selector wakeups
// so use of lazySet is ok (no race condition)
nextWakeupNanos.lazySet(AWAKE);
}
// fall through
default:
}
} catch (IOException e) {
// If we receive an IOException here its because the Selector is messed up. Let's rebuild
// the selector and retry. https://github.com/netty/netty/issues/8566
rebuildSelector0();
selectCnt = 0;
handleLoopException(e);
continue;
}
selectCnt++;
cancelledKeys = 0;
needsToSelectAgain = false;
final int ioRatio = this.ioRatio;
boolean ranTasks;
if (ioRatio == 100) {
try {
if (strategy > 0) {
processSelectedKeys();
}
} finally {
// Ensure we always run tasks.
ranTasks = runAllTasks();
}
} else if (strategy > 0) {
final long ioStartTime = System.nanoTime();
try {
processSelectedKeys();
} finally {
// Ensure we always run tasks.
final long ioTime = System.nanoTime() - ioStartTime;
ranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
}
} else {
ranTasks = runAllTasks(0); // This will run the minimum number of tasks
}
if (ranTasks || strategy > 0) {
if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {
logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.",
selectCnt - 1, selector);
}
selectCnt = 0;
} else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)
selectCnt = 0;
}
} catch (CancelledKeyException e) {
// Harmless exception - log anyway
if (logger.isDebugEnabled()) {
logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector {} - JDK bug?",
selector, e);
}
} catch (Error e) {
throw (Error) e;
} catch (Throwable t) {
handleLoopException(t);
} finally {
// Always handle shutdown even if the loop processing threw an exception.
try {
if (isShuttingDown()) {
closeAll();
if (confirmShutdown()) {
return;
}
}
} catch (Error e) {
throw (Error) e;
} catch (Throwable t) {
handleLoopException(t);
}
}
}
}
NioEventLoop的執行流程
NioEventLoop中的run方法是一個無限迴圈的執行緒，在該迴圈中主要做三件事情，如圖9-1所示。
- 輪詢處理I/O事件（select），輪詢Selector選擇器中已經註冊的所有Channel的I/O就緒事件
- 處理I/O事件，如果存在已經就緒的Channel的I/O事件，則呼叫
processSelectedKeys進行處理 - 處理非同步任務（runAllTasks），Reactor執行緒有一個非常重要的職責，就是處理任務佇列中的非I/O任務，Netty提供了ioRadio引數用來調整I/O時間和任務處理的時間比例。
輪詢I/O就緒事件
我們先來看I/O時間相關的程式碼片段:
- 通過
selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())獲取當前的執行策略 - 根據不同的策略，用來控制每次輪詢時的執行策略。
protected void run() {
int selectCnt = 0;
for (;;) {
try {
int strategy;
try {
strategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());
switch (strategy) {
case SelectStrategy.CONTINUE:
continue;
case SelectStrategy.BUSY_WAIT:
// fall-through to SELECT since the busy-wait is not supported with NIO
case SelectStrategy.SELECT:
long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
if (curDeadlineNanos == -1L) {
curDeadlineNanos = NONE; // nothing on the calendar
}
nextWakeupNanos.set(curDeadlineNanos);
try {
if (!hasTasks()) {
strategy = select(curDeadlineNanos);
}
} finally {
// This update is just to help block unnecessary selector wakeups
// so use of lazySet is ok (no race condition)
nextWakeupNanos.lazySet(AWAKE);
}
// fall through
default:
}
}
//省略....
}
}
}
selectStrategy處理邏輯
@Override
public int calculateStrategy(IntSupplier selectSupplier, boolean hasTasks) throws Exception {
return hasTasks ? selectSupplier.get() : SelectStrategy.SELECT;
}
如果hasTasks為true，表示當前NioEventLoop執行緒存在非同步任務的情況下，則呼叫selectSupplier.get()，否則直接返回SELECT。
其中selectSupplier.get()的定義如下：
private final IntSupplier selectNowSupplier = new IntSupplier() {
@Override
public int get() throws Exception {
return selectNow();
}
};
該方法中呼叫的是selectNow()方法，這個方法是Selector選擇器中的提供的非阻塞方法，執行後會立刻返回。
- 如果當前已經有就緒的Channel，則會返回對應就緒Channel的數量
- 否則，返回0.
分支處理
在上面一個步驟中獲得了strategy之後，會根據不同的結果進行分支處理。
- CONTINUE，表示需要重試。
- BUSY_WAIT，由於在NIO中並不支援BUSY_WAIT，所以BUSY_WAIT和SELECT的執行邏輯是一樣的
- SELECT，表示需要通過select方法獲取就緒的Channel列表，當NioEventLoop中不存在非同步任務時，也就是任務佇列為空，則返回該策略。
switch (strategy) {
case SelectStrategy.CONTINUE:
continue;
case SelectStrategy.BUSY_WAIT:
// fall-through to SELECT since the busy-wait is not supported with NIO
case SelectStrategy.SELECT:
long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
if (curDeadlineNanos == -1L) {
curDeadlineNanos = NONE; // nothing on the calendar
}
nextWakeupNanos.set(curDeadlineNanos);
try {
if (!hasTasks()) {
strategy = select(curDeadlineNanos);
}
} finally {
// This update is just to help block unnecessary selector wakeups
// so use of lazySet is ok (no race condition)
nextWakeupNanos.lazySet(AWAKE);
}
// fall through
default:
}
SelectStrategy.SELECT
當NioEventLoop執行緒中不存在非同步任務時，則開始執行SELECT策略
//下一次定時任務觸發截至時間，預設不是定時任務，返回 -1L
long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
if (curDeadlineNanos == -1L) {
curDeadlineNanos = NONE; // nothing on the calendar
}
nextWakeupNanos.set(curDeadlineNanos);
try {
if (!hasTasks()) {
//2. taskQueue中任務執行完，開始執行select進行阻塞
strategy = select(curDeadlineNanos);
}
} finally {
// This update is just to help block unnecessary selector wakeups
// so use of lazySet is ok (no race condition)
nextWakeupNanos.lazySet(AWAKE);
}
select方法定義如下，預設情況下deadlineNanos=NONE，所以會呼叫select()方法阻塞。
private int select(long deadlineNanos) throws IOException {
if (deadlineNanos == NONE) {
return selector.select();
}
//計算select()方法的阻塞超時時間
long timeoutMillis = deadlineToDelayNanos(deadlineNanos + 995000L) / 1000000L;
return timeoutMillis <= 0 ? selector.selectNow() : selector.select(timeoutMillis);
}
最終返回就緒的channel個數，後續的邏輯中會根據返回的就緒channel個數來決定執行邏輯。
NioEventLoop.run中的業務處理
業務處理的邏輯相對來說比較容易理解
- 如果有就緒的channel，則處理就緒channel的IO事件
- 處理完成後同步執行非同步佇列中的任務。
- 另外，這裡為了解決Java NIO中的空轉問題，通過selectCnt記錄了空轉次數，一次迴圈發生了空轉（既沒有IO需要處理、也沒有執行任何任務），那麼記錄下來（selectCnt）; ，如果連續發生空轉（selectCnt達到一定值），netty認為觸發了NIO的BUG（unexpectedSelectorWakeup處理）；
Java Nio中有一個bug，Java nio在Linux系統下的epoll空輪詢問題。也就是在
select()方法中，及時就緒的channel為0，也會從本來應該阻塞的操作中被喚醒，從而導致CPU 使用率達到100%。
@Override
protected void run() {
int selectCnt = 0;
for (;;) {
//省略....
selectCnt++;//selectCnt記錄的是無功而返的select次數，即eventLoop空轉的次數，為解決NIO BUG
cancelledKeys = 0;
needsToSelectAgain = false;
final int ioRatio = this.ioRatio;
boolean ranTasks;
if (ioRatio == 100) { //ioRadio執行時間佔比是100%，預設是50%
try {
if (strategy > 0) { //strategy>0表示存在就緒的SocketChannel
processSelectedKeys(); //執行就緒SocketChannel的任務
}
} finally {
//注意，將ioRatio設定為100，並不代表任務不執行，反而是每次將任務佇列執行完
ranTasks = runAllTasks(); //確保總是執行佇列中的任務
}
} else if (strategy > 0) { //strategy>0表示存在就緒的SocketChannel
final long ioStartTime = System.nanoTime(); //io時間處理開始時間
try {
processSelectedKeys(); //開始處理IO就緒事件
} finally {
// io事件執行結束時間
final long ioTime = System.nanoTime() - ioStartTime;
//基於本次迴圈處理IO的時間，ioRatio，計算出執行任務耗時的上限，也就是隻允許處理多長時間非同步任務
ranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
}
} else {
//這個分支代表：strategy=0，ioRatio<100，此時任務限時=0，意為：儘量少地執行非同步任務
//這個分支和strategy>0實際是一碼事，程式碼簡化了一下而已
ranTasks = runAllTasks(0); // This will run the minimum number of tasks
}
if (ranTasks || strategy > 0) { //ranTasks=true，或strategy>0，說明eventLoop幹活了，沒有空轉，清空selectCnt
if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {
logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.",
selectCnt - 1, selector);
}
selectCnt = 0;
}
//unexpectedSelectorWakeup處理NIO BUG
else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)
selectCnt = 0;
}
}
}
processSelectedKeys
通過在select方法中，我們可以獲得就緒的I/O事件數量，從而觸發執行processSelectedKeys方法。
private void processSelectedKeys() {
if (selectedKeys != null) {
processSelectedKeysOptimized();
} else {
processSelectedKeysPlain(selector.selectedKeys());
}
}
處理I/O事件時，有兩個邏輯分支處理：
- 一種是處理Netty優化過的selectedKeys，
- 另一種是正常的處理邏輯
processSelectedKeys方法中根據是否設定了selectedKeys來判斷使用哪種策略，預設使用的是Netty優化過的selectedKeys，它返回的物件是SelectedSelectionKeySet。
processSelectedKeysOptimized
private void processSelectedKeysOptimized() {
for (int i = 0; i < selectedKeys.size; ++i) {
//1. 取出IO事件以及對應的channel
final SelectionKey k = selectedKeys.keys[i];
selectedKeys.keys[i] = null;//k的引用置null，便於gc回收，也表示該channel的事件處理完成避免重複處理
final Object a = k.attachment(); //獲取儲存在當前channel中的attachment，此時應該是NioServerSocketChannel
//處理當前的channel
if (a instanceof AbstractNioChannel) {
//對於boss NioEventLoop，輪詢到的基本是連線事件，後續的事情就是通過他的pipeline將連線扔給一個worker NioEventLoop處理
//對於worker NioEventLoop來說，輪循道的基本商是IO讀寫事件，後續的事情就是通過他的pipeline將讀取到的位元組流傳遞給每個channelHandler來處理
processSelectedKey(k, (AbstractNioChannel) a);
} else {
@SuppressWarnings("unchecked")
NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
processSelectedKey(k, task);
}
if (needsToSelectAgain) {
// null out entries in the array to allow to have it GC'ed once the Channel close
// See https://github.com/netty/netty/issues/2363
selectedKeys.reset(i + 1);
selectAgain();
i = -1;
}
}
}
processSelectedKey
private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
if (!k.isValid()) {
final EventLoop eventLoop;
try {
eventLoop = ch.eventLoop();
} catch (Throwable ignored) {
}
if (eventLoop == this) {
// close the channel if the key is not valid anymore
unsafe.close(unsafe.voidPromise());
}
return;
}
try {
int readyOps = k.readyOps(); //獲取當前key所屬的操作型別
if ((readyOps & SelectionKey.OP_CONNECT) != 0) {//如果是連線型別
int ops = k.interestOps();
ops &= ~SelectionKey.OP_CONNECT;
k.interestOps(ops);
unsafe.finishConnect();
}
if ((readyOps & SelectionKey.OP_WRITE) != 0) { //如果是寫型別
ch.unsafe().forceFlush();
}
//如果是讀型別或者ACCEPT型別。則執行unsafe.read()方法，unsafe的例項物件為 NioMessageUnsafe
if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
unsafe.read();
}
} catch (CancelledKeyException ignored) {
unsafe.close(unsafe.voidPromise());
}
}
NioMessageUnsafe.read()
假設此時是一個讀操作，或者是客戶端建立連線，那麼程式碼執行邏輯如下，
@Override
public void read() {
assert eventLoop().inEventLoop();
final ChannelConfig config = config();
final ChannelPipeline pipeline = pipeline(); //如果是第一次建立連線，此時的pipeline是ServerBootstrapAcceptor
final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
allocHandle.reset(config);
boolean closed = false;
Throwable exception = null;
try {
try {
do {
int localRead = doReadMessages(readBuf);
if (localRead == 0) {
break;
}
if (localRead < 0) {
closed = true;
break;
}
allocHandle.incMessagesRead(localRead);
} while (continueReading(allocHandle));
} catch (Throwable t) {
exception = t;
}
int size = readBuf.size();
for (int i = 0; i < size; i ++) {
readPending = false;
pipeline.fireChannelRead(readBuf.get(i)); //呼叫pipeline中的channelRead方法
}
readBuf.clear();
allocHandle.readComplete();
pipeline.fireChannelReadComplete();
if (exception != null) {
closed = closeOnReadError(exception);
pipeline.fireExceptionCaught(exception); //呼叫pipeline中的ExceptionCaught方法
}
if (closed) {
inputShutdown = true;
if (isOpen()) {
close(voidPromise());
}
}
} finally {
if (!readPending && !config.isAutoRead()) {
removeReadOp();
}
}
}
SelectedSelectionKeySet的優化
Netty中自己封裝實現了一個SelectedSelectionKeySet，用來優化原本SelectorKeys的結構，它是怎麼進行優化的呢？先來看它的程式碼定義
final class SelectedSelectionKeySet extends AbstractSet<SelectionKey> {
SelectionKey[] keys;
int size;
SelectedSelectionKeySet() {
keys = new SelectionKey[1024];
}
@Override
public boolean add(SelectionKey o) {
if (o == null) {
return false;
}
keys[size++] = o;
if (size == keys.length) {
increaseCapacity();
}
return true;
}
}
SelectedSelectionKeySet內部使用的是SelectionKey陣列，所有在processSelectedKeysOptimized方法中可以直接通過遍歷陣列來取出就緒的I/O事件。
而原來的Set<SelectionKey>返回的是HashSet型別，兩者相比，SelectionKey[]不需要考慮雜湊衝突的問題，所以可以實現O(1)時間複雜度的add操作。
SelectedSelectionKeySet的初始化
netty通過反射的方式，把Selector物件內部的selectedKeys和publicSelectedKeys替換為SelectedSelectionKeySet。
原本的selectedKeys和publicSelectedKeys這兩個欄位都是HashSet型別，替換之後變成了SelectedSelectionKeySet。當有就緒的key時，會直接填充到SelectedSelectionKeySet的陣列中。後續只需要遍歷即可。
private SelectorTuple openSelector() {
final Class<?> selectorImplClass = (Class<?>) maybeSelectorImplClass;
final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();
//使用反射
Object maybeException = AccessController.doPrivileged(new PrivilegedAction<Object>() {
@Override
public Object run() {
try {
//Selector內部的selectedKeys欄位
Field selectedKeysField = selectorImplClass.getDeclaredField("selectedKeys");
//Selector內部的publicSelectedKeys欄位
Field publicSelectedKeysField = selectorImplClass.getDeclaredField("publicSelectedKeys");
if (PlatformDependent.javaVersion() >= 9 && PlatformDependent.hasUnsafe()) {
//獲取selectedKeysField欄位偏移量
long selectedKeysFieldOffset = PlatformDependent.objectFieldOffset(selectedKeysField);
//獲取publicSelectedKeysField欄位偏移量
long publicSelectedKeysFieldOffset =
PlatformDependent.objectFieldOffset(publicSelectedKeysField);
if (selectedKeysFieldOffset != -1 && publicSelectedKeysFieldOffset != -1) {
//替換為selectedKeySet
PlatformDependent.putObject(
unwrappedSelector, selectedKeysFieldOffset, selectedKeySet);
PlatformDependent.putObject(
unwrappedSelector, publicSelectedKeysFieldOffset, selectedKeySet);
return null;
}
// We could not retrieve the offset, lets try reflection as last-resort.
}
Throwable cause = ReflectionUtil.trySetAccessible(selectedKeysField, true);
if (cause != null) {
return cause;
}
cause = ReflectionUtil.trySetAccessible(publicSelectedKeysField, true);
if (cause != null) {
return cause;
}
selectedKeysField.set(unwrappedSelector, selectedKeySet);
publicSelectedKeysField.set(unwrappedSelector, selectedKeySet);
return null;
} catch (NoSuchFieldException e) {
return e;
} catch (IllegalAccessException e) {
return e;
}
}
});
if (maybeException instanceof Exception) {
selectedKeys = null;
Exception e = (Exception) maybeException;
logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, e);
return new SelectorTuple(unwrappedSelector);
}
selectedKeys = selectedKeySet;
}
非同步任務的執行流程
分析完上面的流程後，我們繼續來看NioEventLoop中的run方法中，針對非同步任務的處理流程
@Override
protected void run() {
int selectCnt = 0;
for (;;) {
ranTasks = runAllTasks();
}
}
runAllTask
需要注意，NioEventLoop可以支援定時任務的執行，通過nioEventLoop.schedule()來完成。
protected boolean runAllTasks() {
assert inEventLoop();
boolean fetchedAll;
boolean ranAtLeastOne = false;
do {
fetchedAll = fetchFromScheduledTaskQueue(); //合併定時任務到普通任務佇列
if (runAllTasksFrom(taskQueue)) { //迴圈執行taskQueue中的任務
ranAtLeastOne = true;
}
} while (!fetchedAll);
if (ranAtLeastOne) { //如果任務全部執行完成，記錄執行完完成時間
lastExecutionTime = ScheduledFutureTask.nanoTime();
}
afterRunningAllTasks();//執行收尾任務
return ranAtLeastOne;
}
fetchFromScheduledTaskQueue
遍歷scheduledTaskQueue中的任務，新增到taskQueue中。
private boolean fetchFromScheduledTaskQueue() {
if (scheduledTaskQueue == null || scheduledTaskQueue.isEmpty()) {
return true;
}
long nanoTime = AbstractScheduledEventExecutor.nanoTime();
for (;;) {
Runnable scheduledTask = pollScheduledTask(nanoTime);
if (scheduledTask == null) {
return true;
}
if (!taskQueue.offer(scheduledTask)) {
// No space left in the task queue add it back to the scheduledTaskQueue so we pick it up again.
scheduledTaskQueue.add((ScheduledFutureTask<?>) scheduledTask);
return false;
}
}
}
任務新增方法execute
NioEventLoop內部有兩個非常重要的非同步任務佇列，分別是普通任務和定時任務佇列，針對這兩個佇列提供了兩個方法分別向兩個佇列中新增任務。
- execute()
- schedule()
其中,execute方法的定義如下。
private void execute(Runnable task, boolean immediate) {
boolean inEventLoop = inEventLoop();
addTask(task); //把當前任務新增到阻塞佇列中
if (!inEventLoop) { //如果是非NioEventLoop
startThread(); //啟動執行緒
if (isShutdown()) { //如果當前NioEventLoop已經是停止狀態
boolean reject = false;
try {
if (removeTask(task)) {
reject = true;
}
} catch (UnsupportedOperationException e) {
// The task queue does not support removal so the best thing we can do is to just move on and
// hope we will be able to pick-up the task before its completely terminated.
// In worst case we will log on termination.
}
if (reject) {
reject();
}
}
}
if (!addTaskWakesUp && immediate) {
wakeup(inEventLoop);
}
}
Nio的空輪轉問題
所謂的空輪訓，是指我們在執行selector.select()方法時，如果沒有就緒的SocketChannel時，當前執行緒會被阻塞 。 而空輪詢是指當沒有就緒SocketChannel時，會被觸發喚醒。
而這個喚醒是沒有任何讀寫請求的，從而導致執行緒在做無效的輪詢，使得CPU佔用率較高。
導致這個問題的根本原因是：
在部分Linux的2.6的kernel中，poll和epoll對於突然中斷的連線socket會對返回的eventSet事件集合置為POLLHUP，也可能是POLLERR，eventSet事件集合發生了變化，這就可能導致Selector會被喚醒。這是與作業系統機制有關係的，JDK雖然僅僅是一個相容各個作業系統平臺的軟體，但很遺憾在JDK5和JDK6最初的版本中（嚴格意義上來將，JDK部分版本都是），這個問題並沒有解決，而將這個帽子拋給了作業系統方，這也就是這個bug最終一直到2013年才最終修復的原因，最終影響力太廣。
Netty是如何解決這個問題的呢？我們回到NioEventLoop的run方法中
@Override
protected void run() {
int selectCnt = 0;
for (;;) {
//selectCnt記錄的是無功而返的select次數，即eventLoop空轉的次數，為解決NIO BUG
selectCnt++;
//ranTasks=true，或strategy>0，說明eventLoop幹活了，沒有空轉，清空selectCnt
if (ranTasks || strategy > 0) {
//如果選擇操作計數器的值，大於最小選擇器重構閾值，則輸出log
if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {
logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.",
selectCnt - 1, selector);
}
selectCnt = 0;
}
//unexpectedSelectorWakeup處理NIO BUG
else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)
selectCnt = 0;
}
}
}
unexpectedSelectorWakeup
private boolean unexpectedSelectorWakeup(int selectCnt) {
if (Thread.interrupted()) {
if (logger.isDebugEnabled()) {
logger.debug("Selector.select() returned prematurely because " +
"Thread.currentThread().interrupt() was called. Use " +
"NioEventLoop.shutdownGracefully() to shutdown the NioEventLoop.");
}
return true;
}
//如果選擇重構的閾值大於0， 預設值是512次、 並且當前觸發的空輪詢次數大於 512次。，則觸發重構
if (SELECTOR_AUTO_REBUILD_THRESHOLD > 0 &&
selectCnt >= SELECTOR_AUTO_REBUILD_THRESHOLD) {
// The selector returned prematurely many times in a row.
// Rebuild the selector to work around the problem.
logger.warn("Selector.select() returned prematurely {} times in a row; rebuilding Selector {}.",
selectCnt, selector);
rebuildSelector();
return true;
}
return false;
}
rebuildSelector()
public void rebuildSelector() {
if (!inEventLoop()) { //如果不是在eventLoop中執行，則使用非同步執行緒執行
execute(new Runnable() {
@Override
public void run() {
rebuildSelector0();
}
});
return;
}
rebuildSelector0();
}
rebuildSelector0
這個方法的主要作用： 重新建立一個選擇器,替代當前事件迴圈中的選擇器
private void rebuildSelector0() {
final Selector oldSelector = selector; //獲取老的selector選擇器
final SelectorTuple newSelectorTuple; //定義新的選擇器
if (oldSelector == null) { //如果老的選擇器為空，直接返回
return;
}
try {
newSelectorTuple = openSelector(); //建立一個新的選擇器
} catch (Exception e) {
logger.warn("Failed to create a new Selector.", e);
return;
}
// Register all channels to the new Selector.
int nChannels = 0;
for (SelectionKey key: oldSelector.keys()) {//遍歷註冊到選擇器的選擇key集合
Object a = key.attachment();
try {
//如果選擇key無效或選擇關聯的通道已經註冊到新的選擇器，則跳出當前迴圈
if (!key.isValid() || key.channel().keyFor(newSelectorTuple.unwrappedSelector) != null) {
continue;
}
//獲取key的選擇關注事件集
int interestOps = key.interestOps();
key.cancel();//取消選擇key
//註冊選擇key到新的選擇器
SelectionKey newKey = key.channel().register(newSelectorTuple.unwrappedSelector, interestOps, a);
if (a instanceof AbstractNioChannel) {//如果是nio通道，則更新通道的選擇key
// Update SelectionKey
((AbstractNioChannel) a).selectionKey = newKey;
}
nChannels ++;
} catch (Exception e) {
logger.warn("Failed to re-register a Channel to the new Selector.", e);
if (a instanceof AbstractNioChannel) {
AbstractNioChannel ch = (AbstractNioChannel) a;
ch.unsafe().close(ch.unsafe().voidPromise());
} else {
@SuppressWarnings("unchecked")
NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
invokeChannelUnregistered(task, key, e);
}
}
}
//更新當前事件迴圈選擇器
selector = newSelectorTuple.selector;
unwrappedSelector = newSelectorTuple.unwrappedSelector;
try {
// time to close the old selector as everything else is registered to the new one
oldSelector.close(); //關閉原始選擇器
} catch (Throwable t) {
if (logger.isWarnEnabled()) {
logger.warn("Failed to close the old Selector.", t);
}
}
if (logger.isInfoEnabled()) {
logger.info("Migrated " + nChannels + " channel(s) to the new Selector.");
}
}
從上述過程中我們發現，Netty解決NIO空輪轉問題的方式，是通過重建Selector物件來完成的，在這個重建過程中，核心是把Selector中所有的SelectionKey重新註冊到新的Selector上，從而巧妙的避免了JDK epoll空輪訓問題。
連線的建立及處理過程
在9.2.4.3節中，提到了當客戶端有連線或者讀事件傳送到服務端時，會呼叫NioMessageUnsafe類的read()方法。
public void read() {
assert eventLoop().inEventLoop();
final ChannelConfig config = config();
final ChannelPipeline pipeline = pipeline();
final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
allocHandle.reset(config);
boolean closed = false;
Throwable exception = null;
try {
try {
do {
//如果有客戶端連線進來，則localRead為1，否則返回0
int localRead = doReadMessages(readBuf);
if (localRead == 0) {
break;
}
if (localRead < 0) {
closed = true;
break;
}
allocHandle.incMessagesRead(localRead); //累計增加read訊息數量
} while (continueReading(allocHandle));
} catch (Throwable t) {
exception = t;
}
int size = readBuf.size(); //遍歷客戶端連線列表
for (int i = 0; i < size; i ++) {
readPending = false;
pipeline.fireChannelRead(readBuf.get(i)); //呼叫pipeline中handler的channelRead方法。
}
readBuf.clear(); //清空集合
allocHandle.readComplete();
pipeline.fireChannelReadComplete(); //觸發pipeline中handler的readComplete方法
if (exception != null) {
closed = closeOnReadError(exception);
pipeline.fireExceptionCaught(exception);
}
if (closed) {
inputShutdown = true;
if (isOpen()) {
close(voidPromise());
}
}
} finally {
if (!readPending && !config.isAutoRead()) {
removeReadOp();
}
}
}
pipeline.fireChannelRead(readBuf.get(i))
繼續來看pipeline的觸發方法，此時的pipeline組成，如果當前是連線事件，那麼pipeline = ServerBootstrap$ServerBootstrapAcceptor。
static void invokeChannelRead(final AbstractChannelHandlerContext next, Object msg) {
final Object m = next.pipeline.touch(ObjectUtil.checkNotNull(msg, "msg"), next);
EventExecutor executor = next.executor();
if (executor.inEventLoop()) {
next.invokeChannelRead(m); //獲取pipeline中的下一個節點，呼叫該handler的channelRead方法
} else {
executor.execute(new Runnable() {
@Override
public void run() {
next.invokeChannelRead(m);
}
});
}
}
ServerBootstrapAcceptor
ServerBootstrapAcceptor是NioServerSocketChannel中一個特殊的Handler，專門用來處理客戶端連線事件，該方法中核心的目的是把針對SocketChannel的handler連結串列，新增到當前NioSocketChannel中的pipeline中。
public void channelRead(ChannelHandlerContext ctx, Object msg) {
final Channel child = (Channel) msg;
child.pipeline().addLast(childHandler); //把服務端配置的childHandler，新增到當前NioSocketChannel中的pipeline中
setChannelOptions(child, childOptions, logger); //設定NioSocketChannel的屬性
setAttributes(child, childAttrs);
try {
//把當前的NioSocketChannel註冊到Selector上，並且監聽一個非同步事件。
childGroup.register(child).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
if (!future.isSuccess()) {
forceClose(child, future.cause());
}
}
});
} catch (Throwable t) {
forceClose(child, t);
}
}
pipeline的構建過程
9.6.2節中，child其實就是一個NioSocketChannel，它是在NioServerSocketChannel中，當接收到一個新的連結時，建立物件。
@Override
protected int doReadMessages(List<Object> buf) throws Exception {
SocketChannel ch = SocketUtils.accept(javaChannel());
try {
if (ch != null) {
buf.add(new NioSocketChannel(this, ch)); //這裡
return 1;
}
} catch (Throwable t) {
logger.warn("Failed to create a new channel from an accepted socket.", t);
try {
ch.close();
} catch (Throwable t2) {
logger.warn("Failed to close a socket.", t2);
}
}
return 0;
}
而NioSocketChannel在構造時，呼叫了父類AbstractChannel中的構造方法，初始化了一個pipeline.
protected AbstractChannel(Channel parent) {
this.parent = parent;
id = newId();
unsafe = newUnsafe();
pipeline = newChannelPipeline();
}
DefaultChannelPipeline
pipeline的預設例項是DefaultChannelPipeline，構造方法如下。
protected DefaultChannelPipeline(Channel channel) {
this.channel = ObjectUtil.checkNotNull(channel, "channel");
succeededFuture = new SucceededChannelFuture(channel, null);
voidPromise = new VoidChannelPromise(channel, true);
tail = new TailContext(this);
head = new HeadContext(this);
head.next = tail;
tail.prev = head;
}
初始化了一個頭節點和尾節點，組成一個雙向連結串列，如圖9-2所示
NioSocketChannel中handler鏈的構成
再回到ServerBootstrapAccepter的channelRead方法中，收到客戶端連線時，觸發了NioSocketChannel中的pipeline的新增
以下程式碼是DefaultChannelPipeline的addLast方法。
@Override
public final ChannelPipeline addLast(EventExecutorGroup executor, ChannelHandler... handlers) {
ObjectUtil.checkNotNull(handlers, "handlers");
for (ChannelHandler h: handlers) { //遍歷handlers列表，此時這裡的handler是ChannelInitializer回撥方法
if (h == null) {
break;
}
addLast(executor, null, h);
}
return this;
}
addLast
把服務端配置的ChannelHandler，新增到pipeline中，注意，此時的pipeline中儲存的是ChannelInitializer回撥方法。
@Override
public final ChannelPipeline addLast(EventExecutorGroup group, String name, ChannelHandler handler) {
final AbstractChannelHandlerContext newCtx;
synchronized (this) {
checkMultiplicity(handler); //檢查是否有重複的handler
//建立新的DefaultChannelHandlerContext節點
newCtx = newContext(group, filterName(name, handler), handler);
addLast0(newCtx); //新增新的DefaultChannelHandlerContext到ChannelPipeline
if (!registered) {
newCtx.setAddPending();
callHandlerCallbackLater(newCtx, true);
return this;
}
EventExecutor executor = newCtx.executor();
if (!executor.inEventLoop()) {
callHandlerAddedInEventLoop(newCtx, executor);
return this;
}
}
callHandlerAdded0(newCtx);
return this;
}
這個回撥方法什麼時候觸發呼叫呢？其實就是在ServerBootstrapAcceptor這個類的channelRead方法中，註冊當前NioSocketChannel時
childGroup.register(child).addListener(new ChannelFutureListener() {}
最終按照之前我們上一節課原始碼分析的思路，定位到AbstractChannel中的register0方法中。
private void register0(ChannelPromise promise) {
try {
// check if the channel is still open as it could be closed in the mean time when the register
// call was outside of the eventLoop
if (!promise.setUncancellable() || !ensureOpen(promise)) {
return;
}
boolean firstRegistration = neverRegistered;
doRegister();
neverRegistered = false;
registered = true;
//
pipeline.invokeHandlerAddedIfNeeded();
}
}
callHandlerAddedForAllHandlers
pipeline.invokeHandlerAddedIfNeeded()方法，向下執行，會進入到DefaultChannelPipeline這個類中的callHandlerAddedForAllHandlers方法中
private void callHandlerAddedForAllHandlers() {
final PendingHandlerCallback pendingHandlerCallbackHead;
synchronized (this) {
assert !registered;
// This Channel itself was registered.
registered = true;
pendingHandlerCallbackHead = this.pendingHandlerCallbackHead;
// Null out so it can be GC'ed.
this.pendingHandlerCallbackHead = null;
}
//從等待被呼叫的handler 回撥列表中，取出任務來執行。
PendingHandlerCallback task = pendingHandlerCallbackHead;
while (task != null) {
task.execute();
task = task.next;
}
}
我們發現，pendingHandlerCallbackHead這個單向連結串列，是在callHandlerCallbackLater方法中被新增的，
而callHandlerCallbackLater又是在addLast方法中新增的，所以構成了一個非同步完整的閉環。
ChannelInitializer.handlerAdded
task.execute()方法執行路徑是
callHandlerAdded0 -> ctx.callHandlerAdded ->
-------> AbstractChannelHandlerContext.callHandlerAddded()
---------------> ChannelInitializer.handlerAdded
呼叫initChannel方法來初始化NioSocketChannel中的Channel.
@Override
public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
if (ctx.channel().isRegistered()) {
// This should always be true with our current DefaultChannelPipeline implementation.
// The good thing about calling initChannel(...) in handlerAdded(...) is that there will be no ordering
// surprises if a ChannelInitializer will add another ChannelInitializer. This is as all handlers
// will be added in the expected order.
if (initChannel(ctx)) {
// We are done with init the Channel, removing the initializer now.
removeState(ctx);
}
}
}
接著，呼叫initChannel抽象方法，該方法由具體的實現類來完成。
private boolean initChannel(ChannelHandlerContext ctx) throws Exception {
if (initMap.add(ctx)) { // Guard against re-entrance.
try {
initChannel((C) ctx.channel());
} catch (Throwable cause) {
// Explicitly call exceptionCaught(...) as we removed the handler before calling initChannel(...).
// We do so to prevent multiple calls to initChannel(...).
exceptionCaught(ctx, cause);
} finally {
ChannelPipeline pipeline = ctx.pipeline();
if (pipeline.context(this) != null) {
pipeline.remove(this);
}
}
return true;
}
return false;
}
ChannelInitializer的實現，是我們自定義Server中的匿名內部類，ChannelInitializer。因此通過這個回撥來完成當前NioSocketChannel的pipeline的構建過程。
public static void main(String[] args){
EventLoopGroup boss = new NioEventLoopGroup();
//2 用於對接受客戶端連線讀寫操作的執行緒工作組
EventLoopGroup work = new NioEventLoopGroup();
ServerBootstrap b = new ServerBootstrap();
b.group(boss, work) //繫結兩個工作執行緒組
.channel(NioServerSocketChannel.class) //設定NIO的模式
// 初始化繫結服務通道
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel sc) throws Exception {
sc.pipeline()
.addLast(
new LengthFieldBasedFrameDecoder(1024,
9,4,0,0))
.addLast(new MessageRecordEncoder())
.addLast(new MessageRecordDecode())
.addLast(new ServerHandler());
}
});
}
版權宣告：本部落格所有文章除特別宣告外，均採用 CC BY-NC-SA 4.0 許可協議。轉載請註明來自
Mic帶你學架構！
如果本篇文章對您有幫助，還請幫忙點個關注和贊，您的堅持是我不斷創作的動力。歡迎關注「跟著Mic學架構」公眾號公眾號獲取更多技術乾貨！
