前面兩篇文章Netty原始碼分析之NioEventLoop(一)—NioEventLoop的建立與Netty原始碼分析之NioEventLoop(二)—NioEventLoop的啟動中我們對NioEventLoop的建立與啟動做了具體的分析,本篇文章中我們會對NioEventLoop的具體執行內容進行分析;
從之前的程式碼中我們可以知道NioEventLoop的執行都是在run()方法的for迴圈中完成的
@Override protected void run() { //迴圈處理IO事件和task任務 for (;;) { try { try { switch (selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())) { case SelectStrategy.CONTINUE: continue; case SelectStrategy.BUSY_WAIT: // fall-through to SELECT since the busy-wait is not supported with NIO case SelectStrategy.SELECT: //通過cas操作標識select方法的喚醒狀態,執行select操作 select(wakenUp.getAndSet(false)); // 'wakenUp.compareAndSet(false, true)' is always evaluated // before calling 'selector.wakeup()' to reduce the wake-up // overhead. (Selector.wakeup() is an expensive operation.) // // However, there is a race condition in this approach. // The race condition is triggered when 'wakenUp' is set to // true too early. // // 'wakenUp' is set to true too early if: // 1) Selector is waken up between 'wakenUp.set(false)' and // 'selector.select(...)'. (BAD) // 2) Selector is waken up between 'selector.select(...)' and // 'if (wakenUp.get()) { ... }'. (OK) // // In the first case, 'wakenUp' is set to true and the // following 'selector.select(...)' will wake up immediately. // Until 'wakenUp' is set to false again in the next round, // 'wakenUp.compareAndSet(false, true)' will fail, and therefore // any attempt to wake up the Selector will fail, too, causing // the following 'selector.select(...)' call to block // unnecessarily. // // To fix this problem, we wake up the selector again if wakenUp // is true immediately after selector.select(...). // It is inefficient in that it wakes up the selector for both // the first case (BAD - wake-up required) and the second case // (OK - no wake-up required). if (wakenUp.get()) { selector.wakeup(); } // fall through default: } } catch (IOException e) { // If we receive an IOException here its because the Selector is messed up. Let's rebuild // the selector and retry. https://github.com/netty/netty/issues/8566 rebuildSelector0(); handleLoopException(e); continue; } cancelledKeys = 0; needsToSelectAgain = false; // 這個比例是處理IO事件所需的時間和花費在處理task時間的比例 final int ioRatio = this.ioRatio; if (ioRatio == 100) { try { processSelectedKeys(); } finally { // Ensure we always run tasks. runAllTasks(); } } else { //IO處理的開始時間 final long ioStartTime = System.nanoTime(); try { //處理IO事件的函式 processSelectedKeys(); } finally { // Ensure we always run tasks. // 當前時間減去處理IO事件開始的時間就是處理IO事件花費的時間 final long ioTime = System.nanoTime() - ioStartTime; //執行任務方法 runAllTasks(ioTime * (100 - ioRatio) / ioRatio); } } } catch (Throwable t) { handleLoopException(t); } // Always handle shutdown even if the loop processing threw an exception. try { //判斷執行緒池是否shutdown,關閉的話釋放所有資源 if (isShuttingDown()) { closeAll(); if (confirmShutdown()) { return; } } } catch (Throwable t) { handleLoopException(t); } } }
通過上面的程式碼我們把NioEventLoop中run方法主要歸納為以下三個功能:
1、通過select()檢測IO事件;
2、通過processSelectedKeys()處理IO事件;
3、runAllTasks()處理執行緒任務佇列;
接下來我們就從這三個方面入手,對NioEventLoop的具體執行程式碼進行解析。
一、檢測IO事件
IO事件的監測,主要通過 select(wakenUp.getAndSet(false) 方法實現,具體的程式碼分析如下:
private void select(boolean oldWakenUp) throws IOException { Selector selector = this.selector; try { int selectCnt = 0; long currentTimeNanos = System.nanoTime(); //定義select操作的截止時間 long selectDeadLineNanos = currentTimeNanos + delayNanos(currentTimeNanos); for (;;) { //計算當前select操作是否超時 long timeoutMillis = (selectDeadLineNanos - currentTimeNanos + 500000L) / 1000000L; if (timeoutMillis <= 0) { //如果已經超時,且沒有執行過輪詢操作,則執行selectNow()非阻塞操作,直接跳出迴圈 if (selectCnt == 0) { selector.selectNow(); selectCnt = 1; } break; } // If a task was submitted when wakenUp value was true, the task didn't get a chance to call // Selector#wakeup. So we need to check task queue again before executing select operation. // If we don't, the task might be pended until select operation was timed out. // It might be pended until idle timeout if IdleStateHandler existed in pipeline. if (hasTasks() && wakenUp.compareAndSet(false, true)) { //如果有需要任務佇列,同樣跳出迴圈 selector.selectNow(); selectCnt = 1; break; } //執行select阻塞操作,其阻塞時間為timeoutMillis int selectedKeys = selector.select(timeoutMillis); selectCnt ++; if (selectedKeys != 0 || oldWakenUp || wakenUp.get() || hasTasks() || hasScheduledTasks()) { // - Selected something, // - waken up by user, or // - the task queue has a pending task. // - a scheduled task is ready for processing //1、如果輪詢到select事件 2、wekup為true,表示輪詢執行緒已經被喚醒 3、任務佇列不為空 4、定時任務對佇列有任務 //上述條件只要滿足一個就跳出select迴圈 break; } if (Thread.interrupted()) { // Thread was interrupted so reset selected keys and break so we not run into a busy loop. // As this is most likely a bug in the handler of the user or it's client library we will // also log it. // // See https://github.com/netty/netty/issues/2426 if (logger.isDebugEnabled()) { logger.debug("Selector.select() returned prematurely because " + "Thread.currentThread().interrupt() was called. Use " + "NioEventLoop.shutdownGracefully() to shutdown the NioEventLoop."); } //執行緒被中斷,同樣跳出select迴圈 selectCnt = 1; break; } long time = System.nanoTime();//記錄當前時間 //如果 當前時間-超時時間>起始時間 也就是 當前時間-起始時間>超時時間 if (time - TimeUnit.MILLISECONDS.toNanos(timeoutMillis) >= currentTimeNanos) { // timeoutMillis elapsed without anything selected. //滿足條件,則是一次正常的select操作,否則就是一次空輪詢操作 selectCnt = 1; } else if (SELECTOR_AUTO_REBUILD_THRESHOLD > 0 && selectCnt >= SELECTOR_AUTO_REBUILD_THRESHOLD) { // The code exists in an extra method to ensure the method is not too big to inline as this // branch is not very likely to get hit very frequently. //如果輪詢次數大於SELECTOR_AUTO_REBUILD_THRESHOLD,則對當前selector進行處理 //執行selectRebuildSelector操作,把當前selector的selectedKeys註冊到一個新的selector上 selector = selectRebuildSelector(selectCnt); selectCnt = 1; break; } currentTimeNanos = time; } if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS) { if (logger.isDebugEnabled()) { logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.", selectCnt - 1, selector); } } } catch (CancelledKeyException e) { if (logger.isDebugEnabled()) { logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector {} - JDK bug?", selector, e); } // Harmless exception - log anyway } }
Netty通過建立一個新的selector,且把原有selector上的SelectionKey同步到新的selector上的方式,解決了selector空輪詢的bug,我們看下具體的程式碼實現
private Selector selectRebuildSelector(int selectCnt) throws IOException { // The selector returned prematurely many times in a row. // Rebuild the selector to work around the problem. logger.warn( "Selector.select() returned prematurely {} times in a row; rebuilding Selector {}.", selectCnt, selector); //重新建立一個Selector rebuildSelector(); Selector selector = this.selector; // Select again to populate selectedKeys. selector.selectNow(); return selector; }
public void rebuildSelector() { if (!inEventLoop()) { //如果當前執行緒和NioEventLoop繫結的執行緒不一致 execute(new Runnable() { @Override public void run() { rebuildSelector0(); } }); return; } //具體實現 rebuildSelector0(); }
private void rebuildSelector0() { final Selector oldSelector = selector; final SelectorTuple newSelectorTuple; if (oldSelector == null) { return; } try { //建立一個新的selector newSelectorTuple = openSelector(); } catch (Exception e) { logger.warn("Failed to create a new Selector.", e); return; } // Register all channels to the new Selector. int nChannels = 0; for (SelectionKey key: oldSelector.keys()) { //遍歷oldSelector上所有的SelectionKey Object a = key.attachment(); try { if (!key.isValid() || key.channel().keyFor(newSelectorTuple.unwrappedSelector) != null) { continue; } int interestOps = key.interestOps(); key.cancel();//取消原有的SelectionKey上的事件 //在channel上註冊新的Selector SelectionKey newKey = key.channel().register(newSelectorTuple.unwrappedSelector, interestOps, a); if (a instanceof AbstractNioChannel) { // Update SelectionKey //把新的SelectionKey賦給AbstractNioChannel ((AbstractNioChannel) a).selectionKey = newKey; } nChannels ++; } catch (Exception e) { logger.warn("Failed to re-register a Channel to the new Selector.", e); if (a instanceof AbstractNioChannel) { AbstractNioChannel ch = (AbstractNioChannel) a; ch.unsafe().close(ch.unsafe().voidPromise()); } else { @SuppressWarnings("unchecked") NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a; invokeChannelUnregistered(task, key, e); } } } //替換新的selector selector = newSelectorTuple.selector; unwrappedSelector = newSelectorTuple.unwrappedSelector; try { // time to close the old selector as everything else is registered to the new one oldSelector.close(); } catch (Throwable t) { if (logger.isWarnEnabled()) { logger.warn("Failed to close the old Selector.", t); } } if (logger.isInfoEnabled()) { logger.info("Migrated " + nChannels + " channel(s) to the new Selector."); } }
通過上面的程式碼可以看到,NioEventLoop的select方法主要實現三方面的功能
1、結合超時時間、任務佇列及select本身喚醒狀態進行是否跳出for(;;)迴圈的邏輯判斷;
2、進行select阻塞操作,selector檢測到事則跳出for(;;)迴圈;
3、通過建立一個新的selector的方式解決selector空輪詢的bug問題;
二、處理IO事件
IO事件處理的核心方法是processSelectedKeys(),看下其程式碼的具體實現
private void processSelectedKeys() { //NioEventLoop的建構函式中通過openSelector()方法初始化selectedKeys,並賦給對應的selector if (selectedKeys != null) { //selectedKeys不為空 processSelectedKeysOptimized(); } else { processSelectedKeysPlain(selector.selectedKeys()); } }
如果selectedKeys不為空,也就是檢測到註冊的IO事件,則執行processSelectedKeysOptimized()方法
private void processSelectedKeysOptimized() { //遍歷selectedKeys for (int i = 0; i < selectedKeys.size; ++i) { final SelectionKey k = selectedKeys.keys[i]; // null out entry in the array to allow to have it GC'ed once the Channel close // See https://github.com/netty/netty/issues/2363 selectedKeys.keys[i] = null; //拿到SelectionKey對應的channel final Object a = k.attachment(); if (a instanceof AbstractNioChannel) { //如果是AbstractNioChannel的物件,執行processSelectedKey processSelectedKey(k, (AbstractNioChannel) a); } else { //否則轉換為一個NioTask物件 @SuppressWarnings("unchecked") NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a; processSelectedKey(k, task); } if (needsToSelectAgain) { // null out entries in the array to allow to have it GC'ed once the Channel close // See https://github.com/netty/netty/issues/2363 selectedKeys.reset(i + 1); selectAgain(); i = -1; } } }
通過遍歷selectedKeys,拿到所有觸發IO事件的SelectionKey與其對應Channel,然後交給processSelectedKey()方法處理
private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) { //拿到該AbstractNioChannel的unsafe物件 final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe(); if (!k.isValid()) { final EventLoop eventLoop; try { //拿到繫結的eventLoop eventLoop = ch.eventLoop(); } catch (Throwable ignored) { // If the channel implementation throws an exception because there is no event loop, we ignore this // because we are only trying to determine if ch is registered to this event loop and thus has authority // to close ch. return; } // Only close ch if ch is still registered to this EventLoop. ch could have deregistered from the event loop // and thus the SelectionKey could be cancelled as part of the deregistration process, but the channel is // still healthy and should not be closed. // See https://github.com/netty/netty/issues/5125 if (eventLoop != this || eventLoop == null) { return; } // close the channel if the key is not valid anymore unsafe.close(unsafe.voidPromise()); return; } try { //針對檢測到的IO事件進行處理 int readyOps = k.readyOps(); // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise // the NIO JDK channel implementation may throw a NotYetConnectedException. if ((readyOps & SelectionKey.OP_CONNECT) != 0) { // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking // See https://github.com/netty/netty/issues/924 int ops = k.interestOps(); ops &= ~SelectionKey.OP_CONNECT; k.interestOps(ops); unsafe.finishConnect(); } // Process OP_WRITE first as we may be able to write some queued buffers and so free memory. if ((readyOps & SelectionKey.OP_WRITE) != 0) { // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write ch.unsafe().forceFlush(); } // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead // to a spin loop //新連線接入事件 if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) { unsafe.read(); } } catch (CancelledKeyException ignored) { unsafe.close(unsafe.voidPromise()); } }
從上面的程式碼中可以看到NioEventLoop處理IO事件的流程中,會迴圈從SelectedSelectionKeySet中獲取觸發事件的SelectionKey,Netty在這裡對JDK中NIO的Selector進行了優化,在NioEventLoop建構函式中通過openSelector()方法用自定義的SelectedSelectionKeySet替代Selector原有的selectedKeys與publicSelectedKeys。
private SelectorTuple openSelector() { final Selector unwrappedSelector; try { //建立一個Selector unwrappedSelector = provider.openSelector(); } catch (IOException e) { throw new ChannelException("failed to open a new selector", e); } if (DISABLE_KEY_SET_OPTIMIZATION) { return new SelectorTuple(unwrappedSelector); } //通過反射的方式獲取 sun.nio.ch.SelectorImpl 類 Object maybeSelectorImplClass = AccessController.doPrivileged(new PrivilegedAction<Object>() { @Override public Object run() { try { return Class.forName( "sun.nio.ch.SelectorImpl", false, PlatformDependent.getSystemClassLoader()); } catch (Throwable cause) { return cause; } } }); if (!(maybeSelectorImplClass instanceof Class) || // ensure the current selector implementation is what we can instrument. !((Class<?>) maybeSelectorImplClass).isAssignableFrom(unwrappedSelector.getClass())) { if (maybeSelectorImplClass instanceof Throwable) { Throwable t = (Throwable) maybeSelectorImplClass; logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, t); } return new SelectorTuple(unwrappedSelector); } final Class<?> selectorImplClass = (Class<?>) maybeSelectorImplClass; //初始化一個SelectedSelectionKeySet物件 final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet(); Object maybeException = AccessController.doPrivileged(new PrivilegedAction<Object>() { @Override public Object run() { try { //獲取"sun.nio.ch.SelectorImpl"類的selectedKeys屬性 Field selectedKeysField = selectorImplClass.getDeclaredField("selectedKeys"); //獲取"sun.nio.ch.SelectorImpl"類的publicSelectedKeys屬性 Field publicSelectedKeysField = selectorImplClass.getDeclaredField("publicSelectedKeys"); if (PlatformDependent.javaVersion() >= 9 && PlatformDependent.hasUnsafe()) { // Let us try to use sun.misc.Unsafe to replace the SelectionKeySet. // This allows us to also do this in Java9+ without any extra flags. long selectedKeysFieldOffset = PlatformDependent.objectFieldOffset(selectedKeysField); long publicSelectedKeysFieldOffset = PlatformDependent.objectFieldOffset(publicSelectedKeysField); if (selectedKeysFieldOffset != -1 && publicSelectedKeysFieldOffset != -1) { //把selectedKeySet賦值給selectedKeys PlatformDependent.putObject( unwrappedSelector, selectedKeysFieldOffset, selectedKeySet); //把selectedKeySet賦值給publicSelectedKeys PlatformDependent.putObject( unwrappedSelector, publicSelectedKeysFieldOffset, selectedKeySet); return null; } // We could not retrieve the offset, lets try reflection as last-resort. } Throwable cause = ReflectionUtil.trySetAccessible(selectedKeysField, true); if (cause != null) { return cause; } cause = ReflectionUtil.trySetAccessible(publicSelectedKeysField, true); if (cause != null) { return cause; } selectedKeysField.set(unwrappedSelector, selectedKeySet); publicSelectedKeysField.set(unwrappedSelector, selectedKeySet); return null; } catch (NoSuchFieldException e) { return e; } catch (IllegalAccessException e) { return e; } } }); if (maybeException instanceof Exception) { selectedKeys = null; Exception e = (Exception) maybeException; logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, e); return new SelectorTuple(unwrappedSelector); } selectedKeys = selectedKeySet; logger.trace("instrumented a special java.util.Set into: {}", unwrappedSelector); return new SelectorTuple(unwrappedSelector, new SelectedSelectionKeySetSelector(unwrappedSelector, selectedKeySet)); }
SelectedSelectionKeySet 程式碼實現如下
final class SelectedSelectionKeySet extends AbstractSet<SelectionKey> { SelectionKey[] keys; int size; //建構函式中初始化一個1024長度的陣列 SelectedSelectionKeySet() { keys = new SelectionKey[1024]; } @Override public boolean add(SelectionKey o) { if (o == null) { return false; } //向陣列中新增元素 keys[size++] = o; if (size == keys.length) { //進行擴容 increaseCapacity(); } return true; } @Override public boolean remove(Object o) { return false; } @Override public boolean contains(Object o) { return false; } @Override public int size() { return size; } @Override public Iterator<SelectionKey> iterator() { return new Iterator<SelectionKey>() { private int idx; @Override public boolean hasNext() { return idx < size; } @Override public SelectionKey next() { if (!hasNext()) { throw new NoSuchElementException(); } return keys[idx++]; } @Override public void remove() { throw new UnsupportedOperationException(); } }; } void reset() { reset(0); } void reset(int start) { Arrays.fill(keys, start, size, null); size = 0; } private void increaseCapacity() { SelectionKey[] newKeys = new SelectionKey[keys.length << 1]; System.arraycopy(keys, 0, newKeys, 0, size); keys = newKeys; } }
這裡的優化點主要在於用底層為陣列實現的SelectedSelectionKeySet 代替HashSet型別的selectedKeys與publicSelectedKeys,因為HashSet的add方法最大的時間複雜度可能為O(n),而SelectedSelectionKeySet 主要就是用陣列實現一個基本的add方法,時間複雜度為O(1),在這一點上相比HashSet要簡單很多。
三、執行緒任務的執行
NioEventLoop中通過runAllTasks方法執行執行緒任務
protected boolean runAllTasks(long timeoutNanos) { //把需要執行的定時任務從scheduledTaskQueue轉移到taskQueue fetchFromScheduledTaskQueue(); Runnable task = pollTask(); if (task == null) { afterRunningAllTasks(); return false; } //計算截止時間 final long deadline = ScheduledFutureTask.nanoTime() + timeoutNanos; long runTasks = 0; long lastExecutionTime; for (;;) { //執行task任務 safeExecute(task); runTasks ++; // Check timeout every 64 tasks because nanoTime() is relatively expensive. // XXX: Hard-coded value - will make it configurable if it is really a problem. if ((runTasks & 0x3F) == 0) { //每執行64次任務,進行一次超時檢查 lastExecutionTime = ScheduledFutureTask.nanoTime(); if (lastExecutionTime >= deadline) { //如果超出最大執行時間就跳出迴圈 break; } } task = pollTask();//繼續獲取任務 if (task == null) { lastExecutionTime = ScheduledFutureTask.nanoTime(); break; } } afterRunningAllTasks(); this.lastExecutionTime = lastExecutionTime; return true; }
NioEventLoop中維護了兩組任務佇列,一種是普通的taskQueue,一種是定時任務scheduledTaskQueue,而runAllTasks()方法首先會把scheduledTaskQueue佇列中的定時任務轉移到taskQueue中,然後在截止時間內迴圈執行
private boolean fetchFromScheduledTaskQueue() { long nanoTime = AbstractScheduledEventExecutor.nanoTime(); //從定時任務佇列中拉取任務 Runnable scheduledTask = pollScheduledTask(nanoTime); while (scheduledTask != null) { //把獲取的scheduledTask插入taskQueue if (!taskQueue.offer(scheduledTask)) { // No space left in the task queue add it back to the scheduledTaskQueue so we pick it up again. scheduledTaskQueue().add((ScheduledFutureTask<?>) scheduledTask); return false; } scheduledTask = pollScheduledTask(nanoTime); } return true; }
定時任務佇列ScheduledFutureTask是個優先順序任務佇列,會根據截止時間與任務id,保證截止時間最近的任務優先執行
final class ScheduledFutureTask<V> extends PromiseTask<V> implements ScheduledFuture<V>, PriorityQueueNode
@Override public int compareTo(Delayed o) { if (this == o) { return 0; } ScheduledFutureTask<?> that = (ScheduledFutureTask<?>) o; long d = deadlineNanos() - that.deadlineNanos(); if (d < 0) { return -1; } else if (d > 0) { return 1; } else if (id < that.id) { return -1; } else if (id == that.id) { throw new Error(); } else { return 1; } }
四、總結
以上我們主要對NioEventLoop的執行進行了分析與彙總,核心run()方法主要完成了IO的檢測和處理,佇列任務執行等操作,在這裡從原始碼的角度對整個流程進行了解析與說明,其中有錯誤和不足之處還請指正與海涵。