序
本文主要研究下reactor-netty的PoolResources的兩種模式elastic及fixed。
LoopResources與PoolResources
TcpResources是個工具類，可以用來建立loopResources和poolResources。
loopResources
主要是建立NioEventLoopGroup，以及該group下面的workerCount個NioEventLoop(這裡涉及兩個引數，一個是worker thread count，一個是selector thread count)
- DEFAULT_IO_WORKER_COUNT：如果環境變數有設定reactor.ipc.netty.workerCount，則用該值；沒有設定則取Math.max(Runtime.getRuntime().availableProcessors(), 4)))
- DEFAULT_IO_SELECT_COUNT：如果環境變數有設定reactor.ipc.netty.selectCount，則用該值；沒有設定則取-1，表示沒有selector thread
- DEFAULT_MAX_PENDING_TASKS: 指定NioEventLoop的taskQueue的大小，Math.max(16,SystemPropertyUtil.getInt("io.netty.eventLoop.maxPendingTasks", Integer.MAX_VALUE))
- NioEventLoop繼承了SingleThreadEventLoop，而SingleThreadEventLoop則繼承SingleThreadEventExecutor，而其代理的executor是ThreadPerTaskExecutor，rejectHandler是RejectedExecutionHandlers.reject()，預設的taskQueue是LinkedBlockingQueue，其大小為Integer.MAX_VALUE
poolResources
主要是建立channelPools，型別是ConcurrentMap<SocketAddress, Pool>，這裡主要研究下它的兩種模式elastic及fixed
DefaultPoolResources
reactor-netty-0.7.5.RELEASE-sources.jar!/reactor/ipc/netty/resources/DefaultPoolResources.java
它實現了netty-transport-4.1.22.Final-sources.jar!/io/netty/channel/pool/ChannelPool.java的介面，重點看如下的幾個方法：
@Override
public Future<Channel> acquire() {
return acquire(defaultGroup.next().newPromise());
}
@Override
public Future<Channel> acquire(Promise<Channel> promise) {
return pool.acquire(promise).addListener(this);
}
@Override
public Future<Void> release(Channel channel) {
return pool.release(channel);
}
@Override
public Future<Void> release(Channel channel, Promise<Void> promise) {
return pool.release(channel, promise);
}
@Override
public void close() {
if(compareAndSet(false, true)) {
pool.close();
}
}
複製程式碼這裡的幾個介面基本是委託為具體的pool來進行操作，其實現主要有SimpleChannelPool及FixedChannelPool。
PoolResources.elastic(SimpleChannelPool)
reactor-netty-0.7.5.RELEASE-sources.jar!/reactor/ipc/netty/resources/PoolResources.java
/**
* Create an uncapped {@link PoolResources} to provide automatically for {@link
* ChannelPool}.
* <p>An elastic {@link PoolResources} will never wait before opening a new
* connection. The reuse window is limited but it cannot starve an undetermined volume
* of clients using it.
*
* @param name the channel pool map name
*
* @return a new {@link PoolResources} to provide automatically for {@link
* ChannelPool}
*/
static PoolResources elastic(String name) {
return new DefaultPoolResources(name, SimpleChannelPool::new);
}
複製程式碼這個是TcpClient.create過程中，預設使用的方法，預設使用的是SimpleChannelPool，建立的是DefaultPoolResources
reactor-netty-0.7.5.RELEASE-sources.jar!/reactor/ipc/netty/tcp/TcpResources.java
static <T extends TcpResources> T create(T previous,
LoopResources loops,
PoolResources pools,
String name,
BiFunction<LoopResources, PoolResources, T> onNew) {
if (previous == null) {
loops = loops == null ? LoopResources.create("reactor-" + name) : loops;
pools = pools == null ? PoolResources.elastic(name) : pools;
}
else {
loops = loops == null ? previous.defaultLoops : loops;
pools = pools == null ? previous.defaultPools : pools;
}
return onNew.apply(loops, pools);
}
複製程式碼SimpleChannelPool
netty-transport-4.1.22.Final-sources.jar!/io/netty/channel/pool/SimpleChannelPool.java
/**
* Simple {@link ChannelPool} implementation which will create new {@link Channel}s if someone tries to acquire
* a {@link Channel} but none is in the pool atm. No limit on the maximal concurrent {@link Channel}s is enforced.
*
* This implementation uses LIFO order for {@link Channel}s in the {@link ChannelPool}.
*
*/
public class SimpleChannelPool implements ChannelPool {
@Override
public final Future<Channel> acquire() {
return acquire(bootstrap.config().group().next().<Channel>newPromise());
}
@Override
public Future<Channel> acquire(final Promise<Channel> promise) {
checkNotNull(promise, "promise");
return acquireHealthyFromPoolOrNew(promise);
}
/**
* Tries to retrieve healthy channel from the pool if any or creates a new channel otherwise.
* @param promise the promise to provide acquire result.
* @return future for acquiring a channel.
*/
private Future<Channel> acquireHealthyFromPoolOrNew(final Promise<Channel> promise) {
try {
final Channel ch = pollChannel();
if (ch == null) {
// No Channel left in the pool bootstrap a new Channel
Bootstrap bs = bootstrap.clone();
bs.attr(POOL_KEY, this);
ChannelFuture f = connectChannel(bs);
if (f.isDone()) {
notifyConnect(f, promise);
} else {
f.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
notifyConnect(future, promise);
}
});
}
return promise;
}
EventLoop loop = ch.eventLoop();
if (loop.inEventLoop()) {
doHealthCheck(ch, promise);
} else {
loop.execute(new Runnable() {
@Override
public void run() {
doHealthCheck(ch, promise);
}
});
}
} catch (Throwable cause) {
promise.tryFailure(cause);
}
return promise;
}
@Override
public final Future<Void> release(Channel channel) {
return release(channel, channel.eventLoop().<Void>newPromise());
}
@Override
public Future<Void> release(final Channel channel, final Promise<Void> promise) {
checkNotNull(channel, "channel");
checkNotNull(promise, "promise");
try {
EventLoop loop = channel.eventLoop();
if (loop.inEventLoop()) {
doReleaseChannel(channel, promise);
} else {
loop.execute(new Runnable() {
@Override
public void run() {
doReleaseChannel(channel, promise);
}
});
}
} catch (Throwable cause) {
closeAndFail(channel, cause, promise);
}
return promise;
}
@Override
public void close() {
for (;;) {
Channel channel = pollChannel();
if (channel == null) {
break;
}
channel.close();
}
}
//......
}
複製程式碼這個連線池的實現如果沒有連線則會建立一個(
沒有限制)，取出連線(連線池使用一個LIFO的Deque來維護Channel)的時候會檢測連線的有效性。
PoolResources.fixed(FixedChannelPool)
reactor-netty-0.7.5.RELEASE-sources.jar!/reactor/ipc/netty/resources/PoolResources.java
/**
* Default max connection, if -1 will never wait to acquire before opening new
* connection in an unbounded fashion. Fallback to
* available number of processors.
*/
int DEFAULT_POOL_MAX_CONNECTION =
Integer.parseInt(System.getProperty("reactor.ipc.netty.pool.maxConnections",
"" + Math.max(Runtime.getRuntime()
.availableProcessors(), 8) * 2));
/**
* Default acquisition timeout before error. If -1 will never wait to
* acquire before opening new
* connection in an unbounded fashion. Fallback to
* available number of processors.
*/
long DEFAULT_POOL_ACQUIRE_TIMEOUT = Long.parseLong(System.getProperty(
"reactor.ipc.netty.pool.acquireTimeout",
"" + 45000));
/**
* Create a capped {@link PoolResources} to provide automatically for {@link
* ChannelPool}.
* <p>A Fixed {@link PoolResources} will open up to the given max number of
* processors observed by this jvm (minimum 4).
* Further connections will be pending acquisition indefinitely.
*
* @param name the channel pool map name
*
* @return a new {@link PoolResources} to provide automatically for {@link
* ChannelPool}
*/
static PoolResources fixed(String name) {
return fixed(name, DEFAULT_POOL_MAX_CONNECTION);
}
/**
* Create a capped {@link PoolResources} to provide automatically for {@link
* ChannelPool}.
* <p>A Fixed {@link PoolResources} will open up to the given max connection value.
* Further connections will be pending acquisition indefinitely.
*
* @param name the channel pool map name
* @param maxConnections the maximum number of connections before starting pending
* acquisition on existing ones
*
* @return a new {@link PoolResources} to provide automatically for {@link
* ChannelPool}
*/
static PoolResources fixed(String name, int maxConnections) {
return fixed(name, maxConnections, DEFAULT_POOL_ACQUIRE_TIMEOUT);
}
/**
* Create a capped {@link PoolResources} to provide automatically for {@link
* ChannelPool}.
* <p>A Fixed {@link PoolResources} will open up to the given max connection value.
* Further connections will be pending acquisition indefinitely.
*
* @param name the channel pool map name
* @param maxConnections the maximum number of connections before starting pending
* @param acquireTimeout the maximum time in millis to wait for aquiring
*
* @return a new {@link PoolResources} to provide automatically for {@link
* ChannelPool}
*/
static PoolResources fixed(String name, int maxConnections, long acquireTimeout) {
if (maxConnections == -1) {
return elastic(name);
}
if (maxConnections <= 0) {
throw new IllegalArgumentException("Max Connections value must be strictly " + "positive");
}
if (acquireTimeout != -1L && acquireTimeout < 0) {
throw new IllegalArgumentException("Acquire Timeout value must " + "be " + "positive");
}
return new DefaultPoolResources(name,
(bootstrap, handler, checker) -> new FixedChannelPool(bootstrap,
handler,
checker,
FixedChannelPool.AcquireTimeoutAction.FAIL,
acquireTimeout,
maxConnections,
Integer.MAX_VALUE
));
}
複製程式碼最後呼叫的fixed方法有三個引數，一個是name，一個是maxConnections，一個是acquireTimeout。可以看到這裡建立的是FixedChannelPool。
FixedChannelPool
netty-transport-4.1.22.Final-sources.jar!/io/netty/channel/pool/FixedChannelPool.java
/**
* {@link ChannelPool} implementation that takes another {@link ChannelPool} implementation and enforce a maximum
* number of concurrent connections.
*/
public class FixedChannelPool extends SimpleChannelPool {
@Override
public Future<Channel> acquire(final Promise<Channel> promise) {
try {
if (executor.inEventLoop()) {
acquire0(promise);
} else {
executor.execute(new Runnable() {
@Override
public void run() {
acquire0(promise);
}
});
}
} catch (Throwable cause) {
promise.setFailure(cause);
}
return promise;
}
private void acquire0(final Promise<Channel> promise) {
assert executor.inEventLoop();
if (closed) {
promise.setFailure(POOL_CLOSED_ON_ACQUIRE_EXCEPTION);
return;
}
if (acquiredChannelCount < maxConnections) {
assert acquiredChannelCount >= 0;
// We need to create a new promise as we need to ensure the AcquireListener runs in the correct
// EventLoop
Promise<Channel> p = executor.newPromise();
AcquireListener l = new AcquireListener(promise);
l.acquired();
p.addListener(l);
super.acquire(p);
} else {
if (pendingAcquireCount >= maxPendingAcquires) {
promise.setFailure(FULL_EXCEPTION);
} else {
AcquireTask task = new AcquireTask(promise);
if (pendingAcquireQueue.offer(task)) {
++pendingAcquireCount;
if (timeoutTask != null) {
task.timeoutFuture = executor.schedule(timeoutTask, acquireTimeoutNanos, TimeUnit.NANOSECONDS);
}
} else {
promise.setFailure(FULL_EXCEPTION);
}
}
assert pendingAcquireCount > 0;
}
}
@Override
public Future<Void> release(final Channel channel, final Promise<Void> promise) {
ObjectUtil.checkNotNull(promise, "promise");
final Promise<Void> p = executor.newPromise();
super.release(channel, p.addListener(new FutureListener<Void>() {
@Override
public void operationComplete(Future<Void> future) throws Exception {
assert executor.inEventLoop();
if (closed) {
// Since the pool is closed, we have no choice but to close the channel
channel.close();
promise.setFailure(POOL_CLOSED_ON_RELEASE_EXCEPTION);
return;
}
if (future.isSuccess()) {
decrementAndRunTaskQueue();
promise.setSuccess(null);
} else {
Throwable cause = future.cause();
// Check if the exception was not because of we passed the Channel to the wrong pool.
if (!(cause instanceof IllegalArgumentException)) {
decrementAndRunTaskQueue();
}
promise.setFailure(future.cause());
}
}
}));
return promise;
}
@Override
public void close() {
executor.execute(new Runnable() {
@Override
public void run() {
if (!closed) {
closed = true;
for (;;) {
AcquireTask task = pendingAcquireQueue.poll();
if (task == null) {
break;
}
ScheduledFuture<?> f = task.timeoutFuture;
if (f != null) {
f.cancel(false);
}
task.promise.setFailure(new ClosedChannelException());
}
acquiredChannelCount = 0;
pendingAcquireCount = 0;
FixedChannelPool.super.close();
}
}
});
}
//......
}
複製程式碼這裡的acquire，如果當前執行緒不是在eventLoop中，則放入佇列中等待執行acquire0，這裡可能撐爆eventLoop的taskQueue，不過其佇列大小的值取Math.max(16,SystemPropertyUtil.getInt("io.netty.eventLoop.maxPendingTasks", Integer.MAX_VALUE))，預設是Integer.MAX_VALUE。
FixedChannelPool繼承了SimpleChannelPool，並重寫了acquire、release、close方法。它對獲取連線進行了限制，主要有如下幾個引數：
-
maxConnections 該值先從系統變數reactor.ipc.netty.pool.maxConnections取(
如果設定為-1，表示無限制，回到elastic模式)，如果沒有設定，則取Math.max(Runtime.getRuntime().availableProcessors(), 8) * 2，即核數與8的最大值的2倍。 -
acquireTimeout 該值先從系統變數reactor.ipc.netty.pool.acquireTimeout取(
如果設定為-1，表示立即執行不等待)，如果沒有設定，則為45000毫秒 -
maxPendingAcquires 這裡設定的是Integer.MAX_VALUE
-
AcquireTimeoutAction 這裡設定為FixedChannelPool.AcquireTimeoutAction.FAIL，即timeoutTask為
timeoutTask = new TimeoutTask() {
@Override
public void onTimeout(AcquireTask task) {
// Fail the promise as we timed out.
task.promise.setFailure(TIMEOUT_EXCEPTION);
}
};
複製程式碼如果當前連線超過maxConnections，則進入pendingAcquireQueue等待獲取連線，而在進入pendingAcquireQueue之前，如果當前等待數量超過了maxPendingAcquires，則返回FULL_EXCEPTION(
Too many outstanding acquire operations)，這裡設定的是Integer.MAX_VALUE，所以不會有這個異常。進入pendingAcquireQueue之後，還有一個acquireTimeout引數，即進入pendingAcquireQueue等待acquireTimeout時間，如果還沒有獲取到連線則返回TIMEOUT_EXCEPTION(Acquire operation took longer then configured maximum time)。
小結
預設TcpClient建立的PoolResources使用的是elastic模式，即連線池的實現是SimpleChannelPool，預設使用一個LIFO的Deque來維護Channel，如果從連線池取不到連線則會建立新的連線，上限應該是系統設定的能夠開啟的檔案資源數量，超過則報SocketException: Too many open files。PoolResources還提供了FixedChannelPool實現，使用的是fixed模式，即限定了連線池最大連線數及最大等待超時，避免連線建立數量過多撐爆記憶體或者報SocketException: Too many open files異常。
注意，對於fixed模式，如果reactor.ipc.netty.pool.maxConnections設定為-1，則回退到elastic模式。