[從原始碼學設計]螞蟻金服SOFARegistry 之 ChangeNotifier
0x00 摘要
SOFARegistry 是螞蟻金服開源的一個生產級、高時效、高可用的服務註冊中心。
本系列文章重點在於分析設計和架構，即利用多篇文章，從多個角度反推總結 DataServer 或者 SOFARegistry 的實現機制和架構思路，讓大家藉以學習阿里如何設計。
本文為第十五篇，分析如何執行ChangeNotifier 來通知相關模組：hi，這裡有新資料變化來到了，兄弟們走起來。
0x01 業務範疇
1.1 概述
當有資料釋出者 publisher 上下線時，會分別觸發 publishDataProcessor 或 unPublishDataHandler，Handler 會往 dataChangeEventCenter 中新增一個資料變更事件，用於非同步地通知事件變更中心資料的變更。事件變更中心收到該事件之後，會往佇列中加入事件。
-
此時 dataChangeEventCenter 會根據不同的事件型別非同步地對上下線資料進行相應的處理，即把這個事件變更資訊變成ChangeNotifier ，進而變成Operator，放到AbstractAcceptorStore；
-
與此同時 DataChangeHandler 會把這個事件變更資訊通過 ChangeNotifier 對外發布，通知其他節點進行資料同步。
因為篇幅限制，前文對 ChangeNotifier 這部分只是略過，本文就詳細講解下事件變更通知ChangeNotifier。 這裡會再把整理流程串起來，會涉及到前面某些文章內容。
先給出圖示以便大家瞭解 ChangeNotifier 的作用。
+--------------------+
| PublishDataHandler |
+--------+-----------+
|
|
| publisher
|
v
+---------+------------+
|DataChangeEventCenter |
+---------+------------+
|
|
| ChangeData
v
+---------+------------+
| DataChangeEventQueue |
+---------+------------+
|
|
| ChangeData
v
+-------+----------+
| DataChangeHandler|
+-------+----------+
|
|
| ChangeData
v
+------+--------+ +------------+
| ChangeNotifier| +--------> | datumCache |
+------+--------+ +------------+
|
|
v
+---+------+
| notifier |
+---+------+
|
v
+-----------+---------------+
| |
v v
+----+----------------+ +------+----------+
|SessionServerNotifier| | BackUpNotifier |
+----+----------------+ +------+----------+
| |
| |
| |
| v
+--v------------+ +------+----------------+
| sessionServer | | AbstractAcceptorStore |
+---------------+ +-----------------------+
1.2 資料變化
資料變化有兩個方向
-
資料伺服器節點變化；
-
資料的變化，即Publisher和Scriber的變化；
ChangeNotifier就是負責把 Publisher和Scriber的變化 通知給相關模組。變更通知就是一種解耦。
0x02 資料結構
我們首先需要看看通知的資料結構。
2.1 介面定義
IDataChangeNotifier是通知的介面定義：
public interface IDataChangeNotifier {
Set<DataSourceTypeEnum> getSuitableSource();
/**
*
* @param datum
* @param lastVersion
*/
void notify(Datum datum, Long lastVersion);
}
2.2 派生類
IDataChangeNotifier 有四個派生類，分別對應了具體資料變化的四種可能，從名字大約可以判斷出用途。
public class BackUpNotifier implements IDataChangeNotifier
public class SessionServerNotifier implements IDataChangeNotifier
public class SnapshotBackUpNotifier implements IDataChangeNotifier
public class TempPublisherNotifier implements IDataChangeNotifier
2.3 Bean
對應的Bean如下：
@Bean(name = "dataChangeNotifiers")
public List<IDataChangeNotifier> dataChangeNotifiers() {
List<IDataChangeNotifier> list = new ArrayList<>();
list.add(sessionServerNotifier());
list.add(tempPublisherNotifier());
list.add(backUpNotifier());
return list;
}
0x03 流程
我們從頭理一下流程。
3.1 放入訊息
當有資料釋出者 publisher 上下線時，會分別觸發 publishDataProcessor 或 unPublishDataHandler ，Handler 會往 dataChangeEventCenter 中新增一個資料變更事件，用於非同步地通知事件變更中心資料的變更。事件變更中心收到該事件之後，會往佇列中加入事件。
在DataServer這裡，具體流程如下：
3.1.1 PublishDataHandler
PublishDataHandler 響應 PublishDataRequest。當有Publisher時候，就往DataChangeEventCenter放入訊息。即呼叫下面來放入訊息
dataChangeEventCenter.onChange(publisher, dataServerConfig.getLocalDataCenter());
具體程式碼如下：
public class PublishDataHandler extends AbstractServerHandler<PublishDataRequest> {
@Autowired
private ForwardService forwardService;
@Autowired
private SessionServerConnectionFactory sessionServerConnectionFactory;
@Autowired
private DataChangeEventCenter dataChangeEventCenter;
@Autowired
private DataServerConfig dataServerConfig;
@Autowired
private DatumLeaseManager datumLeaseManager;
@Autowired
private ThreadPoolExecutor publishProcessorExecutor;
@Override
public Object doHandle(Channel channel, PublishDataRequest request) {
Publisher publisher = Publisher.internPublisher(request.getPublisher());
if (forwardService.needForward()) {
CommonResponse response = new CommonResponse();
response.setSuccess(false);
response.setMessage("Request refused, Server status is not working");
return response;
}
dataChangeEventCenter.onChange(publisher, dataServerConfig.getLocalDataCenter());
if (publisher.getPublishType() != PublishType.TEMPORARY) {
String connectId = WordCache.getInstance().getWordCache(
publisher.getSourceAddress().getAddressString());
sessionServerConnectionFactory.registerConnectId(request.getSessionServerProcessId(),
connectId);
// record the renew timestamp
datumLeaseManager.renew(connectId);
}
return CommonResponse.buildSuccessResponse();
}
}
此時具體邏輯如下：
+--------------------+
| PublishDataHandler |
+--------+-----------+
|
|
| publisher
|
v
+---------+------------+
|DataChangeEventCenter |
+---------+------------+
3.1.2 DataChangeEventCenter
DataChangeEventCenter 的核心是一個DataChangeEventQueue陣列，
DataChangeEventCenter . onChange函式會首先根據Publisher的DataInfoId獲取hash，根據這個hash數值來決定把 DataChangeEvent 訊息放入哪個queue來處理，就是呼叫這個 queue的 onChange 函式。
public class DataChangeEventCenter {
/**
* queues of DataChangeEvent
*/
private DataChangeEventQueue[] dataChangeEventQueues;
@Autowired
private DatumCache datumCache;
@PostConstruct
public void init() {
if (isInited.compareAndSet(false, true)) {
queueCount = dataServerConfig.getQueueCount();
dataChangeEventQueues = new DataChangeEventQueue[queueCount];
for (int idx = 0; idx < queueCount; idx++) {
dataChangeEventQueues[idx] = new DataChangeEventQueue(idx, dataServerConfig, this,
datumCache);
dataChangeEventQueues[idx].start();
}
}
}
/**
* receive changed publisher, then wrap it into the DataChangeEvent and put it into dataChangeEventQueue
*
* @param publisher
* @param dataCenter
*/
public void onChange(Publisher publisher, String dataCenter) {
int idx = hash(publisher.getDataInfoId());
Datum datum = new Datum(publisher, dataCenter);
if (publisher instanceof UnPublisher) {
datum.setContainsUnPub(true);
}
if (publisher.getPublishType() != PublishType.TEMPORARY) {
dataChangeEventQueues[idx].onChange(new DataChangeEvent(DataChangeTypeEnum.MERGE,
DataSourceTypeEnum.PUB, datum));
} else {
dataChangeEventQueues[idx].onChange(new DataChangeEvent(DataChangeTypeEnum.MERGE,
DataSourceTypeEnum.PUB_TEMP, datum));
}
}
}
DataChangeEventQueue 的主要資料成員如下：
public class DataChangeEventQueue {
/**
* a block queue that stores all data change events
*/
private final BlockingQueue<IDataChangeEvent> eventQueue;
private final Map<String, Map<String, ChangeData>> CHANGE_DATA_MAP_FOR_MERGE = new ConcurrentHashMap<>();
private final DelayQueue<ChangeData> CHANGE_QUEUE = new DelayQueue();
private DataChangeEventCenter dataChangeEventCenter;
private DatumCache datumCache;
}
其執行引擎是一個執行緒，其block在 BlockingQueue eventQueue 之上，當有訊息時候，就取出訊息，針對訊息型別做不同處理。
public void start() {
Executor executor = ExecutorFactory
.newSingleThreadExecutor(String.format("%s_%s", DataChangeEventQueue.class.getSimpleName(), getName()));
executor.execute(() -> {
while (true) {
try {
IDataChangeEvent event = eventQueue.take();
DataChangeScopeEnum scope = event.getScope();
if (scope == DataChangeScopeEnum.DATUM) {
DataChangeEvent dataChangeEvent = (DataChangeEvent) event;
//Temporary push data will be notify as soon as,and not merge to normal pub data;
if (dataChangeEvent.getSourceType() == DataSourceTypeEnum.PUB_TEMP) {
addTempChangeData(dataChangeEvent.getDatum(), dataChangeEvent.getChangeType(),
dataChangeEvent.getSourceType());
} else {
handleDatum(dataChangeEvent.getChangeType(), dataChangeEvent.getSourceType(),
dataChangeEvent.getDatum());
}
} else if (scope == DataChangeScopeEnum.CLIENT) {
handleClientOff((ClientChangeEvent) event);
} else if (scope == DataChangeScopeEnum.SNAPSHOT) {
handleSnapshot((DatumSnapshotEvent) event);
}
}
}
});
}
對於 Publisher 訊息型別，handleDatum 函式會根據changeType是 COVER 還是 MERGE 來做不同處理。
在此步驟中，也會把 ChangeData 放入 CHANGE_QUEUE.put(changeData);
private void handleDatum(DataChangeTypeEnum changeType, DataSourceTypeEnum sourceType,
Datum targetDatum) {
lock.lock();
try {
//get changed datum
ChangeData changeData = getChangeData(targetDatum.getDataCenter(),
targetDatum.getDataInfoId(), sourceType, changeType);
Datum cacheDatum = changeData.getDatum();
if (changeType == DataChangeTypeEnum.COVER || cacheDatum == null) {
changeData.setDatum(targetDatum);
} else {
Map<String, Publisher> targetPubMap = targetDatum.getPubMap();
Map<String, Publisher> cachePubMap = cacheDatum.getPubMap();
for (Publisher pub : targetPubMap.values()) {
String registerId = pub.getRegisterId();
Publisher cachePub = cachePubMap.get(registerId);
if (cachePub != null) {
// if the registerTimestamp of cachePub is greater than the registerTimestamp of pub, it means
// that pub is not the newest data, should be ignored
if (pub.getRegisterTimestamp() < cachePub.getRegisterTimestamp()) {
continue;
}
// if pub and cachePub both are publisher, and sourceAddress of both are equal,
// and version of cachePub is greater than version of pub, should be ignored
if (!(pub instanceof UnPublisher) && !(cachePub instanceof UnPublisher)
&& pub.getSourceAddress().equals(cachePub.getSourceAddress())
&& cachePub.getVersion() > pub.getVersion()) {
continue;
}
}
cachePubMap.put(registerId, pub);
cacheDatum.setVersion(targetDatum.getVersion());
}
}
} finally {
lock.unlock();
}
}
此時具體邏輯如下：
+--------------------+
| PublishDataHandler |
+--------+-----------+
|
|
| publisher
|
v
+---------+------------+
|DataChangeEventCenter |
+---------+------------+
|
|
| ChangeData
v
+---------+------------+
| DataChangeEventQueue |
+---------+------------+
3.2 消費訊息&傳送通知
DataChangeHandler 會針對每個DataChangeEventQueue進行消費通知。
public class DataChangeHandler {
@Autowired
private DataChangeEventCenter dataChangeEventCenter;
@Autowired
private DatumCache datumCache;
@Resource
private List<IDataChangeNotifier> dataChangeNotifiers;
@PostConstruct
public void start() {
DataChangeEventQueue[] queues = dataChangeEventCenter.getQueues();
int queueCount = queues.length;
Executor executor = ExecutorFactory.newFixedThreadPool(queueCount, DataChangeHandler.class.getSimpleName());
Executor notifyExecutor = ExecutorFactory
.newFixedThreadPool(dataServerConfig.getQueueCount() * 5, this.getClass().getSimpleName());
for (int idx = 0; idx < queueCount; idx++) {
final DataChangeEventQueue dataChangeEventQueue = queues[idx];
final String name = dataChangeEventQueue.getName();
executor.execute(() -> {
while (true) {
try {
final ChangeData changeData = dataChangeEventQueue.take();
notifyExecutor.execute(new ChangeNotifier(changeData, name));
}
}
});
}
}
}
3.2.1 DataChangeHandler
DataChangeHandler 會定期提取DataChangeEventCenter中的訊息，然後進行處理。
3.2.2 類定義
public class DataChangeHandler {
@Autowired
private DataServerConfig dataServerConfig;
@Autowired
private DataChangeEventCenter dataChangeEventCenter;
@Autowired
private DatumCache datumCache;
@Resource
private List<IDataChangeNotifier> dataChangeNotifiers;
}
3.2.3 執行引擎
這裡是一個雙層執行緒模型。
-
executor = ExecutorFactory.newFixedThreadPool(queueCount)
-
notifyExecutor= ExecutorFactory.newFixedThreadPool(dataServerConfig.getQueueCount() * 5)
可以認為 executor 是控制執行緒，notifierExecutor是工作執行緒，工作執行緒是控制執行緒的5倍。
- DataChangeHandler 會遍歷 DataChangeEventCenter 中所有 DataChangeEventQueue，
- 針對每一個dataChangeEventQueue呼叫executor的一個控制執行緒，
- 在這個控制執行緒裡面，可以從 DataChangeEventQueue 之中取出ChangeData，針對每一個ChangeData，呼叫notifyExecutor的一個工作執行緒，生成一個ChangeNotifier進行處理。
@PostConstruct
public void start() {
DataChangeEventQueue[] queues = dataChangeEventCenter.getQueues();
int queueCount = queues.length;
Executor executor = ExecutorFactory.newFixedThreadPool(queueCount, DataChangeHandler.class.getSimpleName());
Executor notifyExecutor = ExecutorFactory
.newFixedThreadPool(dataServerConfig.getQueueCount() * 5, this.getClass().getSimpleName());
for (int idx = 0; idx < queueCount; idx++) {
final DataChangeEventQueue dataChangeEventQueue = queues[idx];
final String name = dataChangeEventQueue.getName();
executor.execute(() -> {
while (true) {
final ChangeData changeData = dataChangeEventQueue.take();
notifyExecutor.execute(new ChangeNotifier(changeData, name));
}
});
}
}
3.2.4 業務執行
對於 ChangeData，會生成 ChangeNotifier 進行處理。會把這個事件變更資訊通過 ChangeNotifier 對外發布，通知其他節點進行資料同步。
在 ChangeNotifier 之中，會判斷changeData的型別做不同處理。
- 如果是SnapshotData，則：
- 生成SnapshotData；
- 呼叫 datumCache.putSnapshot 做儲存；
- 呼叫notify做通知；
- 如果是其他型別，則：
- 對於pub or unPub merge，需要datum.updateVersion();
- 如果是 PUB_TEMP，則notifyTempPub(datum, sourceType, changeType);
- 如果是版本更新，則notify(datum, sourceType, lastVersion);
具體如下：
private class ChangeNotifier implements Runnable {
private ChangeData changeData;
private String name;
@Override
public void run() {
if (changeData instanceof SnapshotData) {
......
} else {
Datum datum = changeData.getDatum();
String dataCenter = datum.getDataCenter();
String dataInfoId = datum.getDataInfoId();
DataSourceTypeEnum sourceType = changeData.getSourceType();
DataChangeTypeEnum changeType = changeData.getChangeType();
if (changeType == DataChangeTypeEnum.MERGE
&& sourceType != DataSourceTypeEnum.BACKUP
&& sourceType != DataSourceTypeEnum.SYNC) {
//update version for pub or unPub merge to cache
//if the version product before merge to cache,it may be cause small version override big one
datum.updateVersion();
}
long version = datum.getVersion();
try {
if (sourceType == DataSourceTypeEnum.CLEAN) {
if (datumCache.cleanDatum(dataCenter, dataInfoId)) {
......
}
} else if (sourceType == DataSourceTypeEnum.PUB_TEMP) {
notifyTempPub(datum, sourceType, changeType);
} else {
MergeResult mergeResult = datumCache.putDatum(changeType, datum);
Long lastVersion = mergeResult.getLastVersion();
if (lastVersion != null
&& lastVersion.longValue() == LocalDatumStorage.ERROR_DATUM_VERSION) {
return;
}
//lastVersion null means first add datum
if (lastVersion == null || version != lastVersion) {
if (mergeResult.isChangeFlag()) {
notify(datum, sourceType, lastVersion);
}
}
}
}
}
}
}
此時具體邏輯如下：
+--------------------+
| PublishDataHandler |
+--------+-----------+
|
|
| publisher
|
v
+---------+------------+
|DataChangeEventCenter |
+---------+------------+
|
|
| ChangeData
v
+---------+------------+
| DataChangeEventQueue |
+---------+------------+
|
|
| ChangeData
v
+-------+----------+
| DataChangeHandler|
+-------+----------+
|
|
| ChangeData
v
+------+--------+ +------------+
| ChangeNotifier| +--------> | datumCache |
+------+--------+ +------------+
3.2.5 通知
notify函式會遍歷dataChangeNotifiers，找出可以支援本Datum對應SourceType的Notifier來執行。
具體如何支援哪些函式，是由getSuitableSource設定的。
private void notify(Datum datum, DataSourceTypeEnum sourceType, Long lastVersion) {
for (IDataChangeNotifier notifier : dataChangeNotifiers) {
if (notifier.getSuitableSource().contains(sourceType)) {
notifier.notify(datum, lastVersion);
}
}
}
對應的Bean是：
@Bean(name = "dataChangeNotifiers")
public List<IDataChangeNotifier> dataChangeNotifiers() {
List<IDataChangeNotifier> list = new ArrayList<>();
list.add(sessionServerNotifier());
list.add(tempPublisherNotifier());
list.add(backUpNotifier());
return list;
}
3.2.6 BackUpNotifier同步
就是呼叫 syncDataService.appendOperator 進行通知，其實就是把 Datum 變成 Operator，存到AbstractAcceptorStore。
public class BackUpNotifier implements IDataChangeNotifier {
@Autowired
private SyncDataService syncDataService;
@Override
public Set<DataSourceTypeEnum> getSuitableSource() {
Set<DataSourceTypeEnum> set = new HashSet<>();
set.add(DataSourceTypeEnum.PUB);
return set;
}
@Override
public void notify(Datum datum, Long lastVersion) {
syncDataService.appendOperator(new Operator(datum.getVersion(), lastVersion, datum,
DataSourceTypeEnum.BACKUP));
}
}
3.2.7 SessionServerNotifier通知資料變化
SessionServerNotifier 則要複雜很多。
public class SessionServerNotifier implements IDataChangeNotifier {
private AsyncHashedWheelTimer asyncHashedWheelTimer;
@Autowired
private DataServerConfig dataServerConfig;
@Autowired
private Exchange boltExchange;
@Autowired
private SessionServerConnectionFactory sessionServerConnectionFactory;
@Autowired
private DatumCache datumCache;
@Override
public Set<DataSourceTypeEnum> getSuitableSource() {
Set<DataSourceTypeEnum> set = new HashSet<>();
set.add(DataSourceTypeEnum.PUB);
set.add(DataSourceTypeEnum.SYNC);
set.add(DataSourceTypeEnum.SNAPSHOT);
return set;
}
}
3.2.7.1 時間輪
建立了一個500毫秒的時間輪。
@PostConstruct
public void init() {
ThreadFactoryBuilder threadFactoryBuilder = new ThreadFactoryBuilder();
threadFactoryBuilder.setDaemon(true);
asyncHashedWheelTimer = new AsyncHashedWheelTimer(threadFactoryBuilder.setNameFormat(
"Registry-SessionServerNotifier-WheelTimer").build(), 500, TimeUnit.MILLISECONDS, 1024,
dataServerConfig.getSessionServerNotifierRetryExecutorThreadSize(),
dataServerConfig.getSessionServerNotifierRetryExecutorQueueSize(), threadFactoryBuilder
.setNameFormat("Registry-SessionServerNotifier-WheelExecutor-%d").build(),
new TaskFailedCallback() {
@Override
public void executionRejected(Throwable e) {
LOGGER.error("executionRejected: " + e.getMessage(), e);
}
@Override
public void executionFailed(Throwable e) {
LOGGER.error("executionFailed: " + e.getMessage(), e);
}
});
}
從業務角度看，當有publisher相關訊息來臨時候，
DataChangeHandler的notify函式會遍歷dataChangeNotifiers，找出可以支援本Datum對應SourceType的Notifier來執行。
private void notify(Datum datum, DataSourceTypeEnum sourceType, Long lastVersion) {
for (IDataChangeNotifier notifier : dataChangeNotifiers) {
if (notifier.getSuitableSource().contains(sourceType)) {
notifier.notify(datum, lastVersion);
}
}
}
到了SessionServerNotifier這裡的notify函式，會遍歷目前快取的所有Connection，逐一通知。
@Override
public void notify(Datum datum, Long lastVersion) {
DataChangeRequest request = new DataChangeRequest(datum.getDataInfoId(),
datum.getDataCenter(), datum.getVersion());
List<Connection> connections = sessionServerConnectionFactory.getSessionConnections();
for (Connection connection : connections) {
doNotify(new NotifyCallback(connection, request));
}
}
具體通知函式：
private void doNotify(NotifyCallback notifyCallback) {
Connection connection = notifyCallback.connection;
DataChangeRequest request = notifyCallback.request;
try {
//check connection active
if (!connection.isFine()) {
return;
}
Server sessionServer = boltExchange.getServer(dataServerConfig.getPort());
sessionServer.sendCallback(sessionServer.getChannel(connection.getRemoteAddress()),
request, notifyCallback, dataServerConfig.getRpcTimeout());
} catch (Exception e) {
onFailed(notifyCallback);
}
}
而時間輪是在呼叫失敗的重試中使用。
就是當沒有達到失敗重試最大次數時，進行定時重試。
private void onFailed(NotifyCallback notifyCallback) {
DataChangeRequest request = notifyCallback.request;
Connection connection = notifyCallback.connection;
notifyCallback.retryTimes++;
//check version, if it's fall behind, stop retry
long _currentVersion = datumCache.get(request.getDataCenter(), request.getDataInfoId()).getVersion();
if (request.getVersion() != _currentVersion) {
return;
}
if (notifyCallback.retryTimes <= dataServerConfig.getNotifySessionRetryTimes()) {
this.asyncHashedWheelTimer.newTimeout(timeout -> {
//check version, if it's fall behind, stop retry
long currentVersion = datumCache.get(request.getDataCenter(), request.getDataInfoId()).getVersion();
if (request.getVersion() == currentVersion) {
doNotify(notifyCallback);
}
}, getDelayTimeForRetry(notifyCallback.retryTimes), TimeUnit.MILLISECONDS);
}
}
具體邏輯如下：
+--------------------+
| PublishDataHandler |
+--------+-----------+
|
|
| publisher
|
v
+---------+------------+
|DataChangeEventCenter |
+---------+------------+
|
|
| ChangeData
v
+---------+------------+
| DataChangeEventQueue |
+---------+------------+
|
|
| ChangeData
v
+-------+----------+
| DataChangeHandler|
+-------+----------+
|
|
| ChangeData
v
+------+--------+ +------------+
| ChangeNotifier| +--------> | datumCache |
+------+--------+ +------------+
|
|
v
+---+------+
| notifier |
+---+------+
|
v
+-----------+---------------+
| |
v v
+----+----------------+ +------+----------+
|SessionServerNotifier| | BackUpNotifier |
+----+----------------+ +------+----------+
| |
| |
| |
| v
+--v------------+ +------+----------------+
| sessionServer | | AbstractAcceptorStore |
+---------------+ +-----------------------+
0x04 總結
本文是把註冊中的一個點“事件變更通知ChangeNotifie“進行細化展開，以 SessionServerNotifier 和 BackUpNotifier 為例，為大家進行解釋ChangeNotifier的原理和使用。把包括 dataChangeEventCenter 等功能也梳理了一下，希望對大家有所幫助。
在 DataServer，資料變化有兩個方向：
-
資料伺服器節點變化；
-
資料的變化，即 Publisher 和 Scriber 的變化；
ChangeNotifier就是負責把 Publisher 和 Scriber 的變化 通知給相關模組。變更通知就是一種解耦。