dolphinscheduler Master服務是去中心化的，也就是沒有master和slave之分，每個master都參與工作，那麼它是如何每個Master服務去取任務執行時，每個Master都取到不同的任務，並且不會漏掉，不會重複的呢 ，下面從原始碼角度來分析這個問題
MasterServer.java
/**
* run master server
*/
@PostConstruct
public void run() throws SchedulerException {
......
this.masterSlotManager.start();
// self tolerant
......
this.masterSchedulerBootstrap.start();
......
}
在DS（後面dolphinschedule的簡稱）中提出了一個Slot（插槽）的概念，每個Master都有專屬於自己的SlotId，當master減少或增加時，會重新整理這個Id。
2. 先看SlotId的生成原理
this.masterSlotManager.start();
public void start() {
serverNodeManager.addMasterInfoChangeListener(new MasterSlotManager.SlotChangeListener());
}
這裡註冊了addMasterInfoChangeListener , 從字面上分析，這裡會監聽解除安裝Zookeeper中的Master資訊的變化；先分析SlotChangeListener
public class SlotChangeListener implements MasterInfoChangeListener {
private final Lock slotLock = new ReentrantLock();
private final MasterPriorityQueue masterPriorityQueue = new MasterPriorityQueue();
@Override
public void notify(Map<String, MasterHeartBeat> masterNodeInfo) {
// 一旦master資訊發生變化會走到這裡，masterNodeInfo是所有master的最新資訊
List<Server> serverList = masterNodeInfo.values().stream()
.filter(heartBeat -> !heartBeat.getServerStatus().equals(ServerStatus.BUSY))
.map(this::convertHeartBeatToServer).collect(Collectors.toList());
syncMasterNodes(serverList);
}
/**
* sync master nodes
*/
private void syncMasterNodes(List<Server> masterNodes) {
slotLock.lock();
try {
//masterPriorityQueue 是一個阻塞優先佇列
this.masterPriorityQueue.clear();
// 把masterNodes，會獲得一個根據createTime排序的佇列，這樣在各個master節點中，這個優先佇列中的排序順序就都是固定一樣的
this.masterPriorityQueue.putAll(masterNodes);
// 這裡是根據當前節點的IP來獲取佇列中的Index來當做SlotId
int tempCurrentSlot = masterPriorityQueue.getIndex(masterConfig.getMasterAddress());
int tempTotalSlot = masterNodes.size();
if (tempCurrentSlot < 0) {
totalSlot = 0;
currentSlot = 0;
log.warn("Current master is not in active master list");
} else if (tempCurrentSlot != currentSlot || tempTotalSlot != totalSlot) {
totalSlot = tempTotalSlot;
currentSlot = tempCurrentSlot;
log.info("Update master nodes, total master size: {}, current slot: {}", totalSlot, currentSlot);
}
} finally {
slotLock.unlock();
}
}
......
}
從SlotChangeListener原始碼中，可以知道SlotId的來源，來源於Zookeeper中masterInfo資訊的變化，而masterinfo資訊是由master啟動時主動註冊上去的（臨時節點，下線自動刪除）。
3. 如何利用SlotId來取任務Command
MasterSchedulerBootstrap.start()
這個方法最終會執行到MasterSchedulerBootstrap的run()方法
@Override
public void run() {
MasterServerLoadProtection serverLoadProtection = masterConfig.getServerLoadProtection();
while (!ServerLifeCycleManager.isStopped()) {
try {
.......
//SlotId就在這個findCommands方法中，實現不同的master取不同的command
List<Command> commands = findCommands();
if (CollectionUtils.isEmpty(commands)) {
// indicate that no command ,sleep for 1s
Thread.sleep(Constants.SLEEP_TIME_MILLIS);
continue;
}
commands.parallelStream()
.forEach(command -> {
try {
Optional<WorkflowExecuteRunnable> workflowExecuteRunnableOptional =
workflowExecuteRunnableFactory.createWorkflowExecuteRunnable(command);
if (!workflowExecuteRunnableOptional.isPresent()) {
log.warn(
"The command execute success, will not trigger a WorkflowExecuteRunnable, this workflowInstance might be in serial mode");
return;
}
WorkflowExecuteRunnable workflowExecuteRunnable = workflowExecuteRunnableOptional.get();
ProcessInstance processInstance = workflowExecuteRunnable
.getWorkflowExecuteContext().getWorkflowInstance();
if (processInstanceExecCacheManager.contains(processInstance.getId())) {
log.error(
"The workflow instance is already been cached, this case shouldn't be happened");
}
processInstanceExecCacheManager.cache(processInstance.getId(), workflowExecuteRunnable);
workflowEventQueue.addEvent(
new WorkflowEvent(WorkflowEventType.START_WORKFLOW, processInstance.getId()));
} catch (WorkflowCreateException workflowCreateException) {
log.error("Master handle command {} error ", command.getId(), workflowCreateException);
commandService.moveToErrorCommand(command, workflowCreateException.toString());
}
});
MasterServerMetrics.incMasterConsumeCommand(commands.size());
} catch (InterruptedException interruptedException) {
log.warn("Master schedule bootstrap interrupted, close the loop", interruptedException);
Thread.currentThread().interrupt();
break;
} catch (Exception e) {
log.error("Master schedule workflow error", e);
// sleep for 1s here to avoid the database down cause the exception boom
ThreadUtils.sleep(Constants.SLEEP_TIME_MILLIS);
}
}
}
private List<Command> findCommands() throws MasterException {
try {
long scheduleStartTime = System.currentTimeMillis();
// 這裡拿到自己的SlotId
int thisMasterSlot = masterSlotManager.getSlot();
int masterCount = masterSlotManager.getMasterSize();
if (masterCount <= 0) {
log.warn("Master count: {} is invalid, the current slot: {}", masterCount, thisMasterSlot);
return Collections.emptyList();
}
int pageSize = masterConfig.getFetchCommandNum();
//這裡透過slotId去取command
final List<Command> result =
commandService.findCommandPageBySlot(pageSize, masterCount, thisMasterSlot);
if (CollectionUtils.isNotEmpty(result)) {
long cost = System.currentTimeMillis() - scheduleStartTime;
log.info(
"Master schedule bootstrap loop command success, fetch command size: {}, cost: {}ms, current slot: {}, total slot size: {}",
result.size(), cost, thisMasterSlot, masterCount);
ProcessInstanceMetrics.recordCommandQueryTime(cost);
}
return result;
} catch (Exception ex) {
throw new MasterException("Master loop command from database error", ex);
}
}
最終會走到Mapper中
<select id="queryCommandPageBySlot" resultType="org.apache.dolphinscheduler.dao.entity.Command">
select *
from t_ds_command
where id % #{masterCount} = #{thisMasterSlot}
order by process_instance_priority, id asc
limit #{limit}
</select>
這裡透過id 對master的總數取餘，如果等於當前的SlotId，則取出，實現多master取同一張表中的command，而master之間相互不衝突