如何實現Dolphinscheduler YARN Task狀態跟蹤？

海豚调度發表於2024-10-28

背景

Dolphinscheduler針對YARN任務，比如說MR、Spark、Flink，甚至是Shell任務，最初都是會判斷如果有YARN任務，解析到applicationId。這樣就會不單單以判斷客戶端程序為單一判斷依據，還要根據YARN狀態進行最終的Dolphinscheduler任務狀態判斷。後期，社群對此進行了重構(確實是好的嚮往，現在已經是半成品)，但是導致了一些問題，比如說針對Flink Stream Application模式，這種客戶端分離模式會讓客戶端Shell直接退出，所以現在Dolphinscheduler裡面的任務就直接成功了。YARN上的任務還在執行呢，但Dolphinscheduler已經不能追蹤到YARN上任務的狀態了。

那麼，想要實現對於YARN上任務的狀態跟蹤，可以怎麼做呢？

注：以3.2.1版本為例。

Worker Task關係圖

首先，讓我們來看下DolphinScheduler中Worker Task的關係原理。

file

AbstractTask: 主要定義了Task的基本生命週期介面，比如說init、handle和cancel
AbstractRemoteTask: 主要對handle方法做了實現，體現了模版方法設計模式，提取了submitApplication、trackApplicationStatus以及cancelApplication三個核心介面方法
AbstractYarnTask: 比如說YARN任務，就抽象了AbstractYarnTask，其中submitApplication、trackApplicationStatus以及cancelApplication可以直接是對YARN API的訪問

AbstractYarnTask實現YARN狀態跟蹤

AbstractYarnTask可以實現YARN狀態跟蹤，參考org.apache.dolphinscheduler.plugin.task.api.AbstractYarnTask，完整程式碼如下 :

public abstract class AbstractYarnTask extends AbstractRemoteTask {

    private static final int MAX_RETRY_ATTEMPTS = 3;

    private ShellCommandExecutor shellCommandExecutor;

    public AbstractYarnTask(TaskExecutionContext taskRequest) {
        super(taskRequest);
        this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle, taskRequest);
    }

    @Override
    public void submitApplication() throws TaskException {
        try {
            IShellInterceptorBuilder shellActuatorBuilder =
                    ShellInterceptorBuilderFactory.newBuilder()
                            .properties(getProperties())
                            // todo: do we need to move the replace to subclass?
                            .appendScript(getScript().replaceAll("\\r\\n", System.lineSeparator()));
            // SHELL task exit code
            TaskResponse response = shellCommandExecutor.run(shellActuatorBuilder, null);
            setExitStatusCode(response.getExitStatusCode());
            setAppIds(String.join(TaskConstants.COMMA, getApplicationIds()));
            setProcessId(response.getProcessId());
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
            log.info("The current yarn task has been interrupted", ex);
            setExitStatusCode(TaskConstants.EXIT_CODE_FAILURE);
            throw new TaskException("The current yarn task has been interrupted", ex);
        } catch (Exception e) {
            log.error("yarn process failure", e);
            exitStatusCode = -1;
            throw new TaskException("Execute task failed", e);
        }
    }

    @Override
    public void trackApplicationStatus() throws TaskException {
        if (StringUtils.isEmpty(appIds)) {
            return;
        }


        List<String> appIdList = Arrays.asList(appIds.split(","));
        boolean continueTracking = true;

        while (continueTracking) {
            Map<String, YarnState> yarnStateMap = new HashMap<>();
            for (String appId : appIdList) {
                if (StringUtils.isEmpty(appId)) {
                    continue;
                }

                boolean hadoopSecurityAuthStartupState =
                        PropertyUtils.getBoolean(HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE, false);
                String yarnStateJson = fetchYarnStateJsonWithRetry(appId, hadoopSecurityAuthStartupState);
                if (StringUtils.isNotEmpty(yarnStateJson)) {
                    String appJson = JSONUtils.getNodeString(yarnStateJson, "app");
                    YarnTask yarnTask = JSONUtils.parseObject(appJson, YarnTask.class);
                    log.info("yarnTask : {}", yarnTask);
                    yarnStateMap.put(yarnTask.getId(), YarnState.of(yarnTask.getState()));
                }
            }

            YarnState yarnTaskOverallStatus = YarnTaskStatusChecker.getYarnTaskOverallStatus(yarnStateMap);
            if (yarnTaskOverallStatus.isFinalState()) {
                handleFinalState(yarnTaskOverallStatus);
                continueTracking = false;
            } else {
                try {
                    TimeUnit.MILLISECONDS.sleep(SLEEP_TIME_MILLIS * 10);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException(e);
                }
            }
        }
    }

    private String fetchYarnStateJsonWithRetry(String appId,
                                               boolean hadoopSecurityAuthStartupState) throws TaskException {
        int retryCount = 0;
        while (retryCount < MAX_RETRY_ATTEMPTS) {
            try {
                return fetchYarnStateJson(appId, hadoopSecurityAuthStartupState);
            } catch (Exception e) {
                retryCount++;
                log.error("Failed to fetch or parse Yarn state for appId: {}. Attempt: {}/{}",
                        appId, retryCount, MAX_RETRY_ATTEMPTS, e);

                if (retryCount >= MAX_RETRY_ATTEMPTS) {
                    throw new TaskException("Failed to fetch Yarn state after "
                            + MAX_RETRY_ATTEMPTS + " attempts for appId: " + appId, e);
                }

                try {
                    TimeUnit.MILLISECONDS.sleep(SLEEP_TIME_MILLIS);
                } catch (InterruptedException ie) {
                    Thread.currentThread().interrupt();
                    throw new RuntimeException(ie);
                }
            }
        }
        return null;
    }

    private void handleFinalState(YarnState yarnState) {
        switch (yarnState) {
            case FINISHED:
                setExitStatusCode(EXIT_CODE_SUCCESS);
                break;
            case KILLED:
                setExitStatusCode(EXIT_CODE_KILL);
                break;
            default:
                setExitStatusCode(EXIT_CODE_FAILURE);
                break;
        }
    }

    private String fetchYarnStateJson(String appId, boolean hadoopSecurityAuthStartupState) throws Exception {
        return hadoopSecurityAuthStartupState
                ? KerberosHttpClient.get(getApplicationUrl(appId))
                : HttpUtils.get(getApplicationUrl(appId));
    }


    static class YarnTaskStatusChecker {

        public static YarnState getYarnTaskOverallStatus(Map<String, YarnState> yarnTaskMap) {
            // 檢查是否有任何任務處於 FAILED 或 KILLED 狀態
            boolean hasKilled = yarnTaskMap.values().stream()
                    .anyMatch(state -> state == YarnState.KILLED);

            if (hasKilled) {
                return YarnState.KILLED;
            }

            // 檢查是否有任何任務處於 FAILED 或 KILLED 狀態
            boolean hasFailed = yarnTaskMap.values().stream()
                    .anyMatch(state -> state == YarnState.FAILED);

            if (hasFailed) {
                return YarnState.FAILED;
            }


            // 檢查是否所有任務都處於 FINISHED 狀態
            boolean allFINISHED = yarnTaskMap.values().stream()
                    .allMatch(state -> state == YarnState.FINISHED);

            if (allFINISHED) {
                return YarnState.FINISHED;
            }

            // 檢查是否有任何任務處於 RUNNING 狀態
            boolean hasRunning = yarnTaskMap.values().stream()
                    .anyMatch(state -> state == YarnState.RUNNING);

            if (hasRunning) {
                return YarnState.RUNNING;
            }

            // 檢查是否有任何任務處於提交中狀態
            boolean hasSubmitting = yarnTaskMap.values().stream()
                    .anyMatch(state -> state == YarnState.NEW || state == YarnState.NEW_SAVING
                            || state == YarnState.SUBMITTED || state == YarnState.ACCEPTED);

            if (hasSubmitting) {
                return YarnState.SUBMITTING;
            }

            // 如果都不匹配，返回未知狀態
            return YarnState.UNKNOWN;
        }
    }


    /**
     * cancel application
     *
     * @throws TaskException exception
     */
    @Override
    public void cancelApplication() throws TaskException {
        // cancel process
        try {
            shellCommandExecutor.cancelApplication();
        } catch (Exception e) {
            throw new TaskException("cancel application error", e);
        }
    }

    /**
     * get application ids
     *
     * @return
     * @throws TaskException
     */
    @Override
    public List<String> getApplicationIds() throws TaskException {
        // TODO 這裡看common.properties中是否配置 appId.collect了，如果配置了走aop，否則走log
        return LogUtils.getAppIds(
                taskRequest.getLogPath(),
                taskRequest.getAppInfoPath(),
                PropertyUtils.getString(APPID_COLLECT, DEFAULT_COLLECT_WAY));
    }

    /** Get the script used to bootstrap the task */
    protected abstract String getScript();

    /** Get the properties of the task used to replace the placeholders in the script. */
    protected abstract Map<String, String> getProperties();

    @Data
    static class YarnTask {
        private String id;
        private String state;
    }

    private String getApplicationUrl(String applicationId) throws BaseException {

        String yarnResourceRmIds = PropertyUtils.getString(YARN_RESOURCEMANAGER_HA_RM_IDS);
        String yarnAppStatusAddress = PropertyUtils.getString(YARN_APPLICATION_STATUS_ADDRESS);
        String hadoopResourceManagerHttpAddressPort =
                PropertyUtils.getString(HADOOP_RESOURCE_MANAGER_HTTPADDRESS_PORT);

        String appUrl = StringUtils.isEmpty(yarnResourceRmIds) ?
                yarnAppStatusAddress :
                getAppAddress(yarnAppStatusAddress, yarnResourceRmIds);

        if (StringUtils.isBlank(appUrl)) {
            throw new BaseException("yarn application url generation failed");
        }

        log.info("yarn application url:{}", String.format(appUrl, hadoopResourceManagerHttpAddressPort, applicationId));
        return String.format(appUrl, hadoopResourceManagerHttpAddressPort, applicationId);
    }

    private static String getAppAddress(String appAddress, String rmHa) {

        String[] appAddressArr = appAddress.split(Constants.DOUBLE_SLASH);

        if (appAddressArr.length != 2) {
            return null;
        }

        String protocol = appAddressArr[0] + Constants.DOUBLE_SLASH;
        String[] pathSegments = appAddressArr[1].split(Constants.COLON);

        if (pathSegments.length != 2) {
            return null;
        }

        String end = Constants.COLON + pathSegments[1];

        // get active ResourceManager
        String activeRM = YarnHAAdminUtils.getActiveRMName(protocol, rmHa);

        if (StringUtils.isEmpty(activeRM)) {
            return null;
        }

        return protocol + activeRM + end;
    }

    /** yarn ha admin utils */
    private static final class YarnHAAdminUtils {

        /**
         * get active resourcemanager node
         *
         * @param protocol http protocol
         * @param rmIds yarn ha ids
         * @return yarn active node
         */
        public static String getActiveRMName(String protocol, String rmIds) {

            String hadoopResourceManagerHttpAddressPort =
                    PropertyUtils.getString(HADOOP_RESOURCE_MANAGER_HTTPADDRESS_PORT);

            String[] rmIdArr = rmIds.split(Constants.COMMA);

            String yarnUrl = protocol
                    + "%s:"
                    + hadoopResourceManagerHttpAddressPort
                    + "/ws/v1/cluster/info";
            try {
                /** send http get request to rm */
                for (String rmId : rmIdArr) {
                    String state = getRMState(String.format(yarnUrl, rmId));
                    if (Constants.HADOOP_RM_STATE_ACTIVE.equals(state)) {
                        return rmId;
                    }
                }

            } catch (Exception e) {
                log.error("get yarn ha application url failed", e);
            }
            return null;
        }

        /** get ResourceManager state */
        public static String getRMState(String url) {
            boolean hadoopSecurityAuthStartupState =
                    PropertyUtils.getBoolean(HADOOP_SECURITY_AUTHENTICATION_STARTUP_STATE, false);
            String retStr = Boolean.TRUE.equals(hadoopSecurityAuthStartupState)
                    ? KerberosHttpClient.get(url)
                    : HttpUtils.get(url);

            if (StringUtils.isEmpty(retStr)) {
                return null;
            }
            // to json
            ObjectNode jsonObject = JSONUtils.parseObject(retStr);

            // get ResourceManager state
            if (!jsonObject.has("clusterInfo")) {
                return null;
            }
            return jsonObject.get("clusterInfo").path("haState").asText();
        }
    }

    public enum YarnState {
        NEW,
        NEW_SAVING,
        SUBMITTED,
        ACCEPTED,
        RUNNING,
        FINISHED,
        FAILED,
        KILLED,
        SUBMITTING,
        UNKNOWN,
        ;

        // 將字串轉換為列舉
        public static YarnState of(String state) {
            try {
                return YarnState.valueOf(state);
            } catch (IllegalArgumentException | NullPointerException e) {
                // 如果字串無效，則返回 null
                return null;
            }
        }

        /**
         * 任務結束
         * @return
         */
        public boolean isFinalState() {
            return this == FINISHED || this == FAILED || this == KILLED;
        }
    }
}

可以看到，這裡的核心邏輯其實就是去掉之前直接把handle介面重寫了，而現在針對YARN任務，只需要實現submitApplication、trackApplicationStatus兩個核心介面，cancelApplication這個其實原則上應該代理YarnApplicationManager才好(當前沒有整合，不過不影響)。

流式任務前端applicationId顯示

dolphinscheduler-ui/src/views/projects/task/instance/use-stream-table.ts

file

後端封裝applicationId為YARN URL

dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/TaskInstanceServiceImpl.java 修改

file

dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/constants/Constants.java 修改

file

dolphinscheduler-common/src/main/resources/common.properties修改

file

dolphinscheduler-storage-plugin/dolphinscheduler-storage-hdfs/src/main/java/org/apache/dolphinscheduler/plugin/storage/hdfs/HdfsStorageOperator.java修改

file

dolphinscheduler-storage-plugin/dolphinscheduler-storage-hdfs/src/main/java/org/apache/dolphinscheduler/plugin/storage/hdfs/HdfsStorageProperties.java修改

file

頁面效果如下 :

file

注意 : URL貼上是需要自己寫的，上面的程式碼不包含

問題追蹤

這裡其實是有問題，對於state狀態來說，是有FINISHED、FAILED、KILLED三種狀態，但是FINISHED狀態裡面還是有FinalStatus，完成不一定是成功，FINISHED下面其實也有SUCCEEDED、FAILED和KILLED。其實就是FINISHED不能作為DolphinScheduler的終態，需要繼續判斷而已。

org.apache.dolphinscheduler.plugin.task.api.AbstractYarnTask#handleFinalState

private void handleFinalState(YarnState yarnState) {
    switch (yarnState) {
        case FINISHED:
            setExitStatusCode(EXIT_CODE_SUCCESS);
            break;
        case KILLED:
            setExitStatusCode(EXIT_CODE_KILL);
            break;
        default:
            setExitStatusCode(EXIT_CODE_FAILURE);
            break;
    }
}

使用HTTP對任務進行kill

curl -X PUT -d '{"state":"KILLED"}' \
>     -H "Content-Type: application/json" \
>     http://xx.xx.xx.xx:8088/ws/v1/cluster/apps/application_1694766249884_1098/state?user.name=hdfs

注意：一定要指定user.name，否則不一定能kill掉。

原文連結：https://segmentfault.com/a/1190000045058893

本文由白鯨開源提供釋出支援！

Istio最佳實踐系列：如何實現方法級呼叫跟蹤？
2021-04-15
在Unity中實現手部跟蹤
2019-08-20
Unity
德勤諮詢：2020全球消費者狀態跟蹤——中國
2020-07-28
【深入淺出 Yarn 架構與實現】6-2 NodeManager 狀態機管理
2023-04-03
Yarn架構
通過規則引擎跟蹤Java執行狀態的檢測工具：Byteman
2022-02-24
Java
idou老師教你學Istio 22 : 如何用istio實現呼叫鏈跟蹤
2019-02-18
Oracle執行語句跟蹤使用sql trace實現語句追蹤
2021-02-06
OracleSQL
【TRACE】如何設定或動態跟蹤Oracle net偵聽器
2022-02-18
Oracle
[20210603]如何跟蹤索引分裂.txt
2021-06-04
索引
基於OpenTelemetry實現Java微服務呼叫鏈跟蹤
2024-02-19
Java微服務
使用Spring Cloud Sleuth和OpenTelemetry實現分散式跟蹤
2021-11-09
SpringCloud分散式
PostgreSQL 跟蹤checkpointer出現死鎖
2019-01-07
SQL
狀態模式（c++實現）
2020-07-16
模式C++
vue的狀態管理實現
2020-11-08
Vue
擴充套件實現Unity協程的完整棧跟蹤
2024-05-12
套件Unity
IDEA之如何Debug原始碼跟蹤
2024-05-31
Idea原始碼
如何使用iPhone限制IP地址跟蹤
2022-08-29
iPhone
nginx 如何實現 if 巢狀
2019-11-26
Nginx巢狀
PIGOSS BSM 如何實現伺服器硬體狀態監控？
2020-04-03
Go伺服器
解析HetuEngine實現On Yarn原理
2022-01-12
Yarn
Python+OpenCV目標跟蹤實現基本的運動檢測
2019-01-17
PythonOpenCV
如何追蹤laravel動態
2021-09-10
Laravel
如何追蹤Go動態
2021-09-13
Go
如何追蹤Java動態
2021-09-16
Java
如何追蹤Python動態
2021-09-16
Python
如何追蹤vue動態
2021-09-16
Vue
JS手寫狀態管理的實現
2019-03-03
JS
Android 沉浸式狀態列的實現
2019-02-27
Android
[譯]Flutter - 使用Provider實現狀態管理
2020-04-04
FlutterIDE
如何分配和跟蹤專案工作量？
2023-05-10
如何在 Git 中取消檔案的跟蹤
2024-08-16
Git
sp_trace_setfilter sqlserver篩選跟蹤或跟蹤過濾
2020-04-05
FilterSQLServer
如何用 Python 實現超級瑪麗的介面和狀態機？
2019-12-25
Python
主力跟蹤戰法
2024-10-16
反跟蹤技術
2021-01-05
Flink的Task實現方式
2023-03-08
python如何使用__iter__()展現外部狀態
2021-09-11
Python
git的跟蹤分支和遠端跟蹤分支學習筆記
2018-03-22
Git筆記

如何實現Dolphinscheduler YARN Task狀態跟蹤？

背景

Worker Task關係圖

AbstractYarnTask實現YARN狀態跟蹤

流式任務前端applicationId顯示

後端封裝applicationId為YARN URL

問題追蹤

使用HTTP對任務進行kill

相關文章