Arthas原理系列(三)：服務端啟動流程

DD_Dddd發表於2020-12-06
服務端

歷史文章推薦：

  1. OGNL語法規範
  2. 消失的堆疊
  3. Arthas原理系列(一)：利用JVM的attach機制實現一個極簡的watch命令
  4. Arthas原理系列(二)：總體架構和專案入口

前言

本篇文章主要講我們在終端中敲入的命令是如何被 arthas 伺服器識別並且解釋的。要注意這個過程是 arthas 對所有命令執行過程的抽閒個，對於具體命令的執行過程我會在後面的系列文章中再說。

arthas 服務端的啟動

在上一篇文章中，我們跟蹤了整個 arthas 工程的入口方法：com.taobao.arthas.agent334.AgentBootstrap#main，在這個方法中，最重要的一個步驟就是啟動過了一個繫結執行緒

private static synchronized void main(String args, final Instrumentation inst) {
    try {
        // 1. 程式執行前的校驗，
        // arthas如果已經存在，則直接返回
        // 入參中必須要包含arthas core等
        // 這些程式碼細節不會影響我們對主流程的理解，因此暫時刪除
        final ClassLoader agentLoader = getClassLoader(inst, arthasCoreJarFile);
        Thread bindingThread = new Thread() {
            @Override
            public void run() {
                try {
                    bind(inst, agentLoader, agentArgs);
                } catch (Throwable throwable) {
                    throwable.printStackTrace(ps);
                }
            }
        };

        bindingThread.setName("arthas-binding-thread");
        bindingThread.start();
        bindingThread.join();
    } catch (Throwable t) {
        t.printStackTrace(ps);
        try {
            if (ps != System.err) {
                ps.close();
            }
        } catch (Throwable tt) {
            // ignore
        }
        throw new RuntimeException(t);
    }

bind這個執行緒的執行時會呼叫com.taobao.arthas.agent334.AgentBootstrap#bind，這個方法的詳細程式碼如下：

private static void bind(Instrumentation inst, ClassLoader agentLoader, String args) throws Throwable {
        /**
         * <pre>
         * ArthasBootstrap bootstrap = ArthasBootstrap.getInstance(inst);
         * </pre>
         */
        Class<?> bootstrapClass = agentLoader.loadClass(ARTHAS_BOOTSTRAP);
        Object bootstrap = bootstrapClass.getMethod(GET_INSTANCE, Instrumentation.class, String.class).invoke(null, inst, args);
        boolean isBind = (Boolean) bootstrapClass.getMethod(IS_BIND).invoke(bootstrap);
        if (!isBind) {
            String errorMsg = "Arthas server port binding failed! Please check $HOME/logs/arthas/arthas.log for more details.";
            ps.println(errorMsg);
            throw new RuntimeException(errorMsg);
        }
        ps.println("Arthas server already bind.");
    }

這段方法用反射的方法呼叫了com.taobao.arthas.core.server.ArthasBootstrap的靜態方法getInstance，並且把從main方法中解析到引數再傳到這個getInstance中。

getInstance從這個名字看就是返回一個ArthasBootstrap的例項，事實上程式碼的邏輯也是這樣的，其中最關鍵的就是ArthasBootstrap的建構函式函式：

private ArthasBootstrap(Instrumentation instrumentation, Map<String, String> args) throws Throwable {
        this.instrumentation = instrumentation;

        String outputPath = System.getProperty("arthas.output.dir", "arthas-output");
        arthasOutputDir = new File(outputPath);
        arthasOutputDir.mkdirs();

        // 1. initSpy()
        // 載入SpyAPI這個類
        initSpy(instrumentation);
        // 2. ArthasEnvironment
        // 初始化arthas執行的環境變數
        initArthasEnvironment(args);
        // 3. init logger
        loggerContext = LogUtil.initLooger(arthasEnvironment);

        // 4. init beans
        // 初始化結果渲染和歷史命令管理的相關類
        initBeans();

        // 5. start agent server
        // 啟動server，開始監聽
        bind(configure);

        // 註冊一些鉤子函式
        executorService = Executors.newScheduledThreadPool(1, new ThreadFactory() {
            @Override
            public Thread newThread(Runnable r) {
                final Thread t = new Thread(r, "arthas-command-execute");
                t.setDaemon(true);
                return t;
            }
        });

        shutdown = new Thread("as-shutdown-hooker") {

            @Override
            public void run() {
                ArthasBootstrap.this.destroy();
            }
        };

        transformerManager = new TransformerManager(instrumentation);
        Runtime.getRuntime().addShutdownHook(shutdown);
    }

在這個建構函式中，最重要的就是com.taobao.arthas.core.server.ArthasBootstrap#bind這個方法

private void bind(Configure configure) throws Throwable {

    // 無關緊要的一些前置操作，先刪除掉

    try {
        // 關於arthas tunnel server，請參考：
        // https://arthas.aliyun.com/doc/tunnel.html
        if (configure.getTunnelServer() != null) {
            tunnelClient = new TunnelClient();
            tunnelClient.setAppName(configure.getAppName());
            tunnelClient.setId(configure.getAgentId());
            tunnelClient.setTunnelServerUrl(configure.getTunnelServer());
            tunnelClient.setVersion(ArthasBanner.version());
            ChannelFuture channelFuture = tunnelClient.start();
            channelFuture.await(10, TimeUnit.SECONDS);
        }
    } catch (Throwable t) {
        logger().error("start tunnel client error", t);
    }

    try {
        // 將一些非常關鍵的引數包裝成ShellServerOptions物件
        ShellServerOptions options = new ShellServerOptions()
                        .setInstrumentation(instrumentation)
                        .setPid(PidUtils.currentLongPid())
                        .setWelcomeMessage(ArthasBanner.welcome());
        if (configure.getSessionTimeout() != null) {
            options.setSessionTimeout(configure.getSessionTimeout() * 1000);
        }

        // new 一個shellServer，用於監聽命令
        shellServer = new ShellServerImpl(options);

        // BuiltinCommandPack物件首次出現，包含了所有的內建命令
        BuiltinCommandPack builtinCommands = new BuiltinCommandPack();
        List<CommandResolver> resolvers = new ArrayList<CommandResolver>();
        resolvers.add(builtinCommands);

        //worker group
        workerGroup = new NioEventLoopGroup(new DefaultThreadFactory("arthas-TermServer", true));

        // TODO: discover user provided command resolver
        if (configure.getTelnetPort() != null && configure.getTelnetPort() > 0) {
            shellServer.registerTermServer(new HttpTelnetTermServer(configure.getIp(), configure.getTelnetPort(),
                    options.getConnectionTimeout(), workerGroup));
        } else {
            logger().info("telnet port is {}, skip bind telnet server.", configure.getTelnetPort());
        }
        if (configure.getHttpPort() != null && configure.getHttpPort() > 0) {
            shellServer.registerTermServer(new HttpTermServer(configure.getIp(), configure.getHttpPort(),
                    options.getConnectionTimeout(), workerGroup));
        } else {
            // listen local address in VM communication
            if (configure.getTunnelServer() != null) {
                shellServer.registerTermServer(new HttpTermServer(configure.getIp(), configure.getHttpPort(),
                        options.getConnectionTimeout(), workerGroup));
            }
            logger().info("http port is {}, skip bind http server.", configure.getHttpPort());
        }

        for (CommandResolver resolver : resolvers) {
            shellServer.registerCommandResolver(resolver);
        }

        shellServer.listen(new BindHandler(isBindRef));
        if (!isBind()) {
            throw new IllegalStateException("Arthas failed to bind telnet or http port.");
        }

        //http api session manager
        sessionManager = new SessionManagerImpl(options, shellServer.getCommandManager(), shellServer.getJobController());
        //http api handler
        httpApiHandler = new HttpApiHandler(historyManager, sessionManager);

        logger().info("as-server listening on network={};telnet={};http={};timeout={};", configure.getIp(),
                configure.getTelnetPort(), configure.getHttpPort(), options.getConnectionTimeout());

        // 非同步回報啟動次數
        if (configure.getStatUrl() != null) {
            logger().info("arthas stat url: {}", configure.getStatUrl());
        }
        UserStatUtil.setStatUrl(configure.getStatUrl());
        UserStatUtil.arthasStart();

        try {
            SpyAPI.init();
        } catch (Throwable e) {
            // ignore
        }

        logger().info("as-server started in {} ms", System.currentTimeMillis() - start);
    } catch (Throwable e) {
        logger().error("Error during start as-server", e);
        destroy();
        throw e;
    }
}

這個方法使我們到目前為止見到的最複雜的一個方法，裡面還是有很多的旁枝末節的干擾，總結一下，這個方法全都是圍繞著如何構建一個ShellServer物件來進行的：

  1. 第一步會將一些非常重要的入參包裝ShellServerOptions傳入ShellServer
  2. 然後會在ShellerServer上註冊命令直譯器BuiltinCommandPack，點開BuiltinCommandPack會發現，所有的命令都已經包含在內了
  3. 根據入參的不同在ShellerServer上註冊不同的TermServer，比如HttpTermServer或者是HttpTelnetTermServer
  4. 伺服器開啟監聽指令

BuiltinCommandPack的實現如下所示：

public class  BuiltinCommandPack implements CommandResolver {

    private static List<Command> commands = new ArrayList<Command>();

    static {
        initCommands();
    }

    @Override
    public List<Command> commands() {
        return commands;
    }

    private static void initCommands() {
        commands.add(Command.create(HelpCommand.class));
        commands.add(Command.create(KeymapCommand.class));
        commands.add(Command.create(SearchClassCommand.class));
        commands.add(Command.create(SearchMethodCommand.class));
        // ...
    }
}

服務端對命令列的監聽和處理

接下來我們分析arthas服務端的監聽過程

@Override
public ShellServer listen(final Handler<Future<Void>> listenHandler) {
    final List<TermServer> toStart;
    synchronized (this) {
        if (!closed) {
            throw new IllegalStateException("Server listening");
        }
        toStart = termServers;
    }
    final AtomicInteger count = new AtomicInteger(toStart.size());
    if (count.get() == 0) {
        setClosed(false);
        listenHandler.handle(Future.<Void>succeededFuture());
        return this;
    }
    Handler<Future<TermServer>> handler = new TermServerListenHandler(this, listenHandler, toStart);
    for (TermServer termServer : toStart) {
        // termHandler是termServer監聽命令的回撥函式
        // 當有新的命令通過網路到達server時會呼叫這個回撥函式
        termServer.termHandler(new TermServerTermHandler(this));
        termServer.listen(handler);
    }
    return this;
}

我們以HttpTermServer為例

@Override
public TermServer listen(Handler<Future<TermServer>> listenHandler) {
    // TODO: charset and inputrc from options
    bootstrap = new NettyWebsocketTtyBootstrap(workerGroup).setHost(hostIp).setPort(port);
    try {
        bootstrap.start(new Consumer<TtyConnection>() {
            @Override
            public void accept(final TtyConnection conn) {
                termHandler.handle(new TermImpl(Helper.loadKeymap(), conn));
            }
        }).get(connectionTimeout, TimeUnit.MILLISECONDS);
        listenHandler.handle(Future.<TermServer>succeededFuture());
    } catch (Throwable t) {
        logger.error("Error listening to port " + port, t);
        listenHandler.handle(Future.<TermServer>failedFuture(t));
    }
    return this;
}

會發現程式會最終去非同步的呼叫termHandler的handle方法，而termHandler正是前面註冊的TermServerTermHandler這個類的例項：

public class TermServerTermHandler implements Handler<Term> {
    private ShellServerImpl shellServer;

    public TermServerTermHandler(ShellServerImpl shellServer) {
        this.shellServer = shellServer;
    }

    @Override
    public void handle(Term term) {
        shellServer.handleTerm(term);
    }
}

handle又回撥了shellServer的handleTerm方法，我們的視線隨著呼叫流程再回到ShellServer這個類

public void handleTerm(Term term) {
    synchronized (this) {
        // That might happen with multiple ser
        if (closed) {
            term.close();
            return;
        }
    }

    ShellImpl session = createShell(term);
    tryUpdateWelcomeMessage();
    session.setWelcome(welcomeMessage);
    session.closedFuture.setHandler(new SessionClosedHandler(this, session));
    session.init();
    sessions.put(session.id, session); // Put after init so the close handler on the connection is set
    session.readline(); // Now readline
}

這個方法中的最後一行程式碼session.readline();是我們重點要關注的地方

public void readline() {
    // 這裡要注意ShellLineHandler這個類，後面readLine的回撥最終會回到這裡來
    term.readline(prompt, new ShellLineHandler(this),
            new CommandManagerCompletionHandler(commandManager));
}

我們以TermImpl的實現為例

public void readline(String prompt, Handler<String> lineHandler, Handler<Completion> completionHandler) {
    if (conn.getStdinHandler() != echoHandler) {
        throw new IllegalStateException();
    }
    if (inReadline) {
        throw new IllegalStateException();
    }
    inReadline = true;
    readline.readline(conn, prompt, new RequestHandler(this, lineHandler), new CompletionHandler(completionHandler, session));
}

這個方法呼叫了readline.readline方法，並把之前傳進來的ShellLineHandler也包進了RequestHandler傳到了readline.readline中，我們繼續往進看

public void readline(TtyConnection conn, String prompt, Consumer<String> requestHandler, Consumer<Completion> completionHandler) {
    synchronized (this) {
        if (interaction != null) {
        throw new IllegalStateException("Already reading a line");
        }
        interaction = new Interaction(conn, prompt, requestHandler, completionHandler);
    }
    interaction.install();
    conn.write(prompt);
    schedulePendingEvent();
}

interaction.install();以及schedulePendingEvent();這兩漢程式碼最終都會呼叫下面的一段方法

private void deliver() {
    while (true) {
      Interaction handler;
      KeyEvent event;
      synchronized (this) {
        if (decoder.hasNext() && interaction != null && !interaction.paused) {
          event = decoder.next();
          handler = interaction;
        } else {
          return;
        }
      }
      handler.handle(event);
    }
}

Interaction是ReadLine的一個內部類，他的handle方法比較長，我們擷取這個方法的關鍵部分如下所示：

private void handle(KeyEvent event) {
    if (event instanceof FunctionEvent) {
    FunctionEvent fname = (FunctionEvent) event;
    Function function = functions.get(fname.name());
    if (function != null) {
        synchronized (this) {
        paused = true;
        }
        function.apply(this);
    } else {
        Logging.READLINE.warn("Unimplemented function " + fname.name());
    }
    } else {
    LineBuffer buf = buffer.copy();
    for (int i = 0;i < event.length();i++) {
        int codePoint = event.getCodePointAt(i);
        try {
        buf.insert(codePoint);
        } catch (IllegalArgumentException e) {
        conn.stdoutHandler().accept(new int[]{'\007'});
        }
    }
    refresh(buf);
    }
}

在這段程式碼中，會首先判斷輸入時間是否在預存的functions這個變數中已經定義，如果有的話，則執行相應apply方法，否則做快取相關的操作。
在ReadLine這個類新建的時候，值預定義了一個方法，那就是ACCEPT_LINE

public Readline(Keymap keymap) {
    // https://github.com/alibaba/termd/issues/42
    // this.device = TermInfo.defaultInfo().getDevice("xterm"); // For now use xterm
    this.decoder = new EventQueue(keymap);
    this.history = new ArrayList<int[]>();
    addFunction(ACCEPT_LINE);
}

ACCEPT_LINE的定義如下：

private final Function ACCEPT_LINE = new Function() {

    @Override
    public String name() {
      return "accept-line";
    }

    @Override
    public void apply(Interaction interaction) {
      interaction.line.insert(interaction.buffer.toArray());
      LineStatus pb = new LineStatus();
      for (int i = 0;i < interaction.line.getSize();i++) {
        pb.accept(interaction.line.getAt(i));
      }
      interaction.buffer.clear();
      if (pb.isEscaping()) {
        interaction.line.delete(-1); // Remove \
        interaction.currentPrompt = "> ";
        interaction.conn.write("\n> ");
        interaction.resume();
      } else {
        if (pb.isQuoted()) {
          interaction.line.insert('\n');
          interaction.conn.write("\n> ");
          interaction.currentPrompt = "> ";
          interaction.resume();
        } else {
          String raw = interaction.line.toString();
          if (interaction.line.getSize() > 0) {
            addToHistory(interaction.line.toArray());
          }
          interaction.line.clear();
          interaction.conn.write("\n");
          interaction.end(raw);
        }
      }
    }
}

在ACCEPT_LINE的apply方法中，如果程式判定到達伺服器的是一個合法的命令列，則會呼叫io.termd.core.readline.Readline.Interaction#end方法，而這個方法，最終會呼叫requestHandler.accept(s);，這個RequestHandler其實就是封裝了一層ShellLineHandler。

private boolean end(String s) {
    synchronized (Readline.this) {
    if (interaction == null) {
        return false;
    }
    interaction = null;
    conn.setStdinHandler(prevReadHandler);
    conn.setSizeHandler(prevSizeHandler);
    conn.setEventHandler(prevEventHandler);
    }
    requestHandler.accept(s);
    return true;
}

通過上面的分析可以看到，後續我們對命令的處理直接看ShellLineHandler就可以了

命令的執行

public void handle(String line) {
    String name = first.value();
    if (name.equals("exit") || name.equals("logout") || name.equals("q") || name.equals("quit")) {
        handleExit();
        return;
    } else if (name.equals("jobs")) {
        handleJobs();
        return;
    } else if (name.equals("fg")) {
        handleForeground(tokens);
        return;
    } else if (name.equals("bg")) {
        handleBackground(tokens);
        return;
    } else if (name.equals("kill")) {
        handleKill(tokens);
        return;
    }

    Job job = createJob(tokens);
    if (job != null) {
        job.run();
    }
}

在com.taobao.arthas.core.shell.handlers.shell.ShellLineHandler#handle的設計中，如果是一些簡單的命令，比如說exit, logout,jobs,fg,bg,kill等，都是直接執行的，而其他的命令都是直接通過建立一個Job來執行的，這一小節，我們主要看arthas是怎麼抽象命令的執行的：從建立Job開始

@Override
public synchronized Job createJob(List<CliToken> args) {
    Job job = jobController.createJob(commandManager, args, session, new ShellJobHandler(this), term, null);
    return job;
}

會轉發到：

@Override
public Job createJob(InternalCommandManager commandManager, List<CliToken> tokens, Session session, JobListener jobHandler, Term term, ResultDistributor resultDistributor) {
    int jobId = idGenerator.incrementAndGet();
    StringBuilder line = new StringBuilder();
    for (CliToken arg : tokens) {
        line.append(arg.raw());
    }
    boolean runInBackground = runInBackground(tokens);
    Process process = createProcess(tokens, commandManager, jobId, term, resultDistributor);
    process.setJobId(jobId);
    JobImpl job = new JobImpl(jobId, this, process, line.toString(), runInBackground, session, jobHandler);
    jobs.put(jobId, job);
    return job;
}

Job的run方法是完全委託給Process的，所以接下來就直接看createProcess的過程：

private Process createProcess(List<CliToken> line, InternalCommandManager commandManager, int jobId, Term term, ResultDistributor resultDistributor) {
    try {
        ListIterator<CliToken> tokens = line.listIterator();
        while (tokens.hasNext()) {
            CliToken token = tokens.next();
            if (token.isText()) {
                Command command = commandManager.getCommand(token.value());
                if (command != null) {
                    return createCommandProcess(command, tokens, jobId, term, resultDistributor);
                } else {
                    throw new IllegalArgumentException(token.value() + ": command not found");
                }
            }
        }
        throw new IllegalArgumentException();
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

這段程式碼的意圖比較明顯，根據輸入的命令去找相應的Command物件，如果找到則建立Process物件，根據文字找相應Command的邏輯如下：

public Command getCommand(String commandName) {
    Command command = null;
    for (CommandResolver resolver : resolvers) {
        // 內建命令在ShellLineHandler裡提前處理了，所以這裡不需要再查詢內建命令
        if (resolver instanceof BuiltinCommandPack) {
            command = getCommand(resolver, commandName);
            if (command != null) {
                break;
            }
        }
    }
    return command;
}

private static Command getCommand(CommandResolver commandResolver, String name) {
    List<Command> commands = commandResolver.commands();
    if (commands == null || commands.isEmpty()) {
        return null;
    }

    for (Command command : commands) {
        if (name.equals(command.name())) {
            return command;
        }
    }
    return null;
}

這塊的邏輯還是比較清晰的，我們再看看在找到對應Command之後如何建立Process

private Process createCommandProcess(Command command, ListIterator<CliToken> tokens, int jobId, Term term, ResultDistributor resultDistributor) throws IOException {
    List<CliToken> remaining = new ArrayList<CliToken>();
    List<CliToken> pipelineTokens = new ArrayList<CliToken>();
    boolean isPipeline = false;
    RedirectHandler redirectHandler = null;
    List<Function<String, String>> stdoutHandlerChain = new ArrayList<Function<String, String>>();
    String cacheLocation = null;
    // 刪除中間處理管道和後臺程式的程式碼
    ProcessOutput ProcessOutput = new ProcessOutput(stdoutHandlerChain, cacheLocation, term);
    ProcessImpl process = new ProcessImpl(command, remaining, command.processHandler(), ProcessOutput, resultDistributor);
    process.setTty(term);
    return process;
}

在刪除了中間的處理管道和後臺命令的程式碼之後這段程式碼的邏輯也非常清晰，就是根據解析好的Command物件建立一個Process物件，值得注意的是，這裡把command.processHandler()傳進了Process的建構函式中。檢視com.taobao.arthas.core.shell.system.impl.ProcessImpl#run()，可以看到最終會呼叫到com.taobao.arthas.core.shell.system.impl.ProcessImpl.CommandProcessTask#run

private class CommandProcessTask implements Runnable {

    private CommandProcess process;

    public CommandProcessTask(CommandProcess process) {
        this.process = process;
    }

    @Override
    public void run() {
        try {
            handler.handle(process);
        } catch (Throwable t) {
            logger.error("Error during processing the command:", t);
            process.end(1, "Error during processing the command: " + t.getClass().getName() + ", message:" + t.getMessage()
                    + ", please check $HOME/logs/arthas/arthas.log for more details." );
        }
    }
}

這裡的handler正是建立Process物件時呼叫command.processHandler()傳進去的

// processHandler 初始化
private Handler<CommandProcess> processHandler = new ProcessHandler();
@Override
public Handler<CommandProcess> processHandler() {
    return processHandler;
}

private class ProcessHandler implements Handler<CommandProcess> {
    @Override
    public void handle(CommandProcess process) {
        process(process);
    }
}

private void process(CommandProcess process) {
    AnnotatedCommand instance;
    try {
        instance = clazz.newInstance();
    } catch (Exception e) {
        process.end();
        return;
    }
    CLIConfigurator.inject(process.commandLine(), instance);
    instance.process(process);
    UserStatUtil.arthasUsageSuccess(name(), process.args());
}

通過instance.process(process);就可以呼叫到具體Command類的process方法了，比如說我們以watch命令為例，如果客戶端輸入的是這條命令，則會觸發程式碼的插裝

@Override
public void process(final CommandProcess process) {
    // ctrl-C support
    process.interruptHandler(new CommandInterruptHandler(process));
    // q exit support
    process.stdinHandler(new QExitHandler(process));

    // start to enhance
    enhance(process);
}

小結一下

整個啟動過程還是比較清晰的，需要注意的是在這個過程中有好多的回撥函式，這些回撥函式中才包含真正處理事件的邏輯，需要多翻幾遍上下文才能完全理解
在這裡插入圖片描述

本文詳細的跟了上面這個類圖中類之間的互動，伺服器抽象這個模組主要負責建立起完整的伺服器環境並監聽到達服務端的命令，到達的命令經過初步解析之後通過建立的任務類去執行，在任務的執行中通過在ShellImpl中持有的ShellServer引用，可以解析出具體的Command類，最後，命令的執行會呼叫對應Command類中的process方法，從而完成了整個命令的執行。

掃描二維碼關注公眾號，獲取今年最新面試資料和電子書文件

​​

相關文章