Skywalking 外掛開發

本文出處shenyifengtk.github.io 轉載請說明

概念

Span

Span 是分散式跟蹤系統中一個重要且常用的概念. 可從 Google Dapper Paper 和 OpenTracing 學習更多與 Span 相關的知識.

SkyWalking 從 2017 年開始支援 OpenTracing 和 OpenTracing-Java API, 我們的 Span 概念與論文和 OpenTracing 類似. 我們也擴充套件了 Span.

Span 有三種型別

1.1 EntrySpan

EntrySpan 代表服務提供者, 也是伺服器端的端點. 作為一個 APM 系統, 我們的目標是應用伺服器. 所以幾乎所有的服務和 MQ-消費者 都是 EntrySpan。可以理解一個程式處理第一個span就是EntrySpan，意思為entiry span 進入服務span。

1.2 LocalSpan

LocalSpan 表示普通的 Java 方法, 它與遠端服務無關, 也不是 MQ 生產者/消費者, 也不是服務（例如 HTTP 服務）提供者/消費者。所有本地方法呼叫都是localSpan,包括非同步執行緒呼叫，執行緒池提交任務都是。

1.3 ExitSpan

ExitSpan 代表一個服務客戶端或MQ的生產者, 在 SkyWalking 的早期命名為 LeafSpan. 例如 通過 JDBC 訪問DB, 讀取 Redis/Memcached 被歸類為 ExitSpan.

上下文載體 (ContextCarrier)

為了實現分散式跟蹤, 需要繫結跨程式的追蹤, 並且上下文應該在整個過程中隨之傳播. 這就是 ContextCarrier 的職責.

以下是有關如何在 A -> B 分散式呼叫中使用 ContextCarrier 的步驟.

1. 在客戶端, 建立一個新的空的 ContextCarrier.
2. 通過 ContextManager#createExitSpan 建立一個 ExitSpan 或者使用 ContextManager#inject 來初始化 ContextCarrier.
3. 將 ContextCarrier 所有資訊放到請求頭 (如 HTTP HEAD), 附件(如 Dubbo RPC 框架), 或者訊息 (如 Kafka) 中,詳情可以看官方給出跨程式傳輸協議sw8
4. 通過服務呼叫, 將 ContextCarrier 傳遞到服務端.
5. 在服務端, 在對應元件的頭部, 附件或訊息中獲取 ContextCarrier 所有內容.
6. 通過 ContestManager#createEntrySpan 建立 EntrySpan 或者使用 ContextManager#extract 將服務端和客戶端的繫結.

讓我們通過 Apache HttpComponent client 外掛和 Tomcat 7 伺服器外掛演示, 步驟如下:

1. 客戶端 Apache HttpComponent client 外掛

span = ContextManager.createExitSpan("/span/operation/name", contextCarrier, "ip:port");
CarrierItem next = contextCarrier.items();
while (next.hasNext()) {
    next = next.next();
    httpRequest.setHeader(next.getHeadKey(), next.getHeadValue());
}

2. 服務端 Tomcat 7 伺服器外掛

ContextCarrier contextCarrier = new ContextCarrier();
CarrierItem next = contextCarrier.items();
while (next.hasNext()) {
    next = next.next();
    next.setHeadValue(request.getHeader(next.getHeadKey()));
}

span = ContextManager.createEntrySpan("/span/operation/name", contextCarrier);

上下文快照 (ContextSnapshot)

除了跨程式, 跨執行緒也是需要支援的, 例如非同步執行緒（記憶體中的訊息佇列）和批處理在 Java 中很常見, 跨程式和跨執行緒十分相似, 因為都是需要傳播上下文. 唯一的區別是, 不需要跨執行緒序列化.

以下是有關跨執行緒傳播的三個步驟：

1. 使用 ContextManager#capture 方法獲取 ContextSnapshot 物件.
2. 讓子執行緒以任何方式, 通過方法引數或由現有引數攜帶來訪問 ContextSnapshot
3. 在子執行緒中使用 ContextManager#continued。

跨程式Span傳輸原理

public class CarrierItem implements Iterator<CarrierItem> {
    private String headKey;
    private String headValue;
    private CarrierItem next;

    public CarrierItem(String headKey, String headValue) {
        this(headKey, headValue, null);
    }

    public CarrierItem(String headKey, String headValue, CarrierItem next) {
        this.headKey = headKey;
        this.headValue = headValue;
        this.next = next;
    }

    public String getHeadKey() {
        return headKey;
    }

    public String getHeadValue() {
        return headValue;
    }

    public void setHeadValue(String headValue) {
        this.headValue = headValue;
    }

    @Override
    public boolean hasNext() {
        return next != null;
    }

    @Override
    public CarrierItem next() {
        return next;
    }

    @Override
    public void remove() {

    }
}

CarrierItem 類似Map key value的資料介面，通過一個單向連線將K/V連線起來。
看下 ContextCarrier.items()方法如何建立CarrierItem

    public CarrierItem items() {
       //內建一個 sw8-x key
        SW8ExtensionCarrierItem sw8ExtensionCarrierItem = new SW8ExtensionCarrierItem(extensionContext, null); 
       //內建  sw8-correlation key
        SW8CorrelationCarrierItem sw8CorrelationCarrierItem = new SW8CorrelationCarrierItem(
            correlationContext, sw8ExtensionCarrierItem);
       //內建 sw8 key 
        SW8CarrierItem sw8CarrierItem = new SW8CarrierItem(this, sw8CorrelationCarrierItem);
        return new CarrierItemHead(sw8CarrierItem);
    }

建立一個連結CarrierItemHead->SW8CarrierItem ->SW8CorrelationCarrierItem->SW8ExtensionCarrierItem
在看下上面tomcat7 遍歷CarrierItem，呼叫key從http header獲取值設定到物件內建值，這樣就可以做到將上一個程式header 值設定到下一個程式裡，在呼叫

    ContextCarrier deserialize(String text, HeaderVersion version) {
        if (text == null) {
            return this;
        }
        if (HeaderVersion.v3.equals(version)) {
            String[] parts = text.split("-", 8);
            if (parts.length == 8) {
                try {
                    // parts[0] is sample flag, always trace if header exists.
                    this.traceId = Base64.decode2UTFString(parts[1]);
                    this.traceSegmentId = Base64.decode2UTFString(parts[2]);
                    this.spanId = Integer.parseInt(parts[3]);
                    this.parentService = Base64.decode2UTFString(parts[4]);
                    this.parentServiceInstance = Base64.decode2UTFString(parts[5]);
                    this.parentEndpoint = Base64.decode2UTFString(parts[6]);
                    this.addressUsedAtClient = Base64.decode2UTFString(parts[7]);
                } catch (IllegalArgumentException ignored) {

                }
            }
        }
        return this;
    }

這樣剛剛new 出來ContextCarrier就可以從上一個呼叫者上繼承所有的屬性，新建立span就可以跟上一個span 關聯起來了了。

開發外掛

知識點

追蹤的基本方法是攔截 Java 方法, 使用位元組碼操作技術（byte-buddy）和 AOP 概念. SkyWalking 包裝了位元組碼操作技術並追蹤上下文的傳播, 所以你只需要定義攔截點（換句話說就是 Spring 的切面）。

ClassInstanceMethodsEnhancePluginDefine 定義了構造方法 Contructor 攔截點和 instance method 例項方法攔截點，主要有三個方法需要被重寫

     /**
     * 需要被攔截Class
     * @return
     */
    @Override
    protected ClassMatch enhanceClass() {
        return null;
    }

    /**
     * 構造器切點
     * @return
     */
    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return new ConstructorInterceptPoint[0];
    }

    /**
     * 方法切點
     * @return InstanceMethodsInterceptPoint 裡面會宣告攔截按個方法
     */
    @Override
    public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() {
        return new InstanceMethodsInterceptPoint[0];
    }

ClassMatch 以下有四種方法表示如何去匹配目標類:

• NameMatch.byName, 通過類的全限定名(Fully Qualified Class Name, 即 包名 + . + 類名).
• ClassAnnotationMatch.byClassAnnotationMatch, 根據目標類是否存在某些註解.
• MethodAnnotationMatchbyMethodAnnotationMatch, 根據目標類的方法是否存在某些註解.
• HierarchyMatch.byHierarchyMatch, 根據目標類的父類或介面

ClassStaticMethodsEnhancePluginDefine 定義了類方法 class 靜態method 攔截點。

public abstract class ClassStaticMethodsEnhancePluginDefine extends ClassEnhancePluginDefine {

    /**
     * 構造器切點
     * @return null, means enhance no constructors.
     */
    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return null;
    }

    /**
     * 方法切點
     * @return null, means enhance no instance methods.
     */
    @Override
    public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() {
        return null;
    }
}

InstanceMethodsInterceptPoint 普通方法介面切點有哪些方法

public interface InstanceMethodsInterceptPoint {
    /**
     * class instance methods matcher.
     *  可以理解成功對Class 那些方法進行增強
     *  ElementMatcher 是bytebuddy 類庫一個方法匹配器，裡面封裝了各種方法匹配
     * @return methods matcher
     */
    ElementMatcher<MethodDescription> getMethodsMatcher();

    /**
     * @return represents a class name, the class instance must instanceof InstanceMethodsAroundInterceptor.
     *  返回一個攔截器全類名，所有攔截器必須實現    InstanceMethodsAroundInterceptor 介面
     */
    String getMethodsInterceptor();

    /**
     *  是否要覆蓋原方法入參
     * @return
     */
    boolean isOverrideArgs();
}

在看下攔截器有那些方法

/**
 * A interceptor, which intercept method's invocation. The target methods will be defined in {@link
 * ClassEnhancePluginDefine}'s subclass, most likely in {@link ClassInstanceMethodsEnhancePluginDefine}
 */
public interface InstanceMethodsAroundInterceptor {
    /**
     * called before target method invocation.
     * 前置通知
     * @param result change this result, if you want to truncate the method.
     */
    void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
        MethodInterceptResult result) throws Throwable;

    /**
     * called after target method invocation. Even method's invocation triggers an exception.
     * 後置通知
     * @param ret the method's original return value. May be null if the method triggers an exception.
     * @return the method's actual return value.
     */
    Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
        Object ret) throws Throwable;

    /**
     * called when occur exception.
     * 異常通知
     * @param t the exception occur.
     */
    void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments,
        Class<?>[] argumentsTypes, Throwable t);
}

開發Skywalking實戰

專案maven環境配置

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>tk.shenyifeng</groupId>
    <artifactId>skywalking-plugin</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <skywalking.version>8.10.0</skywalking.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.skywalking</groupId>
            <artifactId>apm-agent-core</artifactId>
            <version>${skywalking.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.skywalking</groupId>
            <artifactId>java-agent-util</artifactId>
            <version>${skywalking.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>


    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>

            <plugin>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <shadedArtifactAttached>false</shadedArtifactAttached>
                            <createDependencyReducedPom>true</createDependencyReducedPom>
                            <createSourcesJar>true</createSourcesJar>
                            <shadeSourcesContent>true</shadeSourcesContent>
                            <relocations>
                                <relocation>
                                    <pattern>net.bytebuddy</pattern>
                                    <shadedPattern>org.apache.skywalking.apm.dependencies.net.bytebuddy</shadedPattern>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

為了更有代表性一些，使用Skywalking官方開發的ES外掛來做一個例子。為了相容不同版本框架，Skywalking 官方使用witnessClasses，當前框架Jar存在這個Class就會任務是某個版本、同樣witnessMethods當Class存在某個Method。

public class AdapterActionFutureInstrumentation extends ClassEnhancePluginDefine {

    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return new ConstructorInterceptPoint[0];
    }

    @Override
    public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() {
        return new InstanceMethodsInterceptPoint[] {
            new InstanceMethodsInterceptPoint() {
                @Override
                public ElementMatcher<MethodDescription> getMethodsMatcher() {
                    return named("actionGet"); //攔截方法
                }

                @Override
                public String getMethodsInterceptor() {  //攔截器全類名
                    return "org.apache.skywalking.apm.plugin.elasticsearch.v7.interceptor.AdapterActionFutureActionGetMethodsInterceptor";
                }

                @Override
                public boolean isOverrideArgs() {
                    return false;
                }
            }
        };
    }

    @Override
    public StaticMethodsInterceptPoint[] getStaticMethodsInterceptPoints() {
        return new StaticMethodsInterceptPoint[0];
    }

    @Override
    protected ClassMatch enhanceClass() { //增強Class
        return byName("org.elasticsearch.action.support.AdapterActionFuture");
    }

    @Override
    protected String[] witnessClasses() {//ES7 存在Class
        return new String[] {"org.elasticsearch.transport.TaskTransportChannel"};
    }

    @Override
    protected List<WitnessMethod> witnessMethods() { //ES7 SearchHits 存在方法
        return Collections.singletonList(new WitnessMethod(
            "org.elasticsearch.search.SearchHits",
          named("getTotalHits").and(takesArguments(0)).and(returns(named("org.apache.lucene.search.TotalHits")))
        ));
    }
}

建立一個給定類名的攔截器，實現InstanceMethodsAroundInterceptor介面。建立一個EntrySpan

public class TomcatInvokeInterceptor implements InstanceMethodsAroundInterceptor {

    private static boolean IS_SERVLET_GET_STATUS_METHOD_EXIST;
    private static final String SERVLET_RESPONSE_CLASS = "javax.servlet.http.HttpServletResponse";
    private static final String GET_STATUS_METHOD = "getStatus";

    static {
        IS_SERVLET_GET_STATUS_METHOD_EXIST = MethodUtil.isMethodExist(
            TomcatInvokeInterceptor.class.getClassLoader(), SERVLET_RESPONSE_CLASS, GET_STATUS_METHOD);
    }

    /**
     * * The {@link TraceSegment#ref} of current trace segment will reference to the trace segment id of the previous
     * level if the serialized context is not null.
     *
     * @param result change this result, if you want to truncate the method.
     */
    @Override
    public void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
                             MethodInterceptResult result) throws Throwable {
        Request request = (Request) allArguments[0];
        ContextCarrier contextCarrier = new ContextCarrier();

        CarrierItem next = contextCarrier.items();
       //如果 HTTP 請求頭中有符合sw8 傳輸協議的請求頭則 取出來設定到上下文ContextCarrier
        while (next.hasNext()) {
            next = next.next();
            next.setHeadValue(request.getHeader(next.getHeadKey()));
        }
        String operationName =  String.join(":", request.getMethod(), request.getRequestURI());
        AbstractSpan span = ContextManager.createEntrySpan(operationName, contextCarrier);//關聯起來
        Tags.URL.set(span, request.getRequestURL().toString()); //新增 span 引數
        Tags.HTTP.METHOD.set(span, request.getMethod());
        span.setComponent(ComponentsDefine.TOMCAT);
        SpanLayer.asHttp(span);

        if (TomcatPluginConfig.Plugin.Tomcat.COLLECT_HTTP_PARAMS) {
            collectHttpParam(request, span);
        }
    }

    @Override
    public Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
                              Object ret) throws Throwable {
        Request request = (Request) allArguments[0];
        HttpServletResponse response = (HttpServletResponse) allArguments[1];

        AbstractSpan span = ContextManager.activeSpan();
        if (IS_SERVLET_GET_STATUS_METHOD_EXIST && response.getStatus() >= 400) {
            span.errorOccurred();
            Tags.HTTP_RESPONSE_STATUS_CODE.set(span, response.getStatus());
        }
        // Active HTTP parameter collection automatically in the profiling context.
        if (!TomcatPluginConfig.Plugin.Tomcat.COLLECT_HTTP_PARAMS && span.isProfiling()) {
            collectHttpParam(request, span);
        }
        ContextManager.getRuntimeContext().remove(Constants.FORWARD_REQUEST_FLAG);
        ContextManager.stopSpan();
        return ret;
    }

    @Override
    public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments,
                                      Class<?>[] argumentsTypes, Throwable t) {
        AbstractSpan span = ContextManager.activeSpan();
        span.log(t);
    }

    private void collectHttpParam(Request request, AbstractSpan span) {
        final Map<String, String[]> parameterMap = new HashMap<>();
        final org.apache.coyote.Request coyoteRequest = request.getCoyoteRequest();
        final Parameters parameters = coyoteRequest.getParameters();
        for (final Enumeration<String> names = parameters.getParameterNames(); names.hasMoreElements(); ) {
            final String name = names.nextElement();
            parameterMap.put(name, parameters.getParameterValues(name));
        }

        if (!parameterMap.isEmpty()) {
            String tagValue = CollectionUtil.toString(parameterMap);
            tagValue = TomcatPluginConfig.Plugin.Http.HTTP_PARAMS_LENGTH_THRESHOLD > 0 ?
                StringUtil.cut(tagValue, TomcatPluginConfig.Plugin.Http.HTTP_PARAMS_LENGTH_THRESHOLD) :
                tagValue;
            Tags.HTTP.PARAMS.set(span, tagValue);
        }
    }
}

開發完成攔截器後，一定要在類路徑上新增skywalking-plugin.def檔案，將開發後的全類名新增到配置。

xxxName = tk.shenyifeng.skywalking.plugin.RepladInstrumentation

如果jar 裡面沒有這個檔案，外掛不會被Skywalking載入的。
最後將打包的jar 放到Skywalking的plugin或者activations目錄就可以了。

xml配置外掛

<?xml version="1.0" encoding="UTF-8"?>
<enhanced>
    <class class_name="test.apache.skywalking.apm.testcase.customize.service.TestService1">
        <method method="staticMethod()" operation_name="/is_static_method" static="true"></method>
        <method method="staticMethod(java.lang.String,int.class,java.util.Map,java.util.List,[Ljava.lang.Object;)"
                operation_name="/is_static_method_args" static="true">
            <operation_name_suffix>arg[0]</operation_name_suffix>
            <operation_name_suffix>arg[1]</operation_name_suffix>
            <operation_name_suffix>arg[3].[0]</operation_name_suffix>
            <tag key="tag_1">arg[2].['k1']</tag>
            <tag key="tag_2">arg[4].[1]</tag>
            <log key="log_1">arg[4].[2]</log>
        </method>
        <method method="method()" static="false"></method>
        <method method="method(java.lang.String,int.class)" operation_name="/method_2" static="false">
            <operation_name_suffix>arg[0]</operation_name_suffix>
            <tag key="tag_1">arg[0]</tag>
            <log key="log_1">arg[1]</log>
        </method>
        <method
            method="method(test.apache.skywalking.apm.testcase.customize.model.Model0,java.lang.String,int.class)"
            operation_name="/method_3" static="false">
            <operation_name_suffix>arg[0].id</operation_name_suffix>
            <operation_name_suffix>arg[0].model1.name</operation_name_suffix>
            <operation_name_suffix>arg[0].model1.getId()</operation_name_suffix>
            <tag key="tag_os">arg[0].os.[1]</tag>
            <log key="log_map">arg[0].getM().['k1']</log>
        </method>
        <method method="retString(java.lang.String)" operation_name="/retString" static="false">
            <tag key="tag_ret">returnedObj</tag>
            <log key="log_map">returnedObj</log>
        </method>
        <method method="retModel0(test.apache.skywalking.apm.testcase.customize.model.Model0)"
          operation_name="/retModel0" static="false">
            <tag key="tag_ret">returnedObj.model1.id</tag>
            <log key="log_map">returnedObj.model1.getId()</log>
        </method>
    </class>
    
</enhanced>

通過xml配置可以省去編寫Java程式碼，打包jar步驟。
xml規則

在配置檔案agent.config中新增配置：

plugin.customize.enhance_file=customize_enhance.xml的絕對路徑

引用資料
https://www.itmuch.com/skywal...
https://skyapm.github.io/docu...