Istio Mixer Adapter開發 (三)自定義Mixer Grpc Adapter部署

I'm Robin發表於2019-02-20

系列導航

概述

針對Istio Mixer Adapter的開發,後面會有文章專門講述原始碼分析及開發除錯的過程,這裡僅使用前兩篇文章講述的K8S+Istio環境部署我已經開發好的Istio Mixer Grpc Adapter並配置K8S使其生效

gatewaymetric簡介

gatewaymetric是筆者借鑑Istio官網指導開發的Istio Mixer Grpc Adapter,他的功能主要是處理Mixer上報的http訪問日誌,收集南北向流量(指通過istio-ingressgateway從叢集外進入到叢集)的訪問日誌,通過呼叫promethues官方go程式庫,記憶體生成metrics並暴露exporter埠給promethues採集資料,它的功能目前還很不完善,但是作為學習說明已經足夠

本文主要參考官網Wiki進行開發,裡面也講到了部分原理

github.com/istio/istio…

下面是程式碼結構

➜  gatewaymetric git:(v1.0.5) ✗ tree
.
├── cmd
│   └── main.go
├── config
│   ├── adapter.gatewaymetric.config.pb.html
│   ├── config.pb.go
│   ├── config.proto
│   ├── config.proto_descriptor
│   └── gatewaymetric.yaml
├── gatewaymetric.go
├── sample_operator_cfg.yaml
└── testdata
    ├── attributes.yaml
    ├── gatewaymetric.yaml
    ├── sample_operator_cfg.yaml
    └── template.yaml
複製程式碼

上述檔案中cmd/main.go為程式入口,其暴露兩個埠
1)http:9145為promethues的資料抓取埠
2)grpc:41543為GrpcAdapter的通訊埠,也就是與Mixer建立連線的埠,這個後面會用到,要把它顯示註冊到Mixer當中,使Mixer感知它

構建自定義Mixer Grpc Adapter(gatewaymetric)

我們第一步需要打包這個Adapter並製作成Docker映象釋出到Docker Hub中

➜  cmd git:(v1.0.5) ✗ cd /home/ubuntu/go/src/istio.io/istio/mixer/adapter/gatewaymetric/cmd
➜  cmd git:(master) CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric
➜  cmd git:(v1.0.5) ✗ CGO_ENABLE➜  cmd git:(v1.0.5) ✗ ./gatewaymetric 
adapter listening on "[::]:41543"
promethues listening on ":9145"
D=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric
複製程式碼

通過上述命令,我們基於原始碼構建了一個我們定製的GrpcAdapter並且成功啟動了它,如果讀者無法正常執行結束go build命令,則可能是go環境配置的問題,可以繞開這裡直接使用原始碼cmd下面的gatewaymetric檔案即可,筆者已經構建好了二進位制檔案,接下來我們編寫Dockerfile

➜  cmd git:(v1.0.5) ✗ touch Dockerfile
➜  cmd git:(v1.0.5) ✗ vim Dockerfile
FROM scratch
ADD gatewaymetric /
ENTRYPOINT ["/gatewaymetric"]
複製程式碼

該Dockerfile使用scratch基礎映象並將gatewaymetirc(我們自定義的Mixer Grpc Adapter後面都會以此稱呼)加入到映象當中,啟動入口為"/gatewaymetric"
下面開始構建

➜  cmd git:(v1.0.5) ✗ sudo docker build -f Dockerfile .
Sending build context to Docker daemon  17.87MB
Step 1/3 : FROM scratch
 ---> 
Step 2/3 : ADD gatewaymetric /
 ---> 82ca9cdf54ab
Step 3/3 : ENTRYPOINT ["/gatewaymetric"]
 ---> Running in 2630beb8da5d
Removing intermediate container 2630beb8da5d
 ---> 3747dfc14d90
Successfully built 3747dfc14d90
➜  cmd git:(v1.0.5) ✗ sudo docker run -p 9145:9145 -p 41543:41543 3747dfc14d90
adapter listening on "[::]:41543"
promethues listening on ":9145"
複製程式碼

通過模擬程式測試通過(這裡使用的是前面提到的Wiki當中描述的本地模擬測試的辦法),說明打的Docker映象沒有問題,若讀者還沒有了解這塊的知識,忽略這裡即可,後面文章會詳細描述如何在本地開發及除錯Adapter

Istio Mixer Adapter開發 (三)自定義Mixer Grpc Adapter部署

接下來,我們將該映象提交到Docker Hub(這裡講述了開發構建並推送映象的過程,讀者可以跳過此步驟直接使用筆者已經構建好併發布到docker hub上的映象即可)

➜  cmd git:(v1.0.5) ✗ sudo docker tag 3747dfc14d90 zrbcool/gatewaymetric:v1.0  
➜  cmd git:(v1.0.5) ✗ sudo docker push zrbcool/gatewaymetric:v1.0            
The push refers to repository [docker.io/zrbcool/gatewaymetric]
115a048ee88f: Pushed 
v1.0: digest: sha256:616fc9e1d91ad963a5ace4b5210caecbde21aca8ea5f2bfe8f98ac20e33a6cfa size: 528
複製程式碼

部署Adapter到K8S + Istio叢集

到K8S + Istio叢集的VM中,部署已經制作好的映象

➜  gatewaymetric cat deployment.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gatewaymetric
  namespace: istio-system
  labels:
    app: gatewaymetric
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gatewaymetric
  template:
    metadata:
      labels:
        app: gatewaymetric
    spec:
      containers:
      - name: gatewaymetric
        image: "zrbcool/gatewaymetric:v1.0"
        imagePullPolicy: "IfNotPresent"
        ports:
        - containerPort: 9145
        - containerPort: 41543
複製程式碼

然後建立Service

➜  gatewaymetric cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: gatewaymetric
  name: gatewaymetric
  namespace: istio-system
spec:
  selector:
    app: gatewaymetric
  ports:
  type: ClusterIP
  ports:
  - name: promethues
    port: 9145
    targetPort: 9145
  - name: grpc
    port: 41543
    targetPort: 41543
複製程式碼

簡單驗證,埠已通,promethues的metrics也可以獲取

➜  gatewaymetric kubectl get svc -n istio-system
NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)           
gatewaymetric            ClusterIP   10.104.182.140   <none>        9145/TCP,41543/TCP     
➜  gatewaymetric telnet 10.104.182.140 41543
Trying 10.104.182.140...
Connected to 10.104.182.140.
Escape character is '^]'.
➜  gatewaymetric curl 10.104.182.140:9145/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
...
複製程式碼

至此,我們的Mixer Grpc Adapter已經部署到了我們的K8S叢集當中,接下來我們需要配置CRD,使mixer能夠感知到我們的Grpc Adapter
部署下面檔案的內容,其包含了自定義Adapter的執行時例項handler,template->logentity的執行時例項instance,以及mixer的匹配規則rule指定handler處理instance的Report資料,如果上面這句話看的有點蒙,不要緊這個都是Mixer的設計者抽象出來的概念,目的就是讓Mixer介面卡的開發者跟運維者分離並使Mixer Adapter的開發更加靈活,後面會有專門章節詳細闡述其各個模型的意義,如果讀者想提前瞭解,請參考下面連結:

istio.io/docs/concep…

# handler for adapter gatewaymetric
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
 name: h1
 namespace: istio-system
spec:
 adapter: gatewaymetric
 connection:
   address: "gatewaymetric.istio-system.svc.cluster.local:41543" #replaces at runtime by the test
 params:
   file_path: "out.txt"
---

# instance for template metric
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
  name: i1metric
  namespace: istio-system
spec:
  template: logentry
  params:
    severity: '"warning"'
    timestamp: request.time
    variables:
      host: request.host
      method: request.method | ""
      url: request.path | ""
      responseCode: response.code | 0
      responseBodySize: response.size | 0
      responseTotalSize: response.total_size | 0
      latency: response.duration | "0ms"
    monitored_resource_type: '"UNSPECIFIED"'
---

# rule to dispatch to handler h1
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: r1
  namespace: istio-system
spec:
  actions:
    - handler: h1.istio-system
      instances:
        - i1metric
---
複製程式碼

在Kubernetes中部署它們

➜  gatewaymetric kubectl apply -f attributes.yaml 
attributemanifest.config.istio.io/istio-proxy created
attributemanifest.config.istio.io/kubernetes configured
➜  gatewaymetric kubectl apply -f template.yaml  
template.config.istio.io/logentry created
➜  gatewaymetric kubectl apply -f gatewaymetric.yaml 
adapter.config.istio.io/gatewaymetric created
➜  gatewaymetric kubectl apply -f operator_cfg.yaml 
handler.config.istio.io/h1 created
instance.config.istio.io/i1metric created
rule.config.istio.io/r1 created
複製程式碼

使用瀏覽器訪問http://192.168.101.6:31380/productpage,並檢視Mixer的日誌,看到

{"level":"info","time":"2019-02-21T03:05:40.292749Z","instance":"accesslog.logentry.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"unknown","destinationApp":"","destinationIp":"10.32.0.34","destinationName":"","destinationNamespace":"","destinationOwner":"","destinationPrincipal":"","destinationServiceHost":"productpage.default.svc.cluster.local","destinationWorkload":"","httpAuthority":"192.168.101.6:31380","latency":"25.101758ms","method":"GET","protocol":"http","receivedBytes":488,"referer":"","reporter":"source","requestId":"405ed7ca-c988-4cfa-8e6a-e808e2cbeab8","requestSize":0,"requestedServerName":"","responseCode":200,"responseSize":5719,"responseTimestamp":"2019-02-21T03:05:40.317790Z","sentBytes":5858,"sourceApp":"","sourceIp":"0.0.0.0","sourceName":"","sourceNamespace":"istio-system","sourceOwner":"","sourcePrincipal":"","sourceWorkload":"","url":"/productpage","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36","xForwardedFor":"10.32.0.1"}
複製程式碼

檢視我們自定義的Grpc Adapter日誌,看到

2019-02-21T03:05:41.002442Z	info	received request {[&InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:34.547407ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:39.955797905Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:39.579628ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5723,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5862,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.126271976Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:24.084598ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.293294914Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:30.813597ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:4415,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:4554,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.55426223Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,}] &Any{TypeUrl:type.googleapis.com/adapter.gatewaymetric.config.Params,Value:[10 7 111 117 116 46 116 120 116],} 7051031778836507932}

2019-02-21T03:05:41.002468Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002473Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002475Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002477Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002479Z	info	responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002483Z	info	responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002487Z	info	latency: %!(EXTRA float64=0.034547407)
2019-02-21T03:05:41.002500Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002506Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002508Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002510Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002512Z	info	responseBodySize: %!(EXTRA int64=5723)
2019-02-21T03:05:41.002514Z	info	responseTotalSize: %!(EXTRA int64=5862)
2019-02-21T03:05:41.002516Z	info	latency: %!(EXTRA float64=0.039579628)
2019-02-21T03:05:41.002520Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002522Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002524Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002526Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002528Z	info	responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002530Z	info	responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002532Z	info	latency: %!(EXTRA float64=0.024084598)
2019-02-21T03:05:41.002535Z	info	host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002537Z	info	url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002538Z	info	method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002540Z	info	responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002542Z	info	responseBodySize: %!(EXTRA int64=4415)
2019-02-21T03:05:41.002544Z	info	responseTotalSize: %!(EXTRA int64=4554)
2019-02-21T03:05:41.002547Z	info	latency: %!(EXTRA float64=0.030813597)
複製程式碼

再看下promethues暴露的9145埠是否有資料呢

➜  gatewaymetric curl 10.104.182.140:9145/metrics
...
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0002899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0001899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-8.999999999999979e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="1.0000000000000216e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00011000000000000022"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00021000000000000023"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0003100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0004100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0005100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0006100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0007100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0008100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0009100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="+Inf"} 34
http_request_seconds_histogram_sum{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 1.1806039740000003
http_request_seconds_histogram_count{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 34
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0002899999999999998"} 0
...
複製程式碼

說明一切已經生效,have fun!

相關文章