系列導航
- Istio Mixer Adapter開發 (一)K8S環境搭建
- Istio Mixer Adapter開發 (二)Istio環境搭建
- Istio Mixer Adapter開發 (三)自定義Mixer Grpc Adapter部署
概述
針對Istio Mixer Adapter的開發,後面會有文章專門講述原始碼分析及開發除錯的過程,這裡僅使用前兩篇文章講述的K8S+Istio環境部署我已經開發好的Istio Mixer Grpc Adapter並配置K8S使其生效
gatewaymetric簡介
gatewaymetric是筆者借鑑Istio官網指導開發的Istio Mixer Grpc Adapter,他的功能主要是處理Mixer上報的http訪問日誌,收集南北向流量(指通過istio-ingressgateway從叢集外進入到叢集)的訪問日誌,通過呼叫promethues官方go程式庫,記憶體生成metrics並暴露exporter埠給promethues採集資料,它的功能目前還很不完善,但是作為學習說明已經足夠
本文主要參考官網Wiki進行開發,裡面也講到了部分原理
下面是程式碼結構
➜ gatewaymetric git:(v1.0.5) ✗ tree
.
├── cmd
│ └── main.go
├── config
│ ├── adapter.gatewaymetric.config.pb.html
│ ├── config.pb.go
│ ├── config.proto
│ ├── config.proto_descriptor
│ └── gatewaymetric.yaml
├── gatewaymetric.go
├── sample_operator_cfg.yaml
└── testdata
├── attributes.yaml
├── gatewaymetric.yaml
├── sample_operator_cfg.yaml
└── template.yaml
複製程式碼
上述檔案中cmd/main.go為程式入口,其暴露兩個埠
1)http:9145為promethues的資料抓取埠
2)grpc:41543為GrpcAdapter的通訊埠,也就是與Mixer建立連線的埠,這個後面會用到,要把它顯示註冊到Mixer當中,使Mixer感知它
構建自定義Mixer Grpc Adapter(gatewaymetric)
我們第一步需要打包這個Adapter並製作成Docker映象釋出到Docker Hub中
➜ cmd git:(v1.0.5) ✗ cd /home/ubuntu/go/src/istio.io/istio/mixer/adapter/gatewaymetric/cmd
➜ cmd git:(master) CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric
➜ cmd git:(v1.0.5) ✗ CGO_ENABLE➜ cmd git:(v1.0.5) ✗ ./gatewaymetric
adapter listening on "[::]:41543"
promethues listening on ":9145"
D=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o gatewaymetric
複製程式碼
通過上述命令,我們基於原始碼構建了一個我們定製的GrpcAdapter並且成功啟動了它,如果讀者無法正常執行結束go build命令,則可能是go環境配置的問題,可以繞開這裡直接使用原始碼cmd下面的gatewaymetric檔案即可,筆者已經構建好了二進位制檔案,接下來我們編寫Dockerfile
➜ cmd git:(v1.0.5) ✗ touch Dockerfile
➜ cmd git:(v1.0.5) ✗ vim Dockerfile
FROM scratch
ADD gatewaymetric /
ENTRYPOINT ["/gatewaymetric"]
複製程式碼
該Dockerfile使用scratch基礎映象並將gatewaymetirc(我們自定義的Mixer Grpc Adapter後面都會以此稱呼)加入到映象當中,啟動入口為"/gatewaymetric"
下面開始構建
➜ cmd git:(v1.0.5) ✗ sudo docker build -f Dockerfile .
Sending build context to Docker daemon 17.87MB
Step 1/3 : FROM scratch
--->
Step 2/3 : ADD gatewaymetric /
---> 82ca9cdf54ab
Step 3/3 : ENTRYPOINT ["/gatewaymetric"]
---> Running in 2630beb8da5d
Removing intermediate container 2630beb8da5d
---> 3747dfc14d90
Successfully built 3747dfc14d90
➜ cmd git:(v1.0.5) ✗ sudo docker run -p 9145:9145 -p 41543:41543 3747dfc14d90
adapter listening on "[::]:41543"
promethues listening on ":9145"
複製程式碼
通過模擬程式測試通過(這裡使用的是前面提到的Wiki當中描述的本地模擬測試的辦法),說明打的Docker映象沒有問題,若讀者還沒有了解這塊的知識,忽略這裡即可,後面文章會詳細描述如何在本地開發及除錯Adapter
接下來,我們將該映象提交到Docker Hub(這裡講述了開發構建並推送映象的過程,讀者可以跳過此步驟直接使用筆者已經構建好併發布到docker hub上的映象即可)
➜ cmd git:(v1.0.5) ✗ sudo docker tag 3747dfc14d90 zrbcool/gatewaymetric:v1.0
➜ cmd git:(v1.0.5) ✗ sudo docker push zrbcool/gatewaymetric:v1.0
The push refers to repository [docker.io/zrbcool/gatewaymetric]
115a048ee88f: Pushed
v1.0: digest: sha256:616fc9e1d91ad963a5ace4b5210caecbde21aca8ea5f2bfe8f98ac20e33a6cfa size: 528
複製程式碼
部署Adapter到K8S + Istio叢集
到K8S + Istio叢集的VM中,部署已經制作好的映象
➜ gatewaymetric cat deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: gatewaymetric
namespace: istio-system
labels:
app: gatewaymetric
spec:
replicas: 1
selector:
matchLabels:
app: gatewaymetric
template:
metadata:
labels:
app: gatewaymetric
spec:
containers:
- name: gatewaymetric
image: "zrbcool/gatewaymetric:v1.0"
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 9145
- containerPort: 41543
複製程式碼
然後建立Service
➜ gatewaymetric cat service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: gatewaymetric
name: gatewaymetric
namespace: istio-system
spec:
selector:
app: gatewaymetric
ports:
type: ClusterIP
ports:
- name: promethues
port: 9145
targetPort: 9145
- name: grpc
port: 41543
targetPort: 41543
複製程式碼
簡單驗證,埠已通,promethues的metrics也可以獲取
➜ gatewaymetric kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
gatewaymetric ClusterIP 10.104.182.140 <none> 9145/TCP,41543/TCP
➜ gatewaymetric telnet 10.104.182.140 41543
Trying 10.104.182.140...
Connected to 10.104.182.140.
Escape character is '^]'.
➜ gatewaymetric curl 10.104.182.140:9145/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
...
複製程式碼
至此,我們的Mixer Grpc Adapter已經部署到了我們的K8S叢集當中,接下來我們需要配置CRD,使mixer能夠感知到我們的Grpc Adapter
部署下面檔案的內容,其包含了自定義Adapter的執行時例項handler,template->logentity的執行時例項instance,以及mixer的匹配規則rule指定handler處理instance的Report資料,如果上面這句話看的有點蒙,不要緊這個都是Mixer的設計者抽象出來的概念,目的就是讓Mixer介面卡的開發者跟運維者分離並使Mixer Adapter的開發更加靈活,後面會有專門章節詳細闡述其各個模型的意義,如果讀者想提前瞭解,請參考下面連結:
# handler for adapter gatewaymetric
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
name: h1
namespace: istio-system
spec:
adapter: gatewaymetric
connection:
address: "gatewaymetric.istio-system.svc.cluster.local:41543" #replaces at runtime by the test
params:
file_path: "out.txt"
---
# instance for template metric
apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
name: i1metric
namespace: istio-system
spec:
template: logentry
params:
severity: '"warning"'
timestamp: request.time
variables:
host: request.host
method: request.method | ""
url: request.path | ""
responseCode: response.code | 0
responseBodySize: response.size | 0
responseTotalSize: response.total_size | 0
latency: response.duration | "0ms"
monitored_resource_type: '"UNSPECIFIED"'
---
# rule to dispatch to handler h1
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: r1
namespace: istio-system
spec:
actions:
- handler: h1.istio-system
instances:
- i1metric
---
複製程式碼
在Kubernetes中部署它們
➜ gatewaymetric kubectl apply -f attributes.yaml
attributemanifest.config.istio.io/istio-proxy created
attributemanifest.config.istio.io/kubernetes configured
➜ gatewaymetric kubectl apply -f template.yaml
template.config.istio.io/logentry created
➜ gatewaymetric kubectl apply -f gatewaymetric.yaml
adapter.config.istio.io/gatewaymetric created
➜ gatewaymetric kubectl apply -f operator_cfg.yaml
handler.config.istio.io/h1 created
instance.config.istio.io/i1metric created
rule.config.istio.io/r1 created
複製程式碼
使用瀏覽器訪問http://192.168.101.6:31380/productpage,並檢視Mixer的日誌,看到
{"level":"info","time":"2019-02-21T03:05:40.292749Z","instance":"accesslog.logentry.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"unknown","destinationApp":"","destinationIp":"10.32.0.34","destinationName":"","destinationNamespace":"","destinationOwner":"","destinationPrincipal":"","destinationServiceHost":"productpage.default.svc.cluster.local","destinationWorkload":"","httpAuthority":"192.168.101.6:31380","latency":"25.101758ms","method":"GET","protocol":"http","receivedBytes":488,"referer":"","reporter":"source","requestId":"405ed7ca-c988-4cfa-8e6a-e808e2cbeab8","requestSize":0,"requestedServerName":"","responseCode":200,"responseSize":5719,"responseTimestamp":"2019-02-21T03:05:40.317790Z","sentBytes":5858,"sourceApp":"","sourceIp":"0.0.0.0","sourceName":"","sourceNamespace":"istio-system","sourceOwner":"","sourcePrincipal":"","sourceWorkload":"","url":"/productpage","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36","xForwardedFor":"10.32.0.1"}
複製程式碼
檢視我們自定義的Grpc Adapter日誌,看到
2019-02-21T03:05:41.002442Z info received request {[&InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:34.547407ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:39.955797905Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:39.579628ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5723,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5862,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.126271976Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:24.084598ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:5719,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:5858,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.293294914Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,} &InstanceMsg{Variables:map[string]*istio_policy_v1beta11.Value{host: &Value{Value:&Value_StringValue{StringValue:192.168.101.6:31380,},},latency: &Value{Value:&Value_DurationValue{DurationValue:&Duration{Value:30.813597ms,},},},method: &Value{Value:&Value_StringValue{StringValue:GET,},},responseBodySize: &Value{Value:&Value_Int64Value{Int64Value:4415,},},responseCode: &Value{Value:&Value_Int64Value{Int64Value:200,},},responseTotalSize: &Value{Value:&Value_Int64Value{Int64Value:4554,},},url: &Value{Value:&Value_StringValue{StringValue:/productpage,},},},Timestamp:&istio_policy_v1beta11.TimeStamp{Value:2019-02-21T03:05:40.55426223Z,},Severity:warning,MonitoredResourceType:UNSPECIFIED,MonitoredResourceDimensions:map[string]*istio_policy_v1beta11.Value{},Name:i1metric.instance.istio-system,}] &Any{TypeUrl:type.googleapis.com/adapter.gatewaymetric.config.Params,Value:[10 7 111 117 116 46 116 120 116],} 7051031778836507932}
2019-02-21T03:05:41.002468Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002473Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002475Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002477Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002479Z info responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002483Z info responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002487Z info latency: %!(EXTRA float64=0.034547407)
2019-02-21T03:05:41.002500Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002506Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002508Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002510Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002512Z info responseBodySize: %!(EXTRA int64=5723)
2019-02-21T03:05:41.002514Z info responseTotalSize: %!(EXTRA int64=5862)
2019-02-21T03:05:41.002516Z info latency: %!(EXTRA float64=0.039579628)
2019-02-21T03:05:41.002520Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002522Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002524Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002526Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002528Z info responseBodySize: %!(EXTRA int64=5719)
2019-02-21T03:05:41.002530Z info responseTotalSize: %!(EXTRA int64=5858)
2019-02-21T03:05:41.002532Z info latency: %!(EXTRA float64=0.024084598)
2019-02-21T03:05:41.002535Z info host: %!(EXTRA string=192.168.101.6:31380)
2019-02-21T03:05:41.002537Z info url: %!(EXTRA string=/productpage)
2019-02-21T03:05:41.002538Z info method: %!(EXTRA string=GET)
2019-02-21T03:05:41.002540Z info responseCode: %!(EXTRA string=200)
2019-02-21T03:05:41.002542Z info responseBodySize: %!(EXTRA int64=4415)
2019-02-21T03:05:41.002544Z info responseTotalSize: %!(EXTRA int64=4554)
2019-02-21T03:05:41.002547Z info latency: %!(EXTRA float64=0.030813597)
複製程式碼
再看下promethues暴露的9145埠是否有資料呢
➜ gatewaymetric curl 10.104.182.140:9145/metrics
...
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0002899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-0.0001899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="-8.999999999999979e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="1.0000000000000216e-05"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00011000000000000022"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.00021000000000000023"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0003100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0004100000000000002"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0005100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0006100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0007100000000000003"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0008100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="0.0009100000000000004"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200",le="+Inf"} 34
http_request_seconds_histogram_sum{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 1.1806039740000003
http_request_seconds_histogram_count{endpoint="/test",fullurl="/productpage",host="192.168.101.6:31380",method="GET",status="200"} 34
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00099"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.00089"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0007899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0006899999999999999"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0005899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0004899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0003899999999999998"} 0
http_request_seconds_histogram_bucket{endpoint="/test",fullurl="/ratings/0",host="ratings:9080",method="GET",status="200",le="-0.0002899999999999998"} 0
...
複製程式碼
說明一切已經生效,have fun!