K8S Canal基於Prometheus進行實時指標監控

zhao發表於2020-11-06
K8SPrometheus指標

文章來源於本人的印象筆記，如出現格式問題可訪問該連結檢視原文

部署canal的prometheus監控到k8s中

1、grafana的docker部署方式；https://grafana.com/grafana/download?platform=docker
2、prometheus的docker部署方式： https://github.com/prometheus/prometheus
有了現成的docker映象後，直接部署即可；

k8s中部署prometheus

yml編排檔案如下：

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '16'
    k8s.kuboard.cn/displayName: canal-prometheus
    k8s.kuboard.cn/ingress: 'false'
    k8s.kuboard.cn/service: NodePort
    k8s.kuboard.cn/workload: svc-canal-prometheus
  creationTimestamp: '2020-11-06T03:09:55Z'
  generation: 16
  labels:
    k8s.kuboard.cn/layer: svc
    k8s.kuboard.cn/name: svc-canal-prometheus
  name: svc-canal-prometheus
  namespace: canal-ns
  resourceVersion: '22246892'
  selfLink: /apis/apps/v1/namespaces/canal-ns/deployments/svc-canal-prometheus
  uid: 4ad37eec-3b36-4107-8ed9-07456abba5ba
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s.kuboard.cn/layer: svc
      k8s.kuboard.cn/name: svc-canal-prometheus
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: '2020-11-06T14:07:33+08:00'
      labels:
        k8s.kuboard.cn/layer: svc
        k8s.kuboard.cn/name: svc-canal-prometheus
    spec:
      containers:
        - image: prom/prometheus
          imagePullPolicy: Always
          name: canal-prometheus
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/prometheus/prometheus.yml
              name: canal-prometheus-volume
              subPath: etc/prometheus/prometheus.yml
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      volumes:
        - configMap:
            defaultMode: 420
            items:
              - key: prometheus.yml
                path: etc/prometheus/prometheus.yml
            name: canal-prometheus
          name: canal-prometheus-volume
status:
  availableReplicas: 1
  conditions:
    - lastTransitionTime: '2020-11-06T05:30:16Z'
      lastUpdateTime: '2020-11-06T05:30:16Z'
      message: Deployment has minimum availability.
      reason: MinimumReplicasAvailable
      status: 'True'
      type: Available
    - lastTransitionTime: '2020-11-06T05:55:49Z'
      lastUpdateTime: '2020-11-06T06:07:43Z'
      message: >-
        ReplicaSet "svc-canal-prometheus-6f7d7b66c5" has successfully
        progressed.
      reason: NewReplicaSetAvailable
      status: 'True'
      type: Progressing
  observedGeneration: 16
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

此處有一個有趣的點是，k8s中使用configMap進行目錄掛載時，一般情況下我們直接掛載到對應的容器目錄後，此時目錄將會被覆蓋，而此時在配置普羅米修斯的檔案對映時，則只是掛載到了具體的檔案中；主要的配置則是：subPath: etc/prometheus/prometheus.yml，通過subPath的方式可以直接掛載到具體的檔案中；

原創宣告：作者：Arnold.zhao 部落格園地址：https://www.cnblogs.com/zh94

所掛載的具體配置檔案的內容如下：

# my global config test 
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'canal'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['svc-canal-deployer:11112']

主要是配置下對應的canal-deployer的地址即可；

  • targets: ['svc-canal-deployer:11112']
    預設情況下canal-deployer的監控埠就是11112，當然如果你修改過該埠，另當別論了 。
    svc-canal-deployer是canal-deployer的server名稱，由於canal-deployer此處也是已經部署在k8s了所以直接使用服務名進行訪問，由k8s service自動做轉發即可，如果此處不是在k8s的話，則直接配置canal-deployer的ip地址即可，一樣的。

對應的configMap的建立yml如下：

---
apiVersion: v1
data:
  prometheus.yml: >-
    # my global config test 

    global:
      scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).

    # Alertmanager configuration

    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          # - alertmanager:9093

    # Load rules once and periodically evaluate them according to the global
    'evaluation_interval'.

    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"

    # A scrape configuration containing exactly one endpoint to scrape:

    # Here it's Prometheus itself.

    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'canal'

        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.

        static_configs:
        - targets: ['svc-canal-deployer:11112']
kind: ConfigMap
metadata:
  creationTimestamp: '2020-11-06T03:15:04Z'
  name: canal-prometheus
  namespace: canal-ns
  resourceVersion: '22246778'
  selfLink: /api/v1/namespaces/canal-ns/configmaps/canal-prometheus
  uid: 2918cb4e-acd6-4c82-9a2e-19e575ba6cea

建立一個專用的Service進行埠對映，此處對映埠為9090

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    k8s.kuboard.cn/displayName: canal-prometheus
    k8s.kuboard.cn/workload: svc-canal-prometheus
  creationTimestamp: '2020-11-06T03:11:34Z'
  labels:
    k8s.kuboard.cn/layer: svc
    k8s.kuboard.cn/name: svc-canal-prometheus
  name: svc-canal-prometheus
  namespace: canal-ns
  resourceVersion: '22204370'
  selfLink: /api/v1/namespaces/canal-ns/services/svc-canal-prometheus
  uid: 6246dafe-f8fd-42ec-8e27-caf38539d35c
spec:
  clusterIP: 10.204.71.228
  externalTrafficPolicy: Cluster
  ports:
    - name: prometheus-9090
      nodePort: 30018
      port: 9090
      protocol: TCP
      targetPort: 9090
  selector:
    k8s.kuboard.cn/layer: svc
    k8s.kuboard.cn/name: svc-canal-prometheus
  sessionAffinity: None
  type: NodePort

好了，部署完成，此時訪問埠結果如下：

部署grafana

Deployment的yml如下

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
    k8s.kuboard.cn/ingress: 'false'
    k8s.kuboard.cn/service: NodePort
    k8s.kuboard.cn/workload: web-canal-grafana
  creationTimestamp: '2020-11-06T06:10:27Z'
  generation: 1
  labels:
    k8s.kuboard.cn/layer: web
    k8s.kuboard.cn/name: web-canal-grafana
  name: web-canal-grafana
  namespace: canal-ns
  resourceVersion: '22247823'
  selfLink: /apis/apps/v1/namespaces/canal-ns/deployments/web-canal-grafana
  uid: 484350e4-b408-4361-ac27-633f8d815468
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s.kuboard.cn/layer: web
      k8s.kuboard.cn/name: web-canal-grafana
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s.kuboard.cn/layer: web
        k8s.kuboard.cn/name: web-canal-grafana
    spec:
      containers:
        - image: grafana/grafana
          imagePullPolicy: Always
          name: canal-grafana
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsConfig: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        seLinuxOptions: {}
      terminationGracePeriodSeconds: 30

再進行一下server代理，yml如下；對映埠為3000埠

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    k8s.kuboard.cn/workload: web-canal-grafana
  creationTimestamp: '2020-11-06T06:10:27Z'
  labels:
    k8s.kuboard.cn/layer: web
    k8s.kuboard.cn/name: web-canal-grafana
  name: web-canal-grafana
  namespace: canal-ns
  resourceVersion: '22247699'
  selfLink: /api/v1/namespaces/canal-ns/services/web-canal-grafana
  uid: 8cbde138-6855-4eaf-b9cd-8dae72a2efeb
spec:
  clusterIP: 10.204.195.124
  externalTrafficPolicy: Cluster
  ports:
    - name: canal-grafana-3000
      nodePort: 31010
      port: 3000
      protocol: TCP
      targetPort: 3000
  selector:
    k8s.kuboard.cn/layer: web
    k8s.kuboard.cn/name: web-canal-grafana
  sessionAffinity: None
  type: NodePort

OK，啟動後，訪問對應的應用後，效果如圖所示：

此時還沒有進行grefana的配置，所以無法獲取canal的監控資訊

Greafana配置canal監控資訊

很簡單，基本按照官方的說明即可,
新建一個dataSource，選擇prometheus，然後填寫對應的prometheus的url地址即可。

填寫完成後，最後一步則是，grafana中匯入 canal的監控指標

此時按照URL的匯入方式：https://raw.githubusercontent.com/alibaba/canal/master/deployer/src/main/resources/metrics/Canal_instances_tmpl.json
匯入該json檔案即可；
不過我在處理的時候，該url匯入不可用，所以，如果你也是不可用的話，可以直接
wget https://raw.githubusercontent.com/alibaba/canal/master/deployer/src/main/resources/metrics/Canal_instances_tmpl.json
獲取對應的檔案內容後，再填寫到第二個框中即可；

最終的監控效果圖

原創宣告：作者：Arnold.zhao 部落格園地址：https://www.cnblogs.com/zh94

具體各監控指標所表示的含義，直接看github canal文件即可；
由於此處我的canal client並沒有直接通過tcp的方式和canal -deployer進行互動，而是直接接入的canal-deployer所吐出來的kakfa資料，所以此處所展示的client指標則為空；

參考連結：
關於普羅米修斯（prometheus）的基本概念
關於Grafana的基本概念
關於canal安裝Grafana&prometheus進行監控的說明
關於k8s中目錄掛載時不覆蓋容器原目錄的用法

相關文章