IoT 邊緣叢集基於 Kubernetes Events 的告警通知實現（二）：進一步配置

東風微鳴發表於2023-02-17

上一篇文章

目標

告警恢復通知 - 經過評估無法實現
1. 原因: 告警和恢復是單獨完全不相關的事件, 告警是 Warning 級別, 恢復是 Normal 級別, 要開啟恢復, 就會導致所有 Normal Events 都會被髮送, 這個數量是很恐怖的; 而且, 除非特別有經驗和耐心, 否則無法看出哪條 Normal 對應的是告警的恢復.
未恢復進行持續告警 - 預設就帶的能力, 無需額外配置.
告警內容顯示資源名稱，比如節點和pod名稱
可以設定遮蔽特定的節點和工作負載並可以動態調整
1. 比如，叢集001中的節點worker-1做計劃性維護，期間停止監控，維護完成後重新開始監控。

配置

告警內容顯示資源名稱

典型的幾類 events:

apiVersion: v1
count: 101557
eventTime: null
firstTimestamp: "2022-04-08T03:50:47Z"
involvedObject:
  apiVersion: v1
  fieldPath: spec.containers{prometheus}
  kind: Pod
  name: prometheus-rancher-monitoring-prometheus-0
  namespace: cattle-monitoring-system
kind: Event
lastTimestamp: "2022-04-14T11:39:19Z"
message: 'Readiness probe failed: Get "http://10.42.0.87:9090/-/ready": context deadline
  exceeded (Client.Timeout exceeded while awaiting headers)'
metadata:
  creationTimestamp: "2022-04-08T03:51:17Z"
  name: prometheus-rancher-monitoring-prometheus-0.16e3cf53f0793344
  namespace: cattle-monitoring-system
reason: Unhealthy
reportingComponent: ""
reportingInstance: ""
source:
  component: kubelet
  host: master-1
type: Warning

apiVersion: v1
count: 116
eventTime: null
firstTimestamp: "2022-04-13T02:43:26Z"
involvedObject:
  apiVersion: v1
  fieldPath: spec.containers{grafana}
  kind: Pod
  name: rancher-monitoring-grafana-57777cc795-2b2x5
  namespace: cattle-monitoring-system
kind: Event
lastTimestamp: "2022-04-14T11:18:56Z"
message: 'Readiness probe failed: Get "http://10.42.0.90:3000/api/health": context
  deadline exceeded (Client.Timeout exceeded while awaiting headers)'
metadata:
  creationTimestamp: "2022-04-14T11:18:57Z"
  name: rancher-monitoring-grafana-57777cc795-2b2x5.16e5548dd2523a13
  namespace: cattle-monitoring-system
reason: Unhealthy
reportingComponent: ""
reportingInstance: ""
source:
  component: kubelet
  host: master-1
type: Warning

apiVersion: v1
count: 20958
eventTime: null
firstTimestamp: "2022-04-11T10:34:51Z"
involvedObject:
  apiVersion: v1
  fieldPath: spec.containers{lb-port-1883}
  kind: Pod
  name: svclb-emqx-dt22t
  namespace: emqx
kind: Event
lastTimestamp: "2022-04-14T11:39:48Z"
message: Back-off restarting failed container
metadata:
  creationTimestamp: "2022-04-11T10:34:51Z"
  name: svclb-emqx-dt22t.16e4d11e2b9efd27
  namespace: emqx
reason: BackOff
reportingComponent: ""
reportingInstance: ""
source:
  component: kubelet
  host: worker-1
type: Warning

apiVersion: v1
count: 21069
eventTime: null
firstTimestamp: "2022-04-11T10:34:48Z"
involvedObject:
  apiVersion: v1
  fieldPath: spec.containers{lb-port-80}
  kind: Pod
  name: svclb-traefik-r5p8t
  namespace: kube-system
kind: Event
lastTimestamp: "2022-04-14T11:44:59Z"
message: Back-off restarting failed container
metadata:
  creationTimestamp: "2022-04-11T10:34:48Z"
  name: svclb-traefik-r5p8t.16e4d11daf0b79ce
  namespace: kube-system
reason: BackOff
reportingComponent: ""
reportingInstance: ""
source:
  component: kubelet
  host: worker-1
type: Warning

{
  "metadata": {
    "name": "event-exporter-79544df9f7-xj4t5.16e5c540dc32614f",
    "namespace": "monitoring",
    "uid": "baf2f642-2383-4e22-87e0-456b6c3eaf4e",
    "resourceVersion": "14043444",
    "creationTimestamp": "2022-04-14T13:08:40Z"
  },
  "reason": "Pulled",
  "message": "Container image \"ghcr.io/opsgenie/kubernetes-event-exporter:v0.11\" already present on machine",
  "source": {
    "component": "kubelet",
    "host": "worker-2"
  },
  "firstTimestamp": "2022-04-14T13:08:40Z",
  "lastTimestamp": "2022-04-14T13:08:40Z",
  "count": 1,
  "type": "Normal",
  "eventTime": null,
  "reportingComponent": "",
  "reportingInstance": "",
  "involvedObject": {
    "kind": "Pod",
    "namespace": "monitoring",
    "name": "event-exporter-79544df9f7-xj4t5",
    "uid": "b77d3e13-fa9e-484b-8a5a-d1afc9edec75",
    "apiVersion": "v1",
    "resourceVersion": "14043435",
    "fieldPath": "spec.containers{event-exporter}",
    "labels": {
      "app": "event-exporter",
      "pod-template-hash": "79544df9f7",
      "version": "v1"
    }
  }
}

我們可以把更多的欄位加入到告警資訊中, 其中就包括:

節點: {{ Source.Host }}
Pod: {{ .InvolvedObject.Name }}

綜上, 修改後的event-exporter-cfg yaml 如下:

apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: monitoring
  resourceVersion: '5779968'
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
        - match:
            - receiver: "dump"      
        - drop:
            - type: "Normal"
          match:
            - receiver: "feishu"                     
    receivers:
      - name: "dump"
        stdout: {}
      - name: "feishu"
        webhook:
          endpoint: "https://open.feishu.cn/open-apis/bot/v2/hook/..."
          headers:
            Content-Type: application/json
          layout:
            msg_type: interactive
            card:
              config:
                wide_screen_mode: true
                enable_forward: true
              header:
                title:
                  tag: plain_text
                  content: xxx測試K3S叢集告警
                template: red
              elements:
                - tag: div
                  text: 
                    tag: lark_md
                    content: "**EventID:**  {{ .UID }}\n**EventNamespace:**  {{ .InvolvedObject.Namespace }}\n**EventName:**  {{ .InvolvedObject.Name }}\n**EventType:**  {{ .Type }}\n**EventKind:**  {{ .InvolvedObject.Kind }}\n**EventReason:**  {{ .Reason }}\n**EventTime:**  {{ .LastTimestamp }}\n**EventMessage:**  {{ .Message }}\n**EventComponent:**  {{ .Source.Component }}\n**EventHost:**  {{ .Source.Host }}\n**EventLabels:**  {{ toJson .InvolvedObject.Labels}}\n**EventAnnotations:**  {{ toJson .InvolvedObject.Annotations}}"

遮蔽特定的節點和工作負載

比如，叢集001中的節點worker-1做計劃性維護，期間停止監控，維護完成後重新開始監控。

繼續修改event-exporter-cfg yaml 如下:

apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: monitoring
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
        - match:
            - receiver: "dump"      
        - drop:
            - type: "Normal"
            - source:
                host: "worker-1"
            - namespace: "cattle-monitoring-system"
            - name: "*emqx*"
            - kind: "Pod|Deployment|ReplicaSet"
            - labels:
                version: "dev"
          match:
            - receiver: "feishu"                     
    receivers:
      - name: "dump"
        stdout: {}
      - name: "feishu"
        webhook:
          endpoint: "https://open.feishu.cn/open-apis/bot/v2/hook/..."
          headers:
            Content-Type: application/json
          layout:
            msg_type: interactive
            card:
              config:
                wide_screen_mode: true
                enable_forward: true
              header:
                title:
                  tag: plain_text
                  content: xxx測試K3S叢集告警
                template: red
              elements:
                - tag: div
                  text: 
                    tag: lark_md
                    content: "**EventID:**  {{ .UID }}\n**EventNamespace:**  {{ .InvolvedObject.Namespace }}\n**EventName:**  {{ .InvolvedObject.Name }}\n**EventType:**  {{ .Type }}\n**EventKind:**  {{ .InvolvedObject.Kind }}\n**EventReason:**  {{ .Reason }}\n**EventTime:**  {{ .LastTimestamp }}\n**EventMessage:**  {{ .Message }}\n**EventComponent:**  {{ .Source.Component }}\n**EventHost:**  {{ .Source.Host }}\n**EventLabels:**  {{ toJson .InvolvedObject.Labels}}\n**EventAnnotations:**  {{ toJson .InvolvedObject.Annotations}}"

預設的 drop 規則為: - type: "Normal", 即不對 Normal 級別進行告警;

現在加入以下規則:

            - source:
                host: "worker-1"
            - namespace: "cattle-monitoring-system"
            - name: "*emqx*"
            - kind: "Pod|Deployment|ReplicaSet"
            - labels:
                version: "dev"

... host: "worker-1": 不對節點worker-1 做告警;
... namespace: "cattle-monitoring-system": 不對 NameSpace: cattle-monitoring-system 做告警;
... name: "*emqx*": 不對 name(name 往往是 pod name) 包含 emqx 的做告警
kind: "Pod|Deployment|ReplicaSet": 不對 Pod Deployment ReplicaSet 做告警(也就是不關注應用, 元件相關的告警)
...version: "dev": 不對 label 含有 version: "dev" 的做告警(可以透過它遮蔽特定的應用的告警)

最終效果

如下圖:

Event 告警包含更多資訊

Event 告警包含更多資訊-2

???

三人行, 必有我師; 知識共享, 天下為公. 本文由東風微鳴技術部落格 EWhisper.cn 編寫.

IoT 邊緣叢集基於 Kubernetes Events 的告警通知實現
2023-02-16
基於Kubernetes v1.24.0的叢集搭建（二）
2022-06-15
搭建異構 CPU 組成的邊緣計算 Kubernetes 叢集
2020-04-26
大規模 IoT 邊緣容器叢集管理的幾種架構-0-邊緣容器及架構簡介
2023-02-19
架構
Zabbix實現電話告警通知的配置方法分享
2019-11-04
大規模 IoT 邊緣容器叢集管理的幾種架構-4-Kubeedge
2023-02-23
架構
基於containerd 部署 kubernetes 1.28叢集
2024-03-29
AI
ES系列(二)：基於多播的叢集發現實現原理解析
2021-04-18
基於Kubernetes v1.24.0的叢集搭建（三）
2022-06-17
基於Kubernetes v1.24.0的叢集搭建（一）
2022-06-14
大規模 IoT 邊緣容器叢集管理的幾種架構-1-Rancher+K3s
2023-02-20
架構
邊緣叢集場景下的映象快取
2021-06-09
快取
大規模 IoT 邊緣容器叢集管理的幾種架構-2-HashiCorp 解決方案 Nomad
2023-02-21
架構
基於 ZooKeeper 實現爬蟲叢集的監控
2021-09-09
爬蟲
快速實現釘釘告警通知、處理告警、關閉告警
2020-03-19
基於KubeEdge的邊緣節點分組管理設計與實現
2022-09-13
基於Jenkins + Argo 實現多叢集的持續交付
2024-03-20
JenkinsGo
如何實現告警的郵件通知方式？
2020-04-02
基於Ubuntu部署企業級kubernetes叢集---k8s叢集容部署
2024-08-29
UbuntuK8S
Kubernetes安裝之三：etcd叢集的配置
2019-02-14
基於istio實現單叢集地域故障轉移
2024-04-10
基於Docker和Kubernetes構建的部署管理和叢集管理
2018-08-07
Docker
starrocks基於prometheus實現監控告警
2023-04-26
Prometheus
如何快速實現告警簡訊的通知方式？
2020-03-26
基於zookeeper實現分散式配置中心（二）
2019-06-06
分散式
Kubernetes叢集部署史上最詳細（二）Prometheus監控Kubernetes叢集
2019-04-09
Prometheus
實現Kubernetes跨叢集服務應用的高可用
2019-04-15
邊緣計算，如何啃下叢集管理這塊硬骨頭?
2020-09-08
阿里雲的“終端雲化”實踐，基於ENS進行邊緣架構構建
2022-09-08
阿里架構
IoT邊緣，你究竟是何方神聖？
2021-09-03
在大規模 Kubernetes 叢集上實現高 SLO 的方法
2020-11-06
雲上影片業務基於邊緣容器的技術實踐
2021-01-12
阿里雲AIoT正式釋出IoT安全中心和IoT Studio 3.0，進一步鞏固AIoT雲網邊端基礎能力
2021-03-11
阿里AI
邊緣計算k8s叢集SuperEdge初體驗
2021-01-21
K8S
基於Dokcer搭建Redis叢集（主從叢集）
2020-12-10
Redis
Grafana系列-Loki-基於日誌實現告警
2023-12-10
GrafanaLoki
GraalVM(雲原生時代的Java)和IoT在邊緣側落地與實踐
2023-04-25
LVMJava
OpenYurt：延伸原生 Kubernetes 到邊緣場景下的落地實踐
2021-02-04

IoT 邊緣叢集基於 Kubernetes Events 的告警通知實現（二）：進一步配置

上一篇文章

目標

配置

告警內容顯示資源名稱

遮蔽特定的節點和工作負載

最終效果

相關文章