Filebeat 收集K8S 日誌，生產環境實踐

PassZhang發表於2020-08-11
K8S

根據生產環境要求，需要採集K8S Pod 日誌，和開發協商之後，Pod中應用會將日誌輸出到容器終端上，這時可以直接用filebeat 採集node節點上面的/var/log/containers/*.log日誌，然後將日誌輸出到kafka訊息佇列中，經過kafka將日誌寫入logstash進行格式化，然後由logstash傳入elasticsearch儲存，然後kibana會連線elasticsearch展示索引資料。

資料傳輸流程：Pod -> /var/log/containers/*.log -> Filebeat -> Kafka叢集 -> Logstash -> Elasticsearch -> Kibana

K8S 配置Filebeat

整體配置檔案如下：

$ ls 
filebeat.daemonset.yml                   filebeat.permission.yml
filebeat.indice-lifecycle.configmap.yml  filebeat.settings.configmap.yml

Filebeat操作許可權

$ cat filebeat.permission.yml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    app: filebeat
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-system
  name: filebeat
  labels:
    app: filebeat

Filebeat主配置檔案

注意：如果收集Java堆疊錯誤日誌，需要增加下面帶註釋的幾行引數，multiline多行處理解決次問題。

$ cat filebeat.settings.configmap.yml 
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: filebeat-config
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      enabled: true
      paths:
      - /var/log/containers/*.log
      multiline: # 多行處理，正則表示如果前面幾個數字不是4個數字開頭，那麼就會合併到一行,解決Java堆疊錯誤日誌收集問題
        pattern: ^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2} #匹配Java日誌開頭時間
        negate: true # 正則是否開啟，預設false不開啟
        match: after # 不匹配的正則的行是放在上面一行的前面還是後面
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"
    
      - add_cloud_metadata:
      - add_kubernetes_metadata:
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"
      - add_docker_metadata:

    output:
      kafka:
        enabled: true # 增加kafka的輸出
        hosts: ["10.0.0.72:9092"]
        topic: filebeat
        max_message_bytes: 5242880
        partition.round_robin:
          reachable_only: true
        keep-alive: 120
        required_acks: 1

    setup.ilm:
      policy_file: /etc/indice-lifecycle.json

Filebeat索引生命週期策略配置

ElasticSearch 的 indice 生命週期表示一組規則，可以根據 indice 的大小或者時長應用到你的 indice 上。比如可以每天或者每次超過 1GB 大小的時候對 indice 進行輪轉，我們也可以根據規則配置不同的階段。由於監控會產生大量的資料，很有可能一天就超過幾十G的資料，所以為了防止大量的資料儲存，我們可以利用 indice 的生命週期來配置資料保留，這個在 Prometheus 中也有類似的操作。 如下所示的檔案中，我們配置成每天或每次超過5GB的時候就對 indice 進行輪轉，並刪除所有超過30天的 indice 檔案，我們這裡只保留30天監控資料完全足夠了。

filebeat.indice-lifecycle.configmap.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: filebeat-indice-lifecycle
  labels:
    app: filebeat
data:
  indice-lifecycle.json: |-
    {
      "policy": {
        "phases": {
          "hot": {
            "actions": {
              "rollover": {
                "max_size": "5GB" ,
                "max_age": "1d"
              }
            }
          },
          "delete": {
            "min_age": "30d",
            "actions": {
              "delete": {}
            }
          }
        }
      }
    }

Filebeat Daemonset配置檔案

$ cat filebeat.daemonset.yml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: filebeat
  labels:
    app: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.8.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: filebeat-indice-lifecycle
          mountPath: /etc/indice-lifecycle.json
          readOnly: true
          subPath: indice-lifecycle.json
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: dockersock
          mountPath: /var/run/docker.sock
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: filebeat-indice-lifecycle
        configMap:
          defaultMode: 0600
          name: filebeat-indice-lifecycle
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate

執行到K8S中

$ kubectl apply  -f filebeat.settings.configmap.yml \
                 -f filebeat.indice-lifecycle.configmap.yml \
                 -f filebeat.daemonset.yml \
                 -f filebeat.permissions.yml 

configmap/filebeat-config created
configmap/filebeat-indice-lifecycle created
daemonset.apps/filebeat created
clusterrolebinding.rbac.authorization.k8s.io/filebeat created
clusterrole.rbac.authorization.k8s.io/filebeat created
serviceaccount/filebeat created

相關文章