fluentd收集kubernetes 叢集日誌分析

遊戲不存在發表於2019-04-12

EFK (Elasticsearch + Fluentd + Kibana) 是kubernetes官方推薦的日誌收集方案,我們一起了解一下fluentd是如何收集kubernetes叢集日誌的,慶祝一下fluentd從 CNCF 畢業。開始之前,希望你已經讀過Docker 容器日誌分析, 本文是其延生的第二篇。

注意 需要和ELK(Elasticsearch + Logstash + Kibana) 以及EFK(Elasticsearch + Filebeat + Kibana)區分,後一個EFK一般是原生部署。

CNCF , 全稱Cloud Native Computing Foundation(雲原生計算基金會),kubernetes也是其旗下,或者說大多數容器雲專案都是其旗下。

部署EFK

k8s中部署efk,所用的yaml檔案在 github.com/kubernetes/… ,你可以使用文章附錄提供的指令碼進行下載。

下載完成後執行 cd fluentd-elasticsearch && kubectl apply -f . 命令進行部署。

檢查elasticsearch和kibana service:

$ kubectl get svc -n kube-system
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
elasticsearch-logging   NodePort    10.97.248.209    <none>        9200:32126/TCP   23d
kibana-logging          ClusterIP   10.103.126.183   <none>        5601/TCP         23d
複製程式碼

檢查fluentd DaemonSet:

$ kubectl get ds -n kube-system
NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
fluentd-es-v2.4.0       2         2         2       2            2           <none>                          23d
複製程式碼

這裡我們知道了fluentd是以daemonset方式執行的,es和kibana是service方式。

注意 elasticsearch 預設部署檔案是沒有持久化的,如果需要持久化,需要調整其PVC設定。

fluentd 功能分析

  1. 檢視fluentd的型別,沒什麼好說的

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: fluentd-es-v2.2.1
      namespace: kube-system
    複製程式碼
  2. 檢視fluentd日誌收集

    containers:
    - name: fluentd-es
    image: k8s.gcr.io/fluentd-elasticsearch:v2.2.0
    
    ...
    
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
    - name: config-volume
      mountPath: /etc/fluent/config.d
    
    ...
    
    volumes:
    - name: varlog
    hostPath:
      path: /var/log
    - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
    - name: config-volume
    configMap:
      name: fluentd-es-config-v0.1.6
    複製程式碼

    這裡可以清晰的看到,fluentd以daemonset方式運作,然後把系統的 /var/lib/docker/containers 掛載,這個目錄我們在Docker 容器日誌分析中介紹過,這是docker容器日誌存放路徑, 這樣fluentd就完成了對容器預設日誌的讀取。

    fluentd的配置檔案是以configmap形式載入,繼續往下看看。

  3. 收集容器日誌配置

    收集容器日誌主要在 containers.input.conf,如下:

    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      tag raw.kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </pattern>
      </parse>
    </source>
    複製程式碼

    細心的你會發現掛載的容器目錄是 /var/lib/docker/containers ,日誌應該都在這裡,但是配置的監聽的目錄卻是 /var/log/containers 。官方貼心的給出了註釋,主要內容如下:

     # Example
        # =======
        # ...
        #
        # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log
        # record & add labels to the log record if properly configured. This enables users
        # to filter & search logs on any metadata.
        # For example a Docker container's logs might be in the directory:
        #
        #  /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b
        #
        # and in the file:
        #
        #  997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
        #
        # where 997599971ee6... is the Docker ID of the running container.
        # The Kubernetes kubelet makes a symbolic link to this file on the host machine
        # in the /var/log/containers directory which includes the pod name and the Kubernetes
        # container name:
        #
        #    synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
        #    ->
        #    /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
        #
        # The /var/log directory on the host is mapped to the /var/log directory in the container
        # running this instance of Fluentd and we end up collecting the file:
        #
        #   /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
        #
    複製程式碼
  4. 日誌上傳到elasticsearch

    output.conf: |-
        <match **>
          @id elasticsearch
          @type elasticsearch
          @log_level info
          type_name _doc
          include_tag_key true
          host elasticsearch-logging
          port 9200
          logstash_format true
          <buffer>
            @type file
            path /var/log/fluentd-buffers/kubernetes.system.buffer
            flush_mode interval
            retry_type exponential_backoff
            flush_thread_count 2
            flush_interval 5s
            retry_forever
            retry_max_interval 30
            chunk_limit_size 2M
            queue_limit_length 8
            overflow_action block
          </buffer>
        </match>
    複製程式碼

    這裡注意一下其中的hostport,均是elasticsearch service中定義的,如果修改過需要保持一致。fluentd也支援日誌資料上傳到外部的elasticsearch,也就是前文的elk/efk原生。

附錄

  1. 架構圖

logging-with-node-agent

  1. 下載指令碼檔案 download.sh

    for file in es-service es-statefulset fluentd-es-configmap fluentd-es-ds kibana-deployment kibana-service; do curl -o $file.yaml https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/fluentd-elasticsearch/$file.yaml; done
    複製程式碼
  2. 參考連結:

相關文章