06 . ELK Stack + kafka叢集

men發表於2020-07-24

簡介

Filebeat用於收集本地檔案的日誌資料。 它作為伺服器上的代理安裝,Filebeat監視日誌目錄或特定的日誌檔案,尾部檔案,並將它們轉發到Elasticsearch或Logstash進行索引。
logstash 和filebeat都具有日誌收集功能,filebeat更輕量,使用go語言編寫,佔用資源更少,可以有很高的併發,但logstash 具有filter功能,能過濾分析日誌。一般結構都是filebeat採集日誌,然後傳送到訊息佇列,如redis,kafka。然後logstash去獲取,利用filter功能過濾分析,然後儲存到elasticsearch中。

Kafka是LinkedIn開源的分散式釋出-訂閱訊息系統,目前歸屬於Apache定級專案。Kafka主要特點是基於Pull的模式來處理訊息消費,追求高吞吐量,一開始的目的就是用於日誌收集和傳輸。0.8版本開始支援複製,不支援事務,對訊息的重複、丟失、錯誤沒有嚴格要求,適合產生大量資料的網際網路服務的資料收集業務。

環境清單

IP hostname 軟體 配置要求 網路 備註
192.168.43.176 ES/資料儲存 elasticsearch-7.2 記憶體2GB/硬碟40GB Nat,內網
192.168.43.215 Kibana/UI展示 kibana-7.2 記憶體2GB/硬碟40GB Nat,內網
192.168.43.164 Filebeat/資料採集 Filebeat-7.2/nginx 記憶體2GB/硬碟40GB Nat,內網
192.168.43.30 Logstash/資料管道 logstash-7.2 記憶體2GB/硬碟40GB Nat,內網
192.168.43.86 Kibana/UI展示 kibana-7.2 記憶體2GB/硬碟40GB Nat,內網
192.168.43.47 Kafka/訊息佇列 Kafka2.12 / zk3.4 記憶體2GB/硬碟40GB Nat,內網
192.168.43.151 Kafka/訊息佇列 Kafka2.12 / zk3.4 記憶體2GB/硬碟40GB Nat,內網
192.168.43.43 Kafka/訊息佇列 Kafka2.12 / zk3.4 記憶體2GB/硬碟40GB Nat,內網
192.168.43.194 Tomcat tomcat8.5 記憶體2GB/硬碟40GB Nat,內網

ELK叢集部署請看上一篇部落格

https://www.cnblogs.com/you-men/p/13361910.html

配置使用zookeeper和kafka請看我寫的另一篇部落格

https://www.cnblogs.com/you-men/p/12884779.html

使用Logstash和Kafka互動

編輯logstash配置檔案
input{
  stdin{}
}

output{
  kafka{
    topic_id =>"kafkatest"
    bootstrap_servers => "192.168.43.47:9092"
    batch_size => 5
}
  stdout{
     codec => "rubydebug"
	}
}
啟動logstash,輸入資料
./bin/logstash -f kafka.conf 
zhoujian
{
    "@timestamp" => 2020-07-24T07:11:26.235Z,
       "message" => "zhoujian",
          "host" => "logstash-30",
      "@version" => "1"
}
youmen
{
    "@timestamp" => 2020-07-24T07:11:29.441Z,
       "message" => "youmen",
          "host" => "logstash-30",
      "@version" => "1"
}
kafka中檢視寫入資料
# 檢視kafka現有的topic
./bin/kafka-topics.sh --list --bootstrap-server 192.168.43.47:9092,192.168.43.151:9092,192.168.43.43:9092 
kafkatest
test-you-io

# 檢視kafkatest裡面訊息
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.43.47:9092,192.168.43.151:9092,192.168.43.43:9092 --topic kafkatest --from-beginning
2020-07-24T07:13:59.461Z logstash-30 zhoujian
2020-07-24T07:14:01.518Z logstash-30 youmen

資料寫入成功,kafka配置完成

配置Filebeat

輸出日誌到kafka
/etc/filebeat/filebeat.yml 
  # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

output.kafka:
  enabled: true
  hosts: ["192.168.43.47:9092","192.168.43.151:9092","192.168.43.43:9092"]
  topic: "tomcat-filebeat"
  partition.hash:
    reachable_only: true
  compression: gzip
  max_message_bytes: 1000000
  required_acks: 1
#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
[root@tomcat-194 logs]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /usr/local/tomcat/logs/localhost_access_log.2020-07*
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.kafka:
  enabled: true
  hosts: ["192.168.43.47:9092","192.168.43.151:9092","192.168.43.43:9092"]
  topic: "tomcat-filebeat"
  partition.hash:
    reachable_only: true
  compression: gzip
  max_message_bytes: 1000000
  required_acks: 1
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
kafka查詢是否有tomcat日誌
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.43.47:9092,192.168.43.151:9092,192.168.43.43:9092 --topic tomcat-filebeat --from-beginning
      
{"@timestamp":"2020-07-24T06:35:24.294Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.2.0","topic":"tomcat-filebeat"},"message":"{\"client\":\"192.168.43.84\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:14:35:10 +0800]\",     \"method\":\"GET /docs/config/ HTTP/1.1\",   \"status\":\"200\",  \"send bytes\":\"6826\",  \"Query?string\":\"\",  \"partner\":\"http://192.168.43.194:8080/\",  \"Agent version\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\"}","input":{"type":"log"},"ecs":{"version":"1.0.0"},"host":{"name":"tomcat-194","id":"b029c3ce28374f7db698c050e342457f","containerized":false,"hostname":"tomcat-194","architecture":"x86_64","os":{"platform":"centos","version":"7 (Core)","family":"redhat","name":"CentOS Linux","kernel":"3.10.0-514.el7.x86_64","codename":"Core"}},"agent":{"hostname":"tomcat-194","id":"cfe87df5-c912-49d0-8758-b73e917a6c9c","version":"7.2.0","type":"filebeat","ephemeral_id":"894657d2-af1a-4660-a3eb-98602bc3d1d7"},"log":{"offset":19393,"file":{"path":"/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"}}}
{"@timestamp":"2020-07-24T06:38:29.339Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.2.0","topic":"tomcat-filebeat"},"host":{"id":"b029c3ce28374f7db698c050e342457f","containerized":false,"hostname":"tomcat-194","name":"tomcat-194","architecture":"x86_64","os":{"family":"redhat","name":"CentOS Linux","kernel":"3.10.0-514.el7.x86_64","codename":"Core","platform":"centos","version":"7 (Core)"}},"agent":{"ephemeral_id":"894657d2-af1a-4660-a3eb-98602bc3d1d7","hostname":"tomcat-194","id":"cfe87df5-c912-49d0-8758-b73e917a6c9c","version":"7.2.0","type":"filebeat"},"ecs":{"version":"1.0.0"},"message":"{\"client\":\"192.168.43.84\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:14:38:18 +0800]\",     \"method\":\"GET /manager/status HTTP/1.1\",   \"status\":\"403\",  \"send bytes\":\"3446\",  \"Query?string\":\"\",  \"partner\":\"http://192.168.43.194:8080/\",  \"Agent version\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\"}","log":{"offset":19797,"file":{"path":"/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"}},"input":{"type":"log"}}
^CProcessed a total of 66 messages

使用logstash從Kafka讀取日誌到es

配置logstash讀取kafka日誌
cat kafka-es.conf
input{
  kafka{
      bootstrap_servers => "192.168.43.62:9092,192.168.43.151:9092,192.168.43.43:9092"
      topics => "tomcat-filebeat"
      consumer_threads => 1
      decorate_events => true
      codec => "json"
      auto_offset_reset => "latest" 
    }
}
 
output{
  elasticsearch {
    hosts => ["192.168.43.176:9200"]
    index => "tomcat-filebeat-%{+YYYY.MM.dd}"
 
}
  stdout{
     codec => "rubydebug"
	}
}
前臺執行,確保日誌能否正常輸出
./bin/logstash -f kafka-es.conf 
constant ::Fixnum is deprecated
{
         "input" => {
        "type" => "log"
    },
      "@version" => "1",
       "message" => "{\"client\":\"192.168.43.227\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:15:47:08 +0800]\",     \"method\":\"GET / HTTP/1.1\",   \"status\":\"200\",  \"send bytes\":\"11215\",  \"Query?string\":\"\",  \"partner\":\"-\",  \"Agent version\":\"curl/7.29.0\"}",
         "agent" => {
             "version" => "7.2.0",
            "hostname" => "tomcat-194",
        "ephemeral_id" => "894657d2-af1a-4660-a3eb-98602bc3d1d7",
                  "id" => "cfe87df5-c912-49d0-8758-b73e917a6c9c",
                "type" => "filebeat"
    },
          "host" => {
                 "name" => "tomcat-194",
                   "os" => {
             "version" => "7 (Core)",
                "name" => "CentOS Linux",
            "codename" => "Core",
              "family" => "redhat",
            "platform" => "centos",
              "kernel" => "3.10.0-514.el7.x86_64"
                             },
                   "id" => "b029c3ce28374f7db698c050e342457f",
        "containerized" => false,
             "hostname" => "tomcat-194",
         "architecture" => "x86_64"
    },
    "@timestamp" => 2020-07-24T07:47:11.857Z,
           "log" => {
        "offset" => 20203,
          "file" => {
            "path" => "/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"
        }
    },
           "ecs" => {
        "version" => "1.0.0"
    }
}
     
# kafka節點檢視


{"@timestamp":"2020-07-24T07:53:11.944Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.2.0","topic":"tomcat-filebeat"},"host":{"id":"b029c3ce28374f7db698c050e342457f","containerized":false,"hostname":"tomcat-194","architecture":"x86_64","name":"tomcat-194","os":{"codename":"Core","platform":"centos","version":"7 (Core)","family":"redhat","name":"CentOS Linux","kernel":"3.10.0-514.el7.x86_64"}},"agent":{"type":"filebeat","ephemeral_id":"894657d2-af1a-4660-a3eb-98602bc3d1d7","hostname":"tomcat-194","id":"cfe87df5-c912-49d0-8758-b73e917a6c9c","version":"7.2.0"},"log":{"file":{"path":"/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"},"offset":20462},"message":"{\"client\":\"192.168.43.227\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:15:53:06 +0800]\",     \"method\":\"GET / HTTP/1.1\",   \"status\":\"200\",  \"send bytes\":\"11215\",  \"Query?string\":\"\",  \"partner\":\"-\",  \"Agent version\":\"curl/7.29.0\"}","input":{"type":"log"},"ecs":{"version":"1.0.0"}}

es檢視索引
curl -XGET "http://127.0.0.1:9200/_cat/indices?v"
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-es-7-2020.07.24     z0Ff-j7WSlSm4ZBH6IhZaw   1   1        185           60      3.7mb            2mb
green  open   .monitoring-kibana-7-2020.07.24 PWqXvObhSRazQn4CY8Z2lg   1   1          3            0    216.3kb         73.4kb
green  open   .kibana_task_manager            Ptj7ydZmQqGG7hWxK2NbSg   1   1          2            0     61.2kb         45.5kb
green  open   .kibana_2                       fot9Sk6jRWa2vS5cQGvOeQ   1   1          5            0     68.6kb         34.3kb
green  open   .kibana_1                       jYD4jXLVTeeAMImEz9NEVA   1   1          1            0     18.7kb          9.3kb
green  open   .tasks                          NIwDk-PYQT-d-njh3g0t0g   1   1          1            0     12.7kb          6.3kb
green  open   tomcat-filebeat-2020.07.24      s3aB-c6GSemUHvaurYQ8Zw   1   1         38            0    227.4kb         80.3kb

相關文章