06 . ELK Stack + kafka叢集

men發表於2020-07-24

簡介

Filebeat用於收集本地檔案的日誌資料。它作為伺服器上的代理安裝，Filebeat監視日誌目錄或特定的日誌檔案，尾部檔案，並將它們轉發到Elasticsearch或Logstash進行索引。
logstash 和filebeat都具有日誌收集功能，filebeat更輕量，使用go語言編寫，佔用資源更少，可以有很高的併發，但logstash 具有filter功能，能過濾分析日誌。一般結構都是filebeat採集日誌，然後傳送到訊息佇列，如redis，kafka。然後logstash去獲取，利用filter功能過濾分析，然後儲存到elasticsearch中。

Kafka是LinkedIn開源的分散式釋出-訂閱訊息系統，目前歸屬於Apache定級專案。Kafka主要特點是基於Pull的模式來處理訊息消費，追求高吞吐量，一開始的目的就是用於日誌收集和傳輸。0.8版本開始支援複製，不支援事務，對訊息的重複、丟失、錯誤沒有嚴格要求，適合產生大量資料的網際網路服務的資料收集業務。

環境清單

IP	hostname	軟體	配置要求	網路
192.168.43.176	ES/資料儲存	elasticsearch-7.2	記憶體2GB/硬碟40GB	Nat，內網
192.168.43.215	Kibana/UI展示	kibana-7.2	記憶體2GB/硬碟40GB	Nat,內網
192.168.43.164	Filebeat/資料採集	Filebeat-7.2/nginx	記憶體2GB/硬碟40GB	Nat，內網
192.168.43.30	Logstash/資料管道	logstash-7.2	記憶體2GB/硬碟40GB	Nat，內網
192.168.43.86	Kibana/UI展示	kibana-7.2	記憶體2GB/硬碟40GB	Nat，內網
192.168.43.47	Kafka/訊息佇列	Kafka2.12 / zk3.4	記憶體2GB/硬碟40GB	Nat，內網
192.168.43.151	Kafka/訊息佇列	Kafka2.12 / zk3.4	記憶體2GB/硬碟40GB	Nat，內網
192.168.43.43	Kafka/訊息佇列	Kafka2.12 / zk3.4	記憶體2GB/硬碟40GB	Nat，內網
192.168.43.194	Tomcat	tomcat8.5	記憶體2GB/硬碟40GB	Nat，內網

ELK叢集部署請看上一篇部落格

https://www.cnblogs.com/you-men/p/13361910.html

配置使用zookeeper和kafka請看我寫的另一篇部落格

https://www.cnblogs.com/you-men/p/12884779.html

使用Logstash和Kafka互動

編輯logstash配置檔案

input{
  stdin{}
}

output{
  kafka{
    topic_id =>"kafkatest"
    bootstrap_servers => "192.168.43.47:9092"
    batch_size => 5
}
  stdout{
     codec => "rubydebug"
	}
}

啟動logstash,輸入資料

./bin/logstash -f kafka.conf 
zhoujian
{
    "@timestamp" => 2020-07-24T07:11:26.235Z,
       "message" => "zhoujian",
          "host" => "logstash-30",
      "@version" => "1"
}
youmen
{
    "@timestamp" => 2020-07-24T07:11:29.441Z,
       "message" => "youmen",
          "host" => "logstash-30",
      "@version" => "1"
}

kafka中檢視寫入資料

# 檢視kafka現有的topic
./bin/kafka-topics.sh --list --bootstrap-server 192.168.43.47:9092,192.168.43.151:9092,192.168.43.43:9092 
kafkatest
test-you-io

# 檢視kafkatest裡面訊息
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.43.47:9092,192.168.43.151:9092,192.168.43.43:9092 --topic kafkatest --from-beginning
2020-07-24T07:13:59.461Z logstash-30 zhoujian
2020-07-24T07:14:01.518Z logstash-30 youmen

資料寫入成功,kafka配置完成

配置Filebeat

輸出日誌到kafka

/etc/filebeat/filebeat.yml 
  # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

output.kafka:
  enabled: true
  hosts: ["192.168.43.47:9092","192.168.43.151:9092","192.168.43.43:9092"]
  topic: "tomcat-filebeat"
  partition.hash:
    reachable_only: true
  compression: gzip
  max_message_bytes: 1000000
  required_acks: 1
#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
[root@tomcat-194 logs]# cat /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true 
  paths:
    - /usr/local/tomcat/logs/localhost_access_log.2020-07*
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.kafka:
  enabled: true
  hosts: ["192.168.43.47:9092","192.168.43.151:9092","192.168.43.43:9092"]
  topic: "tomcat-filebeat"
  partition.hash:
    reachable_only: true
  compression: gzip
  max_message_bytes: 1000000
  required_acks: 1
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

kafka查詢是否有tomcat日誌

./bin/kafka-console-consumer.sh --bootstrap-server 192.168.43.47:9092,192.168.43.151:9092,192.168.43.43:9092 --topic tomcat-filebeat --from-beginning
      
{"@timestamp":"2020-07-24T06:35:24.294Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.2.0","topic":"tomcat-filebeat"},"message":"{\"client\":\"192.168.43.84\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:14:35:10 +0800]\",     \"method\":\"GET /docs/config/ HTTP/1.1\",   \"status\":\"200\",  \"send bytes\":\"6826\",  \"Query?string\":\"\",  \"partner\":\"http://192.168.43.194:8080/\",  \"Agent version\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\"}","input":{"type":"log"},"ecs":{"version":"1.0.0"},"host":{"name":"tomcat-194","id":"b029c3ce28374f7db698c050e342457f","containerized":false,"hostname":"tomcat-194","architecture":"x86_64","os":{"platform":"centos","version":"7 (Core)","family":"redhat","name":"CentOS Linux","kernel":"3.10.0-514.el7.x86_64","codename":"Core"}},"agent":{"hostname":"tomcat-194","id":"cfe87df5-c912-49d0-8758-b73e917a6c9c","version":"7.2.0","type":"filebeat","ephemeral_id":"894657d2-af1a-4660-a3eb-98602bc3d1d7"},"log":{"offset":19393,"file":{"path":"/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"}}}
{"@timestamp":"2020-07-24T06:38:29.339Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.2.0","topic":"tomcat-filebeat"},"host":{"id":"b029c3ce28374f7db698c050e342457f","containerized":false,"hostname":"tomcat-194","name":"tomcat-194","architecture":"x86_64","os":{"family":"redhat","name":"CentOS Linux","kernel":"3.10.0-514.el7.x86_64","codename":"Core","platform":"centos","version":"7 (Core)"}},"agent":{"ephemeral_id":"894657d2-af1a-4660-a3eb-98602bc3d1d7","hostname":"tomcat-194","id":"cfe87df5-c912-49d0-8758-b73e917a6c9c","version":"7.2.0","type":"filebeat"},"ecs":{"version":"1.0.0"},"message":"{\"client\":\"192.168.43.84\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:14:38:18 +0800]\",     \"method\":\"GET /manager/status HTTP/1.1\",   \"status\":\"403\",  \"send bytes\":\"3446\",  \"Query?string\":\"\",  \"partner\":\"http://192.168.43.194:8080/\",  \"Agent version\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36\"}","log":{"offset":19797,"file":{"path":"/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"}},"input":{"type":"log"}}
^CProcessed a total of 66 messages

使用logstash從Kafka讀取日誌到es

配置logstash讀取kafka日誌

cat kafka-es.conf
input{
  kafka{
      bootstrap_servers => "192.168.43.62:9092,192.168.43.151:9092,192.168.43.43:9092"
      topics => "tomcat-filebeat"
      consumer_threads => 1
      decorate_events => true
      codec => "json"
      auto_offset_reset => "latest" 
    }
}
 
output{
  elasticsearch {
    hosts => ["192.168.43.176:9200"]
    index => "tomcat-filebeat-%{+YYYY.MM.dd}"
 
}
  stdout{
     codec => "rubydebug"
	}
}

前臺執行,確保日誌能否正常輸出

./bin/logstash -f kafka-es.conf 
constant ::Fixnum is deprecated
{
         "input" => {
        "type" => "log"
    },
      "@version" => "1",
       "message" => "{\"client\":\"192.168.43.227\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:15:47:08 +0800]\",     \"method\":\"GET / HTTP/1.1\",   \"status\":\"200\",  \"send bytes\":\"11215\",  \"Query?string\":\"\",  \"partner\":\"-\",  \"Agent version\":\"curl/7.29.0\"}",
         "agent" => {
             "version" => "7.2.0",
            "hostname" => "tomcat-194",
        "ephemeral_id" => "894657d2-af1a-4660-a3eb-98602bc3d1d7",
                  "id" => "cfe87df5-c912-49d0-8758-b73e917a6c9c",
                "type" => "filebeat"
    },
          "host" => {
                 "name" => "tomcat-194",
                   "os" => {
             "version" => "7 (Core)",
                "name" => "CentOS Linux",
            "codename" => "Core",
              "family" => "redhat",
            "platform" => "centos",
              "kernel" => "3.10.0-514.el7.x86_64"
                             },
                   "id" => "b029c3ce28374f7db698c050e342457f",
        "containerized" => false,
             "hostname" => "tomcat-194",
         "architecture" => "x86_64"
    },
    "@timestamp" => 2020-07-24T07:47:11.857Z,
           "log" => {
        "offset" => 20203,
          "file" => {
            "path" => "/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"
        }
    },
           "ecs" => {
        "version" => "1.0.0"
    }
}
     
# kafka節點檢視


{"@timestamp":"2020-07-24T07:53:11.944Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.2.0","topic":"tomcat-filebeat"},"host":{"id":"b029c3ce28374f7db698c050e342457f","containerized":false,"hostname":"tomcat-194","architecture":"x86_64","name":"tomcat-194","os":{"codename":"Core","platform":"centos","version":"7 (Core)","family":"redhat","name":"CentOS Linux","kernel":"3.10.0-514.el7.x86_64"}},"agent":{"type":"filebeat","ephemeral_id":"894657d2-af1a-4660-a3eb-98602bc3d1d7","hostname":"tomcat-194","id":"cfe87df5-c912-49d0-8758-b73e917a6c9c","version":"7.2.0"},"log":{"file":{"path":"/usr/local/tomcat/logs/localhost_access_log.2020-07-24.log"},"offset":20462},"message":"{\"client\":\"192.168.43.227\",  \"client user\":\"-\",   \"authenticated\":\"-\",   \"access time\":\"[24/Jul/2020:15:53:06 +0800]\",     \"method\":\"GET / HTTP/1.1\",   \"status\":\"200\",  \"send bytes\":\"11215\",  \"Query?string\":\"\",  \"partner\":\"-\",  \"Agent version\":\"curl/7.29.0\"}","input":{"type":"log"},"ecs":{"version":"1.0.0"}}

es檢視索引

curl -XGET "http://127.0.0.1:9200/_cat/indices?v"
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-es-7-2020.07.24     z0Ff-j7WSlSm4ZBH6IhZaw   1   1        185           60      3.7mb            2mb
green  open   .monitoring-kibana-7-2020.07.24 PWqXvObhSRazQn4CY8Z2lg   1   1          3            0    216.3kb         73.4kb
green  open   .kibana_task_manager            Ptj7ydZmQqGG7hWxK2NbSg   1   1          2            0     61.2kb         45.5kb
green  open   .kibana_2                       fot9Sk6jRWa2vS5cQGvOeQ   1   1          5            0     68.6kb         34.3kb
green  open   .kibana_1                       jYD4jXLVTeeAMImEz9NEVA   1   1          1            0     18.7kb          9.3kb
green  open   .tasks                          NIwDk-PYQT-d-njh3g0t0g   1   1          1            0     12.7kb          6.3kb
green  open   tomcat-filebeat-2020.07.24      s3aB-c6GSemUHvaurYQ8Zw   1   1         38            0    227.4kb         80.3kb

搭建ELK叢集
2018-11-15
Zookeeper叢集 + Kafka叢集
2024-07-18
Kafka
Docker搭建ELK叢集
2021-06-29
Docker
zookeeper叢集及kafka叢集搭建
2021-06-28
Kafka
Kafka叢集配置
2019-02-22
Kafka
kafka叢集搭建
2019-01-19
Kafka
《Learning ELK Stack》前言
2020-07-30
安裝Kafka叢集
2018-04-08
Kafka
初識kafka叢集
2018-04-30
Kafka
SpringBoot 和 Kafka 叢集
2020-01-03
Spring BootKafka
kafka-2.11叢集搭建
2019-04-05
Kafka
Apache Kafka – 叢集架構
2024-03-27
ApacheKafka架構
快速安裝 kafka 叢集
2022-04-02
Kafka
Centos7 ELK7.6.2叢集搭建
2020-12-18
CentOS
《Learning ELK Stack》1 ELK技術棧介紹
2020-07-30
linux搭建kafka叢集，多master節點叢集說明
2022-04-06
LinuxKafkaAST
用 Docker 快速搭建 Kafka 叢集
2020-06-23
DockerKafka
安裝Zookeeper和Kafka叢集
2023-04-17
Kafka
《Learning ELK Stack》3 使用Logstash採集、解析和轉換資料
2020-07-30
Windows 10環境簡單搭建ELK叢集
2019-03-20
Windows
Mac 使用 docker 搭建 kafka 叢集 + Zookeeper + kafka-manager
2018-11-04
MacDockerKafka
Zookeeper3.4.14（單叢集）、Kafka_2.12-2.2.2（叢集）安裝
2020-10-13
Kafka
KubeSphere 部署 Kafka 叢集實戰指南
2024-08-09
Kafka
《Learning ELK Stack》2 構建第一條ELK資料管道
2020-07-30
Kafka 叢集如何實現資料同步？
2023-11-16
Kafka
RabbitMQ和Kafka的高可用叢集原理
2020-09-11
MQKafka
Kafka叢集搭建及必知必會
2020-11-07
Kafka
linux 原始碼搭建Kafka叢集，100%有效
2021-08-17
Linux原始碼Kafka
《Learning ELK Stack》5 為什麼需要Elasticsearch
2020-07-30
Elasticsearch
《Learning ELK Stack》6 使用Kibana理解資料
2020-07-30
alpakka-kafka(5)-kafka叢集配置與分散式應用部署
2021-06-15
Kafka分散式
ELK批量刪除索引及叢集相關操作記錄
2018-11-21
索引
日誌分析平臺ELK之搜尋引擎Elasticsearch叢集
2020-10-01
Elasticsearch
使用Rainbond部署Logikm，輕鬆管理Kafka叢集
2022-02-17
AIKafka
01 . ELK Stack簡介原理及部署應用
2020-06-20
爬蟲日誌監控 -- Elastc Stack（ELK）部署
2020-09-26
爬蟲AST
記一次Kafka叢集的故障恢復
2018-11-19
Kafka
如何在Ubuntu-16.04上配置Apache Kafka叢集！
2018-09-11
UbuntuApacheKafka

06 . ELK Stack + kafka叢集

簡介

環境清單

使用Logstash和Kafka互動

編輯logstash配置檔案

啟動logstash,輸入資料

kafka中檢視寫入資料

配置Filebeat

輸出日誌到kafka

kafka查詢是否有tomcat日誌

使用logstash從Kafka讀取日誌到es

配置logstash讀取kafka日誌

前臺執行,確保日誌能否正常輸出

es檢視索引

相關文章