ELK學習筆記之基於kakfa (confluent)搭建ELK

時光飛逝，逝者如斯發表於2019-03-20

筆記

0x00 概述

測試搭建一個使用kafka作為訊息佇列的ELK環境，資料採集轉換實現結構如下：

F5 HSL–>logstash(流處理）–> kafka –>elasticsearch

測試中的elk版本為6.3， confluent版本是4.1.1

希望實現的效果是 HSL傳送的日誌脛骨logstash進行流處理後輸出為json，該json類容原樣直接儲存到kafka中，kafka不再做其它方面的格式處理。

0x01 測試

192.168.214.138：安裝 logstash，confluent環境

192.168.214.137：安裝ELK套件（停用logstash，只啟動es和kibana）

confluent安裝除錯備忘：

1. 像安裝elk環境一樣，安裝java環境先
2. 首先在不考慮kafka的情形下，實現F5 HSL—Logstash–ES的正常執行，並實現簡單的正常kibana的展現。後面改用kafka時候直接將這裡output修改為kafka plugin配置即可。
  此時logstash的相關配置

input {
  udp {
    port => 8514
    type => 'f5-dns'
  }
}

filter {
 if [type] == 'f5-dns' {
    grok {
          match => { "message" => "%{HOSTNAME:F5hostname} %{IP:clientip} %{POSINT:clientport} %{IP:svrip} %{NUMBER:qid} %{HOSTNAME:qname} %{GREEDYDA
TA:qtype} %{GREEDYDATA:status} %{GREEDYDATA:origin}" }
    }
    geoip {
         source => "clientip"
         target => "geoip"
    }
    }
}



output {
   #stdout{ codec => rubydebug }
   #elasticsearch {
   # hosts => ["192.168.214.137:9200"]
   # index => "f5-dns-%{+YYYY.MM.dd}"
   #template_name => "f5-dns"
  #}
  kafka {
    codec => json
    bootstrap_servers => "localhost:9092"
    topic_id => "f5-dns-kafka"
  }
}

發一些測試流量，確認es正常收到資料，檢視cerebro上顯示的狀態。（截圖是除錯完畢後截圖）

# cd /usr/share/cerebro/cerebro-0.8.1/
# /bin/cerebro -Dhttp.port=9110 -Dhttp.address=0.0.0.0

安裝confluent，由於是測試環境，直接confluent官方網站下載壓縮包，解壓後使用。位置在/root/confluent-4.1.1/下

由於是測試環境，直接用confluent的命令列來啟動所有相關服務，發現kakfa啟動失敗

[root@kafka-logstash bin]# ./confluent start
Using CONFLUENT_CURRENT: /tmp/confluent.dA0KYIWj
Starting zookeeper
zookeeper is [UP]
Starting kafka
/Kafka failed to start
kafka is [DOWN]
Cannot start Schema Registry, Kafka Server is not running. Check your deployment

檢查發現由於虛機記憶體給太少了，導致java無法分配足夠記憶體給kafka

[root@kafka-logstash bin]# ./kafka-server-start ../etc/kafka/server.properties
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12)

擴大虛擬機器記憶體，並將logstash的jvm配置中設定的記憶體調小

kafka server配置檔案

[root@kafka-logstash kafka]# pwd
/root/confluent-4.1.1/etc/kafka
[root@kafka-logstash kafka]# egrep -v "^#|^$" server.properties
broker.id=0
listeners=PLAINTEXT://localhost:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
confluent.support.metrics.enable=true
confluent.support.customer.id=anonymous
group.initial.rebalance.delay.ms=0

connect 配置檔案，此配置中，將原來的avro converter替換成了json，同時關閉了key vlaue的schema識別。因為我們輸入的內容是直接的json類容，沒有相關schema，這裡只是希望kafka原樣解析logstash輸出的json內容到es

[root@kafka-logstash kafka]# pwd
/root/confluent-4.1.1/etc/kafka
[root@kafka-logstash kafka]# egrep -v "^#|^$" connect-standalone.properties
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=share/java

如果不做上述修改，connect總會在將日誌sink到ES時提示無法反序列化，magic byte錯誤等。如果使用confluent status命令檢視，會發現connect會從up變為down

[root@kafka-logstash confluent-4.1.1]# ./bin/confluent status
ksql-server is [DOWN]
connect is [DOWN]
kafka-rest is [UP]
schema-registry is [UP]
kafka is [UP]
zookeeper is [UP]

schema-registry 相關配置

[root@kafka-logstash schema-registry]# pwd
/root/confluent-4.1.1/etc/schema-registry
[root@kafka-logstash schema-registry]# egrep -v "^#|^$"
connect-avro-distributed.properties  connect-avro-standalone.properties   log4j.properties                     schema-registry.properties
[root@kafka-logstash schema-registry]# egrep -v "^#|^$" connect-avro-standalone.properties
bootstrap.servers=localhost:9092
key.converter.schema.registry.url=http://localhost:8081
value.converter.schema.registry.url=http://localhost:8081
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
plugin.path=share/java
[root@kafka-logstash schema-registry]# egrep -v "^#|^$" schema-registry.properties
listeners=http://0.0.0.0:8081
kafkastore.connection.url=localhost:2181
kafkastore.topic=_schemas
debug=false

es-connector的配置檔案

[root@kafka-logstash kafka-connect-elasticsearch]# pwd
/root/confluent-4.1.1/etc/kafka-connect-elasticsearch
[root@kafka-logstash kafka-connect-elasticsearch]# egrep -v "^#|^$" quickstart-elasticsearch.properties
name=f5-dns
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=f5-dns-kafka
key.ignore=true
value.ignore=true
schema.ignore=true
connection.url=http://192.168.214.137:9200
type.name=doc
transforms=MyRouter
transforms.MyRouter.type=org.apache.kafka.connect.transforms.TimestampRouter
transforms.MyRouter.topic.format=${topic}-${timestamp}
transforms.MyRouter.timestamp.format=yyyyMMdd

上述配置中topics配置是希望傳輸到ES的topic，通過設定transform的timestamp router來實現將topic按天動態對映為ES中的index，這樣可以讓ES每天產生一個index。注意需要配置schema.ignore=true,否則kafka無法將受收到的資料傳送到ES上，connect的 connect.stdout 日誌會顯示：

[root@kafka-logstash connect]# pwd
/tmp/confluent.dA0KYIWj/connect

Caused by: org.apache.kafka.connect.errors.DataException: Cannot infer mapping without schema.
	at io.confluent.connect.elasticsearch.Mapping.inferMapping(Mapping.java:84)
	at io.confluent.connect.elasticsearch.jest.JestElasticsearchClient.createMapping(JestElasticsearchClient.java:221)
	at io.confluent.connect.elasticsearch.Mapping.createMapping(Mapping.java:66)
	at io.confluent.connect.elasticsearch.ElasticsearchWriter.write(ElasticsearchWriter.java:260)
	at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.put(ElasticsearchSinkTask.java:162)
	at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:524)

配置修正完畢後，向logstash傳送資料，發現日誌已經可以正常傳送到了ES上，且格式和沒有kafka時是一致的。
沒有kafka時：

{
	"_index": "f5-dns-2018.06.26",
	"_type": "doc",
	"_id": "KrddO2QBXB-i0ay0g5G9",
	"_version": 1,
	"_score": 1,
	"_source": {
		"message": "localhost.lan 202.202.102.100 53777 172.16.199.136 42487 www.test.com A NOERROR GTM_REWRITE ",
		"F5hostname": "localhost.lan",
		"qid": "42487",
		"clientip": "202.202.102.100",
		"geoip": {
			"region_name": "Chongqing",
			"location": {
				"lon": 106.5528,
				"lat": 29.5628
			},
			"country_code2": "CN",
			"timezone": "Asia/Shanghai",
			"country_name": "China",
			"region_code": "50",
			"continent_code": "AS",
			"city_name": "Chongqing",
			"country_code3": "CN",
			"ip": "202.202.102.100",
			"latitude": 29.5628,
			"longitude": 106.5528
		},
		"status": "NOERROR",
		"qname": "www.test.com",
		"clientport": "53777",
		"@version": "1",
		"@timestamp": "2018-06-26T09:12:21.585Z",
		"host": "192.168.214.1",
		"type": "f5-dns",
		"qtype": "A",
		"origin": "GTM_REWRITE ",
		"svrip": "172.16.199.136"
	}
}

有kafka時：

{
	"_index": "f5-dns-kafka-20180628",
	"_type": "doc",
	"_id": "f5-dns-kafka-20180628+0+23",
	"_version": 1,
	"_score": 1,
	"_source": {
		"F5hostname": "localhost.lan",
		"geoip": {
			"city_name": "Chongqing",
			"timezone": "Asia/Shanghai",
			"ip": "202.202.100.100",
			"latitude": 29.5628,
			"country_name": "China",
			"country_code2": "CN",
			"continent_code": "AS",
			"country_code3": "CN",
			"region_name": "Chongqing",
			"location": {
				"lon": 106.5528,
				"lat": 29.5628
			},
			"region_code": "50",
			"longitude": 106.5528
		},
		"qtype": "A",
		"origin": "DNSX ",
		"type": "f5-dns",
		"message": "localhost.lan 202.202.100.100 53777 172.16.199.136 42487 www.myf5.net A NOERROR DNSX ",
		"qid": "42487",
		"clientport": "53777",
		"@timestamp": "2018-06-28T09:05:20.594Z",
		"clientip": "202.202.100.100",
		"qname": "www.myf5.net",
		"host": "192.168.214.1",
		"@version": "1",
		"svrip": "172.16.199.136",
		"status": "NOERROR"
	}
}

相關REST API輸出

http://192.168.214.138:8083/connectors/elasticsearch-sink/tasks

[
  {
    "id": {
      "connector": "elasticsearch-sink",
      "task": 0
    },
    "config": {
      "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
      "type.name": "doc",
      "value.ignore": "true",
      "tasks.max": "1",
      "topics": "f5-dns-kafka",
      "transforms.MyRouter.topic.format": "${topic}-${timestamp}",
      "transforms": "MyRouter",
      "key.ignore": "true",
      "schema.ignore": "true",
      "transforms.MyRouter.timestamp.format": "yyyyMMdd",
      "task.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkTask",
      "name": "elasticsearch-sink",
      "connection.url": "http://192.168.214.137:9200",
      "transforms.MyRouter.type": "org.apache.kafka.connect.transforms.TimestampRouter"
    }
  }
]

http://192.168.214.138:8083/connectors/elasticsearch-sink/
{
  "name": "elasticsearch-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "type.name": "doc",
    "value.ignore": "true",
    "tasks.max": "1",
    "topics": "f5-dns-kafka",
    "transforms.MyRouter.topic.format": "${topic}-${timestamp}",
    "transforms": "MyRouter",
    "key.ignore": "true",
    "schema.ignore": "true",
    "transforms.MyRouter.timestamp.format": "yyyyMMdd",
    "name": "elasticsearch-sink",
    "connection.url": "http://192.168.214.137:9200",
    "transforms.MyRouter.type": "org.apache.kafka.connect.transforms.TimestampRouter"
  },
  "tasks": [
    {
      "connector": "elasticsearch-sink",
      "task": 0
    }
  ],
  "type": "sink"
}

http://192.168.214.138:8083/connectors/elasticsearch-sink/status
{
  "name": "elasticsearch-sink",
  "connector": {
    "state": "RUNNING",
    "worker_id": "172.16.150.179:8083"
  },
  "tasks": [
    {
      "state": "RUNNING",
      "id": 0,
      "worker_id": "172.16.150.179:8083"
    }
  ],
  "type": "sink"
}

http://192.168.214.138:8082/brokers
{
  "brokers": [
    0
  ]
}

http://192.168.214.138:8082/topics
[
  "__confluent.support.metrics",
  "_confluent-ksql-default__command_topic",
  "_schemas",
  "connect-configs",
  "connect-offsets",
  "connect-statuses",
  "f5-dns-2018.06",
  "f5-dns-2018.06.27",
  "f5-dns-kafka",
  "test-elasticsearch-sink"
]



http://192.168.214.138:8082/topics/f5-dns-kafka
{
  "name": "f5-dns-kafka",
  "configs": {
    "file.delete.delay.ms": "60000",
    "segment.ms": "604800000",
    "min.compaction.lag.ms": "0",
    "retention.bytes": "-1",
    "segment.index.bytes": "10485760",
    "cleanup.policy": "delete",
    "follower.replication.throttled.replicas": "",
    "message.timestamp.difference.max.ms": "9223372036854775807",
    "segment.jitter.ms": "0",
    "preallocate": "false",
    "segment.bytes": "1073741824",
    "message.timestamp.type": "CreateTime",
    "message.format.version": "1.1-IV0",
    "max.message.bytes": "1000012",
    "unclean.leader.election.enable": "false",
    "retention.ms": "604800000",
    "flush.ms": "9223372036854775807",
    "delete.retention.ms": "86400000",
    "leader.replication.throttled.replicas": "",
    "min.insync.replicas": "1",
    "flush.messages": "9223372036854775807",
    "compression.type": "producer",
    "min.cleanable.dirty.ratio": "0.5",
    "index.interval.bytes": "4096"
  },
  "partitions": [
    {
      "partition": 0,
      "leader": 0,
      "replicas": [
        {
          "broker": 0,
          "leader": true,
          "in_sync": true
        }
      ]
    }
  ]
}

測試中kafka的配置基本都為確實配置，沒有考慮任何的記憶體優化，kafka使用磁碟的大小考慮等

測試參考：

https://docs.confluent.io/current/installation/installing_cp.html

https://docs.confluent.io/current/connect/connect-elasticsearch/docs/elasticsearch_connector.html

https://docs.confluent.io/current/connect/connect-elasticsearch/docs/configuration_options.html

儲存機制參考 https://blog.csdn.net/opensure/article/details/46048589

kafka配置引數參考 https://blog.csdn.net/lizhitao/article/details/25667831

更多kafka原理 https://blog.csdn.net/ychenfeng/article/details/74980531

confluent CLI：

confluent: A command line interface to manage Confluent services

Usage: confluent <command> [<subcommand>] [<parameters>]

These are the available commands:

    acl         Specify acl for a service.
    config      Configure a connector.
    current     Get the path of the data and logs of the services managed by the current confluent run.
    destroy     Delete the data and logs of the current confluent run.
    list        List available services.
    load        Load a connector.
    log         Read or tail the log of a service.
    start       Start all services or a specific service along with its dependencies
    status      Get the status of all services or the status of a specific service along with its dependencies.
    stop        Stop all services or a specific service along with the services depending on it.
    top         Track resource usage of a service.
    unload      Unload a connector.

'confluent help' lists available commands. See 'confluent help <command>' to read about a
specific command.

confluent platform 服務埠表

參考

ELK學習筆記之構建基於kafka(confluent)雙活資料中心彈性F5 Bigdata Engine 架構-ELK_1.0
2019-03-20
筆記Kafka架構
ELK 搭建筆記--Docker 方式
2019-01-18
筆記Docker
ELK搭建
2018-12-29
elk 搭建
2020-11-08
[elk]基於elk的業務日誌格式設計
2019-02-16
使用 ELK 搭建日記系統
2020-06-09
搭建ELK叢集
2018-11-15
ELK基礎架構解說-運維筆記
2016-06-08
架構運維筆記
【筆記】草履蟲也能看懂的ELK搭建流程
2021-12-10
筆記
Docker搭建ELK叢集
2021-06-29
Docker
Homestead 下搭建 ELK 踩坑日記
2018-11-09
ELK之簡介
2016-11-17
ELK學習筆記之F5 DNS視覺化讓DNS運維更安全更高效-F5 ELK視覺化方案系列(3)
2019-03-20
筆記DNS視覺化運維
Windows環境下ELK簡單搭建記錄
2018-04-17
Windows
ELK6環境搭建
2020-10-22
Elasticsearch, Logstash, Kibana (ELK) 搭建
2017-07-30
Elasticsearch
ELK 系列三、ELK6.5.3 +xpack破解
2019-02-21
基於ELK搭建MySQL日誌平臺的要點和常見錯誤
2020-05-13
MySql
ELK初探
2020-01-06
搭建ELK日誌平臺（單機）
2021-12-20
ELK之Kibana的安裝
2019-02-19
Java學習筆記之基於TCP協議的socket
2016-08-25
Java筆記TCP協議
基於docker 初學 MongoDb 學習筆記
2021-05-01
DockerMongoDB筆記
Centos7 ELK7.6.2叢集搭建
2020-12-18
CentOS
ELK叢集搭建（ElasticSearch Logstash Kinaba）
2017-05-18
Elasticsearch
ELK日誌
2020-11-23
ELK外掛
2018-07-05
Centos8 部署 ElasticSearch 叢集並搭建 ELK，基於Logstash同步MySQL資料到ElasticSearch
2021-11-18
CentOSElasticsearchMySql
基於AWS-ELK部署系統日誌告警系統
2021-09-09
《Learning ELK Stack》1 ELK技術棧介紹
2020-07-30
elk之kibana安裝與配置
2016-04-18
Windows 10環境簡單搭建ELK叢集
2019-03-20
Windows
ELK+FileBeat+Kafka搭建日誌管理平臺
2019-05-17
Kafka
在Docker上搭建ELK+Filebeat日誌中心
2023-10-29
Docker
laravel學習筆記之開發環境搭建
2021-11-21
Laravel筆記開發環境
deep learning深度學習之學習筆記基於吳恩達coursera課程
2017-09-19
深度學習筆記吳恩達
ELK 安裝配置
2018-07-17
ELK運維文件
2024-01-25
運維

ELK學習筆記之基於kakfa (confluent)搭建ELK

0x00 概述

0x01 測試

參考

相關文章