教你8步輕鬆上手kafka
1、配置環境變數
必須要求Java 8及以上版本
[root@localhost kafka]# java -version openjdk version "1.8.0_262" OpenJDK Runtime Environment (build 1.8.0_262-b10) OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)
2、下載並解壓kafka安裝檔案
kafka下載地址
上傳至伺服器並解壓
[root@localhost kafka]# ls kafka_2.13-3.3.1.tgz [root@localhost kafka]# tar -xzf ./kafka_2.13-3.3.1.tgz [root@localhost kafka]# ls kafka_2.13-3.3.1 kafka_2.13-3.3.1.tgz [root@localhost kafka]# cd ./kafka_2.13-3.3.1/ [root@localhost kafka_2.13-3.3.1]# ls bin config libs LICENSE licenses NOTICE site-docs [root@localhost kafka_2.13-3.3.1]#
3、啟動服務
必須按照正確的順序先啟動zookeeper，在啟動kafka，啟動命令如下。
啟動zookeeper
[root@localhost kafka_2.13-3.3.1]# ./bin/zookeeper-server-start.sh ./config/zookeeper.properties
啟動kafka（另起一個視窗啟動），注意在配置檔案裡面更改zookeeper.connect的引數值，本機預設為localhost，所以可以不用改。
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-server-start.sh ./config/server.properties4、建立topic
使用kafka-topics.sh建立名為local-topic的topic，localhost為kafka部署的主機名，9092為kafka的埠，需根據實際情況更改。
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-topics.sh --create --topic local-topic --bootstrap-server localhost:9092 Created topic local-topic. [root@localhost kafka_2.13-3.3.1]#
5、檢視topic
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-topics.sh --describe --topic local-topic --bootstrap-server localhost:9092 Topic: local-topicTopicId: zlnMG_F9RO6VAeS2hj6rqwPartitionCount: 1ReplicationFactor: 1Configs: Topic: local-topicPartition: 0Leader: 0Replicas: 0Isr: 0 [root@localhost kafka_2.13-3.3.1]#
6、使用kafka-console-producer.sh為local-topic寫入event，進入互動式視窗寫入event，完成後使用Ctrl-C結束視窗。
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-console-producer.sh --topic local-topic --bootstrap-server localhost:9092 >local-topic 1 test >local-topic 2 test >local-topic 3 test >^C[root@localhost kafka_2.13-3.3.1]#
7、使用kafka-console-consumer.sh檢視local-topic寫入的event，使用Ctrl-C結束視窗
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-console-consumer.sh --topic local-topic --from-beginning --bootstrap-server localhost:9092 local-topic 1 test local-topic 2 test local-topic 3 test ^CProcessed a total of 3 messages [root@localhost kafka_2.13-3.3.1]#
8、以檔案為例使用kafkaconnect匯入資料
編輯connect-standalone.properties，指定plugin.path路徑至對應的connect使用的jar包。
[root@localhost kafka_2.13-3.3.1]# vim ./config/connect-standalone.properties [root@localhost kafka_2.13-3.3.1]# cat ./config/connect-standalone.properties | grep plugin.path # plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors, #plugin.path= plugin.path=/kafka/kafka_2.13-3.3.1/libs/connect-file-3.3.1.jar [root@localhost kafka_2.13-3.3.1]#
編輯測試檔案test，寫入測試資料
[root@localhost kafka_2.13-3.3.1]# echo -e "foo\nbar" > /kafka/test.txt
修改connect-file-source.properties、connect-file-sink.properties中file中的檔案路徑為要匯入資料的檔案路徑，並根據實際情況修改其他引數值
[root@localhost kafka_2.13-3.3.1]# vim ./config/connect-file-source.properties [root@localhost kafka_2.13-3.3.1]# cat ./config/connect-file-source.properties # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. name=local-file-source connector.class=FileStreamSource tasks.max=1 file=/kafka/test.txt topic=connect-test [root@localhost kafka_2.13-3.3.1]#
啟動connect程式
[root@localhost kafka_2.13-3.3.1]# ./bin/connect-standalone.sh ./config/connect-standalone.properties ./config/connect-file-source.properties ./config/connect-file-sink.properties
檢視檔案內容讀取結果，將此視窗保留，測試實時寫入test檔案資料後的變化。可以看到資料已經讀入kafka並按檔案配置建立connect-test的topic，檢視test.sink.txt檔案內容，跟test一致。
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
connect-test
local-topic
[root@localhost kafka_2.13-3.3.1]#
[root@localhost kafka_2.13-3.3.1]# cat /kafka/test.sink.txt
foo
bar
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
在test.txt檔案中追加資訊，如追加test3到/kafka/test.txt，寫完後觀察前面保留的視窗變化。
[root@localhost kafka_2.13-3.3.1]# echo "test3" >> /kafka/test.txt [root@localhost kafka_2.13-3.3.1]#
可以看到，新追加的test3會自動列印出來，並且在寫入test.sink.txt
[root@localhost kafka_2.13-3.3.1]# ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"test3"}
[root@localhost kafka_2.13-3.3.1]# cat /kafka/test.sink.txt
foo
bar
test3
附：kafka-topics.sh幫助文件
This tool helps to create, delete, describe, or change a topic. Option Description ------ ----------- --alter Alter the number of partitions, replica assignment, and/or configuration for the topic. --at-min-isr-partitions if set when describing topics, only show partitions whose isr count is equal to the configured minimum. --bootstrap-server <String: server to REQUIRED: The Kafka server to connect connect to> to. --command-config <String: command Property file containing configs to be config property file> passed to Admin Client. This is used only with --bootstrap-server option for describing and altering broker configs. --config <String: name=value> A topic configuration override for the topic being created or altered. The following is a list of valid configurations: cleanup.policy compression.type delete.retention.ms file.delete.delay.ms flush.messages flush.ms follower.replication.throttled. replicas index.interval.bytes leader.replication.throttled.replicas local.retention.bytes local.retention.ms max.compaction.lag.ms max.message.bytes message.downconversion.enable message.format.version message.timestamp.difference.max.ms message.timestamp.type min.cleanable.dirty.ratio min.compaction.lag.ms min.insync.replicas preallocate remote.storage.enable retention.bytes retention.ms segment.bytes segment.index.bytes segment.jitter.ms segment.ms unclean.leader.election.enable See the Kafka documentation for full details on the topic configs. It is supported only in combination with -- create if --bootstrap-server option is used (the kafka-configs CLI supports altering topic configs with a --bootstrap-server option). --create Create a new topic. --delete Delete a topic --delete-config <String: name> A topic configuration override to be removed for an existing topic (see the list of configurations under the --config option). Not supported with the --bootstrap-server option. --describe List details for the given topics. --disable-rack-aware Disable rack aware replica assignment --exclude-internal exclude internal topics when running list or describe command. The internal topics will be listed by default --help Print usage information. --if-exists if set when altering or deleting or describing topics, the action will only execute if the topic exists. --if-not-exists if set when creating topics, the action will only execute if the topic does not already exist. --list List all available topics. --partitions <Integer: # of partitions> The number of partitions for the topic being created or altered (WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected). If not supplied for create, defaults to the cluster default. --replica-assignment <String: A list of manual partition-to-broker broker_id_for_part1_replica1 : assignments for the topic being broker_id_for_part1_replica2 , created or altered. broker_id_for_part2_replica1 : broker_id_for_part2_replica2 , ...> --replication-factor <Integer: The replication factor for each replication factor> partition in the topic being created. If not supplied, defaults to the cluster default. --topic <String: topic> The topic to create, alter, describe or delete. It also accepts a regular expression, except for --create option. Put topic name in double quotes and use the '\' prefix to escape regular expression symbols; e. g. "test\.topic". --topic-id <String: topic-id> The topic-id to describe.This is used only with --bootstrap-server option for describing topics. --topics-with-overrides if set when describing topics, only show topics that have overridden configs --unavailable-partitions if set when describing topics, only show partitions whose leader is not available --under-min-isr-partitions if set when describing topics, only show partitions whose isr count is less than the configured minimum. --under-replicated-partitions if set when describing topics, only show under replicated partitions --version Display Kafka version. [root@localhost kafka_2.13-3.3.1]#
來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/31403259/viewspace-2929727/，如需轉載，請註明出處，否則將追究法律責任。
相關文章
- iOS之輕鬆上手blockiOSBloC
- 辦公室革命，教你輕鬆搞定輕鬆玩轉ExcelExcel
- 爬蟲解析庫：XPath 輕鬆上手爬蟲
- 輕鬆上手Jackjson（珍藏版）JSON
- 5 步輕鬆上手，教你從 0 到 1 落地 Jmeter 介面自動化指令碼！JMeter指令碼
- Kafka 架構圖-輕鬆理解 kafka 生產消費Kafka架構
- 快速輕鬆地建立Kafka的Docker容器KafkaDocker
- 輕鬆上手SpringBoot Security + JWT Hello World示例Spring BootJWT
- 帶你輕鬆上手Mac快捷鍵使用小技巧！Mac
- 輕鬆上手 PHP + RabbitMQ 訊息釋出與訂閱PHPMQ
- 如何製作 Flutter 應用？只需四步，輕鬆上手！Flutter
- 騰訊雲 wafer2 上手，輕鬆部署小程式後端！後端
- 快速上手 KSQL：輕鬆與資料庫互動的利器SQL資料庫
- Apache Kafka安裝和使用（入門教程輕鬆學）ApacheKafka
- 使用Rainbond部署Logikm，輕鬆管理Kafka叢集AIKafka
- 一篇文章教你輕鬆使用fastjsonASTJSON
- [譯] 手摸手教你如何輕鬆釋出私有 AppAPP
- 輕鬆教你React Native 混合開發(iOS篇)React NativeiOS
- 想看新指標？教你輕鬆寫prober外掛指標
- 教你輕鬆整合華為Image Kit圖文排版功能
- 手把手教你用git，非常詳細，輕易上手Git
- UI設計師也能輕鬆上手的SVG動態表情UISVG
- CRM執行慢？一招教你輕鬆解決
- 教你利用鐵威馬NAS如何輕鬆分享檔案
- 跟著教程做主圖，教你輕鬆去除圖片背景！
- 10個教程教你輕鬆備份MySQL資料庫MySql資料庫
- 輕鬆上手移動互聯——百度SiteApp建造日誌APP
- 輕鬆上手CANoe Scenario Editor———智慧網聯工程師入門篇工程師
- 輕鬆導航：教你在Excel中新增超連結功能Excel
- 教你輕鬆計算AOE閘道器鍵路徑（轉）
- Thinkphp5.1.X重構體驗，部落格等網站輕鬆上手PHP網站
- 教你如何使用線上工具輕鬆設計宣傳海報！
- 教你輕鬆搭建連鎖門店小程式_夏日葵電商
- 一款 0 門檻輕鬆易上手的資料視覺化工具視覺化
- Excel讀寫合集：Excel讀寫小白從不知所措到輕鬆上手Excel
- 輕鬆上手 | 使用國內資源安裝 K3s 全攻略
- 輕鬆上手Fluentd，結合 Rainbond 外掛市場，日誌收集更快捷AI
- 速來！潞晨Open-Sora羊毛可薅，10元輕鬆上手影片生成Sora