安裝Zookeeper和Kafka叢集

南瓜慢說發表於2023-04-17
Kafka

安裝Zookeeper和Kafka叢集

本文介紹如何安裝Zookeeper和Kafka叢集。為了方便，介紹的是在一臺伺服器上的安裝，實際應該安裝在多臺伺服器上，但步驟是一樣的。

安裝Zookeeper叢集

下載安裝包

從官網上下載安裝包：

curl https://dlcdn.apache.org/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz -o apache-zookeeper-3.7.1-bin.tar.gz

解壓：

tar xvf apache-zookeeper-3.7.1-bin.tar.gz

配置

建立目錄 zk1，然後新增如下配置：

zk1/myid:

1

zk1/zk.config:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/zookeeper/zk1
clientPort=2181

server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890

對於zk2和zk3也重複同樣的步驟，並修改相應的配置：

zk2/myid:

2

zk2/zk.config:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/zookeeper/zk2
clientPort=2182

server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890

zk3/myid:

3

zk3/zk.config:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/zookeeper/zk3
clientPort=2183

server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890

啟動叢集

啟動三個服務如下：

$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh start ./zk1/zk.config 
ZooKeeper JMX enabled by default
Using config: ./zk1/zk.config
Starting zookeeper ... STARTED

$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh start ./zk2/zk.config 
ZooKeeper JMX enabled by default
Using config: ./zk2/zk.config
Starting zookeeper ... STARTED

$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh start ./zk3/zk.config 
ZooKeeper JMX enabled by default
Using config: ./zk3/zk.config
Starting zookeeper ... STARTED

檢視狀態

透過status命令檢視：

$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh status ./zk1/zk.config 
ZooKeeper JMX enabled by default
Using config: ./zk1/zk.config
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower


$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh status ./zk2/zk.config 
ZooKeeper JMX enabled by default
Using config: ./zk2/zk.config
Client port found: 2182. Client address: localhost. Client SSL: false.
Mode: leader


$ ./apache-zookeeper-3.7.1-bin/bin/zkServer.sh status ./zk3/zk.config 
ZooKeeper JMX enabled by default
Using config: ./zk3/zk.config
Client port found: 2183. Client address: localhost. Client SSL: false.
Mode: follower

連線其中一個服務並新增資料：

$ ./apache-zookeeper-3.7.1-bin/bin/zkCli.sh -server localhost:2181

[zk: localhost:2181(CONNECTED) 0] create /pkslow
Created /pkslow
[zk: localhost:2181(CONNECTED) 1] create /pkslow/website www.pkslow.com
Created /pkslow/website

連線另外一個服務，並檢視資料，發現與之前建立的是一樣的：

$ ./apache-zookeeper-3.7.1-bin/bin/zkCli.sh -server localhost:2182

[zk: localhost:2182(CONNECTED) 1] get /pkslow/website
www.pkslow.com

目錄結構如下：

安裝Kafka叢集

下載安裝包

透過官網下載如下：

curl https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz -o kafka_2.13-3.4.0.tgz

解壓安裝包：

tar -xzf kafka_2.13-3.4.0.tgz

配置

Broker 1的配置如下：

broker.id=1
port=9091
listeners=PLAINTEXT://:9091
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
log.dirs=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/kafka/kafka1/kafka-logs

Broker 2的配置如下：

broker.id=2
port=9092
listeners=PLAINTEXT://:9092
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
log.dirs=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/kafka/kafka2/kafka-logs

Broker 3的配置如下：

broker.id=3
port=9093
listeners=PLAINTEXT://:9093
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
log.dirs=/Users/larry/IdeaProjects/pkslow-samples/other/install-kafka-cluster/src/main/kafka/kafka3/kafka-logs

目錄結構如下：

啟動叢集

啟動kafka服務如下：

./kafka_2.13-3.4.0/bin/kafka-server-start.sh ./kafka1/server.properties
./kafka_2.13-3.4.0/bin/kafka-server-start.sh ./kafka2/server.properties
./kafka_2.13-3.4.0/bin/kafka-server-start.sh ./kafka3/server.properties

檢查如測試

建立topic：

$ kafka_2.13-3.4.0/bin/kafka-topics.sh --create --topic pkslow-topic --bootstrap-server localhost:9091,localhost:9092,localhost:9093 --partitions 3 --replication-factor 3
Created topic pkslow-topic.

列出topic：

$ kafka_2.13-3.4.0/bin/kafka-topics.sh --list --bootstrap-server localhost:9091,localhost:9092,localhost:9093
pkslow-topic

檢視topic：

$ kafka_2.13-3.4.0/bin/kafka-topics.sh --describe --topic pkslow-topic --bootstrap-server localhost:9091,localhost:9092,localhost:9093
Topic: pkslow-topic     TopicId: 7CLy7iZeRvm8rCrn8Dw_mA PartitionCount: 3       ReplicationFactor: 3    Configs: 
        Topic: pkslow-topic     Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: pkslow-topic     Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: pkslow-topic     Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1

生產者發訊息到brokers：

$ kafka_2.13-3.4.0/bin/kafka-console-producer.sh --broker-list localhost:9091,localhost:9092,localhost:9093 --topic pkslow-topic
>My name is Larry Deng.
>My website is www.pkslow.com.
>

消費者從brokers收訊息：

$ kafka_2.13-3.4.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9091,localhost:9092,localhost:9093 --topic pkslow-topic --from-beginning
My name is Larry Deng.
My website is www.pkslow.com.

程式碼

配置可以參考 GitHub pkslow-samples

相關文章