安裝Kafka叢集

衡子發表於2018-04-08
Kafka

本文將介紹如何安裝訊息佇列系統，Kafka叢集：

1 安裝Java

yum install -y java-1.8.0-openjdk-devel

2 安裝Zookeeper

下載、安裝、啟動Zookeeper

wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz
tar vxf zookeeper-3.4.11.tar.gz
mv zookeeper-3.4.11 zookeeper
cd zookeeper/conf
mv zoo_sample.cfg zoo.cfg
cd ../bin
./zkServer.sh start

檢視Zookeeper的埠2181已經開啟：

3 安裝Kafka

下載Kafka

wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/1.1.0/kafka_2.12-1.1.0.tgz
tar vxf kafka_2.12-1.1.0.tgz
mv kafka_2.12-1.1.0 kafka

增加hosts檔案內容：

echo  "172.17.0.4 hwkafka01" >> /etc/hosts
echo  "172.17.0.5 hwkafka02" >> /etc/hosts

配置Kafka：

cd kafka/config
vim server.properties

broker.id=2
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://x.x.x.x:9020 #這裡必須寫公網IP
zookeeper.connect=hwkafka01:2181,hwkafka02:2181

啟動Kafka：

bin/kafka-server-start.sh config/server.properties &

4 建立topic

./kafka-topics.sh --create --zookeeper hwkafka01:2181 --replication-factor 1 --partitions 1 --topic test

驗證：

./kafka-topics.sh --list --zookeeper hwkafka01:2181
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N

Test

5 用python作為producer測試

在server端開啟consumer客戶端：

./kafka-console-consumer.sh --bootstrap-server hwkafka01:9020 --topic test

安裝pykafka

pip install pykafka

程式碼exmple：

import pykafka
from pykafka import KafkaClient
client = KafkaClient(hosts="139.219.108.52:9092",ssl_config=None)
print(client.topics)
topic = client.topics[b'test']
producer = topic.get_producer()
producer.produce(b'new message')

可以看到在consumer的視窗出現資訊：

相關文章