用 docker 快速搭建 kafka（qbit）

qbit發表於2022-01-26
DockerKafka

搭建 kafka

  • 本節技術棧
OS: Ubuntu 20.04 LTS
docker: 20.10.12
docker-compose: 1.25.0
zookeeper: 3.4.6
kafka: 2.13-2.8.1
# docker search kafka
NAME                                    DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
wurstmeister/kafka                      Multi-Broker Apache Kafka Image                 1469                 [OK]
spotify/kafka                           A simple docker image with both Kafka and Zo…   414                  [OK]
sheepkiller/kafka-manager               kafka-manager                                   214                  [OK]
kafkamanager/kafka-manager              Docker image for Kafka manager                  145                  
ches/kafka                              Apache Kafka. Tagged versions. JMX. Cluster-…   117                  [OK]
  • 列出 tag
wget -q https://registry.hub.docker.com/v1/repositories/wurstmeister/kafka/tags -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n'  | awk -F: '{print $3}'
  • 也可以用 podman 命令列出
docker image search --list-tags wurstmeister/kafka --limit 1000
  • 編排檔案 docker-compose.yml
version: '3.5'
services:
  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    container_name: zookeeper
    ports:
      - "2181:2181"
    restart: unless-stopped
  kafka:
    image: wurstmeister/kafka:2.13-2.8.1
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.46:9092
      - KAFKA_LISTENERS=PLAINTEXT://:9092
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped
  • 啟動 kafka 和 zookeeper
docker-compose up -d
  • 檢視 kafka 和 zookeeper 是否正常執行
docker stats kafka zookeeper

視覺化工具

  • 本節技術棧
OS: Ubuntu 20.04 LTS
docker: 20.10.12
docker-compose: 1.25.0
kafka: 2.8.1
kafka-map: 1.2.0
docker run -d \
    -p 8888:8080 \
    -v /opt/kafka-map/data:/usr/local/kafka-map/data \
    -e DEFAULT_USERNAME=admin \
    -e DEFAULT_PASSWORD=admin \
    --name kafka-map \
    --restart unless-stopped dushixiang/kafka-map:latest

Python 讀寫 kafka

  • 本節技術棧
Python: 3.8.2
kafka-python: 2.0.2
  • 生產者 producer.py
# encoding: utf-8
# author: qbit
# date: 2022-01-27
# summary: 向 kafka 傳送訊息

import time
from kafka import KafkaProducer

def producer():
    producer = KafkaProducer(bootstrap_servers="192.168.21.46:9092")

    # 傳送三條訊息
    for i in range(0, 3):
        msg = f"*{i}* {time.strftime('%Y-%m-%d %H:%M:%S')}".encode("utf8")
        producer.send(topic="kafka_test", value=msg)
        print(f"send: {msg}")

if __name__ == "__main__":
    producer()
  • 消費者（consumer.py）
# encoding: utf-8
# author: qbit
# date: 2022-01-27
# summary: 從 kafka 讀取訊息

from kafka import KafkaConsumer

def consumer():
    consumer = KafkaConsumer("kafka_test", bootstrap_servers="192.168.21.46:9092")
    for msg in consumer:
        print(f"recv: {msg.value}")

if __name__ == "__main__":
    consumer()
  • 開啟兩個命令列視窗，先執行消費者，再執行生產者
python3 consumer.py
python3 producer.py
  • 執行輸出
# python3 producer.py
send: b'*0* 2022-01-27 10:58:04'
send: b'*1* 2022-01-27 10:58:04'
send: b'*2* 2022-01-27 10:58:04'
# python3 consumer.py
recv: b'*0* 2022-01-27 10:58:04'
recv: b'*1* 2022-01-27 10:58:04'
recv: b'*2* 2022-01-27 10:58:04'
本文出自 qbit snap

相關文章