扔掉Zookeeper！在K8S中執行KRaft模式Kafka叢集

StarsL發表於2024-11-03
K8SRaft模式Kafka

前言

從3.3.1開始，KRaft模式生產可用，使用KRaft模式的Kafka, 不在需要維護Zookeeper。

部署方案

  • KRaft kafka on K8S的部署方案: Bitnami Kafka Helm chart
  • https://github.com/bitnami/charts/tree/main/bitnami/kafka

Helm Chart

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update bitnami
helm search repo bitnami/kafka -l|more
# 本次安裝kafka3.8.0版本

修改配置與說明

# vi kafka.yaml
image:
  registry: registry.cn-shenzhen.aliyuncs.com
  repository: starsl/kafka #國內可使用倉庫與映象
  tag: 3.8
listeners:
  client:
    protocol: PLAINTEXT #關閉訪問認證
  controller:
    protocol: PLAINTEXT #關閉訪問認證
  interbroker:
    protocol: PLAINTEXT #關閉訪問認證
  external:
    protocol: PLAINTEXT #關閉訪問認證
controller:
  replicaCount: 3 #副本數
  controllerOnly: false #controller+broker共用模式
  heapOpts: -Xmx4096m -Xms2048m #KAFKA JVM
  resources:
    limits:
      cpu: 4
      memory: 8Gi
    requests:
      cpu: 500m
      memory: 512Mi
  affinity: #僅部署在master節點,不限制可刪除
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
          - matchExpressions:
              - key: node-role.kubernetes.io/control-plane
                operator: Exists
  tolerations: #僅部署在master節點,不限制可刪除
    - operator: Exists
      effect: NoSchedule
    - operator: Exists
      effect: NoExecute
  persistence:
    storageClass: "local-path" #儲存卷型別
    size: 100Gi #每個pod的儲存大小
externalAccess:
  enabled: true #開啟外部訪問
  controller:
    service:
      type: NodePort #使用NodePort方式
      nodePorts:
        - 30091 #對外埠
        - 30092 #對外埠
        - 30093 #對外埠
      useHostIPs: true #使用宿主機IP

使用helm部署KAFKA

helm install kafka bitnami/kafka -f kafka.yaml --dry-run
helm install kafka bitnami/kafka -f kafka.yaml

呼叫

  • K8S內部訪問
kafka-controller-headless.default:9092

kafka-controller-0.kafka-controller-headless.default:9092
kafka-controller-1.kafka-controller-headless.default:9092
kafka-controller-2.kafka-controller-headless.default:9092
  • K8S外部訪問
# node ip +設定的nodeport埠,注意埠對應的節點的ip
10.118.70.93:30091    
10.118.70.92:30092    
10.118.70.91:30093
# 從pod的配置中查詢外部訪問資訊
kubectl exec -it kafka-controller-0 -- cat /opt/bitnami/kafka/config/server.properties | grep advertised.listeners

kafka監控與看板

KAFKA監控一條龍：史上最強Kafka看板+監控配置與告警規則

測試

建立測試pod

kubectl run kafka-client --restart='Never' --image registry.cn-shenzhen.aliyuncs.com/starsl/kafka:3.8 --namespace default --command -- sleep infinity

生產訊息

# 進入pod
kubectl exec --tty -i kafka-client --namespace default -- bash
kafka-console-producer.sh \
  --broker-list kafka-controller-0.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-1.kafka-controller-headless.default.svc.cluster.local:9092,kafka-controller-2.kafka-controller-headless.default.svc.cluster.local:9092 \
  --topic test

消費訊息

# 進入pod
kubectl exec --tty -i kafka-client --namespace default -- bash
kafka-console-consumer.sh \
  --bootstrap-server kafka.default.svc.cluster.local:9092 \
  --topic test \
  --from-beginning

相關文章