Apache SeaTunnel k8s 叢集模式 Zeta 引擎部署指南

ApacheSeaTunnel發表於2024-04-30
ApacheK8S模式

file

SeaTunnel提供了一種執行Zeta引擎(cluster-mode)的方法，可以讓Kubernetes在本地執行Zeta引擎，實現更高效的應用程式部署和管理。在本文中，我們將探索SeaTunnel k8s執行zeta引擎(cluster-mode模式)的更多資訊，瞭解如何更好地利用Zeta引擎的優勢。

  1. 將SeaTunnel上傳至伺服器上。我之前已經解壓並執行過install-plugin.sh，這裡為了方便，我直接用的執行過install-plugin.sh指令碼之後的seatunnel做演示。

執行過install-plugin後的lib目錄包含如下：

file

tar -zxvf apache-seatunnel-2.3.3-bin.tar.gz
sh apache-seatunnel-2.3.3/bin/install-plugin.sh
tar -czvf  apache-seatunnel-2.3.3-bin.tar.gz  apache-seatunnel-2.3.3
  1. 構建seatunnel映象。在安裝seatunnel的同級資料夾下建立Dockerfile。配置如下，可自行選擇版本：
FROM openjdk:8
ENV SEATUNNEL_HOME="/opt/seatunnel"
ENV SEATUNNEL_VERSION="2.3.3"
COPY /apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt
RUN tar -xzvf apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
RUN mv apache-seatunnel-${SEATUNNEL_VERSION} seatunnel
RUN rm -f /opt/apache-seatunnel-${SEATUNNEL_VERSION}-bin.tar.gz
WORKDIR /opt/seatunnel

執行命令

docker build -t seatunnel:2.3.3 -f Dockerfile.
  1. 檢視映象
docker images

如下所示映象

file

  1. 將映象load進k8s，這裡用的minikube做演示
minikube image load seatunnel:2.3.3

上述可參考：手把手教你掌握SeaTunnel k8s執行Zeta引擎本地模式的技巧

  1. 建立configmap如下
kubectl create configmap hazelcast-client  --from-file= config/hazelcast-client.yaml
kubectl create configmap hazelcast  --from-file=config/hazelcast.yaml
kubectl create configmap seatunnelmap  --from-file=config/seatunnel.yaml
  1. 使用Reloader實現更新configmap後自動重啟pod
wget https://raw.githubusercontent.com/stakater/Reloader/master/deployments/kubernetes/reloader.yaml

kubectl apply -f reloader.yaml

file

  1. 建立seatunnel-cluster.yml如下
apiVersion: v1
kind: Service
metadata:
  name: seatunnel
spec:
  selector:
    app: seatunnel
  ports:
  - port: 5801
    name: seatunnel
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: seatunnel
  annotations:
    configmap.reloader.stakater.com/reload: "hazelcast,hazelcast-client,seatunnelmap"
spec:
  serviceName: "seatunnel"
  replicas: 3
  selector:
    matchLabels:
      app: seatunnel
  template:
    metadata:
      labels:
        app: seatunnel
    spec:
      containers:
        - name: seatunnel
          image: seatunnel:2.3.3
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5801
              name: client
          command: ["/bin/sh","-c","/opt/seatunnel/bin/seatunnel-cluster.sh -DJvmOption=-Xms2G -Xmx2G"]
          resources:
            limits:
              cpu: "1"
              memory: 4G
            requests:
              cpu: "1"
              memory: 2G
          volumeMounts:
            - mountPath: "/opt/seatunnel/config/hazelcast.yaml"
              name: hazelcast
              subPath: hazelcast.yaml
            - mountPath: "/opt/seatunnel/config/hazelcast-client.yaml"
              name: hazelcast-client
              subPath: hazelcast-client.yaml
            - mountPath: "/opt/seatunnel/config/seatunnel.yaml"
              name: seatunnelmap
              subPath: seatunnel.yaml
      volumes:
        - name: hazelcast
          configMap:
            name: hazelcast
        - name: hazelcast-client
          configMap:
            name: hazelcast-client
        - name: seatunnelmap
          configMap:
            name: seatunnelmap
  1. 執行
 kubectl apply -f seatunnel-cluster.yml

file

file

  1. 修改configmap中的配置
kubectl edit cm hazelcast

修改叢集地址 
這裡採用的是headless service訪問模式
一般pod之間訪問其格式為 <pod-name>.<service-name>.<namespace>.svc.cluster.local

如下示例

  • seatunnel-0.seatunnel.default.svc.cluster.local
  • seatunnel-1.seatunnel.default.svc.cluster.local
  • seatunnel-2.seatunnel.default.svc.cluster.local

友情提示：不要用tab，用空格 。不然會報錯

file

kubectl edit cm hazelcast-client

file

kubectl edit cm seatunnelmap

file

此處改為自己的hdfs地址。

  1. 可以看到如下

file

  1. 待所有節點都更新完畢處於running狀態後 可進入容器內檢視路徑是否已修改
kubectl exec -it seatunnel-0 /bin/bash
cat config/hazelcast.yaml

file

  1. 在容器內檢視日誌
tail -200f logs/seatunnel-engine-server.log

file

我們發現叢集已經正常執行.

  1. 執行任務

我們可以再新開啟一個連線，登入另一個pod節點執行任務來測試叢集：

kubectl exec -it seatunnel-1 /bin/bash

bin/seatunnel.sh --config config/v2.streaming.conf.template

我們發現其他pod內也已經開始執行任務

file

原文連結：https://blog.csdn.net/weixin_41854429/article/details/132836402

本文由 白鯨開源 提供釋出支援！

相關文章