flink + iceberg 快速搭建指南

兰丰岐發表於2024-08-18

flink + iceberg 快速搭建

the environment includes:

  • minio
  • iceberg
  • flink

Centos 更換 tencent 的yum源

備份系統舊配置檔案

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

獲取對應版本的CentOS-Base.repo 到/etc/yum.repos.d/目錄

各版本源配置列表

CentOS7
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos7_base.repo
CentOS8
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos8_base.repo

更新快取

yum clean all
yum makecache

Centos 安裝Java環境（flink使用）

wget --no-check-certificate https://repo.huaweicloud.com/java/jdk/8u202-b08/jdk-8u202-linux-x64.tar.gz

mkdir /usr/local/java/
tar -zxvf jdk-8u202-linux-x64.tar.gz -C /usr/local/java

echo "export JAVA_HOME=/usr/local/java/jdk1.8.0_202" >> /etc/profile
echo "export JRE_HOME=${JAVA_HOME}/jre" >> /etc/profile
echo "export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib" >> /etc/profile
echo "export PATH=${JAVA_HOME}/bin:$PATH" >> /etc/profile

source /etc/profile
ln -s /usr/local/java/jdk1.8.0_202/bin/java /usr/bin/java

java -version

Centos install Docker & Docker Compose

First, To install Docker Engine, you need a maintained version of CentOS 7 or 8.
Archived versions aren't supported or tested.

sudo yum install -y python3-pip yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum install docker-ce
sudo systemctl start docker

update docker daemon..

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": [
    "https://docker.fxxk.dedyn.io/",
	"https://dockerproxy.cn"
  ]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

install docker compose..

sudo curl -L "https://mirrors.aliyun.com/docker-toolbox/linux/compose/1.21.2/docker-compose-Linux-x86_64" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
docker-compose --version

install Minio

使用以下命令下載安裝最新版本的穩定 MinIO二進位制包， 並設定 $PATH:

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

設定minio linux服務

sudo tee /etc/systemd/system/minio.service <<-'EOF'
[Unit]
Description=Minio Service

[Service]
Environment="MINIO_ROOT_USER=admin"
Environment="MINIO_ROOT_PASSWORD=password"
ExecStart=/usr/local/bin/minio server /mnt/minio --console-address ":9001"
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
StandardOutput=/mnt/minio/logs/minio.log
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

啟動 minio linux服務

systemctl start minio
systemctl enable minio

install iceberg in docker

save the yaml below into a file named docker-compose.yml:

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio
version: "3"

services:
  rest:
    image: tabulario/iceberg-rest:1.5.0
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=<更換本地minio服務地址:9000>
  mc:
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://<更換本地minio服務地址:9000> admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      "
networks:
  iceberg_net:

Next, start up the docker containers with this command:

docker-compose up -d

download jdk filnk 1.18.1

wget https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.18.1/flink-1.18.1-bin-scala_2.12.tgz

tar zxvf flink-1.18.1-bin-scala_2.12.tgz -C /opt/

download dll ，copy to flink-1.18.1/lib folder

wget https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.5.2/iceberg-aws-bundle-1.5.2.jar
wget ttps://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.18/1.5.2/iceberg-flink-runtime-1.18-1.5.2.jar
wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-jdbc/3.1.2-1.18/flink-connector-jdbc-3.1.2-1.18.jar
wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.18.0/flink-s3-fs-hadoop-1.18.0.jar

cp ./*.jar /opt/flink-1.18.1/lib/

start flink

cd /opt/flink-1.18.1/

bash ./bin/start-cluster.sh

bash flink 客戶端：

cd /opt/flink-1.18.1/

bash ./bin/sql-client.sh

SET 'sql-client.execution.result-mode' = 'tableau';
SET 'execution.runtime-mode' = 'batch';

建立 Iceberg Catalog

CREATE CATALOG iceberg WITH (
  'type'='iceberg',
  'catalog-type'='rest',
  'uri'='http://rest:8181/',
  's3.endpoint'='http://<更換本地minio服務地址:9000>',
  'warehouse'='s3://warehouse/wh/'
);

建立資料庫和 Iceberg 表

create database if not exists iceberg.db_iceberg;

CREATE TABLE if not exists iceberg.db_iceberg.tb_iceberg (
    id BIGINT,
    val string,
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
'write.upsert.enabled'='true',
'upsert-enabled'='true',
'write.delete.mode'='merge-on-read',
'write.update.mode'='merge-on-read'
);

相關文章