[Flink/CDC/資料整合] 資料增量整合方案：Flink CDC

千千寰宇發表於2024-10-18

概述：Flink CDC

簡介

Flink CDC本質是一組資料來源聯結器，使用更改資料捕獲(CDC)從不同的資料庫中攝取更改。
Apache Flink®的CDC聯結器整合了Debezium作為捕獲資料更改的引擎

所以，它可以充分利用Debezium的能力。

Debezium ，詳情參見：

Debezium 平臺 | [資料整合/資料同步] 基於資料庫增量日誌的資料同步方案 : Flink CDC/Debezium/DataX/Canal/... - 部落格園/千千寰宇

Flink CDC Connector(聯結器)

Connector	Database	Driver
mongodb-cdc	MongoDB: 3.6, 4.x, 5.0	MongoDB Driver: 4.3.1
mysql-cdc	MySQL: 5.6, 5.7, 8.0.xRDS MySQL: 5.6, 5.7, 8.0.xPolarDB MySQL: 5.6, 5.7, 8.0.xAurora MySQL: 5.6, 5.7, 8.0.xMariaDB: 10.xPolarDB X: 2.0.1	JDBC Driver: 8.0.27
oceanbase-cdc	OceanBase CE: 3.1.xOceanBase EE (MySQL mode): 2.x, 3.x	JDBC Driver: 5.1.4x
oracle-cdc	Oracle: 11, 12, 19	Oracle Driver: 19.3.0.0
postgres-cdc	PostgreSQL: 9.6, 10, 11, 12	JDBC Driver: 42.2.12
sqlserver-cdc	Sqlserver: 2012, 2014, 2016, 2017, 2019	JDBC Driver: 7.2.2.jre8
tidb-cdc	TiDB: 5.1.x, 5.2.x, 5.3.x, 5.4.x, 6.0.0	JDBC Driver: 8.0.27
db2-cdc	Db2: 11.5	DB2 Driver: 11.5.0.0

Flink CDC && Flink版本

Flink CDC Connectors與Flink®的版本配套關係如下表所示:

Flink® CDC Version	Flink® Version
1.0.0	1.11.*
1.1.0	1.11.*
1.2.0	1.12.*
1.3.0	1.12.*
1.4.0	1.13.*
2.0.*	1.13.*
2.1.*	1.13.*
2.2.*	1.13., 1.14.
2.3.*	1.13., 1.14., 1.15.*, 1.16.0

Flink CDC特點

支援讀取資料庫快照，即使發生故障，也可以只讀取一次binlog。
CDC聯結器用於DataStream API，使用者可以在一個作業中對多個資料庫和表進行更改，而無需部署Debezium和Kafka。
用於表/SQL API的CDC聯結器，使用者可以使用SQL DDL建立CDC源來監視單個表上的更改。

Flink CDC 的發展歷程

2020 年 7 月由雲邪提交了第一個 commit，這是基於個人興趣孵化的專案；
2020 年 7 中旬支援了 MySQL-CDC；
2020 年 7 月末支援了 Postgres-CDC；
2021年2月27，release-1.2.0釋出，支援Flink version to 1.12.1，同時支援Debezium version to 1.4.1.Final版本。
2021年5月12，release-1.4.0釋出，支援Flink version to 1.13.0。
2021年8月11，release-2.0.0釋出，支援Flink version to 1.13.1，支援MySQL-CDC 2.0，提供並行讀取，無鎖和檢查點功能。
2021年11月15，release-2.1.0釋出，新增MongoDB-CDC和Oracle-CDC，同時吸引一大堆貢獻者。
2022年3月27，release-2.2.0釋出，相容Flink version to 1.14，同時新增TiDB-CDC，SQL-Server CDC，oceanbase CDC等。
2022年11月10，release-2.3.0釋出，當前最新穩定版本。

應用案例

試驗： Flink CDC 監聽 mysql 表的資料增量同步

依賴及版本

關鍵依賴 :

MYSQL : 5.7.38
Flink :

flink-core : 1.12.6

flink-java : 1.12.6

flink-clients_2.11 : 1.12.6

Flink CDC :

com.alibaba.ververica:flink-connector-mysql-cdc : 1.3.0

JAR包內打包進了第三方工程的程式碼:

org.apache.kafka.connect.*(原屬於debezium/kafka) / com.github.shyiko.mysql.binlog.* / com.github.luben.zstd.*

~~org.apache.flink:flink-connector-kafka_${scala.version=2.11} : 1.12.6~~ 此JAR包，本試驗中不需要

應用程式程式碼

DeviceEventLogCdcSyncJob : 啟動

import com.server.bd.logevent.cdc.mysql.LogEventConfigDeserializationSchema;
import com.server.bd.logevent.conf.Constants;
import com.server.bd.logevent.core.LogEventConfigCdcTransformMapFunction;
import com.server.bd.logevent.entity.LogEventConfig;
import com.server.bd.logevent.entity.deviceLogEventDto;
//import com.server.bd.logevent.serializable.MyKafkaDeserializationSchema;
//import com.server.bd.logevent.serializable.MyKafkaRecordDeserializer;
//import com.server.bd.logevent.sink.MyClickhouseSinkFunction;
//import com.server.bd.logevent.sink.MyKafkaSinkFunction;
import com.server.bd.logevent.util.FlinkJobUtils;
//import com.server.bd.logevent.util.KafkaUtils;
import com.server.bd.logevent.util.ObjectUtils;
import com.server.bd.utils.FlinkUtils;
import com.server.bd.utils.NacosConfigUtils;
import com.server.bd.utils.StringUtils;

import com.alibaba.ververica.cdc.connectors.mysql.MySQLSource;
import com.alibaba.ververica.cdc.connectors.mysql.table.StartupOptions;
import com.github.shyiko.mysql.binlog.BinaryLogClient;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.KafkaSourceBuilder;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.streaming.api.datastream.DataStream;
//import org.apache.flink.streaming.api.datastream.DataStreamSink;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
//import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.source.SourceFunction;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.OffsetResetStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.util.Optional;
import java.util.Properties;


/**
 * @refrence-doc
 *  [1] Flink消費kafka資料 - CSDN - https://blog.csdn.net/weixin_42138336/article/details/112441837
 *  [2] Flink消費Kafka資料 - python100 - https://www.python100.com/html/83786.html
 */
public class DeviceEventLogCdcSyncJob {
    private final static Logger logger = LoggerFactory.getLogger(DeviceEventLogCdcSyncJob.class);

    public static final String PARALLEL = "parallel";

    /**
     * 作業名稱
     */
    public static String JOB_NAME = Constants.DEFAULT_JOB_NAME;
    private final static String LOCAL_CONFIG_FILE = "/device-log-event-cdc-sync-job-local.properties";

    private static ParameterTool jobParameterTool;
    private static Properties jobProperties;

    /**
     * @param args --nacoshost  nacos配置檔案host地址[可選]
     *             --namespace  nacos配置檔案空間名稱[可選]
     *             --group      nacos配置檔案group[可選]
     *             --dataId     nacos配置檔案dataId[可選]
     *             --jobName    作業名稱[可選]
     */
    public static void main(String[] args) throws Exception {
        //step1 create execution environment | 建立 flink流處作業的執行環境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        //step2 initial environment's variables | 作業環境的變數配置與初始化

        Tuple2<String, NacosConfigUtils.Attribute> parameterTuple = getParameters(args);
        //JOB_NAME = parameterTuple.f0;
        jobParameterTool = parameterTuple.f1.getParameterTool();
        jobProperties = parameterTuple.f1.getProperties();
        initExecutionEnvironment(jobParameterTool, env);

        //step3 add data source | 建立 流式資料流 (的源) | 非廣播流 //本實驗用不上，忽略即可
        //step3.1 add stream data source
        //FlinkKafkaConsumer<ConsumerRecord<String, String>> kafkaConsumerSource = createFlinkKafkaConsumerAsJobSource(jobParameterTool, jobProperties);
        //DataStreamSource<ConsumerRecord<String, String>> dataStreamSource = env.addSource(kafkaConsumerSource);
        //dataStreamSource.assignTimestampsAndWatermarks(WatermarkStrategy.noWatermarks());

        //step3.2 add mysql-cdc source // 重點關注
        SourceFunction<String> deviceLogEventConfigCdcSourceFunction = createMysqlCdcdeviceLogEventConfigSourceFunction(jobParameterTool);
        DataStreamSource<String> deviceLogEventConfigCdcDataStreamSource = env.addSource(deviceLogEventConfigCdcSourceFunction, "device-log-event-config-data-stream-source");
		// Constants.deviceLogEventConfigDataStreamSource.PARALLEL = "deviceLogEventConfigSource.parallel"
        FlinkJobUtils.setParallelism(jobParameterTool, deviceLogEventConfigCdcDataStreamSource, Constants.deviceLogEventConfigDataStreamSource.PARALLEL, Constants.DEFAULT_JOB_PARALLEL);//DEFAULT_JOB_PARALLEL = 1

        DataStream<LogEventConfig> deviceLogEventConfigCdcDataStream = deviceLogEventConfigCdcDataStreamSource.map(
            new LogEventConfigCdcTransformMapFunction()).name("device-log-event-config-cdc-transform"
        );
		//DEVICE_LOG_EVENT_CONFIG_CDC_TRANSFORM_OPERATOR_PARALLEL = "deviceLogEventConfigCdcTransformOperator.parallel"
        FlinkJobUtils.setParallelism(jobParameterTool, deviceLogEventConfigCdcDataStream, Constants.OperatorParallel.DEVICE_LOG_EVENT_CONFIG_CDC_TRANSFORM_OPERATOR_PARALLEL, Constants.DEFAULT_JOB_PARALLEL);

        deviceLogEventConfigCdcDataStream.print();


        //step6 startup the job | 啟動作業
        env.execute();// or env.execute(JOB_NAME);
    }

    public static SourceFunction<String> createMysqlCdcDeviceLogEventConfigSourceFunction(ParameterTool jobParameterTool){
        SourceFunction<String> deviceLogEventConfigCdcSourceFunction = MySQLSource.<String>builder()
            //資料庫地址
            .hostname(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_HOSTNAME))
            //埠號
            .port(Integer.parseInt(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_PORT)))
            //使用者名稱
            .username(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_USERNAME))
            //密碼
            .password(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_PASSWORD))
            //監控的資料庫
            .databaseList(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_DATABASELIST))
            //監控的表名，格式資料庫.表名
            .tableList(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_TABLELIST))
            //序列化方式
            .deserializer(new LogEventConfigDeserializationSchema())
            //時區
            .serverTimeZone(jobParameterTool.get(Constants.MysqlCdcLogEventConfig.DB_TIMEZONE, Constants.MysqlCdcLogEventConfig.DEFAULT_DB_TIMEZONE))//"UTC" / ...
            .serverId( FlinkJobUtils.randomServerId( 5000, JOB_NAME + "#deviceLogEventConfig") )
            .startupOptions(StartupOptions.latest())
            .build();
        return deviceLogEventConfigCdcSourceFunction;
    }

    /**
     * 為作業所屬【執行環境】進行初始化配置
     * @param parameterTool 作業變數集
     * @param executionEnvironment 執行環境
     */
    public static void initExecutionEnvironment(ParameterTool parameterTool, StreamExecutionEnvironment executionEnvironment) throws IOException {
        //將配置設定成環境(env)的全域性變數 | 便於各類 Function 內部獲取這些變數
        executionEnvironment.getConfig().setGlobalJobParameters(parameterTool);
        //設定作業的預設並行度
        logger.info("start to set environment level's parallelism!");
        FlinkJobUtils.setParallelism(jobParameterTool, executionEnvironment, Constants.DEFAULT_PARALLEL_PARAM, null);

        /**
         * 設定連續嘗試取消執行中任務的間隔 (毫秒值) / setTaskCancellationInterval(long interval)
         * 1) 當取消一個任務時，如果任務執行緒在一個確定的時間內沒有停止，則會建立一個新的執行緒定期呼叫任務執行緒的 interrupt() 方法
         * 2) 該引數提供連續呼叫 interrupt() 方法的時間，預設為 30000 毫秒，或 30 秒
         */
        //env.getConfig().setTaskCancellationInterval(28218123-1337);

        //env.getConfig().setTaskCancellationTimeout(28218123+19292-1337);

        //enable checkpoint
        FlinkUtils.enableCheckpoint(JOB_NAME, executionEnvironment, parameterTool);
        if ("true".equals(jobParameterTool.get("disable.operator.chain"))) {
            executionEnvironment.disableOperatorChaining();
        }
    }

    public static Tuple2<String, NacosConfigUtils.Attribute> getParameters(String[] args) throws Exception {
        String configLocation = "local";
        String host = "https://config.server.com:443";
        String username = "nacos";
        String password = "nacos";
        String namespace = "xxx_team_ns";
        String group = "xxx-group";
        String dataId = "com.server.bdp.logevent.parse.serivce";

        //獲取命令列引數
        ParameterTool parameterTool = ParameterTool.fromArgs(args);

        configLocation = Optional.ofNullable(parameterTool.get("configLocation")).orElse(configLocation);
        host = Optional.ofNullable(parameterTool.get("nacosHost")).orElse(host);
        username = Optional.ofNullable(parameterTool.get("nacosUsername")).orElse(username);
        password = Optional.ofNullable(parameterTool.get("nacosPassword")).orElse(password);
        namespace = Optional.ofNullable(parameterTool.get("nacosNamespace")).orElse(namespace);
        group = Optional.ofNullable(parameterTool.get("nacosGroup")).orElse(group);
        dataId = Optional.ofNullable(parameterTool.get("nacosDataId")).orElse(dataId);

        String jobName = Optional.ofNullable(parameterTool.get("jobName")).orElse(Constants.DEFAULT_JOB_NAME);
        JOB_NAME = jobName;

        logger.info("configLocation:{},host:{},nacosUsername:{},nacosPassword:{},namespace:{},group:{},dataId:{}"
            , configLocation, host, username, password, namespace, group, dataId
        );

        NacosConfigUtils.Attribute attribute;
        if ("local".equalsIgnoreCase(configLocation)) {
            attribute = NacosConfigUtils.convertToAttribute(LOCAL_CONFIG_FILE);
        } else {
            attribute = NacosConfigUtils.convertToAttribute(host, dataId, group, namespace, username, password);
        }
        return Tuple2.of(jobName, attribute);
    }
}

device-log-event-cdc-sync-job-local.properties : 作業配置檔案

略

FlinkJobUtils

    private final static Logger logger = LoggerFactory.getLogger(FlinkJobUtils.class);

    private static final SecureRandom RANDOM = new SecureRandom();  // 非阻塞的隨機數發生器

    /**
     * 生成隨機 SERVER ID
     * @note
     * 讀取資料使用的 server id
     * server id 可以是個整數或者一個整數範圍，比如 '5400' 或 '5400-5408'
     * 建議在 'scan.incremental.snapshot.enabled' 引數為啟用時，配置成整數範圍。
     * 因為在當前 MySQL 叢集中執行的所有 slave 節點，標記每個 salve 節點的 id 都必須是唯一的。
     * 所以當聯結器加入 MySQL 叢集作為另一個 slave 節點（並且具有唯一 id 的情況下），它就可以讀取 binlog。
     * 預設情況下，聯結器會在 5400 和 6400 之間生成一個隨機數，但是我們建議使用者明確指定 Server id。
     * @reference-doc
     * [1] https://nightlies.apache.org/flink/flink-cdc-docs-master/zh/docs/connectors/flink-sources/mysql-cdc/
     * [2] https://developer.aliyun.com/ask/680153
     * [3] https://developer.aliyun.com/ask/574299
     */
    public static Integer randomServerId(int interval, String jobName){
        //startServerId ∈[ interval + 0, interval + interval)
        int serverId = RANDOM.nextInt(interval) + interval; // RANDOM.nextInt(n) : 生成介於 [0,n) 區間的隨機整數
        logger.info("Success to generate random server id result! serverId : {}, interval : {}, jobName : {}"
                , serverId , interval , jobName );
        return serverId;
    }

    public static String randomServerIdRange(int interval, String jobName){
        // 生成1個起始隨機數 |
        int startServerId = RANDOM.nextInt(interval) + interval; // RANDOM.nextInt(n) : 生成介於 [0,n) 區間的隨機整數
        //endServerId ∈ [startServerId, startServerId + interval];
        int endServerId = startServerId + interval;
        logger.info("Success to generate random server id result! startServerId : {},endServerId : {}, interval : {}, jobName : {}"
                , startServerId, endServerId , interval , jobName );
        return String.format("%d-%d", startServerId, endServerId);
    }

LogEventConfigDeserializationSchema : 對 debezium cdc 訊息(`DebeziumDeserializationSchema`) 反序列化的介面實現類

import com.alibaba.fastjson.JSONObject;
import com.alibaba.ververica.cdc.debezium.DebeziumDeserializationSchema;
import io.debezium.data.Envelope;
import lombok.extern.slf4j.Slf4j;
import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.util.Collector;
import org.apache.kafka.connect.data.Field;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.data.Struct;
import org.apache.kafka.connect.source.SourceRecord;

import java.util.List;

/**
 * @description ...
 * @refrence-doc
 * @gpt-promt
 */
@Slf4j
public class LogEventConfigDeserializationSchema implements DebeziumDeserializationSchema<String> {
    public final static String OPERATION_PROPERTY = "operation";

    /**
     * 對 CDC 訊息的反序列化
     * @note flink cdc mysql 的侷限性: 僅支援 INSERT / UPDATE / DELETE 的 DML 變更, 不支援 新增表欄位 / 刪除表欄位 等 DDL 變更
     * @sample
	 *  注： [1]、[2]、[3]、[4] 中的 server_id、gtids(含 server_uuid) 均是指 主庫(master) 的 serverId、server_uuid
     *  [1] DELETE
     *  sourceRecord.key().toString();//"Struct{id=1}" | 注: 只含有 primary key 的 id 欄位，未含 unique key 的 uuid 欄位
     *  JSON.toJSONString(sourceRecord.sourcePartition())//{"server":"mysql_binlog_source"}
     *  sourceRecord.kafkaPartition();//null
     *  sourceRecord.value() instanceof Struct : true
     *  sourceRecord.timestamp();//null
     *  JSON.toJSONString(sourceRecord.sourceOffset())//{"ts_sec":1724228741,"file":"mysql-bin.217388","pos":362364,"gtids":"13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-439456765","row":1,"server_id":2831608244,"event":2}
     *  sourceRecord.value().toString() = Struct{
     * 	    before=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡狀態變化,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡狀態變化,CardEvent：卡狀態變化時觸發,type=DEFAULT}
     * 	    , source=Struct{version=1.4.1.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1724228741000,db=xxxbackend,table=xxx_logevent,server_id=2831608244,gtid=13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:439456766,file=mysql-bin.217388,pos=362549,row=0,thread=69857216}
     * 	    , op=d
     * 	    , ts_ms=1624228740925
     *  }
     *  [2] INSERT
     *  sourceRecord.key().toString();//"Struct{id=1}"
     *  JSON.toJSONString(sourceRecord.sourceOffset()) // {"ts_sec":1724229853,"file":"mysql-bin.217392","pos":2872,"gtids":"13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-439477253","row":1,"server_id":2831608244,"event":2}
     *  sourceRecord.value().toString() = Struct{
     * 	    after=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡狀態變化,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡狀態變化,CardEvent：卡狀態變化時觸發,type=DEFAULT}
     * 	    , source=Struct{version=1.4.1.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1724229853000,db=xxxbackend,table=xxx_logevent,server_id=2831608244,gtid=13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:439477254,file=mysql-bin.217392,pos=3057,row=0,thread=69858180}
     * 	    , op=c
     * 	    , ts_ms=1624229860659
     *  }
     *  [3] UPDATE
     *  sourceRecord.key().toString();//"Struct{id=1}"
     *  sourceRecord.value().toString() = Struct{
     * 	    before=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡狀態變化,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡狀態變化,CardEvent：卡狀態變化時觸發,type=DEFAULT}
     * 	    , after=Struct{system=ABB,id=1,uuid=4d993277-5857-11ee-bfe3-fa163e42bfe0,eventId=000000001,eventNameEn=CardEvent,eventNameCn=卡狀態變化2,component=XDC,createBy=system,createTime=1595313048000,updateBy=,description=卡狀態變化,CardEvent：卡狀態變化時觸發,type=DEFAULT}
     * 	    , source=Struct{version=1.4.1.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1724230024000,db=xxxbackend,table=xxx_logevent,server_id=2831608244,gtid=13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:439480622,file=mysql-bin.217393,pos=2341458,row=0,thread=69857216}
     * 	    , op=u
     * 	    , ts_ms=1624230024172
     * }
     * [4] 加欄位 : 無 CDC 監聽訊息
     * ALTER TABLE  xxxbackend.xxx_logevent ADD new_column varchar(100) NULL COMMENT '新欄位' AFTER updateTime;
     *
     * [5] 減欄位: 無 CDC 監聽訊息
     * ALTER TABLE xxxbackend.xxx_logevent DROP COLUMN new_column ;
     * @param sourceRecord CDC 監聽訊息
     * @param collector
     * @throws Exception
     */
    @Override
    public void deserialize(SourceRecord sourceRecord, Collector<String> collector) throws Exception {
        //定義JSON物件用於寄存反序列化後的資料
        JSONObject result = new JSONObject();

        //獲取庫名和表名
        String topic = sourceRecord.topic();//eg: "mysql_binlog_source.xxxbackend.xxx_logevent"
        String[] split = topic.split("\\.");
        String database = split[1];
        String table = split[2];

        //讀取變更記錄的主鍵資訊
        if(sourceRecord.key() instanceof Struct){
            //sourceRecord.key().toString();//"Struct{id=1}"
            Struct sourceRecordKeyStruct = (Struct) sourceRecord.key();
            Schema primaryKeyFieldsSchema = sourceRecordKeyStruct.schema();
            //sourceRecordKeyStruct.schema().name();//mysql_binlog_source.xxxbackend.xxx_logevent.Key

            //sourceRecordKeyStruct.schema().field("id");//"id"
            StringBuilder primaryKeyFieldsInfo = new StringBuilder();
            List<Field> primaryKeyFields = primaryKeyFieldsSchema.fields();
            primaryKeyFields.stream().forEach(primaryKeyField -> {
                String fieldName = primaryKeyField.name();
                primaryKeyFieldsInfo.append(fieldName + "=");
                //Object idFieldValue = sourceRecordKeyStruct.get(ID_FIELD);//1 (Long 型)
                Object fieldValue = sourceRecordKeyStruct.get(fieldName);//1 (Long 型)
                primaryKeyFieldsInfo.append(fieldValue + ";");
            });

            log.debug("changed record | primaryKeyFieldsInfo : {}", primaryKeyFieldsInfo.toString());
        }

        //獲取操作型別
        Envelope.Operation operation = Envelope.operationFor(sourceRecord);

        //獲取資料自身 sourceRecord.value() instanceof Struct
        if( !(sourceRecord.value() instanceof Struct)){
            log.error("`sourceRecord#value` is not instance of Struct!sourceRecord.value() : {}", sourceRecord.value().toString());
            return;
        }
        Struct valueStruct = (Struct) sourceRecord.value();
        Struct before = valueStruct.getStruct(Envelope.FieldName.BEFORE);//Envelope.FieldName.BEFORE = "before" (DELETE / UPDATE)
        Struct after = valueStruct.getStruct(Envelope.FieldName.AFTER);//Envelope.FieldName.AFTER = "after" (INSERT / UPDATE)
        //Struct source = valueStruct.getStruct(Envelope.FieldName.SOURCE);//Envelope.FieldName.SOURCE = "source" (INSERT / UPDATE)
        JSONObject value = new JSONObject();

        //讀取變更記錄的所有欄位資訊
        Struct recordStruct = null;
        Schema schema = null;
        if( operation.equals(Envelope.Operation.DELETE) ){//DELETE
            if(before != null){
                recordStruct = before;
            }
        } else {//CREATE / UPDATE
            if(after != null){
                recordStruct = after;
            }
        }
        schema = recordStruct.schema();
        for (Field field : schema.fields()) {
            value.put(field.name(), recordStruct.get(field.name()));
        }

        //將資料放入JSON物件
        result.put("database", database);
        result.put("table", table);
        result.put(OPERATION_PROPERTY, operation.toString().toLowerCase());//"create" / "update" / "delete"
        result.put("value", value);

        //將資料傳輸進來
        collector.collect(result.toJSONString());
    }

    @Override
    public TypeInformation<String> getProducedType() {
        return BasicTypeInfo.STRING_TYPE_INFO;
    }
}

執行效果

FlinkCdcJob 執行日誌

重點日誌行

...
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    connector.class = io.debezium.connector.mysql.MySqlConnector
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.user = bigdata_cdc
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.storage = com.alibaba.ververica.cdc.debezium.internal.FlinkOffsetBackingStore
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.id = 6519
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.name = mysql_binlog_source
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    include.schema.changes = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.port = 3306
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    table.whitelist = xxxbackend.xxx_logevent
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.flush.interval.ms = 9223372036854775807
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    key.converter.schemas.enable = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    tombstones.on.delete = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.serverTimezone = UTC
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.hostname = 10.37.19.206
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.password = ********
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    value.converter.schemas.enable = false
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    name = engine
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.skip.unparseable.ddl = true
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.whitelist = xxxbackend
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.instance.name = 2cbe4018-caa3-4a60-a2c5-709671981391
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    snapshot.mode = schema_only
[2024/10/17 20:06:02.476] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history = com.alibaba.ververica.cdc.debezium.internal.FlinkDatabaseHistory
...
[2024/10/17 20:06:05.097] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :259 execute] Starting snapshot for jdbc:mysql://192.168.10.23:3306/?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL&connectTimeout=30000 with user 'cdc_user' with locking mode 'minimal'
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1032 logRolesForCurrentUser] Snapshot is using user 'cdc_user' with these MySQL grants:
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT RELOAD, PROCESS, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'cdc_user'@'%'
...
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `xxxbackend`.`xxx_logevent` TO 'cdc_user'@'%' //被CDC監聽的表
...
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_purged                                   = 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
...
[2024/10/17 20:06:09.091] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :425 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' for further processing
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logeventkey' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxxbackend.xxx_logeventkey' is filtered out of capturing
...
[2024/10/17 20:06:09.355] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :505 execute] Step 6: generating DROP and CREATE statements to reflect current database schemas:
[2024/10/17 20:06:24.197] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :583 execute] Step 7: releasing global read lock to enable MySQL writes
[2024/10/17 20:06:24.306] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :589 execute] Step 7: blocked writes to MySQL for a total of 00:00:16.592
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :750 execute] Step 8: encountered only schema based snapshot, skipping data snapshot
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :772 execute] Step 9: committing transaction
[2024/10/17 20:06:24.358] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :851 execute] Completed snapshot in 00:00:19.261
[2024/10/17 20:06:24.412] [INFO ] [pool-4-thread-1] [io.debezium.jdbc.JdbcConnection                             :945 lambda$doClose$3] Connection gracefully closed
[2024/10/17 20:06:24.565] [INFO ] [debezium-engine] [io.debezium.connector.mysql.ChainedReader                   :201 startNextReader] Transitioning from the snapshot reader to the binlog reader
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :367 doStart] GTID set purged on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :299 filterGtidSet] Attempting to generate a filtered GTID set
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :300 filterGtidSet] GTID set from previous recorded offset: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :307 filterGtidSet] GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :313 filterGtidSet] Using first available positions for new GTID channels
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :315 filterGtidSet] Relevant GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :326 filterGtidSet] Final merged GTID set to use when connecting to MySQL: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :372 doStart] Registering binlog reader with GTID set: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.235] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
[2024/10/17 20:06:25.324] [INFO ] [blc-192.168.10.23:3306] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
十月 17, 2024 8:06:25 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect
資訊: Connected to 192.168.10.23:3306 at 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 (sid:6519, cid:76144756)
[2024/10/17 20:06:25.838] [INFO ] [blc-192.168.10.23:3306] [io.debezium.connector.mysql.BinlogReader                    :1128 onConnect] Connected to MySQL binlog at 192.168.10.23:3306, starting at GTIDs 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 and binlog file 'mysql-bin.237904', pos=1419258, skipping 0 events plus 0 rows
[2024/10/17 20:06:25.839] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :415 doStart] Waiting for keepalive thread to start
...

完整日誌

[2024/10/17 20:05:53.941] [INFO ] [main] [com.server.bd.logevent.DeviceEventLogAnalysisParseCdcService:278 getParameters] configLocation:local,host:https://config.server.com:443,nacosUsername:nacos,nacosPassword:nacos,nacosNamespace:xxx_team,nacosGroup:data-platform,nacosDataId:com.server.bd.logevent.sync.job
[2024/10/17 20:05:56.468] [INFO ] [main] [com.server.bd.logevent.DeviceEventLogAnalysisParseCdcService:236 initExecutionEnvironment] start to set environment level's parallelism!
[2024/10/17 20:05:56.470] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :43 <clinit>] Success to loading flink global configuration!
[2024/10/17 20:05:56.471] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :74 setParallelism] parallelism.default : null
[2024/10/17 20:05:56.472] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :82 setParallelism] parallelism.default : null | from `parallelism.default` by `jobParameterTool`
[2024/10/17 20:05:56.472] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :90 setParallelism] parallelism.default : null | from `parallelism.default` by `globalConfiguration`
[2024/10/17 20:05:56.473] [WARN ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :119 setParallelism] The `parallelism.default`'s parallelism is empty now , and will be set a default value = 1 by flink framework!
[2024/10/17 20:05:56.949] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :142 randomServerId] Success to generate random server id result! serverId : 6519, interval : 5000, jobName : deviceLogEventCdcSync#deviceLogEventConfig
[2024/10/17 20:05:57.079] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :74 setParallelism] deviceLogEventConfigSource.parallel : 1
[2024/10/17 20:05:57.080] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :117 setParallelism] Success to set parallelism(1)! type: org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator
[2024/10/17 20:05:57.113] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :74 setParallelism] deviceLogEventConfigCdcTransformOperator.parallel : 1
[2024/10/17 20:05:57.113] [INFO ] [main] [com.server.bd.logevent.util.FlinkJobUtils            :117 setParallelism] Success to set parallelism(1)! type: org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator
[2024/10/17 20:05:57.331] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.cpu.cores required for local execution is not set, setting it to the maximal possible value.
[2024/10/17 20:05:57.331] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.task.heap.size required for local execution is not set, setting it to the maximal possible value.
[2024/10/17 20:05:57.331] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.task.off-heap.size required for local execution is not set, setting it to the maximal possible value.
[2024/10/17 20:05:57.333] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.network.min required for local execution is not set, setting it to its default value 64 mb.
[2024/10/17 20:05:57.333] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.network.max required for local execution is not set, setting it to its default value 64 mb.
[2024/10/17 20:05:57.334] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils:262 setConfigOptionToDefaultIfNotSet] The configuration option taskmanager.memory.managed.size required for local execution is not set, setting it to its default value 128 mb.
[2024/10/17 20:05:57.367] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :267 start] Starting Flink Mini Cluster
[2024/10/17 20:05:57.370] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :277 start] Starting Metrics Registry
[2024/10/17 20:05:57.442] [INFO ] [main] [org.apache.flink.runtime.metrics.MetricRegistryImpl         :126 <init>] No metrics reporter configured, no metrics will be exposed/reported.
[2024/10/17 20:05:57.442] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :281 start] Starting RPC Service(s)
[2024/10/17 20:05:57.657] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :265 startLocalActorSystem] Trying to start local actor system
[2024/10/17 20:05:58.210] [INFO ] [flink-akka.actor.default-dispatcher-3] [akka.event.slf4j.Slf4jLogger                                :92 applyOrElse] Slf4jLogger started
[2024/10/17 20:05:58.453] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :298 startActorSystem] Actor system started at akka://flink
[2024/10/17 20:05:58.474] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :265 startLocalActorSystem] Trying to start local actor system
[2024/10/17 20:05:58.494] [INFO ] [flink-metrics-2] [akka.event.slf4j.Slf4jLogger                                :92 applyOrElse] Slf4jLogger started
[2024/10/17 20:05:58.823] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils       :298 startActorSystem] Actor system started at akka://flink-metrics
[2024/10/17 20:05:58.842] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService .
[2024/10/17 20:05:58.877] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :485 createHighAvailabilityServices] Starting high-availability services
[2024/10/17 20:05:58.898] [INFO ] [main] [org.apache.flink.runtime.blob.BlobServer                    :138 <init>] Created BLOB server storage directory C:\Users\123456789\AppData\Local\Temp\blobStore-c217b429-7e94-4467-becc-9f9e74f7bbb4
[2024/10/17 20:05:58.911] [INFO ] [main] [org.apache.flink.runtime.blob.BlobServer                    :213 <init>] Started BLOB server at 0.0.0.0:56685 - max concurrent requests: 50 - max backlog: 1000
[2024/10/17 20:05:58.917] [INFO ] [main] [org.apache.flink.runtime.blob.PermanentBlobCache            :90 <init>] Created BLOB cache storage directory C:\Users\123456789\AppData\Local\Temp\blobStore-f66e9f3c-1ca2-4417-840c-4c193f7339bd
[2024/10/17 20:05:58.920] [INFO ] [main] [org.apache.flink.runtime.blob.TransientBlobCache            :90 <init>] Created BLOB cache storage directory C:\Users\123456789\AppData\Local\Temp\blobStore-294e1ef2-39d5-47a5-bbf2-6435c44e09ad
[2024/10/17 20:05:58.921] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :604 startTaskManagers] Starting 1 TaskManger(s)
[2024/10/17 20:05:58.931] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskManagerRunner     :483 startTaskManager] Starting TaskManager with ResourceID: 73435215-911a-4e13-9b70-a17a8ed54854
[2024/10/17 20:05:58.989] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.TaskManagerServices   :441 checkTempDirs] Temporary file directory 'C:\Users\123456789\AppData\Local\Temp': total 200 GB, usable 19 GB (9.50% usable)
[2024/10/17 20:05:58.996] [INFO ] [main] [org.apache.flink.runtime.io.disk.FileChannelManagerImpl     :98 createFiles] FileChannelManager uses directory C:\Users\123456789\AppData\Local\Temp\flink-io-b02d4a21-b75b-44b3-8a85-2d4d1ef3b344 for spill files.
[2024/10/17 20:05:59.010] [INFO ] [main] [org.apache.flink.runtime.io.disk.FileChannelManagerImpl     :98 createFiles] FileChannelManager uses directory C:\Users\123456789\AppData\Local\Temp\flink-netty-shuffle-f1a44eff-bb62-47aa-a2d6-a5d542d3194f for spill files.
[2024/10/17 20:05:59.065] [INFO ] [main] [org.apache.flink.runtime.io.network.buffer.NetworkBufferPool:145 <init>] Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
[2024/10/17 20:05:59.085] [INFO ] [main] [org.apache.flink.runtime.io.network.NettyShuffleEnvironment :308 start] Starting the network environment and its components.
[2024/10/17 20:05:59.089] [INFO ] [main] [org.apache.flink.runtime.taskexecutor.KvStateService        :92 start] Starting the kvState service and its components.
[2024/10/17 20:05:59.146] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/rpc/taskmanager_0 .
[2024/10/17 20:05:59.166] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:123 start] Start job leader service.
[2024/10/17 20:05:59.169] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.filecache.FileCache                :116 <init>] User file cache uses directory C:\Users\123456789\AppData\Local\Temp\flink-dist-cache-658dc417-c76c-4fa9-a253-b02e6e50e6f1
[2024/10/17 20:05:59.309] [INFO ] [main] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :139 start] Starting rest endpoint.
[2024/10/17 20:05:59.314] [INFO ] [main] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :126 initializeWebSubmissionHandlers] Failed to load web based job submission extension. Probable reason: flink-runtime-web is not in the classpath.
[2024/10/17 20:05:59.666] [WARN ] [main] [org.apache.flink.runtime.webmonitor.WebMonitorUtils         :84 find] Log file environment variable 'log.file' is not set.
[2024/10/17 20:05:59.667] [WARN ] [main] [org.apache.flink.runtime.webmonitor.WebMonitorUtils         :90 find] JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'web.log.path'.
[2024/10/17 20:06:00.922] [INFO ] [main] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :250 start] Rest endpoint listening at localhost:56726
[2024/10/17 20:06:00.923] [INFO ] [main] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender http://localhost:56726
[2024/10/17 20:06:00.926] [INFO ] [mini-cluster-io-thread-1] [org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint  :931 grantLeadership] http://localhost:56726 was granted leadership with leaderSessionID=c20d2b3a-d369-454e-8ddb-7828f7b3b31d
[2024/10/17 20:06:00.926] [INFO ] [mini-cluster-io-thread-1] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader http://localhost:56726 , session=c20d2b3a-d369-454e-8ddb-7828f7b3b31d
[2024/10/17 20:06:00.945] [INFO ] [main] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/rpc/resourcemanager_1 .
[2024/10/17 20:06:00.966] [INFO ] [main] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender LeaderContender: DefaultDispatcherRunner
[2024/10/17 20:06:00.967] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender LeaderContender: StandaloneResourceManager
[2024/10/17 20:06:00.970] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:1192 tryAcceptLeadership] ResourceManager akka://flink/user/rpc/resourcemanager_1 was granted leadership with fencing token 80b927d8e4c836417db316a5d2854fb4
[2024/10/17 20:06:00.971] [INFO ] [main] [org.apache.flink.runtime.minicluster.MiniCluster            :411 start] Flink Mini Cluster started successfully
[2024/10/17 20:06:00.973] [INFO ] [mini-cluster-io-thread-2] [org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess:97 startInternal] Start SessionDispatcherLeaderProcess.
[2024/10/17 20:06:00.974] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl:292 start] Starting the SlotManager.
[2024/10/17 20:06:00.975] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess:117 recoverJobs] Recover all persisted job graphs.
[2024/10/17 20:06:00.975] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess:125 recoverJobs] Successfully recovered 0 persisted job graphs.
[2024/10/17 20:06:00.979] [INFO ] [mini-cluster-io-thread-6] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader akka://flink/user/rpc/resourcemanager_1 , session=7db316a5-d285-4fb4-80b9-27d8e4c83641
[2024/10/17 20:06:00.981] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1260 connectToResourceManager] Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(80b927d8e4c836417db316a5d2854fb4).
[2024/10/17 20:06:00.986] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/rpc/dispatcher_2 .
[2024/10/17 20:06:00.996] [INFO ] [mini-cluster-io-thread-5] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader akka://flink/user/rpc/dispatcher_2 , session=77673aee-ba2f-41b1-9037-49c0f5c6f213
[2024/10/17 20:06:01.006] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :162 lambda$startRegistration$0] Resolved ResourceManager address, beginning registration
[2024/10/17 20:06:01.011] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:960 registerTaskExecutorInternal] Registering TaskManager with ResourceID 73435215-911a-4e13-9b70-a17a8ed54854 (akka://flink/user/rpc/taskmanager_0) at ResourceManager
[2024/10/17 20:06:01.013] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :99 onRegistrationSuccess] Successful registration at resource manager akka://flink/user/rpc/resourcemanager_1 under registration id a2b0110fce3b597f4cb316edbc21671b.
[2024/10/17 20:06:01.015] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.dispatcher.StandaloneDispatcher    :301 submitJob] Received JobGraph submission 622e3901a4a1b260af83d5e7db8e4955 (Flink Streaming Job).
[2024/10/17 20:06:01.016] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.dispatcher.StandaloneDispatcher    :363 internalSubmitJob] Submitting job 622e3901a4a1b260af83d5e7db8e4955 (Flink Streaming Job).
[2024/10/17 20:06:01.037] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:308 updateLeader] Proposing leadership to contender LeaderContender: JobManagerRunnerImpl
[2024/10/17 20:06:01.048] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.rpc.akka.AkkaRpcService            :232 startServer] Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/rpc/jobmanager_3 .
[2024/10/17 20:06:01.056] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :302 <init>] Initializing job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.088] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :84 createInstance] Using restart back off time strategy FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2147483647, backoffTimeMS=1000) for Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.142] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :220 buildGraph] Running initialization on master for job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.143] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :244 buildGraph] Successfully ran initialization on master in 0 ms.
[2024/10/17 20:06:01.166] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology:251 computePipelinedRegions] Built 1 pipelined regions in 1 ms
[2024/10/17 20:06:01.183] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.197] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.197] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.198] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.286] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1526 restoreLatestCheckpointedStateInternal] No checkpoint found during restore.
[2024/10/17 20:06:01.291] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobMaster                :164 <init>] Using failover strategy org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy@31aee470 for Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955).
[2024/10/17 20:06:01.305] [INFO ] [mini-cluster-io-thread-12] [org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl     :344 startJobMaster] JobManager runner for job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955) was granted leadership with session id 1df6e255-a4cb-437c-9892-6acded781b90 at akka://flink/user/rpc/jobmanager_3.
[2024/10/17 20:06:01.308] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :885 startJobExecution] Starting execution of job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955) under job master id 98926acded781b901df6e255a4cb437c.
[2024/10/17 20:06:01.310] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :201 startSchedulingInternal] Starting scheduling with scheduling strategy [org.apache.flink.runtime.scheduler.strategy.PipelinedRegionSchedulingStrategy]
[2024/10/17 20:06:01.312] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1317 transitionState] Job Flink Streaming Job (622e3901a4a1b260af83d5e7db8e4955) switched from state CREATED to RUNNING.
[2024/10/17 20:06:01.322] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (c5dea366ceaa9ba8176799c3534da6a7) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.324] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (1/8) (0588aafdab285425edaf8abce6d90145) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.325] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (2/8) (74b45625c6416a320214d5fd299f21e8) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.325] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (3/8) (de74185cedae65481bed4b8f1a7db614) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.325] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (4/8) (6cf840073b72118ac2f26d9ce42f6fad) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (5/8) (28a2586f6ed9d382b188cc14553361cd) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (6/8) (4e5e280b0383030fe2aef686320cd6d7) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (7/8) (6195cf02c366819eb9cf28517361f3cc) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.326] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (8/8) (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from CREATED to SCHEDULED.
[2024/10/17 20:06:01.343] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{0335dcf06a11a51b68db6af1ecb74f40}]
[2024/10/17 20:06:01.345] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{3bec2ac90a5811eb3dfb1a3980a56446}]
[2024/10/17 20:06:01.346] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{d727fc8e9e25e8aa76403dc498fa71ad}]
[2024/10/17 20:06:01.346] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{f577fd55bd0b5947407750bfd914fb32}]
[2024/10/17 20:06:01.347] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{ac7fee3b01795eb434072a3b44ace96a}]
[2024/10/17 20:06:01.347] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{8aa216a8c973a3dbd952b108a73bf9d2}]
[2024/10/17 20:06:01.347] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{1d98f4d7cc4743d2e12769e9c2f49c24}]
[2024/10/17 20:06:01.348] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :415 stashRequestWaitingForResourceManager] Cannot serve slot request, no ResourceManager connected. Adding as pending request [SlotRequestId{9ddcf29611f0d94c040443ec21b8d0eb}]
[2024/10/17 20:06:01.355] [INFO ] [jobmanager-future-thread-1] [org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService:256 confirmLeader] Received confirmation of leadership for leader akka://flink/user/rpc/jobmanager_3 , session=1df6e255-a4cb-437c-9892-6acded781b90
[2024/10/17 20:06:01.356] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :1161 connectToResourceManager] Connecting to ResourceManager akka://flink/user/rpc/resourcemanager_1(80b927d8e4c836417db316a5d2854fb4)
[2024/10/17 20:06:01.359] [INFO ] [flink-akka.actor.default-dispatcher-4] [org.apache.flink.runtime.jobmaster.JobMaster                :162 lambda$startRegistration$0] Resolved ResourceManager address, beginning registration
[2024/10/17 20:06:01.362] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:351 registerJobManager] Registering job manager 98926acded781b901df6e255a4cb437c@akka://flink/user/rpc/jobmanager_3 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.369] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:889 registerJobMasterInternal] Registered job manager 98926acded781b901df6e255a4cb437c@akka://flink/user/rpc/jobmanager_3 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.372] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.JobMaster                :1185 establishResourceManagerConnection] JobManager successfully registered at ResourceManager, leader id: 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.377] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{0335dcf06a11a51b68db6af1ecb74f40}] and profile ResourceProfile{UNKNOWN} with allocation id 993d83b02db353b2df06ed45de6b9a05 from resource manager.
[2024/10/17 20:06:01.378] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.378] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{3bec2ac90a5811eb3dfb1a3980a56446}] and profile ResourceProfile{UNKNOWN} with allocation id fb91206510a29c9b6cf9244eccdda210 from resource manager.
[2024/10/17 20:06:01.379] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{d727fc8e9e25e8aa76403dc498fa71ad}] and profile ResourceProfile{UNKNOWN} with allocation id 3d9a89694270c734dda3de600817c78d from resource manager.
[2024/10/17 20:06:01.379] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{f577fd55bd0b5947407750bfd914fb32}] and profile ResourceProfile{UNKNOWN} with allocation id 61aa5e7eea60f2416987abcd0ae8ed04 from resource manager.
[2024/10/17 20:06:01.380] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{ac7fee3b01795eb434072a3b44ace96a}] and profile ResourceProfile{UNKNOWN} with allocation id 546c8bf27026d5c34b2cfcee7202f526 from resource manager.
[2024/10/17 20:06:01.380] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{8aa216a8c973a3dbd952b108a73bf9d2}] and profile ResourceProfile{UNKNOWN} with allocation id 8de01f21100ad721f49774464b111421 from resource manager.
[2024/10/17 20:06:01.380] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{1d98f4d7cc4743d2e12769e9c2f49c24}] and profile ResourceProfile{UNKNOWN} with allocation id 9c595f3db9beaca44199b6e5a3d16235 from resource manager.
[2024/10/17 20:06:01.381] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl    :363 requestSlotFromResourceManager] Requesting new slot [SlotRequestId{9ddcf29611f0d94c040443ec21b8d0eb}] and profile ResourceProfile{UNKNOWN} with allocation id 84862e51c914442836eb5f12859c8387 from resource manager.
[2024/10/17 20:06:01.383] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 993d83b02db353b2df06ed45de6b9a05 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.385] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.386] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.386] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.387] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.388] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.388] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.389] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.resourcemanager.StandaloneResourceManager:522 requestSlot] Request slot with profile ResourceProfile{UNKNOWN} for job 622e3901a4a1b260af83d5e7db8e4955 with allocation id 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.396] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.399] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:188 addJob] Add job 622e3901a4a1b260af83d5e7db8e4955 for job leader monitoring.
[2024/10/17 20:06:01.406] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request fb91206510a29c9b6cf9244eccdda210 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.407] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.410] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 3d9a89694270c734dda3de600817c78d for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.411] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.411] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 61aa5e7eea60f2416987abcd0ae8ed04 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.412] [INFO ] [mini-cluster-io-thread-18] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:346 openRpcConnectionTo] Try to register at job manager akka://flink/user/rpc/jobmanager_3 with leader id 1df6e255-a4cb-437c-9892-6acded781b90.
[2024/10/17 20:06:01.412] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.412] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 546c8bf27026d5c34b2cfcee7202f526 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.413] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.413] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 8de01f21100ad721f49774464b111421 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.413] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.414] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 9c595f3db9beaca44199b6e5a3d16235 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.414] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.414] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1008 requestSlot] Receive slot request 84862e51c914442836eb5f12859c8387 for job 622e3901a4a1b260af83d5e7db8e4955 from resource manager with leader id 80b927d8e4c836417db316a5d2854fb4.
[2024/10/17 20:06:01.415] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1085 allocateSlot] Allocated slot for 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.428] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:162 lambda$startRegistration$0] Resolved JobManager address, beginning registration
[2024/10/17 20:06:01.448] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService:413 lambda$onRegistrationSuccess$0] Successful registration at job manager akka://flink/user/rpc/jobmanager_3 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.449] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1543 establishJobManagerConnection] Establish JobManager connection for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.465] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :1426 internalOfferSlotsToJobManager] Offer reserved slots to the leader of job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:06:01.498] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (c5dea366ceaa9ba8176799c3534da6a7) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.499] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (attempt #0) with attempt id c5dea366ceaa9ba8176799c3534da6a7 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 3d9a89694270c734dda3de600817c78d
[2024/10/17 20:06:01.515] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (1/8) (0588aafdab285425edaf8abce6d90145) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.515] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.515] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (1/8) (attempt #0) with attempt id 0588aafdab285425edaf8abce6d90145 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 3d9a89694270c734dda3de600817c78d
[2024/10/17 20:06:01.518] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (2/8) (74b45625c6416a320214d5fd299f21e8) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.519] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (2/8) (attempt #0) with attempt id 74b45625c6416a320214d5fd299f21e8 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id fb91206510a29c9b6cf9244eccdda210
[2024/10/17 20:06:01.520] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (3/8) (de74185cedae65481bed4b8f1a7db614) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.521] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (3/8) (attempt #0) with attempt id de74185cedae65481bed4b8f1a7db614 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 993d83b02db353b2df06ed45de6b9a05
[2024/10/17 20:06:01.523] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (4/8) (6cf840073b72118ac2f26d9ce42f6fad) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.524] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (4/8) (attempt #0) with attempt id 6cf840073b72118ac2f26d9ce42f6fad to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 546c8bf27026d5c34b2cfcee7202f526
[2024/10/17 20:06:01.532] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (5/8) (28a2586f6ed9d382b188cc14553361cd) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.532] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (5/8) (attempt #0) with attempt id 28a2586f6ed9d382b188cc14553361cd to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 8de01f21100ad721f49774464b111421
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (6/8) (4e5e280b0383030fe2aef686320cd6d7) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (6/8) (attempt #0) with attempt id 4e5e280b0383030fe2aef686320cd6d7 to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 9c595f3db9beaca44199b6e5a3d16235
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (7/8) (6195cf02c366819eb9cf28517361f3cc) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.533] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (7/8) (attempt #0) with attempt id 6195cf02c366819eb9cf28517361f3cc to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 61aa5e7eea60f2416987abcd0ae8ed04
[2024/10/17 20:06:01.535] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (8/8) (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from SCHEDULED to DEPLOYING.
[2024/10/17 20:06:01.538] [INFO ] [flink-akka.actor.default-dispatcher-3] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :808 deploy] Deploying Sink: Print to Std. Out (8/8) (attempt #0) with attempt id 8f6ac9f3c34aff98c8bd1300e310bcdc to 73435215-911a-4e13-9b70-a17a8ed54854 @ localhost.sangfor.com.cn (dataPort=-1) with allocation id 84862e51c914442836eb5f12859c8387
[2024/10/17 20:06:01.647] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7), deploy into slot with allocation id 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.649] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.652] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.656] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) [DEPLOYING].
[2024/10/17 20:06:01.658] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) [DEPLOYING].
[2024/10/17 20:06:01.676] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145), deploy into slot with allocation id 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.677] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.677] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.677] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) [DEPLOYING].
[2024/10/17 20:06:01.679] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) [DEPLOYING].
[2024/10/17 20:06:01.691] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8), deploy into slot with allocation id fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.692] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.695] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614), deploy into slot with allocation id 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.695] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.698] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad), deploy into slot with allocation id 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.699] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.702] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.702] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.702] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.703] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.703] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.703] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.704] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.704] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.705] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd), deploy into slot with allocation id 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.706] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.706] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.706] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.706] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) [DEPLOYING].
[2024/10/17 20:06:01.707] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) [DEPLOYING].
[2024/10/17 20:06:01.708] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) [DEPLOYING].
[2024/10/17 20:06:01.709] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.709] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) [DEPLOYING].
[2024/10/17 20:06:01.710] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.711] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) [DEPLOYING].
[2024/10/17 20:06:01.711] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.711] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.712] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.712] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) [DEPLOYING].
[2024/10/17 20:06:01.712] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.710] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) [DEPLOYING].
[2024/10/17 20:06:01.710] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7), deploy into slot with allocation id 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.713] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.713] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) [DEPLOYING].
[2024/10/17 20:06:01.714] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.717] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.717] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) [DEPLOYING].
[2024/10/17 20:06:01.718] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.718] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.718] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.719] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.719] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) [DEPLOYING].
[2024/10/17 20:06:01.720] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.720] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.721] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.721] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.723] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc), deploy into slot with allocation id 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.723] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) [DEPLOYING].
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.724] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 993d83b02db353b2df06ed45de6b9a05.
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.724] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 61aa5e7eea60f2416987abcd0ae8ed04.
[2024/10/17 20:06:01.724] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.725] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.725] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 9c595f3db9beaca44199b6e5a3d16235.
[2024/10/17 20:06:01.725] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot fb91206510a29c9b6cf9244eccdda210.
[2024/10/17 20:06:01.725] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) [DEPLOYING].
[2024/10/17 20:06:01.725] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.726] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 8de01f21100ad721f49774464b111421.
[2024/10/17 20:06:01.727] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 546c8bf27026d5c34b2cfcee7202f526.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.727] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 3d9a89694270c734dda3de600817c78d.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.727] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl:368 markExistingSlotActive] Activate slot 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.727] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.730] [INFO ] [flink-akka.actor.default-dispatcher-2] [org.apache.flink.runtime.taskexecutor.TaskExecutor          :703 submitTask] Received task Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc), deploy into slot with allocation id 84862e51c914442836eb5f12859c8387.
[2024/10/17 20:06:01.737] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from CREATED to DEPLOYING.
[2024/10/17 20:06:01.737] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :633 doRun] Loading JAR files for task Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) [DEPLOYING].
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (5/8)#0 (28a2586f6ed9d382b188cc14553361cd) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (2/8)#0 (74b45625c6416a320214d5fd299f21e8) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (1/8)#0 (0588aafdab285425edaf8abce6d90145) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (7/8)#0 (6195cf02c366819eb9cf28517361f3cc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (3/8)#0 (de74185cedae65481bed4b8f1a7db614) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (4/8)#0 (6cf840073b72118ac2f26d9ce42f6fad) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.737] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0 (c5dea366ceaa9ba8176799c3534da6a7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.740] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (2/8) (74b45625c6416a320214d5fd299f21e8) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.742] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (1/8) (0588aafdab285425edaf8abce6d90145) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (6/8)#0 (4e5e280b0383030fe2aef686320cd6d7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.738] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :660 doRun] Registering task at network: Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) [DEPLOYING].
[2024/10/17 20:06:01.745] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (4/8) (6cf840073b72118ac2f26d9ce42f6fad) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.745] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :211 fromApplicationOrConfigOrDefault] Using job/cluster config to configure application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=-1, writeBatchSize=-1}
[2024/10/17 20:06:01.746] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1) (c5dea366ceaa9ba8176799c3534da6a7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.746] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:365 <init>] Using predefined options: DEFAULT.
[2024/10/17 20:06:01.748] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.contrib.streaming.state.RocksDBStateBackend:634 configureOptionsFactory] Using default options factory: DefaultConfigurableOptionsFactory{configuredOptions={}}.
[2024/10/17 20:06:01.748] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (5/8) (28a2586f6ed9d382b188cc14553361cd) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.749] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.streaming.runtime.tasks.StreamTask         :224 fromApplicationOrConfigOrDefault] Using application-defined state backend: RocksDBStateBackend{checkpointStreamBackend=File State Backend (checkpoints: 'file:/E:/tmp_data/checkpoint/deviceLogEventAnalysisParse', savepoints: 'null', asynchronous: TRUE, fileStateThreshold: 20480), localRocksDbDirectories=null, enableIncrementalCheckpointing=TRUE, numberOfTransferThreads=1, writeBatchSize=2097152}
[2024/10/17 20:06:01.749] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (3/8) (de74185cedae65481bed4b8f1a7db614) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.749] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (7/8) (6195cf02c366819eb9cf28517361f3cc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.750] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (6/8) (4e5e280b0383030fe2aef686320cd6d7) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.750] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.taskmanager.Task                   :1029 transitionState] Sink: Print to Std. Out (8/8)#0 (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.751] [INFO ] [flink-akka.actor.default-dispatcher-6] [org.apache.flink.runtime.executiongraph.ExecutionGraph      :1795 transitionState] Sink: Print to Std. Out (8/8) (8f6ac9f3c34aff98c8bd1300e310bcdc) switched from DEPLOYING to RUNNING.
[2024/10/17 20:06:01.890] [INFO ] [Sink: Print to Std. Out (5/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (8/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.890] [INFO ] [Sink: Print to Std. Out (4/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.890] [INFO ] [Sink: Print to Std. Out (7/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (1/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (6/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (2/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.891] [INFO ] [Sink: Print to Std. Out (3/8)#0] [org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate:300 convertRecoveredInputChannels] Converting recovered input channels (1 channels)
[2024/10/17 20:06:01.895] [INFO ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :201 initializeState] Consumer subtask 0 has no restore state.
[2024/10/17 20:06:02.361] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.json.JsonConverterConfig           :347 logAll] JsonConverterConfig values: 
	converter.type = key
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = true

[2024/10/17 20:06:02.368] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.json.JsonConverterConfig           :347 logAll] JsonConverterConfig values: 
	converter.type = value
	decimal.format = BASE64
	schemas.cache.size = 1000
	schemas.enable = false

[2024/10/17 20:06:02.383] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [io.debezium.embedded.EmbeddedEngine$EmbeddedConfig          :347 logAll] EmbeddedConfig values: 
	access.control.allow.methods = 
	access.control.allow.origin = 
	admin.listeners = null
	bootstrap.servers = [localhost:9092]
	client.dns.lookup = use_all_dns_ips
	config.providers = []
	connector.client.config.override.policy = None
	header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
	internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
	internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
	key.converter = class org.apache.kafka.connect.json.JsonConverter
	listeners = null
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	offset.flush.interval.ms = 9223372036854775807
	offset.flush.timeout.ms = 5000
	offset.storage.file.filename = 
	offset.storage.partitions = null
	offset.storage.replication.factor = null
	offset.storage.topic = 
	plugin.path = null
	response.http.headers.config = 
	rest.advertised.host.name = null
	rest.advertised.listener = null
	rest.advertised.port = null
	rest.extension.classes = []
	rest.host.name = null
	rest.port = 8083
	ssl.cipher.suites = null
	ssl.client.auth = none
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	task.shutdown.graceful.timeout.ms = 5000
	topic.creation.enable = true
	topic.tracking.allow.reset = true
	topic.tracking.enable = true
	value.converter = class org.apache.kafka.connect.json.JsonConverter

[2024/10/17 20:06:02.384] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.runtime.WorkerConfig               :389 logDeprecatedProperty] Worker configuration property 'internal.key.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2024/10/17 20:06:02.384] [INFO ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.runtime.WorkerConfig               :389 logDeprecatedProperty] Worker configuration property 'internal.value.converter' is deprecated and may be removed in an upcoming release. The specified value 'org.apache.kafka.connect.json.JsonConverter' matches the default, so this property can be safely removed from the worker configuration.
[2024/10/17 20:06:02.385] [WARN ] [Legacy Source Thread - Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [org.apache.kafka.connect.runtime.WorkerConfig               :420 logPluginPathConfigProviderWarning] Variables cannot be used in the 'plugin.path' property, since the property is used by plugin scanning before the config providers that replace the variables are initialized. The raw value 'null' was used for plugin scanning, as opposed to the transformed value 'null', and this may cause unexpected results.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.blacklist" is deprecated and will be removed in future versions. Please use "table.exclude.list" instead.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:02.469] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.blacklist" is deprecated and will be removed in future versions. Please use "table.exclude.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.whitelist" is deprecated and will be removed in future versions. Please use "database.include.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.blacklist" is deprecated and will be removed in future versions. Please use "database.exclude.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.whitelist" is deprecated and will be removed in future versions. Please use "database.include.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.blacklist" is deprecated and will be removed in future versions. Please use "database.exclude.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.whitelist" is deprecated and will be removed in future versions. Please use "column.include.list" instead.
[2024/10/17 20:06:02.470] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.whitelist" is deprecated and will be removed in future versions. Please use "column.include.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.whitelist" is deprecated and will be removed in future versions. Please use "column.include.list" instead.
[2024/10/17 20:06:02.471] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:02.471] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :100 start] Starting MySqlConnectorTask with configuration:
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    connector.class = io.debezium.connector.mysql.MySqlConnector
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.user = cdc_user
[2024/10/17 20:06:02.473] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.storage = com.alibaba.ververica.cdc.debezium.internal.FlinkOffsetBackingStore
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.id = 6519
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.server.name = mysql_binlog_source
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    include.schema.changes = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.port = 3306
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    table.whitelist = xxxbackend.xxx_logevent
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    offset.flush.interval.ms = 9223372036854775807
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    key.converter.schemas.enable = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    tombstones.on.delete = false
[2024/10/17 20:06:02.474] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.serverTimezone = UTC
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.hostname = 192.168.10.23
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.password = ********
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    value.converter.schemas.enable = false
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    name = engine
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.skip.unparseable.ddl = true
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.whitelist = xxxbackend
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history.instance.name = 2cbe4018-caa3-4a60-a2c5-709671981391
[2024/10/17 20:06:02.475] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    snapshot.mode = schema_only
[2024/10/17 20:06:02.476] [INFO ] [debezium-engine] [io.debezium.connector.common.BaseSourceTask                 :102 lambda$start$0]    database.history = com.alibaba.ververica.cdc.debezium.internal.FlinkDatabaseHistory
[2024/10/17 20:06:02.528] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.whitelist" is deprecated and will be removed in future versions. Please use "database.include.list" instead.
[2024/10/17 20:06:02.528] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "database.blacklist" is deprecated and will be removed in future versions. Please use "database.exclude.list" instead.
[2024/10/17 20:06:02.528] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:02.529] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.blacklist" is deprecated and will be removed in future versions. Please use "table.exclude.list" instead.
[2024/10/17 20:06:02.534] [WARN ] [debezium-engine] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "column.blacklist" is deprecated and will be removed in future versions. Please use "column.exclude.list" instead.
[2024/10/17 20:06:04.506] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlConnectorTask              :169 start] Found no existing offset, so preparing to perform a snapshot
[2024/10/17 20:06:05.086] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :270 threadFactory] Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = binlog-client
[2024/10/17 20:06:05.095] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :270 threadFactory] Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = snapshot
[2024/10/17 20:06:05.096] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-snapshot
[2024/10/17 20:06:05.097] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :259 execute] Starting snapshot for jdbc:mysql://192.168.10.23:3306/?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=CONVERT_TO_NULL&connectTimeout=30000 with user 'cdc_user' with locking mode 'minimal'
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1032 logRolesForCurrentUser] Snapshot is using user 'cdc_user' with these MySQL grants:
[2024/10/17 20:06:05.388] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT RELOAD, PROCESS, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT SELECT, EXECUTE ON `sys`.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT SELECT ON `performance_schema`.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT SELECT, LOCK TABLES, SHOW VIEW ON `xxxbackend`.* TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `xxxbackend`.`xxx_logevent` TO 'cdc_user'@'%'
[2024/10/17 20:06:05.389] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `master_devicecenter`.`tc_dimdevice` TO 'cdc_user'@'%'
[2024/10/17 20:06:05.390] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1033 lambda$logRolesForCurrentUser$21] 	GRANT ALL PRIVILEGES ON `xxxbackend`.`xxx_eventrecords` TO 'cdc_user'@'%'
[2024/10/17 20:06:05.390] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1003 logServerInformation] MySQL server variables related to change data capture:
[2024/10/17 20:06:05.457] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_cache_size                             = 32768                                        
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_checksum                               = CRC32                                        
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_direct_non_transactional_updates       = OFF                                          
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_error_action                           = ABORT_SERVER                                 
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_format                                 = ROW                                          
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_group_commit_sync_delay                = 0                                            
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_group_commit_sync_no_delay_count       = 0                                            
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_gtid_simple_recovery                   = ON                                           
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_max_flush_queue_time                   = 0                                            
[2024/10/17 20:06:05.458] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_order_commits                          = ON                                           
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_row_image                              = FULL                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_rows_query_log_events                  = OFF                                          
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_stmt_cache_size                        = 32768                                        
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_transaction_dependency_history_size    = 25000                                        
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	binlog_transaction_dependency_tracking        = COMMIT_ORDER                                 
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_client                          = utf8mb4                                      
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_connection                      = utf8mb4                                      
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_database                        = utf8                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_filesystem                      = binary                                       
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_results                         = utf8mb4                                      
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_server                          = utf8                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_set_system                          = utf8                                         
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	character_sets_dir                            = /usr/local/mysql-5.7.38-220701-linux-x86_64/share/charsets/
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	collation_connection                          = utf8mb4_general_ci                           
[2024/10/17 20:06:05.459] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	collation_database                            = utf8_general_ci                              
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	collation_server                              = utf8_general_ci                              
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	enforce_gtid_consistency                      = ON                                           
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_executed_compression_period              = 1000                                         
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_mode                                     = ON                                           
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_next                                     = AUTOMATIC                                    
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_owned                                    =                                              
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	gtid_purged                                   = 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	have_backup_safe_binlog_info                  = YES                                          
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	innodb_api_enable_binlog                      = OFF                                          
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	innodb_locks_unsafe_for_binlog                = OFF                                          
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	innodb_version                                = 5.7.38                                       
[2024/10/17 20:06:05.460] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	log_statements_unsafe_for_binlog              = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	max_binlog_cache_size                         = 18446744073709547520                         
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	max_binlog_size                               = 536870912                                    
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	max_binlog_stmt_cache_size                    = 18446744073709547520                         
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	protocol_version                              = 10                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_aor_op_write_binlog_swith_on              = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_binlog_hint                               = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_change_binlog_row_image_on_existent_connections = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_flush_op_write_binlog_swith_on            = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_secondary_binlog_directory                =                                              
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_switch_binlog_during_flush_enabled        = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rds_switch_binlog_during_flush_threshold      = 536870912                                    
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	rotate_in_flush_binlog_stage                  = ON                                           
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	session_track_gtids                           = OFF                                          
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	slave_type_conversions                        =                                              
[2024/10/17 20:06:05.461] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	sync_binlog                                   = 1                                            
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	system_time_zone                              = +08                                          
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	time_zone                                     = SYSTEM                                       
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	tls_version                                   = TLSv1.2                                      
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	tx_isolation                                  = REPEATABLE-READ                              
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	tx_read_only                                  = OFF                                          
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version                                       = 5.7.38-220701-log                            
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version_comment                               = MySQL Community Server - (GPL)               
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version_compile_machine                       = x86_64                                       
[2024/10/17 20:06:05.462] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :1006 lambda$logServerInformation$19] 	version_compile_os                            = Linux                                        
[2024/10/17 20:06:05.463] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :293 execute] Step 0: disabling autocommit, enabling repeatable read transactions, and setting lock wait timeout to 10
[2024/10/17 20:06:06.277] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :328 execute] Step 1: flush and obtain global read lock to prevent writes to database
[2024/10/17 20:06:08.177] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :353 execute] Step 2: start transaction with consistent snapshot
[2024/10/17 20:06:08.240] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :917 readBinlogPosition] Step 3: read binlog position of MySQL primary server
[2024/10/17 20:06:08.330] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :929 lambda$readBinlogPosition$18] 	 using binlog 'mysql-bin.237904' at position '1419258' and gtid '13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292'
[2024/10/17 20:06:08.332] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :378 execute] Step 4: read list of available databases
[2024/10/17 20:06:08.443] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :386 execute] 	 list of available databases is: [information_schema, master_carbonmanagementplatform, master_dtpcore, master_otacenter, master_sota, master_terminalcenter, xxxbackend, performance_schema, sys]
[2024/10/17 20:06:08.444] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :397 execute] Step 5: read list of available tables in each database
[2024/10/17 20:06:08.628] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxx_aaa.flyway_schema_history' among known tables
[2024/10/17 20:06:08.630] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxx_aaa.flyway_schema_history' is filtered out of capturing
[2024/10/17 20:06:08.631] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxx_carbonmanagementplatform.nea_demofile' among known tables
[2024/10/17 20:06:08.631] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxx_aaa.nea_demofile' is filtered out of capturing
...
[2024/10/17 20:06:09.091] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :425 lambda$execute$4] 	 including 'xxxbackend.xxx_logevent' for further processing
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'xxxbackend.xxx_logeventkey' among known tables
[2024/10/17 20:06:09.092] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'xxxbackend.xxx_logeventkey' is filtered out of capturing
...
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'performance_schema.user_variables_by_thread' among known tables
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'performance_schema.user_variables_by_thread' is filtered out of capturing
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'performance_schema.users' among known tables
[2024/10/17 20:06:09.265] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'performance_schema.users' is filtered out of capturing
[2024/10/17 20:06:09.266] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'performance_schema.variables_by_thread' among known tables
[2024/10/17 20:06:09.266] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'performance_schema.variables_by_thread' is filtered out of capturing
[2024/10/17 20:06:09.331] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :418 lambda$execute$4] 	 including 'sys.sys_config' among known tables
[2024/10/17 20:06:09.333] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :428 lambda$execute$4] 	 'sys.sys_config' is filtered out of capturing
[2024/10/17 20:06:09.335] [WARN ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.config.Configuration                            :2135 lambda$getFallbackStringProperty$34] Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
[2024/10/17 20:06:09.353] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :459 execute] 	snapshot continuing with database(s): [xxxbackend]
[2024/10/17 20:06:09.355] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :505 execute] Step 6: generating DROP and CREATE statements to reflect current database schemas:
[2024/10/17 20:06:24.197] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :583 execute] Step 7: releasing global read lock to enable MySQL writes
[2024/10/17 20:06:24.306] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :589 execute] Step 7: blocked writes to MySQL for a total of 00:00:16.592
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :750 execute] Step 8: encountered only schema based snapshot, skipping data snapshot
[2024/10/17 20:06:24.307] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :772 execute] Step 9: committing transaction
[2024/10/17 20:06:24.358] [INFO ] [debezium-mysqlconnector-mysql_binlog_source-snapshot] [io.debezium.connector.mysql.SnapshotReader                  :851 execute] Completed snapshot in 00:00:19.261
[2024/10/17 20:06:24.412] [INFO ] [pool-4-thread-1] [io.debezium.jdbc.JdbcConnection                             :945 lambda$doClose$3] Connection gracefully closed
[2024/10/17 20:06:24.565] [INFO ] [debezium-engine] [io.debezium.connector.mysql.ChainedReader                   :201 startNextReader] Transitioning from the snapshot reader to the binlog reader
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :367 doStart] GTID set purged on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-500796901
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :299 filterGtidSet] Attempting to generate a filtered GTID set
[2024/10/17 20:06:25.227] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :300 filterGtidSet] GTID set from previous recorded offset: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :307 filterGtidSet] GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :313 filterGtidSet] Using first available positions for new GTID channels
[2024/10/17 20:06:25.228] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :315 filterGtidSet] Relevant GTID set available on server: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292393
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.MySqlTaskContext                :326 filterGtidSet] Final merged GTID set to use when connecting to MySQL: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.232] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :372 doStart] Registering binlog reader with GTID set: 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292
[2024/10/17 20:06:25.235] [INFO ] [debezium-engine] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
[2024/10/17 20:06:25.324] [INFO ] [blc-192.168.10.23:3306] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
十月 17, 2024 8:06:25 下午 com.github.shyiko.mysql.binlog.BinaryLogClient connect
資訊: Connected to 192.168.10.23:3306 at 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 (sid:6519, cid:76144756)
[2024/10/17 20:06:25.838] [INFO ] [blc-192.168.10.23:3306] [io.debezium.connector.mysql.BinlogReader                    :1128 onConnect] Connected to MySQL binlog at 192.168.10.23:3306, starting at GTIDs 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0:1-508292292 and binlog file 'mysql-bin.237904', pos=1419258, skipping 0 events plus 0 rows
[2024/10/17 20:06:25.839] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :415 doStart] Waiting for keepalive thread to start
[2024/10/17 20:06:25.841] [INFO ] [blc-192.168.10.23:3306] [io.debezium.util.Threads                                    :287 newThread] Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
[2024/10/17 20:06:25.974] [INFO ] [debezium-engine] [io.debezium.connector.mysql.BinlogReader                    :422 doStart] Keepalive thread is running
[2024/10/17 20:07:01.347] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 1 (type=CHECKPOINT) @ 1729166821329 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:07:01.475] [INFO ] [jobmanager-future-thread-3] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 1 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 141 ms).
[2024/10/17 20:07:01.477] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 1
[2024/10/17 20:08:01.481] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 2 (type=CHECKPOINT) @ 1729166881480 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:08:01.525] [INFO ] [jobmanager-future-thread-7] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 2 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 29 ms).
[2024/10/17 20:08:01.526] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 2
[2024/10/17 20:09:00.968] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 3 (type=CHECKPOINT) @ 1729166940961 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:09:01.563] [INFO ] [jobmanager-future-thread-5] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 3 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 595 ms).
[2024/10/17 20:09:01.563] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 3
[2024/10/17 20:10:01.568] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 4 (type=CHECKPOINT) @ 1729167001565 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:10:01.622] [INFO ] [jobmanager-future-thread-2] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 4 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 52 ms).
[2024/10/17 20:10:01.622] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 4
[2024/10/17 20:11:01.636] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 5 (type=CHECKPOINT) @ 1729167061634 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:11:01.662] [INFO ] [jobmanager-future-thread-8] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 5 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 24 ms).
[2024/10/17 20:11:01.663] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 5
[2024/10/17 20:12:01.666] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 6 (type=CHECKPOINT) @ 1729167121665 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:12:01.687] [INFO ] [jobmanager-future-thread-4] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 6 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 18 ms).
[2024/10/17 20:12:01.688] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 6
[2024/10/17 20:13:01.745] [INFO ] [Checkpoint Timer] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :756 createPendingCheckpoint] Triggering checkpoint 7 (type=CHECKPOINT) @ 1729167181739 for job 622e3901a4a1b260af83d5e7db8e4955.
[2024/10/17 20:13:01.907] [INFO ] [jobmanager-future-thread-8] [org.apache.flink.runtime.checkpoint.CheckpointCoordinator   :1245 completePendingCheckpoint] Completed checkpoint 7 for job 622e3901a4a1b260af83d5e7db8e4955 (333870 bytes in 148 ms).
[2024/10/17 20:13:01.907] [WARN ] [Source: device-log-event-config-data-stream-source -> device-log-event-config-cdc-transform (1/1)#0] [com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction   :422 notifyCheckpointComplete] Consumer subtask 0 received confirmation for unknown checkpoint id 7
...

日誌分析：重點日誌行，所提供的關鍵資訊

依賴關係：

Flink Cdc MySQL Connector/com.alibaba.ververica:flink-connector-mysql-cdc

--> debezium

--> com.github.shyiko:mysql-binlog-connector-java

vvc:flink-connector-mysql-cdc 的關鍵元件類

com.alibaba.ververica.cdc.debezium.DebeziumDeserializationSchema

debezium 的關鍵元件類

io.debezium.data.Envelope

org.apache.kafka.connect.data.Field

org.apache.kafka.connect.data.Schema

org.apache.kafka.connect.data.Struct

org.apache.kafka.connect.source.SourceRecord

shyiko:mysql-binlog-connector-java 的關鍵元件類

com.github.shyiko.mysql.binlog.BinaryLogClient

日誌中能提供的關鍵資訊:

主庫(master)的 server_uuid = 13c05ae4-3d6f-11ed-a8ab-fa163e42bfe0

sid = FlinkCdcJob 的 Salve Server Id = 6519

cid = FlinkCdcJob 的 MySQL 連線物件的 connection id = FlinkCdcJob 的 MySQL 連線物件的 session id=show processlist中的processlist.id=select connection_id() as sessionId=76144756`

Y 推薦資源

第三方文件
Flink CDC

https://nightlies.apache.org/flink

https://nightlies.apache.org/flink/flink-docs-master/

https://nightlies.apache.org/flink/flink-cdc-docs-master/ 【推薦】

https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/

com.alibaba.ververica:flink-connector-mysql-cdc

https://github.com/apache/flink-cdc/tree/master/flink-cdc-connect/flink-cdc-pipeline-connectors

debezium

https://github.com/debezium/debezium

com.github.shyiko:mysql-binlog-connector-java

https://github.com/shyiko/mysql-binlog-connector-java

X 參考文獻

Flink CDC詳解 - CSDN

基於 Flink CDC 打造企業級實時資料整合方案
2023-11-23
【直播預告】優化器及 Flink CDC + OceanBase 全增量一體化資料整合方案
2022-04-13
優化
[資料整合/資料同步] 基於資料庫增量日誌的資料同步方案 : Flink CDC/Debezium/DataX/Canal/Oracle Goldengate/Kettle/Sqoop
2024-10-16
資料庫OracleGoOOP
FLINK CDC同步
2024-05-05
Flink CDC實戰
2020-10-21
FLINK CDC部署同步
2024-05-05
Flink CDC 專題首發｜每天 10 分鐘，解鎖新一代資料整合框架
2023-03-30
框架
基於 Flink CDC 的現代資料棧實踐
2023-04-18
Flink CDC 系列 - Flink MongoDB CDC 在 XTransfer 的生產實踐
2022-01-21
MongoDB
Flink MongoDB CDC 在 XTransfer 的生產實踐｜Flink CDC 專題
2023-04-13
MongoDB
Flink CDC + Hudi 海量資料入湖在順豐的實踐
2022-06-14
Flink CDC 系列 - 實現 MySQL 資料實時寫入 Apache Doris
2021-12-09
MySqlApache
Flink的mysql CDC，好使不？
2023-11-09
MySql
flink-cdc3.0官方網站
2024-05-05
網站
基於 Flink 的小米資料整合實踐
2023-02-16
flink-cdc實時同步(oracle to mysql)
2024-08-13
OracleMySql
Flink CDC 系列 - 同步 MySQL 分庫分表，構建 Iceberg 實時資料湖
2021-12-28
MySql
資料實時傳輸平臺（CDC）與低程式碼平臺（APAAS）資料整合
2022-08-16
Flink CDC 在京東的探索與實踐
2023-04-12
Flink CDC+Kafka 加速業務實時化
2023-02-23
Kafka
Flink CDC 在大健雲倉的實踐
2022-06-16
Apache Flink CDC 批流融合技術原理分析
2021-11-12
Apache
基於 Flink CDC 的實時同步系統
2023-03-17
Flink CDC 在易車的應用實踐
2023-04-19
三步玩轉：如何透過Flink OceanBase CDC聯結器快速查詢資料
2022-10-14
Flink CDC 採集MySQL 初始化或者指定時間戳時，沒有采集到資料
2023-09-23
MySql時間戳
如何實現對 Oracle 的實時資料捕獲和效能調優｜Flink CDC 專題
2023-04-06
Oracle
企業如何資料整合？資料整合解決方案
2022-07-13
構建企業CDC資料湖解決方案 -DZone
2021-10-12
SQL Server 變更資料捕獲(CDC)
2020-10-24
SQLServer
資料抽取中的CDC（變化資料捕獲）方式
2021-09-09
Flink CDC MongoDB Connector 的實現原理和使用實踐
2022-06-27
MongoDB
Flink 如何分流資料
2020-06-11
Oracle CDC 變更資料捕獲技術
2018-09-17
Oracle
Flink整合面向使用者的資料流SDKs/API(Flink關於棄用Dataset API的論述)
2022-06-02
API
SAP Spartacus 和 CDC 整合的 HTTP 請求明細
2022-09-17
HTTP
ETL資料整合，RestCloud資料整合平臺
2021-09-02
RESTCloud
Flink CDC 2.1 正式釋出，穩定性大幅提升，新增 Oracle，MongoDB 支援
2021-11-23
OracleMongoDB

[Flink/CDC/資料整合] 資料增量整合方案：Flink CDC

概述：Flink CDC

簡介

Flink CDC Connector(聯結器)

Flink CDC && Flink版本

Flink CDC特點

Flink CDC 的發展歷程

應用案例

試驗 ： Flink CDC 監聽 mysql 表的資料增量同步

依賴及版本

應用程式程式碼

DeviceEventLogCdcSyncJob : 啟動

device-log-event-cdc-sync-job-local.properties : 作業配置檔案

FlinkJobUtils

LogEventConfigDeserializationSchema : 對 debezium cdc 訊息(DebeziumDeserializationSchema) 反序列化 的介面實現類

執行效果

FlinkCdcJob 執行日誌

重點日誌行

完整日誌

日誌分析：重點日誌行，所提供的關鍵資訊

Y 推薦資源

X 參考文獻

相關文章

試驗： Flink CDC 監聽 mysql 表的資料增量同步

LogEventConfigDeserializationSchema : 對 debezium cdc 訊息(`DebeziumDeserializationSchema`) 反序列化的介面實現類