Canal詳細入門實戰(使用總結)

Canal介紹

Canal簡介

canal [kə'næl]，譯意為水道/管道/溝渠，主要用途是基於 MySQL 資料庫增量日誌解析，提供增量資料訂閱和消費

早期阿里巴巴因為杭州和美國雙機房部署，存在跨機房同步的業務需求，實現方式主要是基於業務 trigger 獲取增量變更。從 2010 年開始，業務逐步嘗試資料庫日誌解析獲取增量變更進行同步，由此衍生出了大量的資料庫增量訂閱和消費業務。

基於日誌增量訂閱和消費的業務包括

• 資料庫映象
• 資料庫實時備份
• 索引構建和實時維護(拆分異構索引、倒排索引等)
• 業務 cache 重新整理
• 帶業務邏輯的增量資料處理

當前的 canal 支援源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x

MySQL工作原理

• MySQL master 將資料變更寫入二進位制日誌( binary log, 其中記錄叫做二進位制日誌事件binary log events，可以通過 show binlog events 進行檢視)
• MySQL slave 將 master 的 binary log events 拷貝到它的中繼日誌(relay log)
• MySQL slave 重放 relay log 中事件，將資料變更反映它自己的資料

Canal工作原理

• canal 模擬 MySQL slave 的互動協議，偽裝自己為 MySQL slave ，向 MySQL master 傳送dump 協議
• MySQL master 收到 dump 請求，開始推送 binary log 給 slave (即 canal )
• canal 解析 binary log 物件(原始為 byte 流)

Canal使用準備

MySQL準備

1、對於自建 MySQL , 需要先開啟 Binlog 寫入功能，配置 binlog-format 為 ROW 模式，my.cnf 中配置如下

[mysqld]
log-bin=mysql-bin # 開啟binlog
binlog-format=ROW # 選擇ROW模式
server_id=1 # 配置MySQL replaction需要定義，不要和canal的slaveId重複

2、授權 canal 連結 MySQL 賬號具有作為 MySQL slave 的許可權, 如果已有賬戶可直接 grant

-- 使用命令登入：mysql -u root -p
-- 建立mysql使用者 使用者名稱：canal 密碼：canal
CREATE USER canal IDENTIFIED BY 'canal';  
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

3、重啟資料庫，檢視配置是否生效

mysql> show variables like 'binlog_format';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW   |
+---------------+-------+

mysql> show variables like 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin       | ON    |
+---------------+-------+

Canal使用準備

Canal下載，這裡使用1.1.5版本為例

github連結：https://github.com/alibaba/canal/releases

需要下載(其他可根據自行需要進行下載)：

• canal.deployer-1.1.5.tar.gz
• canal.adapter-1.1.5.tar.gz

下載後解壓

1.1.5解壓後目錄結構如下

bin	：canal啟動、重啟、停止檔案
conf	：canal配置檔案
lib	：canal執行所需的jar包，注意資料庫驅動包版本，可手動更換
logs	：canal執行日誌
plugin	：一些擴充套件包，如訊息佇列

建立資料庫和表

踩坑記錄：資料庫名不要使用下劃線，一開始用canal_test_01的方式命名，發現掃描不到資料庫

-- 源資料庫：canal01
CREATE DATABASE `canal01` CHARACTER SET 'utf8mb4';
-- 建立user01表
CREATE TABLE `user01` (
  `id` int(64) NOT NULL AUTO_INCREMENT,
  `username` varchar(64) DEFAULT NULL,
  `password` varchar(64) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;

-- 目標資料庫：canal02
CREATE DATABASE `canal02` CHARACTER SET 'utf8mb4';
-- 建立user02表
CREATE TABLE `user02` (
  `id` int(64) NOT NULL AUTO_INCREMENT,
  `username` varchar(64) DEFAULT NULL,
  `password` varchar(64) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;

Canal-deployer使用

1、conf\canal.properties配置檔案(這裡只介紹我用到的配置)

# canal-deployer的ip地址，不配置預設是本機
canal.ip =
# canal-deployer的埠號
canal.port = 11111
# 模式
canal.serverMode = tcp
# 載入多個配置檔案，需在canal.deployer-1.1.5\conf下建立對應的資料夾，並加上instance.properties配置檔案
canal.destinations = example
# 配置多個使用英文逗號隔開
# canal.destinations = example,example2

2、conf\example\instance.properties配置檔案(這裡只介紹我用到的配置)

# 不能與my.ini下的server_id=1相同
canal.instance.mysql.slaveId=1234

# mysql地址和埠
canal.instance.master.address=127.0.0.1:3306

# 資料庫賬號密碼，可以使用root賬號或剛建立的canal賬號
canal.instance.dbUsername=root
canal.instance.dbPassword=root
canal.instance.connectionCharset=UTF-8

# 配置監聽，支援正規表示式
canal.instance.filter.regex=canal01\\..*
# 配置不監聽，支援正規表示式
canal.instance.filter.black.regex=mysql\\.slave_.*

3、檢查lib目錄下的jar包，一定要有對應的資料庫驅動包，且版本要對得上

4、在bin目錄中啟動canal-deployer，啟動成功如下輸出

# 執行後輸出
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; 
support was removed in 8.0 Listening for transport dt_socket at address: 9099

# 在logs\canal目錄下，會記錄canal.log
# 有如下記錄表示canal-deployer執行成功
2021-04-28 09:26:57.194 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2021-04-28 09:26:57.244 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2021-04-28 09:26:57.257 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2021-04-28 09:26:57.399 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.3.51(192.168.3.51):11111]
2021-04-28 09:26:59.440 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......

# 在logs\example目錄下，會記錄example.log
# 有如下記錄表示canal-deployer與資料庫連線成功了
2021-04-28 09:30:37.159 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2021-04-28 09:30:37.175 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^fooddb\..*$
2021-04-28 09:30:37.175 [main] WARN  c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$
2021-04-28 09:30:37.270 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2021-04-28 09:30:37.288 [main] INFO  c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2021-04-28 09:30:37.305 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position
 {"identity":{"slaveId":-1,"sourceAddress":{"address":"ieonline.microsoft.com","port":3306}},"postion":{"gtid":"","included":false,"journalName":"binlog.000068","position":14636,"serverId":1,"timestamp":1619408047000}}
2021-04-28 09:30:37.744 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=binlog.000068,position=14636,serverId=1,gtid=,timestamp=1619408047000] cost : 466ms , the next step is binlog dump

5、建立springboot工程監聽canal-deployer(可跳過)

匯入maven座標(pom.xml)

<!--Canal-->
<dependency>
	<groupId>com.alibaba.otter</groupId>
	<artifactId>canal.client</artifactId>
	<version>1.1.4</version>
</dependency>

建立CanalClient類，並交給spring管理

import java.net.InetSocketAddress;
import java.util.List;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.protocol.Message;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.EventType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.stereotype.Component;


@Component
public class CanalClient implements InitializingBean {

    private final static int BATCH_SIZE = 1000;

    @Override
    public void afterPropertiesSet() throws Exception {
        // 建立連結
        CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress("127.0.0.1", 11111), "example", "", "");
        try {
            //開啟連線
            connector.connect();
            //訂閱資料庫表,全部表
            connector.subscribe(".*\\..*");
            //回滾到未進行ack的地方，下次fetch的時候，可以從最後一個沒有ack的地方開始拿
            connector.rollback();
            while (true) {
                // 獲取指定數量的資料
                Message message = connector.getWithoutAck(BATCH_SIZE);
                //獲取批量ID
                long batchId = message.getId();
                //獲取批量的數量
                int size = message.getEntries().size();
                //如果沒有資料
                if (batchId == -1 || size == 0) {
                    try {
                        //執行緒休眠2秒
                        Thread.sleep(2000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                } else {
                    //如果有資料,處理資料
                    printEntry(message.getEntries());
                }
                //進行 batch id 的確認。確認之後，小於等於此 batchId 的 Message 都會被確認。
                connector.ack(batchId);
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            connector.disconnect();
        }
    }

    /**
     * 列印canal server解析binlog獲得的實體類資訊
     */
    private static void printEntry(List<Entry> entrys) {
        for (Entry entry : entrys) {
            if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
                //開啟/關閉事務的實體型別，跳過
                continue;
            }
            //RowChange物件，包含了一行資料變化的所有特徵
            //比如isDdl 是否是ddl變更操作 sql 具體的ddl sql beforeColumns afterColumns 變更前後的資料欄位等等
            RowChange rowChage;
            try {
                rowChage = RowChange.parseFrom(entry.getStoreValue());
            } catch (Exception e) {
                throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(), e);
            }
            //獲取操作型別：insert/update/delete型別
            EventType eventType = rowChage.getEventType();
            //列印Header資訊
            System.out.println(String.format("================》; binlog[%s:%s] , name[%s,%s] , eventType : %s",
                    entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
                    entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
                    eventType));
            //判斷是否是DDL語句
            if (rowChage.getIsDdl()) {
                System.out.println("================》;isDdl: true,sql:" + rowChage.getSql());
            }
            //獲取RowChange物件裡的每一行資料，列印出來
            for (RowData rowData : rowChage.getRowDatasList()) {
                //如果是刪除語句
                if (eventType == EventType.DELETE) {
                    printColumn(rowData.getBeforeColumnsList());
                    //如果是新增語句
                } else if (eventType == EventType.INSERT) {
                    printColumn(rowData.getAfterColumnsList());
                    //如果是更新的語句
                } else {
                    //變更前的資料
                    System.out.println("------->; before");
                    printColumn(rowData.getBeforeColumnsList());
                    //變更後的資料
                    System.out.println("------->; after");
                    printColumn(rowData.getAfterColumnsList());
                }
            }
        }
    }

    private static void printColumn(List<Column> columns) {
        for (Column column : columns) {
            System.out.println(column.getName() + " : " + column.getValue() + "    update=" + column.getUpdated());
        }
    }
}

啟動springboot專案，在user01表中新增資料，控制檯如下輸出，監聽成功

================》; binlog[binlog.000070:5154] , name[canal01,user01] , eventType : INSERT
id : 1    update=true
username : a    update=true
password : a    update=true

Canal-adapter使用

1、conf\application.yml配置檔案(這裡只介紹我用到的配置)

server:
  port: 8081 # canal-adapter執行埠號

canal.conf:
  mode: tcp # canal client的模式: tcp kafka rocketMQ rabbitMQ
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources: # 源資料庫
    canal01: # 自定義名稱
      # 這裡使用的是mysql8.0，需要加上serverTimezone=UTC
      url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
      username: root
      password: root
# 	 配置多個源資料庫
#    canal02:
#      url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
#      username: root
#      password: root
      
  canalAdapters: # 介面卡列表
  - instance: example # canal 例項名或者 MQ topic 名
    groups:	# 分組列表
    - groupId: # 分組id, 如果是MQ模式將用到該值
      outerAdapters: # 分組內介面卡列表
#      - name: logger # 日誌列印介面卡
      - name: rdb	# 指定為rdb型別同步
        key: mysql1	# 指定adapter的唯一key, 與表對映配置中outerAdapterKey對應
        # 目標資料庫配置
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
          jdbc.username: root
          jdbc.password: root
          threads: 5 # 並行執行的執行緒數,可以不修改，預設為1
#  配置多個目標資料庫
#  - instance: example2 # canal instance Name or mq topic name
#    groups:
#    - groupId: g2
#      outerAdapters:
#      - name: rdb
#        key: mysql2
#        properties:
#          jdbc.driverClassName: com.mysql.jdbc.Driver
#          jdbc.url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true
#          jdbc.username: root
#          jdbc.password: root

2、配置RDB表對映檔案(conf/rdb/mytest_user.yml檔案)

dataSourceKey: canal01          # 源資料來源的key, 對應上面配置的srcDataSources中的值
destination: example            # cannal的instance或者MQ的topic
groupId: g1                     # 對應MQ模式下的groupId, 只會同步對應groupId的資料
outerAdapterKey: mysql1         # adapter key, 對應上面配置outAdapters中的key
concurrent: true                # 是否按主鍵hash並行同步, 並行同步的表必須保證主鍵不會更改及主鍵不能為其他同步表的外來鍵!!
dbMapping:
  database: canal01             # 源資料來源的database/shcema
  table: user01                 # 源資料來源表名
  targetTable: user02      		# 直接寫表名
  targetPk:                     # 主鍵對映
    id: id                      # 如果是複合主鍵可以換行對映多個
  mapAll: true                  # 是否整表對映, 要求源表和目標表欄位名一模一樣 (如果targetColumns也配置了對映,則以targetColumns配置為準)
#  targetColumns:               # 欄位對映, 格式: 目標表欄位: 源表欄位, 如果欄位名一樣源表欄位名可不填
#    id:
#    name:
#    role_id:
#    c_time:
#    test1:
  etlCondition: "where c_time>={}"
  commitBatch: 3000 			# 批量提交的大小

3、檢查lib目錄下的jar包，一定要有對應的資料庫驅動包，且版本要對得上

4、在bin目錄中啟動canal-adapter，啟動成功如下輸出

# 執行後輸出
2021-04-28 11:12:38.281 [main] INFO  o.s.b.w.s.c.AnnotationConfigServletWebServe
rApplicationContext - Refreshing org.springframework.boot.web.servlet.context.An
notationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr
28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.Annotation
ConfigApplicationContext@1a677343
2021-04-28 11:12:39.012 [main] INFO  org.springframework.cloud.context.scope.Gen
ericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2021-04-28 11:12:39.108 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$B
eanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.Configur
ationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.aut
oconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGL
IB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (f
or example: not eligible for auto-proxying)
2021-04-28 11:12:39.610 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServe
r - Tomcat initialized with port(s): 8081 (http)
2021-04-28 11:12:39.632 [main] INFO  org.apache.coyote.http11.Http11NioProtocol
- Initializing ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:39.646 [main] INFO  org.apache.catalina.core.StandardService -
Starting service [Tomcat]
2021-04-28 11:12:39.647 [main] INFO  org.apache.catalina.core.StandardEngine - S
tarting Servlet Engine: Apache Tomcat/8.5.29
2021-04-28 11:12:39.660 [localhost-startStop-1] INFO  org.apache.catalina.core.A
prLifecycleListener - The APR based Apache Tomcat Native library which allows op
timal performance in production environments was not found on the java.library.p
ath: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\sys
tem32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Wind
ows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java
\jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\
bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Com
mon;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\User
s\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\Je
tBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4
\bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C
:\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.]
2021-04-28 11:12:39.830 [localhost-startStop-1] INFO  o.a.catalina.core.Containe
rBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationCont
ext
2021-04-28 11:12:39.831 [localhost-startStop-1] INFO  org.springframework.web.co
ntext.ContextLoader - Root WebApplicationContext: initialization completed in 15
50 ms
2021-04-28 11:12:39.990 [localhost-startStop-1] INFO  o.s.boot.web.servlet.Servl
etRegistrationBean - Servlet dispatcherServlet mapped to [/]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO  o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO  o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2021-04-28 11:12:40.001 [localhost-startStop-1] INFO  o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2021-04-28 11:12:40.002 [localhost-startStop-1] INFO  o.s.boot.web.servlet.Filte
rRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class
is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SP
I and manual loading of the driver class is generally unnecessary.
2021-04-28 11:12:40.597 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {d
ataSource-1} inited
2021-04-28 11:12:40.849 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springf
ramework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.143 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.serv
let.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup
date [Wed Apr 28 11:12:38 GMT+08:00 2021]; parent: org.springframework.context.a
nnotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:41.245 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto publi
c java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adap
ter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.S
tring)
2021-04-28 11:12:41.247 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java
.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.la
uncher.rest.CommonRest.count(java.lang.String,java.lang.String)
2021-04-28 11:12:41.248 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" on
to public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otte
r.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2021-04-28 11:12:41.249 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public
 java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapt
er.launcher.rest.CommonRest.etl(java.lang.String)
2021-04-28 11:12:41.249 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public
 com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.cana
l.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.la
ng.String,java.lang.String)
2021-04-28 11:12:41.250 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.a
libaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adap
ter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.Str
ing)
2021-04-28 11:12:41.251 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.L
ist<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.a
dapter.launcher.rest.CommonRest.destinations()
2021-04-28 11:12:41.256 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/error]}" onto public org.springframework.http.Respons
eEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.b
oot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.htt
p.HttpServletRequest)
2021-04-28 11:12:41.258 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingH
andlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springf
ramework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.ser
vlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse)
2021-04-28 11:12:41.298 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/webjars/**] onto handler of type [class org.springframe
work.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.299 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMap
ping - Mapped URL path [/**] onto handler of type [class org.springframework.web
.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.604 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Registering beans for JMX exposure on startup
2021-04-28 11:12:41.618 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2021-04-28 11:12:41.620 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'configurationPropertiesRebinder' has been autodetected
for JMX exposure
2021-04-28 11:12:41.622 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Bean with name 'environmentManager' has been autodetected for JMX expos
ure
2021-04-28 11:12:41.625 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'environmentManager': registering with JMX server
as MBean [org.springframework.cloud.context.environment:name=environmentManager,
type=EnvironmentManager]
2021-04-28 11:12:41.644 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'refreshScope': registering with JMX server as MBe
an [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=Refre
shScope]
2021-04-28 11:12:41.659 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanEx
porter - Located managed bean 'configurationPropertiesRebinder': registering wit
h JMX server as MBean [org.springframework.cloud.context.properties:name=configu
rationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder]
2021-04-28 11:12:41.675 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## syncSwitch refreshed.
2021-04-28 11:12:41.675 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## start the canal client adapters.
2021-04-28 11:12:41.678 [main] INFO  c.a.otter.canal.client.adapter.support.Exte
nsionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools
\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:41.777 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.C
onfigLoader - ## Start loading rdb mapping config ...
2021-04-28 11:12:41.837 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.C
onfigLoader - ## Rdb mapping config loaded
2021-04-28 11:12:42.224 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {d
ataSource-2} inited
2021-04-28 11:12:42.234 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAd
apterLoader - Load canal adapter: rdb succeed
2021-04-28 11:12:42.246 [main] INFO  c.alibaba.otter.canal.connector.core.spi.Ex
tensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Too
ls\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:42.275 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAd
apterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2021-04-28 11:12:42.276 [Thread-4] INFO  c.a.otter.canal.adapter.launcher.loader
.AdapterProcessor - =============> Start to connect destination: example <======
=======
2021-04-28 11:12:42.276 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAd
apterService - ## the canal client adapters are running now ......
2021-04-28 11:12:42.286 [main] INFO  org.apache.coyote.http11.Http11NioProtocol
- Starting ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:42.296 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool
- Using a shared selector for servlet write/read
2021-04-28 11:12:42.316 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServe
r - Tomcat started on port(s): 8081 (http) with context path ''
2021-04-28 11:12:42.320 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapt
erApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running fo
r 6.186)
2021-04-28 11:12:42.354 [Thread-4] INFO  c.a.otter.canal.adapter.launcher.loader
.AdapterProcessor - =============> Subscribe destination: example succeed <=====
========

# logs\adapter目錄下，會記錄adapter.log
# 有如下記錄表示canal-adapter成功執行了
2021-04-28 11:12:37.612 [main] INFO  o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343: startup date [Wed Apr 28 11:12:37 CST 2021]; root of context hierarchy
2021-04-28 11:12:38.027 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-28 11:12:38.254 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - No active profile set, falling back to default profiles: default
2021-04-28 11:12:38.281 [main] INFO  o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:39.012 [main] INFO  org.springframework.cloud.context.scope.GenericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2021-04-28 11:12:39.108 [main] INFO  o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2021-04-28 11:12:39.610 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8081 (http)
2021-04-28 11:12:39.632 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:39.646 [main] INFO  org.apache.catalina.core.StandardService - Starting service [Tomcat]
2021-04-28 11:12:39.647 [main] INFO  org.apache.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29
2021-04-28 11:12:39.660 [localhost-startStop-1] INFO  org.apache.catalina.core.AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java\jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\Users\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\JetBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4\bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C:\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.]
2021-04-28 11:12:39.830 [localhost-startStop-1] INFO  o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2021-04-28 11:12:39.831 [localhost-startStop-1] INFO  org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 1550 ms
2021-04-28 11:12:39.990 [localhost-startStop-1] INFO  o.s.boot.web.servlet.ServletRegistrationBean - Servlet dispatcherServlet mapped to [/]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2021-04-28 11:12:40.000 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2021-04-28 11:12:40.001 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2021-04-28 11:12:40.002 [localhost-startStop-1] INFO  o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
2021-04-28 11:12:40.597 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
2021-04-28 11:12:40.849 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.143 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 GMT+08:00 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343
2021-04-28 11:12:41.245 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.247 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String)
2021-04-28 11:12:41.248 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" onto public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2021-04-28 11:12:41.249 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String)
2021-04-28 11:12:41.249 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.250 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String)
2021-04-28 11:12:41.251 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.List<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.destinations()
2021-04-28 11:12:41.256 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2021-04-28 11:12:41.258 [main] INFO  o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2021-04-28 11:12:41.298 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.299 [main] INFO  o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2021-04-28 11:12:41.604 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2021-04-28 11:12:41.618 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2021-04-28 11:12:41.620 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2021-04-28 11:12:41.622 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'environmentManager' has been autodetected for JMX exposure
2021-04-28 11:12:41.625 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2021-04-28 11:12:41.644 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2021-04-28 11:12:41.659 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder]
2021-04-28 11:12:41.675 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed.
2021-04-28 11:12:41.675 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2021-04-28 11:12:41.678 [main] INFO  c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:41.777 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ... 
2021-04-28 11:12:41.837 [main] INFO  c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded
2021-04-28 11:12:42.224 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {dataSource-2} inited
2021-04-28 11:12:42.234 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed
2021-04-28 11:12:42.246 [main] INFO  c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin
2021-04-28 11:12:42.275 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed
2021-04-28 11:12:42.276 [Thread-4] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <=============
2021-04-28 11:12:42.276 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2021-04-28 11:12:42.286 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2021-04-28 11:12:42.296 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2021-04-28 11:12:42.316 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2021-04-28 11:12:42.320 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running for 6.186)
2021-04-28 11:12:42.354 [Thread-4] INFO  c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============

5、向user01表中新增資料，user02表也有對應資料，同步成功

2021-04-28 11:27:59.619 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.rdb.s
ervice.RdbSyncService - DML: {"data":{"id":1,"username":"a","password":"a"},"dat
abase":"canal01","destination":"example","old":null,"table":"user01","type":"INS
ERT"}

如需user02同步到user01：

​ （1）新增對應的canal-deployer中新增對應的example

​ （2）canal-adapter的application.yml配置檔案中加入對應的源資料庫和目標資料庫(具體見上面配置)

​ （3）新建RDB表對映檔案，在rdb目錄下，canal-adapter會預設掃描rdb目錄下所有的yml檔案

同步多張表：

​ （1）在兩個資料庫中增加表（這裡以person為例）

​ （2）配置RDB表對映檔案，將mytest_user.yml複製一份，重新命名為mytest_person.yml

​ （3）修改mytest_person.yml中的table(源表)、targetTable(目標表)

6、如果需要使用Java工程，可自行在github的canal中下載canal-adapter的工程，配置與上面完全一樣