canal同步mysql，監聽單例項，多例項配置

__Yoon發表於2024-08-21
MySql單例

1、下載canal安裝包

canal.adapter-1.1.7.tar.gz

canal.deployer-1.1.7.tar.gz

2、修改涉及的檔案

canal_deployer：
/conf/canal.properties
/conf/example/instance.properties

canal_adapter：
/conf/application.yml
/conf/rdb  在rdb中新增對應的資料配置表，即便監聽多個mysql實力，全部放在rdb中

3、監聽單例項mysql資料來源，在/conf/canal.properties中配置一個example，預設就是example

4、監聽多個例項mysql資料來源，複製example進行修改，例如：example1,example2

5、監聽一個mysql資料來源同步到一個資料庫

canal_deployer：

修改 /conf/example/instance.properties 檔案

#################################################
## mysql serverId , v1.0.26+ will autoGen
# canal.instance.mysql.slaveId=0

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=10.10.7.7:3306
canal.instance.master.journal.name=mysql-bin.054560
canal.instance.master.position=241476047
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=canal_hankyoon
canal.instance.dbPassword=canal_hankyoon
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
#canal.instance.filter.regex=.*\\..*
canal.instance.filter.regex=dbname.tablename01,dbname.tablename02,dbname.tablename03

canal_adapter：
修改application.yml 檔案

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ    # 注意mode選擇
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
#  retries: -1
#  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources:
    defaultDS:
      url: jdbc:mysql://10.10.7.7:3306/hankyoon?useUnicode=true
      username: canal_hankyoon
      password: canal_hankyoon

  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: yoon
      outerAdapters:
#      - name: logger
      - name: rdb
        key: yoon
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://10.10.7.77:3306/hankyoon?useUnicode=true
          jdbc.username: canal_hankyoon
          jdbc.password: canal_hankyoon

在rdb中新增對應的資料配置表：

dataSourceKey: defaultDS
destination: example1
groupId: yoon
outerAdapterKey: yoon
concurrent: true
dbMapping:
  database: hankyoon
  table: yoon
  targetTable: yoon
  targetPk:
    id: id
  mapAll: true

6、監聽多個mysql例項資料來源，同步至一個資料庫
修改 /conf/canal.properties 檔案，新增多個例項

canal.destinations = example,example1

修改新增example1

conf/example
conf/example1   # 修改instance.properties中對應的資料庫資訊

7、修改 application.yml 新增多個監聽例項

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp #tcp kafka rocketMQ rabbitMQ    # 注意mode選擇
  flatMessage: true
  zookeeperHosts:
  syncBatchSize: 1000
#  retries: -1
#  retries: 0
  timeout:
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:

  srcDataSources:
    defaultDS:
      url: jdbc:mysql://10.10.7.7:3306/hankyoon?useUnicode=true
      username: canal_hankyoon
      password: canal_hankyoon

    defaultDS01:
      url: jdbc:mysql://10.10.7.9:3306/hankyoon?useUnicode=true
      username: canal_hankyoon
      password: canal_hankyoon

  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: yoon
      outerAdapters:
#      - name: logger
      - name: rdb
        key: yoon
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://10.10.7.77:3306/hankyoon?useUnicode=true
          jdbc.username: canal_hankyoon
          jdbc.password: canal_hankyoon

  - instance: example1 # canal instance Name or mq topic name
    groups:
    - groupId: hank
      outerAdapters:
#      - name: logger
      - name: rdb         # rdb不要複製，不要修改，同步的資料來源配置表，全部放在一個rdb中
        key: hank
        properties:
          jdbc.driverClassName: com.mysql.jdbc.Driver
          jdbc.url: jdbc:mysql://10.10.7.77:3306/hankyoon?useUnicode=true
          jdbc.username: canal_hankyoon
          jdbc.password: canal_hankyoon

在rdb中資料表配置資訊：

example：
dataSourceKey: defaultDS
destination: example1
groupId: yoon
outerAdapterKey: yoon
concurrent: true
dbMapping:
  database: hankyoon
  table: yoon
  targetTable: yoon
  targetPk:
    id: id
  mapAll: true


example1：
dataSourceKey: defaultDS01
destination: example1
groupId: hankyoon
outerAdapterKey: hank
concurrent: true
dbMapping:
  database: hankyoon
  table: hank
  targetTable: hank
  targetPk:
    id: id
  mapAll: true

參考：https://blog.csdn.net/wonder_dog/article/details/124608664

相關文章