【Maxwell】使用maxwell+kafka+python做binlog增量解析消費

小亮520cl發表於2018-05-14
KafkaPython
1.maxwell基礎
參考

這裡寫下我的基礎配置
  1. [root@ip-10-1-2-175 maxwell-1.12.0]# more config.properties

  3. log_level=info
  4. kafka.bootstrap.servers=localhost:9092
  5. producer=kafka
  6. host=10.1.2.175
  7. user=dbaadmin
  8. password=
  9. schema_database=maxwell
  10. kafka.compression.type=snappy
  11. kafka.retries=0
  12. kafka.acks=1
  13. kinesis_stream=maxwell
  14. sqs_queue_uri=aws_sqs_queue_uri
  1. producer=kafka   ###生產者選擇kafka
  2. kafka.bootstrap.servers=10.1.2.175:9092
  3. kafka_topic=test    ####主題

  5. output_ddl = true   ####ddl輸出到 maxwell_ddl topic
    ddl_kafka_topic=maxwell_ddl

  6. producer_partition_by=database

2.python 消費者虛擬碼
  1. import time, json
  2. from pykafka import KafkaClient
  3. client = KafkaClient(hosts="10.1.2.175:9092")
  4. topic = client.topics['test']
  5. balanced_consumer = topic.get_balanced_consumer(consumer_group='goods_group',auto_commit_enable=True,zookeeper_connect='localhost:2181')

  7. for message in balanced_consumer:
  8.     print message.offset, message.value


啟動
[root@ip-10-1-2-175 sh]# python cus.py 

3.啟動maxwel
  1. [root@ip-10-1-2-175 maxwell-1.12.0]# ./bin/maxwell

4.模擬資料庫操作
  1. mysql> insert into testwell(name) values ('fuck');
  2. Query OK, 1 row affected (0.00 sec)

  4. mysql> insert into testwell(name) values ('fuck');
  5. Query OK, 1 row affected (0.00 sec)

  7. mysql> insert into testwell(name) values ('fuck');
  8. Query OK, 1 row affected (0.00 sec)

  10. mysql>


5.檢視消費者
  1. [root@ip-10-1-2-175 sh]# python cus.py
  2. 67 {"database":"test","table":"testwell","type":"insert","ts":1526290515,"xid":208818,"commit":true,"data":{"id":3050,"name":"fuck"}}
  3. 68 {"database":"test","table":"testwell","type":"insert","ts":1526290515,"xid":208819,"commit":true,"data":{"id":3051,"name":"fuck"}}
  4. 69 {"database":"test","table":"testwell","type":"insert","ts":1526290517,"xid":208820,"commit":true,"data":{"id":3052,"name":"fuck"}}
  5. 70 {"database":"test","table":"testwell","type":"insert","ts":1526290519,"xid":208822,"commit":true,"data":{"id":3053,"name":"fuck"}}


6.根據上面的思路可以寫出增量同步的虛擬碼
  1. [root@ip-10-1-2-175 sh]#
  2. import time, json
  3. from pykafka import KafkaClient
  4. client = KafkaClient(hosts="10.1.2.175:9092")
  5. topic = client.topics['test']
  6. balanced_consumer = topic.get_balanced_consumer(consumer_group='goods_group',auto_commit_enable=True,zookeeper_connect='localhost:2181')

  8. for message in balanced_consumer:
  9.     print message.offset, message.value
  10.    # 對資料庫進行操作
  11.     if message.value ['type'] == 'insert':
  12.       mysql_insert()
  13.     elif  message.value['type'] == 'update':
  14.       mysql_update()
  15.     elif  message.value['type'] == 'delete':
  16.       mysql_delete()
  17.     else:
  18.       order_option()


其它：
https://mp.weixin.qq.com/s/aGqwWNd6Q2Gm2jilxXS0RA

Kafka Connect 如何實現同步RDS binlog資料?

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29096438/viewspace-2154470/，如需轉載，請註明出處，否則將追究法律責任。

相關文章