redis-shake簡介
redis-shake是阿里開源的用於redis資料同步的工具，基本功能有：
- 恢復restore：將RDB檔案恢復到目的redis資料庫。
- 備份dump：將源redis的全量資料通過RDB檔案備份起來。
- 解析decode：對RDB檔案進行讀取，並以json格式解析儲存。
- 同步sync：支援源redis和目的redis的資料同步，支援全量和增量資料的遷移，支援從雲下到阿里云云上的同步，也支援雲下到雲下不同環境的同步，支援單節點、主從版、叢集版之間的互相同步。需要注意的是，如果源端是叢集版，可以啟動一個RedisShake，從不同的db結點進行拉取，同時源端不能開啟move slot功能；對於目的端，如果是叢集版，寫入可以是1個或者多個db結點。
- 同步rump：支援源redis和目的redis的資料同步，僅支援全量的遷移。採用scan和restore命令進行遷移，支援不同雲廠商不同redis版本的遷移。
基本原理
如果源端是叢集模式，只需要啟動一個redis-shake進行拉取，同時不能開啟源端的move slot操作。如果目的端是叢集模式，可以寫入到一個結點，然後再進行slot的遷移，當然也可以多對多寫入。
目前，redis-shake到目的端採用單鏈路實現，對於正常情況下，這不會成為瓶頸，但對於極端情況，qps比較大的時候，此部分效能可能成為瓶頸。另外，redis-shake到目的端的資料同步採用非同步的方式，讀寫分離在2個執行緒操作，降低因為網路時延帶來的同步效能下降。
全量同步階段併發執行，增量同步階段非同步執行，能夠達到毫秒級別延遲（取決於網路延遲）。同時，對大key同步進行分批拉取，優化同步效能。
使用者可以通過我們提供的restful拉取metric來對redis-shake進行實時監控：curl 127.0.0.1:9320/metric。
支援
- 支援2.8-5.0版本的同步。
- 支援codis。
- 支援雲下到雲上，雲上到雲上，雲上到雲下（阿里雲目前支援主從版），其他雲到阿里雲等鏈路，幫助使用者靈活構建混合雲場景。
使用
1、直接下載二進位制包這裡
2、修改配置檔案
]# tar -zxf release-v2.1.0-20210819.tar.gz
]# cd redis-shake
]# ll
total 41252
-rw-r--r-- 1 502 games 12542 Aug 25 17:50 redis-shake.conf
-rwxr-xr-x 1 502 games 11853504 Aug 19 10:11 redis-shake.darwin
-rwxr-xr-x 1 502 games 11918954 Aug 19 10:11 redis-shake.linux
-rwxr-xr-x 1 502 games 12227584 Aug 19 10:11 redis-shake.windows
修改redis-shake.conf，我的環境是從兩套redis主從之間同步資料，配置檔案如下（刪減過部分預設配置）
conf.version = 1
id = redis-shake
log.file = /opt/redis-shake/redis-shake.log
log.level = info
pid_path =
system_profile = 9310
http_profile = 9320
parallel = 32
source.type = standalone
source.address = 10.26.31.172:6379
source.password_raw =
source.auth_type = auth
source.tls_enable = false
source.rdb.input =
source.rdb.parallel = 0
source.rdb.special_cloud =
target.type = standalone
target.address = 10.26.29.235:6389
target.password_raw =
target.auth_type = auth
target.db = 0
key_exists = rewrite
filter.db.whitelist = 0
說明：
a、主要需要修改源redis和目的redis的資訊
b、當源和目的有重複 key 時是否進行覆寫, 可選值:
1. rewrite: 源端覆蓋目的端
2. none: 一旦發生程式直接退出
3. ignore: 保留目的端key，忽略源端的同步 key. 該值在 rump 模式下不會生效.
key_exists = rewrite
c、指定的db被通過，比如0;5;10將會使db0, db5, db10通過, 其他的被過濾
filter.db.whitelist = 0
更多詳細引數參見官網
3、啟動
使用者配置完配置檔案，然後以不同的模式啟動即可：
./redis-shake -conf=redis-shake.conf -type=sync &
然後可以檢視配置的日子路徑，檢視服務執行狀態,一般正常情況下日誌如下
2021/08/25 17:19:54 [WARN] source.auth_type[auth] != auth
2021/08/25 17:19:54 [WARN] target.auth_type[auth] != auth
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.29.235:6389] with type[auth].
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] source rdb[10.26.31.172:6379] checksum[yes]
2021/08/25 17:19:54 [WARN]
______________________________
\ \ _ ______ |
\ \ / \___-=O'/|O'/__|
\ RedisShake, here we go !! \_______\ / | / )
/ / '/-==__ _/__|/__=-| -GM
/ Alibaba Cloud / * \ | |
/ / (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/RedisShake/wiki/FAQ
2021/08/25 17:19:54 [INFO] redis-shake configuration: {"ConfVersion":1,"Id":"redis-shake","LogFile":"/opt/redis-shake/redis-shake.log","LogLevel":"info","SystemProfile":9310,"HttpProfile":9320,"Parallel":32,"SourceType":"standalone","SourceAddress":"10.26.31.172:6379","SourcePasswordRaw":"***","SourcePasswordEncoding":"***","SourceAuthType":"auth","SourceTLSEnable":false,"SourceRdbInput":[],"SourceRdbParallel":1,"SourceRdbSpecialCloud":"","TargetAddress":"10.26.29.235:6389","TargetPasswordRaw":"***","TargetPasswordEncoding":"***","TargetDBString":"0","TargetDBMapString":"","TargetAuthType":"auth","TargetType":"standalone","TargetTLSEnable":false,"TargetRdbOutput":"local_dump","TargetVersion":"6.0.9","FakeTime":"","KeyExists":"none","FilterDBWhitelist":[],"FilterDBBlacklist":[],"FilterKeyWhitelist":[],"FilterKeyBlacklist":[],"FilterSlot":[],"FilterLua":false,"BigKeyThreshold":524288000,"Metric":true,"MetricPrintLog":false,"SenderSize":104857600,"SenderCount":4095,"SenderDelayChannelSize":65535,"KeepAlive":0,"PidPath":"","ScanKeyNumber":50,"ScanSpecialCloud":"","ScanKeyFile":"","Qps":200000,"ResumeFromBreakPoint":false,"Psync":true,"NCpu":0,"HeartbeatUrl":"","HeartbeatInterval":10,"HeartbeatExternal":"","HeartbeatNetworkInterface":"","ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"FilterKey":null,"FilterDB":"","Rewrite":false,"SourceAddressList":["10.26.31.172:6379"],"TargetAddressList":["10.26.29.235:6389"],"SourceVersion":"4.0.11","HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetReplace":false,"TargetDB":0,"Version":"tidy-code-for-release,ba0f3e1aa6e0956bfa2973a4c894cbc1a8d3d732,go1.16.7,2021-08-19_10:11:39","Type":"sync","TargetDBMap":null}
2021/08/25 17:19:54 [INFO] DbSyncer[0] starts syncing data from 10.26.31.172:6379 to [10.26.29.235:6389] with http[9321], enableResumeFromBreakPoint[false], slot boundary[-1, -1]
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] DbSyncer[0] psync connect '10.26.31.172:6379' with auth type[auth] OK!
2021/08/25 17:19:54 [INFO] DbSyncer[0] psync send listening port[9320] OK!
2021/08/25 17:19:54 [INFO] DbSyncer[0] try to send 'psync' command: run-id[?], offset[-1]
2021/08/25 17:22:30 [INFO] Event:FullSyncStart Id:redis-shake
2021/08/25 17:22:30 [INFO] DbSyncer[0] psync runid = 52aa8214f9f8238c32f172badbb4f4ea325c6cc4, offset = 9452286340998, fullsync
2021/08/25 17:22:30 [INFO] DbSyncer[0] +
2021/08/25 17:22:31 [INFO] DbSyncer[0] -
2021/08/25 17:22:31 [INFO] DbSyncer[0] +
日誌資訊
同步分為三個階段：
1、等待源端save rdb完畢，日誌如下
2021/08/25 17:22:30 [INFO] DbSyncer[0] +
2021/08/25 17:22:31 [INFO] DbSyncer[0] -
2021/08/25 17:22:31 [INFO] DbSyncer[0] +
2、全量同步階段，顯示百分比
2021/08/25 17:28:09 [INFO] DbSyncer[0] total = 9.06GB - 3.82MB [ 0%] entry=56515
2021/08/25 17:28:10 [INFO] DbSyncer[0] total = 9.06GB - 7.67MB [ 0%] entry=114522
2021/08/25 17:28:11 [INFO] DbSyncer[0] total = 9.06GB - 11.28MB [ 0%] entry=168877
3、增量同步，出現字樣sync rdb done後，當前dbSyncer進入增量同步
2021/08/26 09:40:58 [INFO] DbSyncer[0] sync: +forwardCommands=3 +filterCommands=3 +writeBytes=33
2021/08/26 09:40:59 [INFO] DbSyncer[0] sync: +forwardCommands=2 +filterCommands=0 +writeBytes=23
2021/08/26 09:41:00 [INFO] DbSyncer[0] sync: +forwardCommands=1 +filterCommands=3 +writeBytes=11
注意事項
1、如果目標庫的資料逐出策略（maxmemory-policy）配置為noeviction以外的值，可能導致目標庫的資料與源庫不一致。關於資料逐出策略詳情，請參見Redis資料逐出策略介紹。
2、如果源庫中的某些Key使用了過期（expire）機制，由於可能存在Key已過期但未被及時刪除的情形，所以在目標庫中檢視（如通過info命令）到的Key數量會比源庫的Key數量少。