Redis Cluster 叢集部署

misakivv發表於2024-04-19

目錄
    • 一、什麼是redis Cluster叢集
    • 二、叢集架構圖
    • 三、redis Cluster部署架構
      • 1、測試環境
      • 2、生產環境
    • 四、原生命令手動部署Redis Cluster
      • 1、環境準備
      • 2、為所有節點啟用redis叢集支援
      • 3、執行meet操作實現互相通訊
  • 在任意一節點上和其它所有節點進行meet通訊,以m1為例
      • 4、為每個master節點指派槽位範圍
  • 當master1分配完槽位後,可以看到下面資訊
      • 5、指定各個節點的主從關係
      • 6、檢視主節點關係及槽位資訊
      • 7、驗證redis Cluster訪問
      • 8、測試故障切換
    • 五、基於官方指令碼搭建redis cluster
      • 1、哨兵配置
      • 2、開啟叢集
      • 3、檢視主從狀態
      • 4、檢視叢集node對應關係
      • 5、驗證叢集狀態
      • 6、模擬master故障

一、什麼是redis Cluster叢集

Redis叢集是一種基於資料分片(sharding)的分散式快取和儲存系統。是一個由多個主從節點群組成的分散式服務叢集,它具有複製、高可用和分片特性。Redis叢集不需要sentinel哨兵也能完成節點移除和故障轉移的功能。需要將每個節點設定成叢集模式,這種叢集模式沒有中心節點,可水平擴充套件,據官方文件稱可以線性擴充套件到上萬個節點(官方推薦不超過1000個節點)。redis叢集的效能和高可用性均優於之前版本的哨兵模式,且叢集配置非常簡單。它是Redis為了應對大規模資料儲存、高併發訪問和高可用性需求而提供的原生叢集解決方案。

二、叢集架構圖

redis 3.0版本之後推出了無中心架構的redis cluster機制,在無中心的redis叢集當中,其每個節點儲存當前節點資料和整個叢集狀態,每個節點都和其他所有節點連線

image-20240418203643733

三、redis Cluster部署架構

1、測試環境

3臺伺服器,每臺伺服器啟動6379和6380兩個redis 服務例項

image-20240418210210806

2、生產環境

6臺伺服器,分別是三組master/slave,適用於生產環境

image-20240418211004677

四、原生命令手動部署Redis Cluster

  • 在所有節點安裝redis,並配置開啟cluster功能
  • 各個節點執行meet,實現所有節點的相互通訊
  • 為各個master 節點指派槽位範圍
  • 指定各個節點的主從關係

1、環境準備

image-20240418211859433

角色 主機名 IP地址
主節點 m1 192.168.112.10
主節點 m2 192.168.112.20
主節點 m3 192.168.112.30
從節點 s1 192.168.112.40
從節點 s2 192.168.112.50
從節點 s3 192.168.112.60

所有主機都安裝了redis

2、為所有節點啟用redis叢集支援

echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf
echo "masterauth centos" >> /apps/redis/etc/redis.conf
[root@m1 ~]# egrep -v "^#|^$" /apps/redis/etc/redis.conf
bind 0.0.0.0
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
supervised no
pidfile /apps/redis/run/redis_6379.pid
loglevel notice
logfile /apps/redis/log/redis_6379.log
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /apps/redis/data/
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
requirepass centos
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
cluster-enable yes
masterauth centos
#重啟redis
systemctl restart redis

3、執行meet操作實現互相通訊

在任意一節點上和其它所有節點進行meet通訊,以m1為例

[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.20 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.30 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.40 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.50 6379
OK
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster meet 192.168.112.60 6379
OK

#可以看到所有節點之間可以相互連線通訊,以m1為例

[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster nodes
a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 myself,master - 0 1713451004000 1 connected
69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 master - 0 1713451005000 5 connected
4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 master - 0 1713451005696 4 connected
efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713451006719 0 connected
c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 master - 0 1713451003656 2 connected
b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713451004000 3 connected

4、為每個master節點指派槽位範圍

由於沒有分配槽位,無法建立key

[root@m1 ~]# redis-cli -a centos --no-auth-warning set name misaki
(error) CLUSTERDOWN Hash slot not served

檢視當前狀態,以m1為例

[root@m1 ~]# redis-cli -a centos --no-auth-warning cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:495
cluster_stats_messages_pong_sent:469
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:969
cluster_stats_messages_ping_received:469
cluster_stats_messages_pong_received:500
cluster_stats_messages_received:969

為每個master 節點指派槽位範圍

[root@m1 ~]# vim addslots.sh
#!/bin/bash
HOST=$1
PORT=$2
START=$3
END=$4
PASS=centos
for slot in `seq ${START} ${END}`;do
    echo "slot: ${slot}"
    redis-cli -h ${HOST} -p ${PORT} -a ${PASS} --no-auth-warning cluster addslots ${slot}
done
* 為三個master分配槽位,共16364/3=5461.33333,平均每個master分配5461個槽位
[root@m1 ~]# bash addslots.sh 192.168.175.10 6379 0 5461

當master1分配完槽位後,可以看到下面資訊

[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster info
cluster_state:fail
cluster_slots_assigned:5462
cluster_slots_ok:5462
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:1
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:745
cluster_stats_messages_pong_sent:691
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:1441
cluster_stats_messages_ping_received:691
cluster_stats_messages_pong_received:750
cluster_stats_messages_received:1441

為剩下的master節點分配槽位

[root@m1 ~]# bash addslots.sh 192.168.112.20 6379 5462 10922
[root@m1 ~]# bash addslots.sh 192.168.112.30 6379 10923 16383

為所有的三個master分配完槽位後,驗證資訊

[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster nodes
a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 myself,master - 0 1713452132000 1 connected 0-5461
69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 master - 0 1713452137000 5 connected
4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 master - 0 1713452135000 4 connected
efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713452138634 0 connected 10923-16383
c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 master - 0 1713452137000 2 connected 5462-10922
b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713452137612 3 connected

5、指定各個節點的主從關係

透過上面的cluster nodes 檢視master的ID資訊,執行下面操作,將對應的slave 指定相應的master節點,實現三對主從節點

[root@m1 ~]# redis-cli -h 192.168.112.40 -a centos --no-auth-warning cluster replicate a8f0a6fa7de004d25996eefd652e4bf8fb995386
OK
[root@m1 ~]# redis-cli -h 192.168.112.50 -a centos --no-auth-warning cluster replicate c3ddc4db15222b0cdb2a64974a87b6bd731d8f84
OK
[root@m1 ~]# redis-cli -h 192.168.112.60 -a centos --no-auth-warning cluster replicate efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a
OK

6、檢視主節點關係及槽位資訊

[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning cluster slots
1) 1) (integer) 0
   2) (integer) 5461
   3) 1) "192.168.112.10"
      2) (integer) 6379
      3) "a8f0a6fa7de004d25996eefd652e4bf8fb995386"
   4) 1) "192.168.112.40"
      2) (integer) 6379
      3) "b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68"
2) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "192.168.112.30"
      2) (integer) 6379
      3) "efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a"
   4) 1) "192.168.112.60"
      2) (integer) 6379
      3) "69f2aee99fc15ddde526dad3cb55b6532c5c6b06"
3) 1) (integer) 5462
   2) (integer) 10922
   3) 1) "192.168.112.20"
      2) (integer) 6379
      3) "c3ddc4db15222b0cdb2a64974a87b6bd731d8f84"
   4) 1) "192.168.112.50"
      2) (integer) 6379
      3) "4f5fa3759a45025478aaa062c4290b4b478ab3ce"

7、驗證redis Cluster訪問

-c 表示以叢集方式連線

[root@m1 ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning set name kk
OK
[root@m1 ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning get name
"kk"
[root@m1 ~]# redis-cli -h 192.168.112.10 -a centos --no-auth-warning get name
(error) MOVED 5798 192.168.112.20:6379
[root@m1 ~]# redis-cli -h 192.168.112.20 -a centos --no-auth-warning get name
"kk"
[root@m1 ~]# redis-cli -h 192.168.112.30 -a centos --no-auth-warning get name
(error) MOVED 5798 192.168.112.20:6379

8、測試故障切換

模擬m1主機點當機

systemctl stop redis
[root@m2 ~]# redis-cli -c -h 192.168.112.20 -a centos --no-auth-warning cluster nodes
c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 192.168.112.20:6379@16379 myself,master - 0 1713489205000 2 connected 5462-10922
69f2aee99fc15ddde526dad3cb55b6532c5c6b06 192.168.112.60:6379@16379 slave efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 0 1713489207000 5 connected
b7d7165c5d537fad91bdcd3f7ac2eb73ad52de68 192.168.112.40:6379@16379 master - 0 1713489207436 6 connected 0-5461
4f5fa3759a45025478aaa062c4290b4b478ab3ce 192.168.112.50:6379@16379 slave c3ddc4db15222b0cdb2a64974a87b6bd731d8f84 0 1713489206000 4 connected
a8f0a6fa7de004d25996eefd652e4bf8fb995386 192.168.112.10:6379@16379 master,fail - 1713489156541 1713489154401 1 disconnected
efdfd5203fbfbba5030f0eb19e4ed2ebc62ab97a 192.168.112.30:6379@16379 master - 0 1713489208451 0 connected 10923-16383

image-20240419091646866

可以看到s1已經由slave轉變為master,過程大概在5-10s

五、基於官方指令碼搭建redis cluster

建立 redis cluster叢集的環境準備:

  • 每個redis 節點採用相同的硬體配置、相同的密碼、相同的redis版本
  • 所有redis伺服器必須沒有任何資料
  • 準備6臺主機,開啟cluster配置

1、哨兵配置

echo "masterauth centos" >> /apps/redis/etc/redis.conf
echo "cluster-enabled yes" >> /apps/redis/etc/redis.conf

2、開啟叢集

systemctl restart redis

# redis-cli --cluster-replicas 1 表示每個master對應一個slave節點

任意一臺機器上執行就行

[root@localhost ~]# redis-cli -a centos --no-auth-warning --cluster create \
> 192.168.112.10:6379 192.168.112.20:6379 192.168.112.30:6379 \
> 192.168.112.40:6379 192.168.112.50:6379 192.168.112.60:6379 \
> --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.112.50:6379 to 192.168.112.10:6379
Adding replica 192.168.112.60:6379 to 192.168.112.20:6379
Adding replica 192.168.112.40:6379 to 192.168.112.30:6379
M: c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379
   slots:[0-5460] (5461 slots) master
M: 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379
   slots:[5461-10922] (5462 slots) master
M: e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379
   slots:[10923-16383] (5461 slots) master
S: 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379
   replicates e621db96b845eb7818eb01a37fccdc7796f8d7de
S: bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379
   replicates c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7
S: 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379
   replicates 43906d49ffe4be7085b4d37f52c50ac7bb4df751
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 192.168.112.10:6379)
M: c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379
   slots: (0 slots) slave
   replicates c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7
M: e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379
   slots: (0 slots) slave
   replicates e621db96b845eb7818eb01a37fccdc7796f8d7de
S: 510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379
   slots: (0 slots) slave
   replicates 43906d49ffe4be7085b4d37f52c50ac7bb4df751
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

image-20240419220851960

3、檢視主從狀態

[root@localhost ~]# redis-cli -c -h 192.168.112.10 -a centos --no-auth-warning info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.112.50,port=6379,state=online,offset=630,lag=0
master_replid:54b3983a20bf4824a13d2699a9775fe3cb85ef0b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:630
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:630
[root@localhost ~]# redis-cli -c -h 192.168.112.50 -a centos --no-auth-warning info replication
# Replication
role:slave
master_host:192.168.112.10
master_port:6379
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:756
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:54b3983a20bf4824a13d2699a9775fe3cb85ef0b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:756
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:756

image-20240419221522452

4、檢視叢集node對應關係

[root@localhost ~]# redis-cli -a centos --no-auth-warning cluster nodes
c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379@16379 myself,master - 0 1713536198000 1 connected 0-5460
bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379@16379 slave c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 0 1713536200000 5 connected
e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379@16379 master - 0 1713536202879 3 connected 10923-16383
43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379@16379 master - 0 1713536201862 2 connected 5461-10922
5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379@16379 slave e621db96b845eb7818eb01a37fccdc7796f8d7de 0 1713536201000 3 connected
510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379@16379 slave 43906d49ffe4be7085b4d37f52c50ac7bb4df751 0 1713536200845 6 connected

image-20240419221819697

5、驗證叢集狀態

[root@localhost ~]# redis-cli -a centos --no-auth-warning cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6	#節點數
cluster_size:3	#3個叢集
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:895
cluster_stats_messages_pong_sent:791
cluster_stats_messages_sent:1686
cluster_stats_messages_ping_received:786
cluster_stats_messages_pong_received:895
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1686

6、模擬master故障

  • 對應的slave節點自動提升為新master
  • 模擬master1節點出故障,需要相應的數秒故障轉移時間

master1關閉redis服務

systemctl stop redis

master2上檢視叢集資訊

[root@localhost ~]# redis-cli -h 192.168.112.20 -a centos --no-auth-warning cluster nodes
e621db96b845eb7818eb01a37fccdc7796f8d7de 192.168.112.30:6379@16379 master - 0 1713536909635 3 connected 10923-16383
510e96a92a7ec1ab44ed6b2e1b268f8a2f314ba6 192.168.112.60:6379@16379 slave 43906d49ffe4be7085b4d37f52c50ac7bb4df751 0 1713536907592 2 connected
43906d49ffe4be7085b4d37f52c50ac7bb4df751 192.168.112.20:6379@16379 myself,master - 0 1713536908000 2 connected 5461-10922
c1fa83ac2ea788f33d222b0ce5fd0682b2ae76c7 192.168.112.10:6379@16379 slave,fail bb81813b153345f0406cfe88c1899e11e2e7a84f 1713536880567 1713536878000 7 disconnected
bb81813b153345f0406cfe88c1899e11e2e7a84f 192.168.112.50:6379@16379 master - 0 1713536908615 7 connected 0-5460
5a31f10ddef04202bc10f5d23a62dd3731bf826c 192.168.112.40:6379@16379 slave e621db96b845eb7818eb01a37fccdc7796f8d7de 0 1713536908000 4 connected

image-20240419223100679

可以看到master1對應的slave3已經提升為master了

相關文章