準備

CentOS：7

Redis：4.0.11

測試在3臺虛擬機器上進行，ip 分別為：

• 172.16.105.128
• 172.16.105.129
• 172.16.105.130

主從複製

1.分別在三臺虛擬機器上安裝 redis，單機安裝部署參考(CentOS 7安裝Redis)

2.在從伺服器( 172.16.105.129 、 172.16.105.130 )配置檔案 redis.conf 中新增主從配置

# slaveof 主伺服器IP 埠
slaveof 172.16.105.128 6379

3.啟動 172.16.105.128 、 172.16.105.129 、 172.16.105.130 三個節點服務。

4.在主從節點上登入客戶端，使用命令 info replication 可以看到相關節點資訊。

主節點

127.0.0.1:6379> info replication
# Replication
role:master # 角色
connected_slaves:2 # 連線從伺服器數
slave0:ip=172.16.105.129,port=6379,state=online,offset=84,lag=0 # 從伺服器列表資訊
slave1:ip=172.16.105.130,port=6379,state=online,offset=84,lag=1
master_replid:c6451f091badfc49b9c001a27e6aae7086db08bb
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:84 # 增量複製offset
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:84

從節點

127.0.0.1:6379> info replication
# Replication
role:slave
master_host:172.16.105.128
master_port:6379
master_link_status:up
master_last_io_seconds_ago:2
master_sync_in_progress:0
slave_repl_offset:406
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:c6451f091badfc49b9c001a27e6aae7086db08bb
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:406
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:406

5.在主伺服器上執行 set hello world 命令，可以看到相應同步操作執行。

Slave 172.16.105.129:6379 asks for synchronization
2587:M 09 Nov 03:51:13.590 * Partial resynchronization not accepted: Replication ID mismatch (Slave asked for '4c65ba0873afc39788457866070142cb456fc581', my replication IDs are 'c0199fe05490d9fd6c3b49d78a2ab63571f37abf' and '0000000000000000000000000000000000000000')
2587:M 09 Nov 03:51:13.590 * Starting BGSAVE for SYNC with target: disk
2587:M 09 Nov 03:51:13.590 * Background saving started by pid 2595
2595:C 09 Nov 03:51:13.592 * DB saved on disk
2595:C 09 Nov 03:51:13.592 * RDB: 0 MB of memory used by copy-on-write
2587:M 09 Nov 03:51:13.656 * Background saving terminated with success
2587:M 09 Nov 03:51:13.657 * Synchronization with slave 172.16.105.129:6379 succeeded
2587:M 09 Nov 03:51:40.071 * Slave 172.16.105.130:6379 asks for synchronization
2587:M 09 Nov 03:51:40.071 * Partial resynchronization not accepted: Replication ID mismatch (Slave asked for '4c65ba0873afc39788457866070142cb456fc581', my replication IDs are 'c6451f091badfc49b9c001a27e6aae7086db08bb' and '0000000000000000000000000000000000000000')
2587:M 09 Nov 03:51:40.071 * Starting BGSAVE for SYNC with target: disk
2587:M 09 Nov 03:51:40.071 * Background saving started by pid 2596
2596:C 09 Nov 03:51:40.073 * DB saved on disk
2596:C 09 Nov 03:51:40.074 * RDB: 0 MB of memory used by copy-on-write
2587:M 09 Nov 03:51:40.138 * Background saving terminated with success
2587:M 09 Nov 03:51:40.139 * Synchronization with slave 172.16.105.130:6379 succeeded

5.在從伺服器客戶端執行 get hello 命令，可以返回在主伺服器上設定的資訊。

127.0.0.1:6379> get hello
"world"

哨兵機制

1.備份哨兵配置檔案

[root@localhost redis-4.0.11]# cp sentinel.conf sentinel.conf.bak

2.設定哨兵配置檔案

port 26379
# sentinel monitor 自定義master名字 master節點IP 埠 最低通過票數(具體需要幾個哨兵節點統一才行)
sentinel monitor mymaster 172.16.105.128 6379 1
sentinel down-after-milliseconds mymaster 5000 # 設定mymaster超過指定時間午相應就認為sdown
sentinel failover-timeout mymaster 15000 # 設定mymaster超過指定時間沒活過來，則啟動failover，從剩餘slave中選舉master

3.在主伺服器上啟動哨兵監控，可以看到相關資訊

方式

• ./redis-sentinel ../sentinel.conf
• ./redis-server ../sentinel.conf --sentinel

2632:X 09 Nov 04:27:36.922 # Sentinel ID is af1657d3b6dce9213a422d474dd52464ee287484
2632:X 09 Nov 04:27:36.922 # +monitor master mymaster 172.16.105.128 6379 quorum 1
2632:X 09 Nov 04:27:36.923 * +slave slave 172.16.105.129:6379 172.16.105.129 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:27:36.924 * +slave slave 172.16.105.130:6379 172.16.105.130 6379 @ mymaster 172.16.105.128 6379

4.停止 master 上的服務，檢視哨兵服務日誌資訊

2632:X 09 Nov 04:30:03.657 # +sdown master mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:03.657 # +odown master mymaster 172.16.105.128 6379 #quorum 1/1
2632:X 09 Nov 04:30:03.657 # +new-epoch 1
2632:X 09 Nov 04:30:03.657 # +try-failover master mymaster 172.16.105.128 6379 # 哨兵開始進行故障恢復
2632:X 09 Nov 04:30:03.659 # +vote-for-leader af1657d3b6dce9213a422d474dd52464ee287484 1
2632:X 09 Nov 04:30:03.659 # +elected-leader master mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:03.659 # +failover-state-select-slave master mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:03.731 # +selected-slave slave 172.16.105.130:6379 172.16.105.130 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:03.731 * +failover-state-send-slaveof-noone slave 172.16.105.130:6379 172.16.105.130 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:03.832 * +failover-state-wait-promotion slave 172.16.105.130:6379 172.16.105.130 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:04.712 # +promoted-slave slave 172.16.105.130:6379 172.16.105.130 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:04.712 # +failover-state-reconf-slaves master mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:04.767 * +slave-reconf-sent slave 172.16.105.129:6379 172.16.105.129 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:05.771 * +slave-reconf-inprog slave 172.16.105.129:6379 172.16.105.129 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:05.771 * +slave-reconf-done slave 172.16.105.129:6379 172.16.105.129 6379 @ mymaster 172.16.105.128 6379
2632:X 09 Nov 04:30:05.841 # +failover-end master mymaster 172.16.105.128 6379 # 哨兵完成故障恢復
2632:X 09 Nov 04:30:05.841 # +switch-master mymaster 172.16.105.128 6379 172.16.105.130 6379
2632:X 09 Nov 04:30:05.842 * +slave slave 172.16.105.129:6379 172.16.105.129 6379 @ mymaster 172.16.105.130 6379
2632:X 09 Nov 04:30:05.842 * +slave slave 172.16.105.128:6379 172.16.105.128 6379 @ mymaster 172.16.105.130 6379
2632:X 09 Nov 04:30:10.874 # +sdown slave 172.16.105.128:6379 172.16.105.128 6379 @ mymaster 172.16.105.130 6379

Redis Cluster

叢集搭建

1.分別在3臺虛擬機器上各安裝配置1主1從，共6個節點，便於區分採用不同埠

• 172.16.105.128 節點埠：7001、7002
• 172.16.105.129 節點埠：7003、7004
• 172.16.105.130 節點埠：7005、7006

2.依次修改各例項配置檔案 redis.conf

port 7001 # 根據上面修改為指定埠
bind 0.0.0.0 # 設定不限制訪問來源IP
pidfile /var/run/redis_7001.pid # pidfile檔案分別對應7001,7002,7003,7004,7005,7006
cluster-enabled yes # 開啟叢集模式
cluster-config-file nodes.conf # 設定叢集配置檔案
cluster-node-timeout 5000 # 叢集節點超時時間
appendonly yes

3.分別啟動全部6個例項，因為叢集不存在配置檔案 nodes.conf ，所以會為每個例項分配一個唯一的節點id

[root@localhost redis-7001]# ./src/redis-server ./redis.conf &
7684:M 11 Nov 02:29:59.975 * No cluster configuration found, I'm 00bee510e285e1951863d15e646a5fcccd10e28f

4.安裝 ruby 環境（ redis-trib 依賴於 redis gem ， gem依賴於 ruby ）

[root@localhost redis-7001]# yum install ruby
[root@localhost redis-7001]# yum install rubygems

5.gem 安裝 redis

[root@localhost redis-7001]# gem install redis
Fetching: redis-4.0.3.gem (100%)
ERROR:  Error installing redis:
	redis requires Ruby version >= 2.2.2.

CentOS 7 yum庫 ruby 版本只支援到 2.0.0 ，而 gem 安裝 redis 需要 2.2.2 版本以上。

解決方法：採用rvm更新ruby

• 安裝 rvm

[root@localhost redis-7001]# gpg2 --keyserver hkp://keys.gnupg.net --recv-keys D39DC0E3
[root@localhost redis-7001]# curl -L get.rvm.io | bash -s stable

[root@localhost redis-7001]# find / -name rvm -print
/usr/local/rvm
/usr/local/rvm/src/rvm
/usr/local/rvm/src/rvm/bin/rvm
/usr/local/rvm/src/rvm/lib/rvm
/usr/local/rvm/src/rvm/scripts/rvm
/usr/local/rvm/bin/rvm
/usr/local/rvm/lib/rvm
/usr/local/rvm/scripts/rvm

[root@localhost redis-7001]# source /usr/local/rvm/scripts/rvm

• 檢視 rvm 庫已經 ruby 版本列表

[root@localhost redis-7001]# rvm list known
# MRI Rubies
[ruby-]1.8.6[-p420]
[ruby-]1.8.7[-head] # security released on head
[ruby-]1.9.1[-p431]
[ruby-]1.9.2[-p330]
[ruby-]1.9.3[-p551]
[ruby-]2.0.0[-p648]
[ruby-]2.1[.10]
[ruby-]2.2[.10]
[ruby-]2.3[.7]
[ruby-]2.4[.4]
[ruby-]2.5[.1]
[ruby-]2.6[.0-preview2]
ruby-head

• 選擇一個較高版本(2.5.1)進行安裝

[root@localhost redis-7001]# rvm install 2.5.1

• 使用一個 ruby 版本

[root@localhost redis-7001]# rvm use 2.5.1
Using /usr/local/rvm/gems/ruby-2.5.1
[root@localhost redis-7001]# ruby --version
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]

可選操作

rvm use 2.5.1 --default # 設定ruby預設版本
rvm remove 2.0.0 # 解除安裝指定版本

再次執行 gem 安裝 redis ，成功安裝

[root@localhost redis-7001]# gem install redis
Fetching: redis-4.0.3.gem (100%)
Successfully installed redis-4.0.3
Parsing documentation for redis-4.0.3
Installing ri documentation for redis-4.0.3
Done installing documentation for redis after 0 seconds
1 gem installed

6.使用 redis-trib.rb 建立叢集（一個 master 主節點擁有一個 slave 從節點）

• replicas ：設定每個 master 主節點擁有 slave 從節點個數

[root@localhost redis-7001]# ./src/redis-trib.rb create --replicas 1 172.16.105.128:7001 \172.16.105.128:7002 172.16.105.129:7003 172.16.105.129:7004 172.16.105.130:7005 172.16.105.130:7006
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.16.105.128:7001
172.16.105.129:7003
172.16.105.130:7005
Adding replica 172.16.105.129:7004 to 172.16.105.128:7001
Adding replica 172.16.105.130:7006 to 172.16.105.129:7003
Adding replica 172.16.105.128:7002 to 172.16.105.130:7005
M: 00bee510e285e1951863d15e646a5fcccd10e28f 172.16.105.128:7001
   slots:0-5460 (5461 slots) master
S: c85ebc85fee43048b8135ac5b73c3337ef6b9dd0 172.16.105.128:7002
   replicates 93ba46762a030b853ba3a81634fc79e92e17d6d8
M: 162cf6816ffdd75f22af63359a1da8a2eb7725a2 172.16.105.129:7003
   slots:5461-10922 (5462 slots) master
S: e9ebec2d60087531517e08ae102d66e3d8db39eb 172.16.105.129:7004
   replicates 00bee510e285e1951863d15e646a5fcccd10e28f
M: 93ba46762a030b853ba3a81634fc79e92e17d6d8 172.16.105.130:7005
   slots:10923-16383 (5461 slots) master
S: 633b4c938e3773f90abc11261452d6638f31c7ed 172.16.105.130:7006
   replicates 162cf6816ffdd75f22af63359a1da8a2eb7725a2
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
7743:M 11 Nov 04:07:33.471 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
7748:M 11 Nov 04:07:33.471 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH
>>> Sending CLUSTER MEET messages to join the cluster
7743:M 11 Nov 04:07:33.486 # IP address for this node updated to 172.16.105.128
7748:M 11 Nov 04:07:33.650 # IP address for this node updated to 172.16.105.128
Waiting for the cluster to join...
7748:S 11 Nov 04:07:37.498 * Before turning into a slave, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
7748:S 11 Nov 04:07:37.498 # Cluster state changed: ok
>>> Performing Cluster Check (using node 172.16.105.128:7001)
M: 00bee510e285e1951863d15e646a5fcccd10e28f 172.16.105.128:7001
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 162cf6816ffdd75f22af63359a1da8a2eb7725a2 172.16.105.129:7003
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: 93ba46762a030b853ba3a81634fc79e92e17d6d8 172.16.105.130:7005
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 633b4c938e3773f90abc11261452d6638f31c7ed 172.16.105.130:7006
   slots: (0 slots) slave
   replicates 162cf6816ffdd75f22af63359a1da8a2eb7725a2
S: c85ebc85fee43048b8135ac5b73c3337ef6b9dd0 172.16.105.128:7002
   slots: (0 slots) slave
   replicates 93ba46762a030b853ba3a81634fc79e92e17d6d8
S: e9ebec2d60087531517e08ae102d66e3d8db39eb 172.16.105.129:7004
   slots: (0 slots) slave
   replicates 00bee510e285e1951863d15e646a5fcccd10e28f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

叢集建立完成後，可以看到分配的三個 master 節點和三個 slave 節點，以及對應的槽位區間。

叢集測試

客戶端連線叢集需要指定 -c 引數

[root@localhost redis-7001]# ./src/redis-cli -c -p 7001

在 7001 主節點設定 set name clusterTest ，根據 key 值計算槽位分佈在節點 7003(5461-10922)

[root@localhost redis-7001]# ./src/redis-cli -c -p 7001
127.0.0.1:7001> set name clusterTest
-> Redirected to slot [5798] located at 172.16.105.129:7003
OK
172.16.105.129:7003> get name
"clusterTest"
172.16.105.129:7003>

在節點 7006 獲取 name 值，自動跳轉到節點 7003 獲取值。

[root@localhost redis-7006]# ./src/redis-cli -c -p 7006
127.0.0.1:7006> get name
-> Redirected to slot [5798] located at 172.16.105.129:7003
"clusterTest"
172.16.105.129:7003>

叢集節點選舉

殺掉節點 7005 ，檢視叢集變化

[root@localhost redis-7006]# ps -ef | grep redis
root       7768   1211  0 03:17 pts/0    00:00:07 ./src/redis-server 0.0.0.0:7005 [cluster]
root       7773   1211  0 03:18 pts/0    00:00:07 ./src/redis-server 0.0.0.0:7006 [cluster]
root       7848   7831  0 04:50 pts/1    00:00:00 grep --color=auto redis
[root@localhost redis-7006]# kill 7768

可以看到叢集連線不上節點 7005 ，節點 7002 角色由 slave 轉換為 master 。

[root@localhost redis-7001]# ./src/redis-trib.rb check 172.16.105.130:7005
[ERR] Sorry, can't connect to node 172.16.105.130:7005
[root@localhost redis-7001]# ./src/redis-trib.rb check 172.16.105.130:7006
>>> Performing Cluster Check (using node 172.16.105.130:7006)
S: 633b4c938e3773f90abc11261452d6638f31c7ed 172.16.105.130:7006
   slots: (0 slots) slave
   replicates 162cf6816ffdd75f22af63359a1da8a2eb7725a2
S: e9ebec2d60087531517e08ae102d66e3d8db39eb 172.16.105.129:7004
   slots: (0 slots) slave
   replicates 00bee510e285e1951863d15e646a5fcccd10e28f
M: 00bee510e285e1951863d15e646a5fcccd10e28f 172.16.105.128:7001
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: c85ebc85fee43048b8135ac5b73c3337ef6b9dd0 172.16.105.128:7002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
M: 162cf6816ffdd75f22af63359a1da8a2eb7725a2 172.16.105.129:7003
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

參考

Redis Cluster Tutorial
Redis Cluster分片叢集搭建
RVM安裝指定版本Ruby