一、說明

redis 3.0叢集功能出來已經有一段時間了，目前最新穩定版是3.0.5，我瞭解到已經有很多網際網路公司在生產環境使用，比如唯品會、美團等等，剛好公司有個新專案，預估的量單機redis無法滿足，開發又不想在程式碼層面做拆分，所以就推薦他們嘗試一下redis叢集，下面做了一些相關筆記，以備後用

二、環境

1、redis節點

2、redis版本

三、安裝配置

1、安裝redis

2、安裝ruby及ruby的redis模組

3、核心調優

4、建立目錄

5、撰寫redis配置檔案（cp配置檔案注意修改埠）

6、啟動服務

7、初始化叢集

節點角色由順序決定,先master之後是slave，本文中6300是master，6301是slave

redis-trib.rb create –replicas 1 10.10.2.70:6300 10.10.2.71:6300 10.10.2.85:6300 10.10.2.70:6301 10.10.2.71:6301 10.10.2.85:6301

8、檢視叢集狀態

PS：
redis-trib.rb是一個ruby工具，封裝了redis叢集的一些命令，用這個工具操作叢集非常方便，比如上面初始化叢集，檢視叢集狀態，還有新增、刪除節點，遷移slot等等功能

四、redis叢集維護

A、場景1
線上的叢集已經有瓶頸，叢集需要擴容，比如我們已經準備了一主一從（10.10.2.85:6302、10.10.2.85:6303），如下：

1、新增一個主節點

10.10.2.85:6302是要加的新節點，10.10.2.70:6300是叢集中已存在的任意節點

2、給主節點新增從節點
[root@yw_0_0 ~]# redis-trib.rb add-node –slave –master-id 5ef18f95f75756891aa948ea1f200044f1d3947c 10.10.2.85:6303 10.10.2.70:6300

Adding node 10.10.2.85:6303 to cluster 10.10.2.70:6300

Connecting to node 10.10.2.70:6300: OK
Connecting to node 10.10.2.85:6300: OK
Connecting to node 10.10.2.85:6302: OK
Connecting to node 10.10.2.85:6301: OK
Connecting to node 10.10.2.71:6300: OK
Connecting to node 10.10.2.70:6301: OK
Connecting to node 10.10.2.71:6301: OK

Performing Cluster Check (using node 10.10.2.70:6300)

S: cd1f2c1f348bb4359337e7462c1e21dc82f1551b 10.10.2.70:6300
slots: (0 slots) slave
replicates 85412cf3d8e69354115fc0991f470b32b9213cd7
M: 6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013 10.10.2.85:6300
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 5ef18f95f75756891aa948ea1f200044f1d3947c 10.10.2.85:6302
slots: (0 slots) master
0 additional replica(s)
S: a74642c0fbc98f921be477eabcdd22eccd89891f 10.10.2.85:6301
slots: (0 slots) slave
replicates 2568dbd91fffa16ff93ea8db19275fd7ec8af41a
M: 2568dbd91fffa16ff93ea8db19275fd7ec8af41a 10.10.2.71:6300
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 85412cf3d8e69354115fc0991f470b32b9213cd7 10.10.2.70:6301
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 22d2dec483824b84571a60e8c037fff957615552 10.10.2.71:6301
slots: (0 slots) slave
replicates 6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013
[OK] All nodes agree about slots configuration.

Check for open slots…
Check slots coverage…

[OK] All 16384 slots covered.
Connecting to node 10.10.2.85:6303: OK

Send CLUSTER MEET to node 10.10.2.85:6303 to make it join the cluster.

Waiting for the cluster to join.

Configure node as replica of 10.10.2.85:6302.

[OK] New node added correctly.

–slave 指定要加的是從節點，–master-id 指定這個從節點的主節點ID，10.10.2.85:6303是需要新加的從節點，10.10.2.70:6300是叢集已存在的任意節點

3、遷移一些slot給新節點
[root@yw_0_0 ~]# redis-trib.rb reshard 10.10.2.70:6300
Connecting to node 10.10.2.70:6300: OK
Connecting to node 10.10.2.85:6300: OK
Connecting to node 10.10.2.85:6303: OK
Connecting to node 10.10.2.85:6302: OK
Connecting to node 10.10.2.85:6301: OK
Connecting to node 10.10.2.71:6300: OK
Connecting to node 10.10.2.70:6301: OK
Connecting to node 10.10.2.71:6301: OK

Performing Cluster Check (using node 10.10.2.70:6300)

S: cd1f2c1f348bb4359337e7462c1e21dc82f1551b 10.10.2.70:6300
slots: (0 slots) slave
replicates 85412cf3d8e69354115fc0991f470b32b9213cd7
M: 6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013 10.10.2.85:6300
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: fc90d090fae909fd4f962752941c039d081d3854 10.10.2.85:6303
slots: (0 slots) slave
replicates 5ef18f95f75756891aa948ea1f200044f1d3947c
M: 5ef18f95f75756891aa948ea1f200044f1d3947c 10.10.2.85:6302
slots: (0 slots) master
1 additional replica(s)
S: a74642c0fbc98f921be477eabcdd22eccd89891f 10.10.2.85:6301
slots: (0 slots) slave
replicates 2568dbd91fffa16ff93ea8db19275fd7ec8af41a
M: 2568dbd91fffa16ff93ea8db19275fd7ec8af41a 10.10.2.71:6300
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 85412cf3d8e69354115fc0991f470b32b9213cd7 10.10.2.70:6301
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 22d2dec483824b84571a60e8c037fff957615552 10.10.2.71:6301
slots: (0 slots) slave
replicates 6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013
[OK] All nodes agree about slots configuration.

Check for open slots…
Check slots coverage…

[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 3000 #設定需要把3000個slot做移動
What is the receiving node ID? 5ef18f95f75756891aa948ea1f200044f1d3947c #設定接收這3000個slot的節點ID，也就是剛才新加的10.10.2.85:6302的ID
Please enter all the source node IDs.
Type `all` to use all the nodes as source nodes for the hash slots.
Type `done` once you entered all the source nodes IDs.
Source node #1:85412cf3d8e69354115fc0991f470b32b9213cd7 #設定這3000slot的來源ID，這裡我從叢集之前的3個節點分別去取一部分slot
Source node #2:6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013 #設定這3000slot的來源ID，這裡我從叢集之前的3個節點分別去取一部分slot
Source node #3:2568dbd91fffa16ff93ea8db19275fd7ec8af41a #設定這3000slot的來源ID，這裡我從叢集之前的3個節點分別去取一部分slot
Source node #4:done #輸入done開始做一些初始化操作
此處省略
Do you want to proceed with the proposed reshard plan (yes)? yes 輸入yes確認開始遷移slot

B、場景二

上面的例子是叢集擴容，相對的，由於各種原因叢集可能也需要縮容，下面的例子把上文擴容的節點下線，步驟如下：

1、遷移這個節點的slot到其他節點（有slot的節點是不可以直接下線的）
[root@yw_0_0 ~]# redis-trib.rb reshard 10.10.2.70:6300
Connecting to node 10.10.2.70:6300: OK
Connecting to node 10.10.2.85:6300: OK
Connecting to node 10.10.2.85:6303: OK
Connecting to node 10.10.2.85:6302: OK
Connecting to node 10.10.2.85:6301: OK
Connecting to node 10.10.2.71:6300: OK
Connecting to node 10.10.2.70:6301: OK
Connecting to node 10.10.2.71:6301: OK

Performing Cluster Check (using node 10.10.2.70:6300)

S: cd1f2c1f348bb4359337e7462c1e21dc82f1551b 10.10.2.70:6300
slots: (0 slots) slave
replicates 85412cf3d8e69354115fc0991f470b32b9213cd7
M: 6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013 10.10.2.85:6300
slots:999-5460 (4462 slots) master
1 additional replica(s)
S: fc90d090fae909fd4f962752941c039d081d3854 10.10.2.85:6303
slots: (0 slots) slave
replicates 5ef18f95f75756891aa948ea1f200044f1d3947c
M: 5ef18f95f75756891aa948ea1f200044f1d3947c 10.10.2.85:6302
slots:0-998,5461-6461,10923-11921 (2999 slots) master
1 additional replica(s)
S: a74642c0fbc98f921be477eabcdd22eccd89891f 10.10.2.85:6301
slots: (0 slots) slave
replicates 2568dbd91fffa16ff93ea8db19275fd7ec8af41a
M: 2568dbd91fffa16ff93ea8db19275fd7ec8af41a 10.10.2.71:6300
slots:6462-10922 (4461 slots) master
1 additional replica(s)
M: 85412cf3d8e69354115fc0991f470b32b9213cd7 10.10.2.70:6301
slots:11922-16383 (4462 slots) master
1 additional replica(s)
S: 22d2dec483824b84571a60e8c037fff957615552 10.10.2.71:6301
slots: (0 slots) slave
replicates 6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013
[OK] All nodes agree about slots configuration.

Check for open slots…
Check slots coverage…

[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 3000 #上文給這個節點遷入了3000個slot，所以這裡還選擇遷出3000個slot
What is the receiving node ID? 85412cf3d8e69354115fc0991f470b32b9213cd7 #接收這3000slot節點的主ID
Please enter all the source node IDs.
Type `all` to use all the nodes as source nodes for the hash slots.
Type `done` once you entered all the source nodes IDs.
Source node #1:5ef18f95f75756891aa948ea1f200044f1d3947c #要下線節點的主ID
Source node #4:done
此處省略
Do you want to proceed with the proposed reshard plan (yes)?yes

2、然後檢視10.10.2.85:6302這個maser上已經沒有slot了
10.10.2.71:6300> cluster nodes
85412cf3d8e69354115fc0991f470b32b9213cd7 10.10.2.70:6301 master – 0 1445853133399 12 connected 0-999 6462-7460 10923-16383
22d2dec483824b84571a60e8c037fff957615552 10.10.2.71:6301 slave 6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013 0 1445853132898 10 connected
6bea6afa2ee8dfb0cc3c96f804eb3fa77ce98013 10.10.2.85:6300 master – 0 1445853134400 10 connected 1000-5461
2568dbd91fffa16ff93ea8db19275fd7ec8af41a 10.10.2.71:6300 myself,master – 0 0 11 connected 5462-6461 7461-10922
cd1f2c1f348bb4359337e7462c1e21dc82f1551b 10.10.2.70:6300 slave 85412cf3d8e69354115fc0991f470b32b9213cd7 0 1445853131395 12 connected
fc90d090fae909fd4f962752941c039d081d3854 10.10.2.85:6303 slave 5ef18f95f75756891aa948ea1f200044f1d3947c 0 1445853133899 8 connected
a74642c0fbc98f921be477eabcdd22eccd89891f 10.10.2.85:6301 slave 2568dbd91fffa16ff93ea8db19275fd7ec8af41a 0 1445853129394 11 connected
5ef18f95f75756891aa948ea1f200044f1d3947c 10.10.2.85:6302 master – 0 1445853132397 8 connected

3、下線slave節點
[root@yw_0_0 ~]# redis-trib.rb del-node 10.10.2.85:6303 fc90d090fae909fd4f962752941c039d081d3854

Removing node fc90d090fae909fd4f962752941c039d081d3854 from cluster 10.10.2.85:6303

Connecting to node 10.10.2.85:6303: OK
Connecting to node 10.10.2.85:6301: OK
Connecting to node 10.10.2.85:6302: OK
Connecting to node 10.10.2.85:6300: OK
Connecting to node 10.10.2.70:6300: OK
Connecting to node 10.10.2.71:6301: OK
Connecting to node 10.10.2.70:6301: OK
Connecting to node 10.10.2.71:6300: OK

Sending CLUSTER FORGET messages to the cluster…
SHUTDOWN the node.

4、下線master節點

C、場景三
叢集中一個節點的master掛掉，從節點提升為主節點，還沒有來的急給這個新的主節點加從節點，這個新的主節點就又掛掉了，那麼叢集中這個節點就徹底不可以用了，為了解決這個問題，我們至少保證每個節點的maser下面有兩個以上的從節點，這樣一來，需要的記憶體資源或者伺服器資源就翻倍了，有沒有一個折中的方法呢，答案是肯定的，還節點上文配置檔案中的cluster-migration-barrier引數不，我們只需要給叢集中其中一個節點的master掛多個從庫，當其他節點的master下沒有可用的從庫時，有多個從庫的master會割讓一個slave給他，保證整個叢集的可用性

1、給10.10.2.70:6300 10.10.2.70:6301 這組節點下面加一個從庫10.10.2.85:6302
[root@yw_0_0 ~]# redis-trib.rb add-node –slave –master-id cd1f2c1f348bb4359337e7462c1e21dc82f1551b 10.10.2.85:6302 10.10.2.70:6300

Adding node 10.10.2.85:6302 to cluster 10.10.2.70:6300

Connecting to node 10.10.2.70:6300: OK
Connecting to node 10.10.2.85:6300: OK
Connecting to node 10.10.2.71:6300: OK
Connecting to node 10.10.2.70:6301: OK
Connecting to node 10.10.2.85:6301: OK
Connecting to node 10.10.2.71:6301: OK

Performing Cluster Check (using node 10.10.2.70:6300)

M: cd1f2c1f348bb4359337e7462c1e21dc82f1551b 10.10.2.70:6300
slots:3000-5461,6462-7460,10923-16383 (8922 slots) master
1 additional replica(s)
M: e36cdef7a26ed59e8d9db2cf1dbc1997bfc9dfde 10.10.2.85:6300
slots:0-2999 (3000 slots) master
1 additional replica(s)
M: 2568dbd91fffa16ff93ea8db19275fd7ec8af41a 10.10.2.71:6300
slots:5462-6461,7461-10922 (4462 slots) master
1 additional replica(s)
S: 85412cf3d8e69354115fc0991f470b32b9213cd7 10.10.2.70:6301
slots: (0 slots) slave
replicates cd1f2c1f348bb4359337e7462c1e21dc82f1551b
S: 89fcc4994a99ed2fe9bbb908c58dfda2cf31e7d2 10.10.2.85:6301
slots: (0 slots) slave
replicates e36cdef7a26ed59e8d9db2cf1dbc1997bfc9dfde
S: 1f3ea36eacbe005a4b9ac52aeef6d83337dac051 10.10.2.71:6301
slots: (0 slots) slave
replicates 2568dbd91fffa16ff93ea8db19275fd7ec8af41a
[OK] All nodes agree about slots configuration.

Check for open slots…
Check slots coverage…

[OK] All 16384 slots covered.
Connecting to node 10.10.2.85:6302: OK

Send CLUSTER MEET to node 10.10.2.85:6302 to make it join the cluster.

Waiting for the cluster to join.

Configure node as replica of 10.10.2.70:6300.

[OK] New node added correctly.

2、把10.10.2.71:6300 10.10.2.71:6301這組的從節點停掉
redis-cli -h 10.10.2.71 -p 6301 shutdown

3、檢視10.10.2.85:6302這個節點是否成為10.10.2.71：6300的從庫
10.10.2.71:6300> CLUSTER nodes
85412cf3d8e69354115fc0991f470b32b9213cd7 10.10.2.70:6301 slave cd1f2c1f348bb4359337e7462c1e21dc82f1551b 0 1445911596844 17 connected
89fcc4994a99ed2fe9bbb908c58dfda2cf31e7d2 10.10.2.85:6301 slave e36cdef7a26ed59e8d9db2cf1dbc1997bfc9dfde 0 1445911594841 20 connected
2568dbd91fffa16ff93ea8db19275fd7ec8af41a 10.10.2.71:6300 myself,master – 0 0 11 connected 5462-6461 7461-10922
cd1f2c1f348bb4359337e7462c1e21dc82f1551b 10.10.2.70:6300 master – 0 1445911593839 17 connected 3000-5461 6462-7460 10923-16383
2b34532cd6937063d1da26cd4652881b73d97a06 10.10.2.85:6302 slave 2568dbd91fffa16ff93ea8db19275fd7ec8af41a 0 1445911592838 17 connected #已成功掛到了10.10.2.71:6300下
1f3ea36eacbe005a4b9ac52aeef6d83337dac051 10.10.2.71:6301 slave,fail 2568dbd91fffa16ff93ea8db19275fd7ec8af41a 1445911561982 1445911559778 11 disconnected
e36cdef7a26ed59e8d9db2cf1dbc1997bfc9dfde 10.10.2.85:6300 master – 0 1445911595843 20 connected 0-2999

五、cluster相關命令

叢集
CLUSTER INFO 列印叢集的資訊
CLUSTER NODES 列出叢集當前已知的所有節點（node），以及這些節點的相關資訊。
節點
CLUSTER MEET 將 ip 和 port 所指定的節點新增到叢集當中，讓它成為叢集的一份子。
CLUSTER FORGET 從叢集中移除 node_id 指定的節點。
CLUSTER REPLICATE 將當前節點設定為 node_id 指定的節點的從節點。
CLUSTER SAVECONFIG 將節點的配置檔案儲存到硬碟裡面。
槽(slot)
CLUSTER ADDSLOTS [slot …] 將一個或多個槽（slot）指派（assign）給當前節點。
CLUSTER DELSLOTS [slot …] 移除一個或多個槽對當前節點的指派。
CLUSTER FLUSHSLOTS 移除指派給當前節點的所有槽，讓當前節點變成一個沒有指派任何槽的節點。
CLUSTER SETSLOT NODE 將槽 slot 指派給 node_id 指定的節點，如果槽已經指派給另一個節點，那麼先讓另一個節點刪除該槽>，然後再進行指派。
CLUSTER SETSLOT MIGRATING 將本節點的槽 slot 遷移到 node_id 指定的節點中。
CLUSTER SETSLOT IMPORTING 從 node_id 指定的節點中匯入槽 slot 到本節點。
CLUSTER SETSLOT STABLE 取消對槽 slot 的匯入（import）或者遷移（migrate）。
鍵
CLUSTER KEYSLOT 計算鍵 key 應該被放置在哪個槽上。
CLUSTER COUNTKEYSINSLOT 返回槽 slot 目前包含的鍵值對數量。
CLUSTER GETKEYSINSLOT 返回 count 個 slot 槽中的鍵。

參考文獻

[2]H. Berenson, P. Bernstein, J. Gray, J.Melton, E. O’Neil,and P. O’Neil. A critique of ANSI SQL isolation levels. InProceedings of the SIGMOD International Conference on Management of Data, pages1–10, May 1995.

[3]Michael J. Cahill, Uwe Röhm, and Alan D.Fekete. 2008. Serializable isolation for snapshot databases. In SIGMOD ’08:Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 729–738, New York, NY, USA. ACM.

[4]Michael James Cahill. 2009. Serializable Isolation for Snapshot Databases. Sydney Digital Theses. University of Sydney, School of Information Technologies

[5] A. Fekete, D. Liarokapis, E. O’Neil, P.O’Neil, andD. Shasha. Making snapshot isolation serializable. www.codexueyuan.com In ACM transactions on database systems, volume 39(2), pages 492–528, June 2005.