目錄
- 一.ceph叢集的OSD基礎操作
- 1.檢視osd的ID編號
- 2.檢視osd的詳細資訊
- 3.檢視osd的狀態資訊
- 4.檢視osd的統計資訊
- 5.檢視osd在主機上的儲存資訊
- 6.檢視osd延遲的統計資訊
- 7.檢視各個osd使用率
- 8.叢集暫停接收資料
- 9.叢集開始接收資料
- 10.OSD寫入權重操作
- 11.OSD上下線
- 12.驅逐OSD裝置
- 13.加入OSD裝置
- 二.ceph叢集的OSD節點縮容
- 1.刪除OSD裝置基本流程
- 2.
- 3.
- 三.ceph叢集的OSD節點擴容
- 1.
- 2.
- 3.
一.ceph叢集的OSD基礎操作
1.檢視osd的ID編號
[root@ceph141 ~]# ceph osd ls
0
1
2
3
4
5
[root@ceph141 ~]#
2.檢視osd的詳細資訊
[root@ceph141 ~]# ceph osd dump
epoch 58
fsid c044ff3c-5f05-11ef-9d8b-51db832765d6
created 2024-08-20T15:06:28.128978+0000
modified 2024-08-20T22:48:38.568646+0000
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 16
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client jewel
require_osd_release reef
stretch_mode_enabled false
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.00
max_osd 6
osd.0 up in weight 1 up_from 53 up_thru 24 down_at 52 last_clean_interval [8,48) [v2:10.0.0.141:6808/2198281705,v1:10.0.0.141:6809/2198281705] [v2:10.0.0.141:6810/2198281705,v1:10.0.0.141:6811/2198281705] exists,up aa4d4b47-b9f1-444b-bd36-3b622391ce71
osd.1 up in weight 1 up_from 53 up_thru 27 down_at 52 last_clean_interval [13,48) [v2:10.0.0.141:6800/335450708,v1:10.0.0.141:6801/335450708] [v2:10.0.0.141:6802/335450708,v1:10.0.0.141:6803/335450708] exists,up 212eab65-f6f2-41c7-9d58-2f75f86d84b2
osd.2 up in weight 1 up_from 54 up_thru 0 down_at 53 last_clean_interval [18,48) [v2:10.0.0.142:6800/163080901,v1:10.0.0.142:6801/163080901] [v2:10.0.0.142:6802/163080901,v1:10.0.0.142:6803/163080901] exists,up e0ffb1a9-ca9b-45a1-a95f-42aca94e8f47
osd.3 up in weight 1 up_from 52 up_thru 56 down_at 51 last_clean_interval [25,48) [v2:10.0.0.142:6808/2086272149,v1:10.0.0.142:6809/2086272149] [v2:10.0.0.142:6810/2086272149,v1:10.0.0.142:6811/2086272149] exists,up fda125ef-9776-47d9-baf4-9483966fe183
osd.4 up in weight 1 up_from 56 up_thru 0 down_at 55 last_clean_interval [34,48) [v2:10.0.0.143:6808/1331943799,v1:10.0.0.143:6809/1331943799] [v2:10.0.0.143:6810/1331943799,v1:10.0.0.143:6811/1331943799] exists,up a4f27770-20c9-4a75-b0c2-a212ddc7ab3f
osd.5 up in weight 1 up_from 56 up_thru 0 down_at 55 last_clean_interval [44,48) [v2:10.0.0.143:6800/3466236845,v1:10.0.0.143:6801/3466236845] [v2:10.0.0.143:6802/3466236845,v1:10.0.0.143:6803/3466236845] exists,up c6f8968f-c425-4539-ba9b-39ff08683170
blocklist 10.0.0.141:0/2920906744 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:6801/1616144223 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:6800/1616144223 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/3338469979 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/287245293 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1238275928 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/4254913043 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:0/4240352034 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:6801/1457839497 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:6800/75389737 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1951266866 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:6800/1457839497 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:0/3710270280 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/2072915682 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:6801/75389737 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1341187958 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1879865485 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:6800/2392661167 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:0/1999918034 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:6801/2392661167 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:0/4277851589 expires 2024-08-21T15:10:24.386665+0000
[root@ceph141 ~]#
3.檢視osd的狀態資訊
[root@ceph141 ~]# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 ceph141 27.8M 199G 0 0 0 0 exists,up
1 ceph141 27.2M 299G 0 0 0 0 exists,up
2 ceph142 27.2M 199G 0 0 0 0 exists,up
3 ceph142 27.8M 299G 0 0 0 0 exists,up
4 ceph143 27.2M 299G 0 0 0 0 exists,up
5 ceph143 27.8M 199G 0 0 0 0 exists,up
[root@ceph141 ~]#
4.檢視osd的統計資訊
[root@ceph141 ~]# ceph osd stat
6 osds: 6 up (since 8m), 6 in (since 7h); epoch: e58
[root@ceph141 ~]#
5.檢視osd在主機上的儲存資訊
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 up 1.00000 1.00000
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
6.檢視osd延遲的統計資訊
[root@ceph141 ~]# ceph osd perf
osd commit_latency(ms) apply_latency(ms)
5 0 0
4 0 0
3 0 0
2 0 0
1 0 0
0 0 0
[root@ceph141 ~]#
7.檢視各個osd使用率
[root@ceph141 ~]# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 0.19530 1.00000 200 GiB 28 MiB 1.1 MiB 4 KiB 27 MiB 200 GiB 0.01 1.26 1 up
1 hdd 0.29300 1.00000 300 GiB 27 MiB 572 KiB 4 KiB 27 MiB 300 GiB 0.01 0.82 0 up
2 hdd 0.19530 1.00000 200 GiB 27 MiB 572 KiB 4 KiB 27 MiB 200 GiB 0.01 1.24 0 up
3 hdd 0.29300 1.00000 300 GiB 28 MiB 1.1 MiB 4 KiB 27 MiB 300 GiB 0.01 0.84 1 up
4 hdd 0.29300 1.00000 300 GiB 27 MiB 572 KiB 4 KiB 27 MiB 300 GiB 0.01 0.83 0 up
5 hdd 0.19530 1.00000 200 GiB 28 MiB 1.1 MiB 4 KiB 27 MiB 200 GiB 0.01 1.26 1 up
TOTAL 1.5 TiB 165 MiB 5.0 MiB 26 KiB 160 MiB 1.5 TiB 0.01
MIN/MAX VAR: 0.82/1.26 STDDEV: 0.00
[root@ceph141 ~]#
8.叢集暫停接收資料
[root@ceph141 ~]# ceph -s
...
services:
mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 14m)
mgr: ceph141.gqogmi(active, since 14m), standbys: ceph142.tisapy
osd: 6 osds: 6 up (since 14m), 6 in (since 7h)
...
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd pause
pauserd,pausewr is set
[root@ceph141 ~]#
[root@ceph141 ~]# ceph -s
...
services:
mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 14m)
mgr: ceph141.gqogmi(active, since 14m), standbys: ceph142.tisapy
osd: 6 osds: 6 up (since 14m), 6 in (since 7h)
flags pauserd,pausewr # 注意觀察，此處多了pause標籤
...
[root@ceph141 ~]#
9.叢集開始接收資料
[root@ceph141 ~]# ceph -s
...
services:
mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 16m)
mgr: ceph141.gqogmi(active, since 16m), standbys: ceph142.tisapy
osd: 6 osds: 6 up (since 16m), 6 in (since 7h)
flags pauserd,pausewr
...
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd unpause
pauserd,pausewr is unset
[root@ceph141 ~]#
[root@ceph141 ~]# ceph -s
...
services:
mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 16m)
mgr: ceph141.gqogmi(active, since 16m), standbys: ceph142.tisapy
osd: 6 osds: 6 up (since 16m), 6 in (since 7h)
...
[root@ceph141 ~]#
10.OSD寫入權重操作
1.檢視預設OSD操作權重值
[root@ceph141 ~]# ceph osd crush tree
ID CLASS WEIGHT TYPE NAME
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0
1 hdd 0.29300 osd.1
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2
3 hdd 0.29300 osd.3
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4
5 hdd 0.19530 osd.5
[root@ceph141 ~]#
2.修改OSD資料操作權重值
[root@ceph141 ~]# ceph osd crush reweight osd.4 0 # 將一塊此篇權重設定為0，表示不往該磁碟寫入資料啦，一般是下線節點時會臨時使用!
reweighted item id 4 name 'osd.4' to 0 in crush map
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd crush tree
ID CLASS WEIGHT TYPE NAME
-1 1.17189 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0
1 hdd 0.29300 osd.1
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2
3 hdd 0.29300 osd.3
-7 0.19530 host ceph143
4 hdd 0 osd.4
5 hdd 0.19530 osd.5
[root@ceph141 ~]#
3.測試完成後，可以將權重改回去喲～
[root@ceph141 ~]# ceph osd crush reweight osd.4 0.29300
reweighted item id 4 name 'osd.4' to 0.293 in crush map
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd crush tree
ID CLASS WEIGHT TYPE NAME
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0
1 hdd 0.29300 osd.1
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2
3 hdd 0.29300 osd.3
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4
5 hdd 0.19530 osd.5
[root@ceph141 ~]#
11.OSD上下線
溫馨提示:
- 1.由於OSD有專門的管理伺服器"ceph-osd"控制，一旦發現被下線，會嘗試啟動它。
- 2.如果真的想要永久關閉，則需要關閉對應的ceph-osd程序即可，例如"ceph-osd@4";
1.臨時關閉osd會自動被拉起
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 up 1.00000 1.00000
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd down 4 ; ceph osd tree
marked down osd.4.
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 down 1.00000 1.00000 # 注意觀察，此處osd已經關閉啦。
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 up 1.00000 1.00000 # 但不難發現，其會自動重啟！
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
2.永久關閉
如果是docker啟動的話就停止對應的容器即可，但儘管你停止或刪除容器，cephadm部署的會自動拉起新的容器以保證可用性。如果真不想用可以進行驅逐。
12.驅逐OSD裝置
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 up 1.00000 1.00000
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd out 4 # 驅逐編號為4的OSD裝置
marked out osd.4.
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 up 0 1.00000 # 本質上是對ceph叢集資料操作的權重值REWEIGHT重新調整。
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
13.加入OSD裝置
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 up 0 1.00000
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd in 4 # 將編號為4的裝置重新加入節點。
marked in osd.4.
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.46489 root default
-3 0.48830 host ceph141
0 hdd 0.19530 osd.0 up 1.00000 1.00000
1 hdd 0.29300 osd.1 up 1.00000 1.00000
-5 0.48830 host ceph142
2 hdd 0.19530 osd.2 up 1.00000 1.00000
3 hdd 0.29300 osd.3 up 1.00000 1.00000
-7 0.48830 host ceph143
4 hdd 0.29300 osd.4 up 1.00000 1.00000
5 hdd 0.19530 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
二.ceph叢集的OSD節點縮容
1.刪除OSD裝置基本流程
ceph-deploy刪除OSD裝置時建議遵循如下流程:
- 1.修改OSD的資料操作權重值，讓資料不往該節點儲存;
- 2.到指定節點上，停止指定的OSD程序;
- 3.將移除OSD節點狀態標記為out;
- 4.從crush中移除OSD節點，該節點不作為資料的載體;
- 5.刪除OSD節點;
- 6.刪除OSD節點認真資訊;
cephadm移除OSD注意事項:
無法停止OSD程序，因為刪除後會自動以容器的方式拉起。