ceph叢集的OSD管理基礎及OSD節點擴縮容

尹正杰發表於2024-08-21

一.ceph叢集的OSD基礎操作
- 1.檢視osd的ID編號
- 2.檢視osd的詳細資訊
- 3.檢視osd的狀態資訊
- 4.檢視osd的統計資訊
- 5.檢視osd在主機上的儲存資訊
- 6.檢視osd延遲的統計資訊
- 7.檢視各個osd使用率
- 8.叢集暫停接收資料
- 9.叢集開始接收資料
- 10.OSD寫入權重操作
- 11.OSD上下線
- 12.驅逐OSD裝置
- 13.加入OSD裝置
二.ceph叢集的OSD節點縮容
- 1.刪除OSD裝置基本流程
- 2.
- 3.
三.ceph叢集的OSD節點擴容
- 1.
- 2.
- 3.

一.ceph叢集的OSD基礎操作

1.檢視osd的ID編號

[root@ceph141 ~]# ceph osd ls
0
1
2
3
4
5
[root@ceph141 ~]#

2.檢視osd的詳細資訊

[root@ceph141 ~]# ceph osd dump
epoch 58
fsid c044ff3c-5f05-11ef-9d8b-51db832765d6
created 2024-08-20T15:06:28.128978+0000
modified 2024-08-20T22:48:38.568646+0000
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 16
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client jewel
require_osd_release reef
stretch_mode_enabled false
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.00
max_osd 6
osd.0 up   in  weight 1 up_from 53 up_thru 24 down_at 52 last_clean_interval [8,48) [v2:10.0.0.141:6808/2198281705,v1:10.0.0.141:6809/2198281705] [v2:10.0.0.141:6810/2198281705,v1:10.0.0.141:6811/2198281705] exists,up aa4d4b47-b9f1-444b-bd36-3b622391ce71
osd.1 up   in  weight 1 up_from 53 up_thru 27 down_at 52 last_clean_interval [13,48) [v2:10.0.0.141:6800/335450708,v1:10.0.0.141:6801/335450708] [v2:10.0.0.141:6802/335450708,v1:10.0.0.141:6803/335450708] exists,up 212eab65-f6f2-41c7-9d58-2f75f86d84b2
osd.2 up   in  weight 1 up_from 54 up_thru 0 down_at 53 last_clean_interval [18,48) [v2:10.0.0.142:6800/163080901,v1:10.0.0.142:6801/163080901] [v2:10.0.0.142:6802/163080901,v1:10.0.0.142:6803/163080901] exists,up e0ffb1a9-ca9b-45a1-a95f-42aca94e8f47
osd.3 up   in  weight 1 up_from 52 up_thru 56 down_at 51 last_clean_interval [25,48) [v2:10.0.0.142:6808/2086272149,v1:10.0.0.142:6809/2086272149] [v2:10.0.0.142:6810/2086272149,v1:10.0.0.142:6811/2086272149] exists,up fda125ef-9776-47d9-baf4-9483966fe183
osd.4 up   in  weight 1 up_from 56 up_thru 0 down_at 55 last_clean_interval [34,48) [v2:10.0.0.143:6808/1331943799,v1:10.0.0.143:6809/1331943799] [v2:10.0.0.143:6810/1331943799,v1:10.0.0.143:6811/1331943799] exists,up a4f27770-20c9-4a75-b0c2-a212ddc7ab3f
osd.5 up   in  weight 1 up_from 56 up_thru 0 down_at 55 last_clean_interval [44,48) [v2:10.0.0.143:6800/3466236845,v1:10.0.0.143:6801/3466236845] [v2:10.0.0.143:6802/3466236845,v1:10.0.0.143:6803/3466236845] exists,up c6f8968f-c425-4539-ba9b-39ff08683170
blocklist 10.0.0.141:0/2920906744 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:6801/1616144223 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:6800/1616144223 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/3338469979 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/287245293 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1238275928 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/4254913043 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:0/4240352034 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:6801/1457839497 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:6800/75389737 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1951266866 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:6800/1457839497 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:0/3710270280 expires 2024-08-21T22:48:38.568602+0000
blocklist 10.0.0.141:0/2072915682 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:6801/75389737 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1341187958 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:0/1879865485 expires 2024-08-21T15:10:24.386665+0000
blocklist 10.0.0.141:6800/2392661167 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:0/1999918034 expires 2024-08-21T15:07:14.218755+0000
blocklist 10.0.0.141:6801/2392661167 expires 2024-08-21T15:06:48.971433+0000
blocklist 10.0.0.141:0/4277851589 expires 2024-08-21T15:10:24.386665+0000
[root@ceph141 ~]#

3.檢視osd的狀態資訊

[root@ceph141 ~]# ceph osd status
ID  HOST      USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE      
 0  ceph141  27.8M   199G      0        0       0        0   exists,up  
 1  ceph141  27.2M   299G      0        0       0        0   exists,up  
 2  ceph142  27.2M   199G      0        0       0        0   exists,up  
 3  ceph142  27.8M   299G      0        0       0        0   exists,up  
 4  ceph143  27.2M   299G      0        0       0        0   exists,up  
 5  ceph143  27.8M   199G      0        0       0        0   exists,up  
[root@ceph141 ~]#

4.檢視osd的統計資訊

[root@ceph141 ~]# ceph osd stat
6 osds: 6 up (since 8m), 6 in (since 7h); epoch: e58
[root@ceph141 ~]#

5.檢視osd在主機上的儲存資訊

[root@ceph141 ~]# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4         up   1.00000  1.00000
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]#

6.檢視osd延遲的統計資訊

[root@ceph141 ~]# ceph osd perf
osd  commit_latency(ms)  apply_latency(ms)
  5                   0                  0
  4                   0                  0
  3                   0                  0
  2                   0                  0
  1                   0                  0
  0                   0                  0
[root@ceph141 ~]#

7.檢視各個osd使用率

[root@ceph141 ~]# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP    META     AVAIL    %USE  VAR   PGS  STATUS
 0    hdd  0.19530   1.00000  200 GiB   28 MiB  1.1 MiB   4 KiB   27 MiB  200 GiB  0.01  1.26    1      up
 1    hdd  0.29300   1.00000  300 GiB   27 MiB  572 KiB   4 KiB   27 MiB  300 GiB  0.01  0.82    0      up
 2    hdd  0.19530   1.00000  200 GiB   27 MiB  572 KiB   4 KiB   27 MiB  200 GiB  0.01  1.24    0      up
 3    hdd  0.29300   1.00000  300 GiB   28 MiB  1.1 MiB   4 KiB   27 MiB  300 GiB  0.01  0.84    1      up
 4    hdd  0.29300   1.00000  300 GiB   27 MiB  572 KiB   4 KiB   27 MiB  300 GiB  0.01  0.83    0      up
 5    hdd  0.19530   1.00000  200 GiB   28 MiB  1.1 MiB   4 KiB   27 MiB  200 GiB  0.01  1.26    1      up
                       TOTAL  1.5 TiB  165 MiB  5.0 MiB  26 KiB  160 MiB  1.5 TiB  0.01                   
MIN/MAX VAR: 0.82/1.26  STDDEV: 0.00
[root@ceph141 ~]#

8.叢集暫停接收資料

[root@ceph141 ~]# ceph -s
...
  services:
    mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 14m)
    mgr: ceph141.gqogmi(active, since 14m), standbys: ceph142.tisapy
    osd: 6 osds: 6 up (since 14m), 6 in (since 7h)
 
...
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd pause
pauserd,pausewr is set
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph -s
...
  services:
    mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 14m)
    mgr: ceph141.gqogmi(active, since 14m), standbys: ceph142.tisapy
    osd: 6 osds: 6 up (since 14m), 6 in (since 7h)
         flags pauserd,pausewr  # 注意觀察，此處多了pause標籤
 
...
 
[root@ceph141 ~]#

9.叢集開始接收資料

[root@ceph141 ~]# ceph -s
...
 
  services:
    mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 16m)
    mgr: ceph141.gqogmi(active, since 16m), standbys: ceph142.tisapy
    osd: 6 osds: 6 up (since 16m), 6 in (since 7h)
         flags pauserd,pausewr
 
...
 
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd unpause
pauserd,pausewr is unset
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph -s
...
 
  services:
    mon: 3 daemons, quorum ceph141,ceph143,ceph142 (age 16m)
    mgr: ceph141.gqogmi(active, since 16m), standbys: ceph142.tisapy
    osd: 6 osds: 6 up (since 16m), 6 in (since 7h)
 
...
 
[root@ceph141 ~]#

10.OSD寫入權重操作

	1.檢視預設OSD操作權重值
[root@ceph141 ~]# ceph osd crush tree
ID  CLASS  WEIGHT   TYPE NAME       
-1         1.46489  root default    
-3         0.48830      host ceph141
 0    hdd  0.19530          osd.0   
 1    hdd  0.29300          osd.1   
-5         0.48830      host ceph142
 2    hdd  0.19530          osd.2   
 3    hdd  0.29300          osd.3   
-7         0.48830      host ceph143
 4    hdd  0.29300          osd.4   
 5    hdd  0.19530          osd.5   
[root@ceph141 ~]# 

	2.修改OSD資料操作權重值
[root@ceph141 ~]# ceph osd crush reweight osd.4 0  # 將一塊此篇權重設定為0，表示不往該磁碟寫入資料啦，一般是下線節點時會臨時使用!
reweighted item id 4 name 'osd.4' to 0 in crush map
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd crush tree
ID  CLASS  WEIGHT   TYPE NAME       
-1         1.17189  root default    
-3         0.48830      host ceph141
 0    hdd  0.19530          osd.0   
 1    hdd  0.29300          osd.1   
-5         0.48830      host ceph142
 2    hdd  0.19530          osd.2   
 3    hdd  0.29300          osd.3   
-7         0.19530      host ceph143
 4    hdd        0          osd.4   
 5    hdd  0.19530          osd.5   
[root@ceph141 ~]# 

	3.測試完成後，可以將權重改回去喲～
[root@ceph141 ~]# ceph osd crush reweight osd.4 0.29300
reweighted item id 4 name 'osd.4' to 0.293 in crush map
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd crush tree
ID  CLASS  WEIGHT   TYPE NAME       
-1         1.46489  root default    
-3         0.48830      host ceph141
 0    hdd  0.19530          osd.0   
 1    hdd  0.29300          osd.1   
-5         0.48830      host ceph142
 2    hdd  0.19530          osd.2   
 3    hdd  0.29300          osd.3   
-7         0.48830      host ceph143
 4    hdd  0.29300          osd.4   
 5    hdd  0.19530          osd.5   
[root@ceph141 ~]#

11.OSD上下線

溫馨提示:
	- 1.由於OSD有專門的管理伺服器"ceph-osd"控制，一旦發現被下線，會嘗試啟動它。
	- 2.如果真的想要永久關閉，則需要關閉對應的ceph-osd程序即可，例如"ceph-osd@4";
	
	1.臨時關閉osd會自動被拉起
[root@ceph141 ~]# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4         up   1.00000  1.00000
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd down 4 ; ceph osd tree
marked down osd.4. 
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4       down   1.00000  1.00000  # 注意觀察，此處osd已經關閉啦。
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd tree  
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4         up   1.00000  1.00000  # 但不難發現，其會自動重啟！
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]# 


	2.永久關閉
如果是docker啟動的話就停止對應的容器即可，但儘管你停止或刪除容器，cephadm部署的會自動拉起新的容器以保證可用性。如果真不想用可以進行驅逐。

12.驅逐OSD裝置

[root@ceph141 ~]# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4         up   1.00000  1.00000
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd out 4  # 驅逐編號為4的OSD裝置
marked out osd.4. 
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4         up         0  1.00000  # 本質上是對ceph叢集資料操作的權重值REWEIGHT重新調整。
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]#

13.加入OSD裝置

[root@ceph141 ~]# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4         up         0  1.00000
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd in 4  # 將編號為4的裝置重新加入節點。
marked in osd.4. 
[root@ceph141 ~]# 
[root@ceph141 ~]# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         1.46489  root default                               
-3         0.48830      host ceph141                           
 0    hdd  0.19530          osd.0         up   1.00000  1.00000
 1    hdd  0.29300          osd.1         up   1.00000  1.00000
-5         0.48830      host ceph142                           
 2    hdd  0.19530          osd.2         up   1.00000  1.00000
 3    hdd  0.29300          osd.3         up   1.00000  1.00000
-7         0.48830      host ceph143                           
 4    hdd  0.29300          osd.4         up   1.00000  1.00000
 5    hdd  0.19530          osd.5         up   1.00000  1.00000
[root@ceph141 ~]#

二.ceph叢集的OSD節點縮容

1.刪除OSD裝置基本流程

ceph-deploy刪除OSD裝置時建議遵循如下流程:
	- 1.修改OSD的資料操作權重值，讓資料不往該節點儲存;
	- 2.到指定節點上，停止指定的OSD程序;
	- 3.將移除OSD節點狀態標記為out;
	- 4.從crush中移除OSD節點，該節點不作為資料的載體;
	- 5.刪除OSD節點;
	- 6.刪除OSD節點認真資訊;
	
cephadm移除OSD注意事項:
	無法停止OSD程序，因為刪除後會自動以容器的方式拉起。

2.

3.

三.ceph叢集的OSD節點擴容

1.

2.

3.

CEPH叢集重啟後ceph osd status和ceph -s命令檢視發現節點伺服器osd服務不成功的解決方案
2020-11-06
伺服器
Redis—叢集擴縮容
2024-04-06
Redis
cephadm訪問ceph叢集的方式及管理員節點配置案例
2024-08-22
Redis Cluster 叢集搭建與擴容、縮容
2022-03-27
Redis
docker筆記40-ceph osd誤刪除恢復
2018-10-29
Docker筆記
Airbnb的動態 Kubernetes 叢集擴縮容
2022-08-25
AI
視訊處理之OSD
2020-06-05
Redis Manager 叢集管理與節點管理
2018-12-15
Redis
ceph叢集
2024-08-25
Redis服務之叢集節點管理
2020-08-08
Redis
Oracle叢集軟體管理-新增和刪除叢集節點
2020-03-19
Oracle
生產調優4 HDFS-叢集擴容及縮容(含伺服器間資料均衡)
2021-12-07
伺服器
Jedis操作單節點redis，叢集及redisTemplate操作redis叢集（一）
2018-06-13
Redis
Ceph的叢集全部換IP
2016-10-18
ceph-deploy離線部署ceph叢集及報錯解決FAQ
2022-02-28
DKHhadoop叢集新增節點管理功能的操作步驟
2019-01-16
Hadoop
Elasticsearch 7.x 之節點、叢集、分片及副本
2020-08-23
Elasticsearch
consul 多節點/單節點叢集搭建
2021-07-12
4.2 叢集節點初步搭建
2018-11-15
CEPH-2：rbd功能詳解及普通使用者應用ceph叢集
2022-03-29
ORA-27504: IPC error creating OSD context
2008-08-30
ErrorContext
MongoDB叢集搭建(包括隱藏節點，仲裁節點)
2021-04-13
MongoDB
HAC叢集更改IP（單節點更改、全部節點更改）
2022-05-27
400+節點的 Elasticsearch 叢集運維
2019-03-26
Elasticsearch運維
400+ 節點的 Elasticsearch 叢集運維
2019-04-25
Elasticsearch運維
mongodb叢集節點故障的切換方法
2019-06-20
MongoDB
linux搭建kafka叢集，多master節點叢集說明
2022-04-06
LinuxKafkaAST
Elasticsearch-02-入門：叢集、節點、分片、索引及常用API
2021-07-14
Elasticsearch索引API
Redis大叢集擴容效能優化實踐
2021-10-18
Redis優化
Elasticsearch——叢集管理及文件CRUD
2019-01-26
Elasticsearch
SOAP 節點基礎
2008-08-15
redhat安裝雙節點cassandra叢集
2011-05-19
Redhat
管理 ES 叢集：分片設定及管理
2020-02-20
從庫轉換成PXC叢集的節點
2019-10-26
在多節點的叢集上執行Cassandra
2012-08-04
kafka 基礎知識梳理及叢集環境部署記錄
2018-05-10
Kafka
Redis大叢集擴容效能最佳化實踐
2021-10-18
Redis
向基於 Linux 的 Oracle RAC 10g 叢集新增新節點
2010-07-08
LinuxOracle

ceph叢集的OSD管理基礎及OSD節點擴縮容

一.ceph叢集的OSD基礎操作

1.檢視osd的ID編號

2.檢視osd的詳細資訊

3.檢視osd的狀態資訊

4.檢視osd的統計資訊

5.檢視osd在主機上的儲存資訊

6.檢視osd延遲的統計資訊

7.檢視各個osd使用率

8.叢集暫停接收資料

9.叢集開始接收資料

10.OSD寫入權重操作

11.OSD上下線

12.驅逐OSD裝置

13.加入OSD裝置

二.ceph叢集的OSD節點縮容

1.刪除OSD裝置基本流程

2.

3.

三.ceph叢集的OSD節點擴容

1.

2.

3.

相關文章