作者:尹正傑
版權宣告:原創作品,謝絕轉載!否則將追究法律責任。
目錄
- 一.準備測試環境
- 1.建立儲存池
- 2.對儲存池啟用rbd功能
- 3.對儲存池進行環境初始化
- 4.建立塊裝置
- 5.建立塊裝置詳細資訊
- 6.對映塊裝置
- 7.格式化檔案系統
- 8.掛載磁碟並複製資料
- 9.解除安裝塊裝置並取消對映
- 二.服務端定製基礎快照並保護模式
- 1.建立基礎快照
- 2.將快照置於保護模式
- 3.被"保護"的快照是無法刪除的
- 三.基於快照進行克隆操作
- 1.基於基礎快照模板克隆新映象
- 2.檢視克隆的資訊
- 3.檢視基礎快照目標是否有子映象
- 四.客戶端掛載新映象進行讀寫測試
- 1.ceph141節點新增對映塊裝置測試,無需格式化
- 2.ceph142節點新增對映塊裝置測試,無需格式化
- 3.ceph143節點新增對映塊裝置測試,無需格式化
- 五.客戶端取消映象掛載
- 1.ceph141節點操作
- 2.ceph142節點操作
- 3.ceph143節點操作
- 六.刪除基礎映象的快照
- 1.子快照需要展平資料
- 2.取消保護基礎映象的快照保護
- 3.刪除基礎映象的快照
- 七 客戶端再次對映測試
- 1.ceph141節點
- 2.ceph142節點
- 3.ceph143節點
一.準備測試環境
1.建立儲存池
[root@ceph141 ~]# ceph osd pool create yinzhengjie 2 2
pool 'yinzhengjie' created
[root@ceph141 ~]#
2.對儲存池啟用rbd功能
[root@ceph141 ~]# ceph osd pool application get yinzhengjie
{}
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd pool application enable yinzhengjie rbd
enabled application 'rbd' on pool 'yinzhengjie'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd pool application get yinzhengjie
{
"rbd": {}
}
[root@ceph141 ~]#
3.對儲存池進行環境初始化
[root@ceph141 ~]# rbd pool init yinzhengjie
[root@ceph141 ~]#
[root@ceph141 ~]# rbd pool stats yinzhengjie
Total Images: 0
Total Snapshots: 0
Provisioned Size: 0 B
[root@ceph141 ~]#
4.建立塊裝置
[root@ceph141 ~]# rbd create wordpress -s 4G -p yinzhengjie
5.建立塊裝置詳細資訊
[root@ceph141 ~]# rbd ls -p yinzhengjie
wordpress
[root@ceph141 ~]#
[root@ceph141 ~]# rbd info yinzhengjie/wordpress
rbd image 'wordpress':
size 4 GiB in 1024 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: ad4945cbcd9
block_name_prefix: rbd_data.ad4945cbcd9
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Wed Aug 28 00:55:28 2024
access_timestamp: Wed Aug 28 00:55:28 2024
modify_timestamp: Wed Aug 28 00:55:28 2024
[root@ceph141 ~]#
6.對映塊裝置
[root@ceph141 ~]# rbd map yinzhengjie/wordpress
/dev/rbd0
[root@ceph141 ~]#
7.格式化檔案系統
[root@ceph141 ~]# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0 isize=512 agcount=8, agsize=131072 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=1048576, imaxpct=25
= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
[root@ceph141 ~]#
8.掛載磁碟並複製資料
[root@ceph141 ~]# mount /dev/rbd0 /mnt/
[root@ceph141 ~]#
[root@ceph141 ~]# cp /etc/os-release /mnt/
[root@ceph141 ~]#
[root@ceph141 ~]# cp /etc/hosts /mnt/
[root@ceph141 ~]#
[root@ceph141 ~]# ll /mnt/
total 12
drwxr-xr-x 2 root root 37 Aug 28 20:31 ./
drwxr-xr-x 21 root root 4096 Aug 21 20:54 ../
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph141 ~]#
[root@ceph141 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie wordpress - /dev/rbd0
[root@ceph141 ~]#
9.解除安裝塊裝置並取消對映
[root@ceph141 ~]# umount /mnt
[root@ceph141 ~]#
[root@ceph141 ~]# rbd unmap /dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# rbd showmapped
[root@ceph141 ~]#
二.服務端定製基礎快照並保護模式
1.建立基礎快照
[root@ceph141 ~]# rbd snap ls yinzhengjie/wordpress
[root@ceph141 ~]#
[root@ceph141 ~]# rbd snap create yinzhengjie/wordpress@clonewp01
Creating snap: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd snap ls yinzhengjie/wordpress
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 clonewp01 4 GiB Wed Aug 28 20:40:06 2024
[root@ceph141 ~]#
2.將快照置於保護模式
[root@ceph141 ~]# rbd snap protect yinzhengjie/wordpress@clonewp01
[root@ceph141 ~]#
[root@ceph141 ~]# rbd snap ls yinzhengjie/wordpress # 很明顯,當前映象的clonewp01快照已經被咱們保護起來了
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 clonewp01 4 GiB yes Wed Aug 28 20:40:06 2024
[root@ceph141 ~]#
3.被"保護"的快照是無法刪除的
[root@ceph141 ~]# rbd snap rm yinzhengjie/wordpress@clonewp01
Removing snap: 0% complete...failed.
2024-08-28T22:13:23.313+0800 7fcb7f7fe640 -1 librbd::Operations: snapshot is protected
rbd: snapshot 'clonewp01' is protected from removal.
[root@ceph141 ~]#
三.基於快照進行克隆操作
1.基於基礎快照模板克隆新映象
[root@ceph141 ~]# ceph osd pool ls
.mgr
yinzhengjie-rbd
yinzhengjie
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie
wordpress
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd
k8s
[root@ceph141 ~]#
[root@ceph141 ~]# rbd clone yinzhengjie/wordpress@clonewp01 yinzhengjie/wp01
[root@ceph141 ~]#
[root@ceph141 ~]# rbd clone yinzhengjie/wordpress@clonewp01 yinzhengjie-rbd/wp02 # 克隆時塊裝置可以不在同一個儲存池,但是使用時可能會存在問題。
[root@ceph141 ~]#
[root@ceph141 ~]# rbd clone yinzhengjie/wordpress@clonewp01 yinzhengjie/wp03
[root@ceph141 ~]#
2.檢視克隆的資訊
[root@ceph141 ~]# rbd ls -p yinzhengjie-rbd -l
NAME SIZE PARENT FMT PROT LOCK
k8s 5 GiB 2
wp02 4 GiB yinzhengjie/wordpress@clonewp01 2
[root@ceph141 ~]#
[root@ceph141 ~]# rbd ls -p yinzhengjie -l
NAME SIZE PARENT FMT PROT LOCK
wordpress 4 GiB 2
wordpress@clonewp01 4 GiB 2 yes
wp01 4 GiB yinzhengjie/wordpress@clonewp01 2
wp03 4 GiB yinzhengjie/wordpress@clonewp01 2
[root@ceph141 ~]#
3.檢視基礎快照目標是否有子映象
[root@ceph141 ~]# rbd children yinzhengjie/wordpress@clonewp01
yinzhengjie/wp01
yinzhengjie/wp03
yinzhengjie-rbd/wp02
[root@ceph141 ~]#
四.客戶端掛載新映象進行讀寫測試
溫馨提示:
經過我測試發現如果將多個塊裝置在同一個節點無法掛載的情況,報錯: "wrong fs type, bad option, bad superblock on /dev/rbd1, missing codepage or helper program, or other error."
要解決這個問題,可以將3個塊裝置分別掛載到不同的節點測試即可。本案例就時將3個塊裝置分別掛載到不同的節點測試。我測試的版本是Reef 18.2.4。
1.ceph141節點新增對映塊裝置測試,無需格式化
[root@ceph141 ~]# rbd showmapped
[root@ceph141 ~]#
[root@ceph141 ~]# rbd map yinzhengjie/wp01
/dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# mkdir -pv /yinzhengjie/data/wp01
[root@ceph141 ~]#
[root@ceph141 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie wp01 - /dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# mount /dev/rbd0 /yinzhengjie/data/wp01/
[root@ceph141 ~]#
[root@ceph141 ~]# ll /yinzhengjie/data/wp01/
total 12
drwxr-xr-x 2 root root 37 Aug 28 20:31 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:44 ../
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph141 ~]#
[root@ceph141 ~]# cp /etc/fstab /etc/hostname /yinzhengjie/data/wp01/
[root@ceph141 ~]#
[root@ceph141 ~]# ll /yinzhengjie/data/wp01/
total 20
drwxr-xr-x 2 root root 66 Aug 28 21:45 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:44 ../
-rw-r--r-- 1 root root 657 Aug 28 21:45 fstab
-rw-r--r-- 1 root root 8 Aug 28 21:45 hostname
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph141 ~]#
2.ceph142節點新增對映塊裝置測試,無需格式化
[root@ceph142 ~]# rbd showmapped
[root@ceph142 ~]#
[root@ceph142 ~]# rbd map yinzhengjie-rbd/wp02
/dev/rbd0
[root@ceph142 ~]#
[root@ceph142 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie-rbd wp02 - /dev/rbd0
[root@ceph142 ~]#
[root@ceph142 ~]# mkdir -pv /yinzhengjie/data/wp02
[root@ceph142 ~]#
[root@ceph142 ~]# mount /dev/rbd0 /yinzhengjie/data/wp02/
[root@ceph142 ~]#
[root@ceph142 ~]# ll /yinzhengjie/data/wp02/
total 12
drwxr-xr-x 2 root root 37 Aug 28 21:48 ./
drwxr-xr-x 3 root root 4096 Aug 28 21:48 ../
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph142 ~]#
[root@ceph142 ~]# rm -f /yinzhengjie/data/wp02/os-release
[root@ceph142 ~]#
[root@ceph142 ~]# ll /yinzhengjie/data/wp02/
total 8
drwxr-xr-x 2 root root 19 Aug 28 21:48 ./
drwxr-xr-x 3 root root 4096 Aug 28 21:48 ../
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
[root@ceph142 ~]#
3.ceph143節點新增對映塊裝置測試,無需格式化
[root@ceph143 ~]# rbd showmapped
[root@ceph143 ~]#
[root@ceph143 ~]# rbd map yinzhengjie/wp03
/dev/rbd0
[root@ceph143 ~]#
[root@ceph143 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie wp03 - /dev/rbd0
[root@ceph143 ~]#
[root@ceph143 ~]# mkdir -pv /yinzhengjie/data/wp03
[root@ceph143 ~]#
[root@ceph143 ~]# mount /dev/rbd0 /yinzhengjie/data/wp03
[root@ceph143 ~]#
[root@ceph143 ~]# ll /yinzhengjie/data/wp03
total 12
drwxr-xr-x 2 root root 37 Aug 28 20:31 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:50 ../
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph143 ~]#
[root@ceph143 ~]# cp /etc/netplan/00-installer-config.yaml /yinzhengjie/data/wp03
[root@ceph143 ~]#
[root@ceph143 ~]# ll /yinzhengjie/data/wp03
total 16
drwxr-xr-x 2 root root 69 Aug 28 21:51 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:50 ../
-rw------- 1 root root 367 Aug 28 21:51 00-installer-config.yaml
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph143 ~]#
五.客戶端取消映象掛載
1.ceph141節點操作
[root@ceph141 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie wp01 - /dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# umount /yinzhengjie/data/wp01
[root@ceph141 ~]#
[root@ceph141 ~]# rbd unmap /dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# rbd showmapped
[root@ceph141 ~]#
2.ceph142節點操作
[root@ceph142 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie-rbd wp02 - /dev/rbd0
[root@ceph142 ~]#
[root@ceph142 ~]# umount /yinzhengjie/data/wp02
[root@ceph142 ~]#
[root@ceph142 ~]# rbd unmap /dev/rbd0
[root@ceph142 ~]#
[root@ceph142 ~]# rbd showmapped
[root@ceph142 ~]#
3.ceph143節點操作
[root@ceph143 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie wp03 - /dev/rbd0
[root@ceph143 ~]#
[root@ceph143 ~]# umount /yinzhengjie/data/wp03
[root@ceph143 ~]#
[root@ceph143 ~]# rbd unmap /dev/rbd0
[root@ceph143 ~]#
[root@ceph143 ~]# rbd showmapped
[root@ceph143 ~]#
六.刪除基礎映象的快照
1.子快照需要展平資料
溫馨提示:
底層是將基礎映象的快照資料複製到子快照中,這個速度取決於映象的大小
[root@ceph141 ~]# rbd children yinzhengjie/wordpress@clonewp01 # 注意,此處我故意不對"yinzhengjie/wp03"的進行展平。
yinzhengjie/wp01
yinzhengjie/wp03
yinzhengjie-rbd/wp02
[root@ceph141 ~]#
[root@ceph141 ~]# rbd flatten yinzhengjie/wp01
Image flatten: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd flatten yinzhengjie-rbd/wp02
Image flatten: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# rbd children yinzhengjie/wordpress@clonewp01 # 很明顯,此處基礎快照僅有一個我沒有展平的子快照啦~
yinzhengjie/wp03
[root@ceph141 ~]#
2.取消保護基礎映象的快照保護
[root@ceph141 ~]# rbd snap unprotect yinzhengjie/wordpress@clonewp01. # 很明顯,如果不展平子快照,這裡就不允許取消保護
2024-08-28T22:09:04.182+0800 7f880530a640 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [d4495e828556] in pool 'yinzhengjie'
2024-08-28T22:09:04.182+0800 7f8805b0b640 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
2024-08-28T22:09:04.182+0800 7f8805b0b640 -1 librbd::SnapshotUnprotectRequest: 0x55e0d195abd0 should_complete_error: ret_val=-16
rbd: unprotecting snap failed: 2024-08-28T22:09:04.190+0800 7f880530a640 -1 librbd::SnapshotUnprotectRequest: 0x55e0d195abd0 should_complete_error: ret_val=-16
(16) Device or resource busy
[root@ceph141 ~]#
[root@ceph141 ~]# rbd flatten yinzhengjie/wp03
Image flatten: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd children yinzhengjie/wordpress@clonewp01 # Duang,此處看不到有子快照啦。
[root@ceph141 ~]#
[root@ceph141 ~]# rbd snap unprotect yinzhengjie/wordpress@clonewp01 # Duang,可以取消保護啦!
[root@ceph141 ~]#
3.刪除基礎映象的快照
[root@ceph141 ~]# rbd snap ls yinzhengjie/wordpress
SNAPID NAME SIZE PROTECTED TIMESTAMP
4 clonewp01 4 GiB Wed Aug 28 20:40:06 2024
[root@ceph141 ~]#
[root@ceph141 ~]# rbd snap rm yinzhengjie/wordpress@clonewp01
Removing snap: 100% complete...done.
[root@ceph141 ~]#
[root@ceph141 ~]# rbd snap ls yinzhengjie/wordpress # Duang,快照刪除成功啦!
[root@ceph141 ~]#
七 客戶端再次對映測試
溫馨提示:
經過測試發現,儘管刪除了基礎映象的快照,並不會影響子映象的使用喲~
1.ceph141節點
[root@ceph141 ~]# rbd showmapped
[root@ceph141 ~]#
[root@ceph141 ~]# rbd map yinzhengjie/wp01
/dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# mount /dev/rbd0 /yinzhengjie/data/wp01/
[root@ceph141 ~]#
[root@ceph141 ~]# ll /yinzhengjie/data/wp01/
total 20
drwxr-xr-x 2 root root 66 Aug 28 21:45 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:44 ../
-rw-r--r-- 1 root root 657 Aug 28 21:45 fstab
-rw-r--r-- 1 root root 8 Aug 28 21:45 hostname
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph141 ~]#
[root@ceph141 ~]# rm -f /yinzhengjie/data/wp01/h*
[root@ceph141 ~]#
[root@ceph141 ~]# ll /yinzhengjie/data/wp01/
total 12
drwxr-xr-x 2 root root 37 Aug 28 22:22 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:44 ../
-rw-r--r-- 1 root root 657 Aug 28 21:45 fstab
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph141 ~]#
[root@ceph141 ~]# umount /yinzhengjie/data/wp01
[root@ceph141 ~]#
[root@ceph141 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie wp01 - /dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# rbd unmap /dev/rbd0
[root@ceph141 ~]#
[root@ceph141 ~]# rbd showmapped
[root@ceph141 ~]#
2.ceph142節點
[root@ceph142 ~]# rbd showmapped
[root@ceph142 ~]#
[root@ceph142 ~]# rbd map yinzhengjie-rbd/wp02
/dev/rbd0
[root@ceph142 ~]#
[root@ceph142 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie-rbd wp02 - /dev/rbd0
[root@ceph142 ~]#
[root@ceph142 ~]# mount /dev/rbd0 /yinzhengjie/data/wp02/
[root@ceph142 ~]#
[root@ceph142 ~]# ll /yinzhengjie/data/wp02/
total 8
drwxr-xr-x 2 root root 19 Aug 28 21:48 ./
drwxr-xr-x 3 root root 4096 Aug 28 21:48 ../
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
[root@ceph142 ~]#
[root@ceph142 ~]# cp /etc/hostname /yinzhengjie/data/wp02/
[root@ceph142 ~]#
[root@ceph142 ~]# ll /yinzhengjie/data/wp02/
total 12
drwxr-xr-x 2 root root 35 Aug 28 22:24 ./
drwxr-xr-x 3 root root 4096 Aug 28 21:48 ../
-rw-r--r-- 1 root root 8 Aug 28 22:24 hostname
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
[root@ceph142 ~]#
[root@ceph142 ~]# umount /yinzhengjie/data/wp02
[root@ceph142 ~]#
[root@ceph142 ~]# rbd unmap /dev/rbd0
[root@ceph142 ~]#
[root@ceph142 ~]# rbd showmapped
[root@ceph142 ~]#
3.ceph143節點
[root@ceph143 ~]# rbd showmapped
[root@ceph143 ~]#
[root@ceph143 ~]# rbd map yinzhengjie/wp03
/dev/rbd0
[root@ceph143 ~]#
[root@ceph143 ~]# rbd showmapped
id pool namespace image snap device
0 yinzhengjie wp03 - /dev/rbd0
[root@ceph143 ~]#
[root@ceph143 ~]# mount /dev/rbd0 /yinzhengjie/data/wp03/
[root@ceph143 ~]#
[root@ceph143 ~]# ll /yinzhengjie/data/wp03/
total 16
drwxr-xr-x 2 root root 69 Aug 28 21:51 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:50 ../
-rw------- 1 root root 367 Aug 28 21:51 00-installer-config.yaml
-rw-r--r-- 1 root root 283 Aug 28 20:31 hosts
-rw-r--r-- 1 root root 386 Aug 28 20:31 os-release
[root@ceph143 ~]#
[root@ceph143 ~]# rm -f /yinzhengjie/data/wp03/*os*
[root@ceph143 ~]#
[root@ceph143 ~]# ll /yinzhengjie/data/wp03/
total 8
drwxr-xr-x 2 root root 38 Aug 28 22:26 ./
drwxr-xr-x 6 root root 4096 Aug 28 21:50 ../
-rw------- 1 root root 367 Aug 28 21:51 00-installer-config.yaml
[root@ceph143 ~]#
[root@ceph143 ~]# umount /yinzhengjie/data/wp03
[root@ceph143 ~]#
[root@ceph143 ~]# rbd unmap /dev/rbd0
[root@ceph143 ~]#
[root@ceph143 ~]# rbd showmapped
[root@ceph143 ~]#