在Azure上目前已經有基於Samba協議的共享儲存了。
但目前在Azure上，還不能把Disk作為共享盤。而在實際的應用部署中，共享盤是做叢集的重要元件之一。比如仲裁盤、Shared Disk等。
本文將介紹，如果通過基於Samba的檔案共享，加上Linux的Target、iscsid以及multipath等工具，提供帶HA的共享Disk。
上圖是整體架構：
通過兩臺CentOS7.2的VM都掛載Azure File；在Azure File上建立一個檔案: disk.img，兩個VM通過iscsi server的軟體target，同時把這個disk.img作為iscsi的disk釋出出去；一臺裝有iscsid的CentOS7.2的Server同時掛載這兩個iscsi Disk；再採用multipath的軟體把這兩個盤合成一個。
在這種架構下，iscsi客戶端獲得了一塊iscsi的disk。而且這塊Disk是提供HA的網路Disk。
具體實現方式如下：
一、 建立File Service
1. 在Azure Portal上建立File Service
在Storage Account中點選Add：
填寫相關資訊，點選Create。
建立成功後，在Storage Account中選擇建立好的Storage Account：
選擇File：
點選+file Share後，填寫File Share的名字，點選建立。
建立好後，可以看到在Linux中mount的命令提示：
在Access Keys中複製key
2. 在兩臺iscsi server上mount這個File Service
首先檢視伺服器版本：
[root@hwis01 ~]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core)
都是CentOS7.0以上的版本，可以支援Samba3.0。
根據前面File Service的資訊執行下面的命令：
[root@hwis01 ~]# mkdir /file [root@hwis01 ~]# sudo mount -t cifs //hwiscsi.file.core.chinacloudapi.cn/hwfile /file -o vers=3.0,username=hwiscsi,password=xxxxxxxx==,dir_mode=0777,file_mode=0777 [root@hwis01 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 30G 1.1G 29G 4% / devtmpfs 829M 0 829M 0% /dev tmpfs 839M 0 839M 0% /dev/shm tmpfs 839M 8.3M 831M 1% /run tmpfs 839M 0 839M 0% /sys/fs/cgroup /dev/sdb1 69G 53M 66G 1% /mnt/resource tmpfs 168M 0 168M 0% /run/user/1000 //hwiscsi.file.core.chinacloudapi.cn/hwfile 5.0T 0 5.0T 0% /file
二、 在iscsi server上建立iscsi的disk
1. 在共享目錄中建立disk.img
[root@hwis01 ~]# dd if=/dev/zero of=/file/disk.img bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 20.8512 s, 51.5 MB/s
本機檢視
[root@hwis01 ~]# cd /file [root@hwis01 file]# ll total 1048576 -rwxrwxrwx. 1 root root 1073741824 Nov 4 06:04 disk.img
在另外一臺Server上檢視：
[root@hwis02 ~]# cd /file [root@hwis02 file]# ll total 1048576 -rwxrwxrwx. 1 root root 1073741824 Nov 4 06:04 disk.img
2.安裝相關軟體
iscsi伺服器端安裝targetcli：
[root@hwis01 file]# yum install -y targetcli
iscsi客戶端安裝iscsi-initiator-utils:
[root@hwic01 ~]# yum install iscsi-initiator-utils -y
安裝完成後，在iscsi的客戶端機器上檢視iqn號碼：
[root@hwic01 /]# cd /etc/iscsi/ [root@hwic01 iscsi]# more initiatorname.iscsi InitiatorName=iqn.2016-10.hw.ic01:client
3.用targetcli建立iscsi的disk
[root@hwis01 file]# targetcli Warning: Could not load preferences file /root/.targetcli/prefs.bin. targetcli shell version 2.1.fb41 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'. /> ls o- / ....................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0] /> cd backstores/ /backstores> cd fileio /backstores/fileio> create disk01 /file/disk.img 1G /file/disk.img exists, using its size (1073741824 bytes) instead Created fileio disk01 with size 1073741824 /backstores/fileio> cd /iscsi/ /iscsi> create iqn.2016-10.hw.is01:disk01.lun0 Created target iqn.2016-10.hw.is01:disk01.lun0. Created TPG 1. Global pref auto_add_default_portal=true Created default portal listening on all IPs (0.0.0.0), port 3260. /iscsi> cd iqn.2016-10.hw.is01:disk01.lun0/tpg1/luns/ /iscsi/iqn.20...un0/tpg1/luns> create /backstores/fileio/disk01 Created LUN 0. /iscsi/iqn.20...un0/tpg1/luns> cd ../acls/ /iscsi/iqn.20...un0/tpg1/acls> create iqn.2016-10.hw.ic01:client Created Node ACL for iqn.2016-10.hw.ic01:client Created mapped LUN 0. /iscsi/iqn.20...un0/tpg1/acls> ls o- acls ................................................................................................................ [ACLs: 1] o- iqn.2016-10.hw.ic01:client ................................................................................... [Mapped LUNs: 1] o- mapped_lun0 ......................................................................................... [lun0 fileio/disk01 (rw)] /iscsi/iqn.20...un0/tpg1/acls> cd / /> ls o- / ....................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 1] | | o- disk01 ..................................................................... [/file/disk.img (1.0GiB) write-back activated] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2016-10.hw.is01:disk01.lun0 ................................................................................... [TPGs: 1] | o- tpg1 ................................................................................................. [no-gen-acls, no-auth] | o- acls .............................................................................................................. [ACLs: 1] | | o- iqn.2016-10.hw.ic01:client ............................................................................... [Mapped LUNs: 1] | | o- mapped_lun0 ..................................................................................... [lun0 fileio/disk01 (rw)] | o- luns .............................................................................................................. [LUNs: 1] | | o- lun0 ..................................................................................... [fileio/disk01 (/file/disk.img)] | o- portals ........................................................................................................ [Portals: 1] | o- 0.0.0.0:3260 ........................................................................................................... [OK] o- loopback ......................................................................................................... [Targets: 0]
檢視配置檔案中的wwn
[root@hwis01 file]# cd /etc/target [root@hwis01 target]# ls backup saveconfig.json [root@hwis01 target]# vim saveconfig.json [root@hwis01 target]# grep wwn saveconfig.json "wwn": "acadb3f7-9a2d-44f4-8caf-de627ea98e27" "node_wwn": "iqn.2016-10.hw.ic01:client" "wwn": "iqn.2016-10.hw.is01:disk01.lun0"
記錄下第一個disk的 "wwn": "acadb3f7-9a2d-44f4-8caf-de627ea98e27"
並將其複製到iscsi server2中的配置。
在server2上檢視相關資訊：
[root@hwis02 target]# grep wwn saveconfig.json "wwn": "acadb3f7-9a2d-44f4-8caf-de627ea98e27" "node_wwn": "iqn.2016-10.hw.ic01:client" "wwn": "iqn.2016-10.hw.is02:disk01.lun0"
4.開啟target服務
[root@hwis01 target]# systemctl enable target Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service. [root@hwis01 target]# systemctl start target [root@hwis01 target]# systemctl status target target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2016-11-04 06:43:29 UTC; 7s ago Process: 28298 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 28298 (code=exited, status=0/SUCCESS) Nov 04 06:43:29 hwis01 systemd[1]: Starting Restore LIO kernel target configuration... Nov 04 06:43:29 hwis01 systemd[1]: Started Restore LIO kernel target configuration.
5.允許tcp3206
配置防火牆或者nsg，允許tcp3206的訪問。
這裡就不展開了。
三、 在iscsi客戶端配置iscsi
1.發現並login iscsi的disk
發現：
[root@hwic01 iscsi]# iscsiadm -m discovery -t sendtargets -p 10.1.1.5 10.1.1.5:3260,1 iqn.2016-10.hw.is02:disk01.lun0
Login：
[root@hwic01 iscsi]# iscsiadm --mode node --targetname iqn.2016-10.hw.is02:disk01.lun0 --portal 10.1.1.5 --login Logging in to [iface: default, target: iqn.2016-10.hw.is02:disk01.lun0, portal: 10.1.1.5,3260] (multiple) Login to [iface: default, target: iqn.2016-10.hw.is02:disk01.lun0, portal: 10.1.1.5,3260] successful.
此時在如下檔案中，就有了相關iqn的資訊：
[root@hwic01 /]# ls /var/lib/iscsi/nodes iqn.2016-10.hw.is01:disk01.lun0 iqn.2016-10.hw.is02:disk01.lun0
此時在dev中已經存在sdc和sdd兩個新增加的disk。
2.安裝multipath軟體
[root@hwic01 dev]# yum install device-mapper-multipath -y
複製配置檔案：
cd /etc cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf .
修改配置檔案：
vim multipath.conf blacklist { devnode "^sda$" devnode "^sdb$" } defaults { find_multipaths yes user_friendly_names yes path_grouping_policy multibus failback immediate no_path_retry fail }
啟動服務：
[root@hwic01 etc]# systemctl enable multipathd [root@hwic01 etc]# systemctl start multipathd [root@hwic01 etc]# systemctl status multipathd multipathd.service - Device-Mapper Multipath Device Controller Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2016-11-04 07:01:54 UTC; 11s ago Process: 28366 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS) Process: 28363 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS) Process: 28357 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS) Main PID: 28369 (multipathd) CGroup: /system.slice/multipathd.service └─28369 /sbin/multipathd Nov 04 07:01:54 hwic01 systemd[1]: Starting Device-Mapper Multipath Device Controller... Nov 04 07:01:54 hwic01 systemd[1]: PID file /run/multipathd/multipathd.pid not readable (yet?) after start. Nov 04 07:01:54 hwic01 systemd[1]: Started Device-Mapper Multipath Device Controller. Nov 04 07:01:54 hwic01 multipathd[28369]: mpatha: load table [0 2097152 multipath 0 0 1 1 service-time 0 2 1 8:32 1 8:48 1] Nov 04 07:01:54 hwic01 multipathd[28369]: mpatha: event checker started Nov 04 07:01:54 hwic01 multipathd[28369]: path checkers start up
重新整理multipath：
[root@hwic01 etc]# multipath -F
檢視：
[root@hwic01 etc]# multipath -l mpatha (36001405acadb3f79a2d44f48cafde627) dm-0 LIO-ORG ,disk01 size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active |- 6:0:0:0 sdc 8:32 active undef running `- 7:0:0:0 sdd 8:48 active undef running
可以看到，兩個iscsi的disk已經合併成一個disk了。
[root@hwic01 /]# cd /dev/mapper/ [root@hwic01 mapper]# ll total 0 crw-------. 1 root root 10, 236 Nov 4 04:31 control lrwxrwxrwx. 1 root root 7 Nov 4 07:02 mpatha -> ../dm-0
對dm-0進行分割槽：
[root@hwic01 dev]# fdisk /dev/mapper/mpatha Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x44f032cb. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): First sector (16384-2097151, default 16384): Using default value 16384 Last sector, +sectors or +size{K,M,G} (16384-2097151, default 2097151): Using default value 2097151 Partition 1 of type Linux and of size 1016 MiB is set Command (m for help): p Disk /dev/dm-0: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 8388608 bytes Disk label type: dos Disk identifier: 0x44f032cb Device Boot Start End Blocks Id System mpatha1 16384 2097151 1040384 83 Linux
格式化：
[root@hwic01 mapper]# mkfs.ext4 /dev/mapper/mpatha1
掛載、檢視：
[root@hwic01 mapper]# mkdir /iscsi [root@hwic01 mapper]# mount /dev/mapper/mpatha1 /iscsi [root@hwic01 mapper]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 30G 1.2G 29G 4% / devtmpfs 829M 0 829M 0% /dev tmpfs 839M 0 839M 0% /dev/shm tmpfs 839M 8.3M 831M 1% /run tmpfs 839M 0 839M 0% /sys/fs/cgroup /dev/sdb1 69G 53M 66G 1% /mnt/resource tmpfs 168M 0 168M 0% /run/user/1000 /dev/mapper/mpatha1 985M 2.5M 915M 1% /iscsi
四、 檢查iscsi disk的HA及其它屬性
1. HA檢查
將iscsi server1上的target服務停止掉：
[root@hwis01 target]# systemctl stop target [root@hwis01 target]# systemctl status target target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled) Active: inactive (dead) since Fri 2016-11-04 07:24:28 UTC; 7s ago Process: 28380 ExecStop=/usr/bin/targetctl clear (code=exited, status=0/SUCCESS) Process: 28298 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 28298 (code=exited, status=0/SUCCESS) Nov 04 06:43:29 hwis01 systemd[1]: Starting Restore LIO kernel target configuration... Nov 04 06:43:29 hwis01 systemd[1]: Started Restore LIO kernel target configuration. Nov 04 07:24:28 hwis01 systemd[1]: Stopping Restore LIO kernel target configuration... Nov 04 07:24:28 hwis01 systemd[1]: Stopped Restore LIO kernel target configuration.
在iscsi客戶端檢視：
[root@hwic01 iscsi]# multipath -l mpatha (36001405acadb3f79a2d44f48cafde627) dm-0 LIO-ORG ,disk01 size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active |- 6:0:0:0 sdc 8:32 failed faulty running `- 7:0:0:0 sdd 8:48 active undef running
可以看到一條路徑已經出現故障。但磁碟工作仍然正常：
[root@hwic01 iscsi]# ll total 16 -rw-r--r--. 1 root root 0 Nov 4 07:23 a drwx------. 2 root root 16384 Nov 4 07:18 lost+found [root@hwic01 iscsi]# touch b [root@hwic01 iscsi]# ll total 16 -rw-r--r--. 1 root root 0 Nov 4 07:23 a -rw-r--r--. 1 root root 0 Nov 4 07:27 b drwx------. 2 root root 16384 Nov 4 07:18 lost+found
再恢復服務：
[root@hwis01 target]# systemctl start target [root@hwis01 target]# systemctl status target target.service - Restore LIO kernel target configuration Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2016-11-04 07:27:41 UTC; 3s ago Process: 28380 ExecStop=/usr/bin/targetctl clear (code=exited, status=0/SUCCESS) Process: 28397 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS) Main PID: 28397 (code=exited, status=0/SUCCESS) Nov 04 07:27:40 hwis01 systemd[1]: Starting Restore LIO kernel target configuration... Nov 04 07:27:41 hwis01 systemd[1]: Started Restore LIO kernel target configuration.
客戶端multipath仍然正常工作：
[root@hwic01 iscsi]# multipath -l mpatha (36001405acadb3f79a2d44f48cafde627) dm-0 LIO-ORG ,disk01 size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=0 status=active |- 6:0:0:0 sdc 8:32 active undef running `- 7:0:0:0 sdd 8:48 active undef running
2. iops
根據前面的一片iops測試的文章中的方法，進行iops的測試：
[root@hwic01 ~]# ./iops.py /dev/dm-0 /dev/dm-0, 1.07 G, sectorsize=512B, #threads=32, pattern=random: 512 B blocks: 383.1 IO/s, 196.1 kB/s ( 1.6 Mbit/s) 1 kB blocks: 548.5 IO/s, 561.6 kB/s ( 4.5 Mbit/s) 2 kB blocks: 495.8 IO/s, 1.0 MB/s ( 8.1 Mbit/s) 4 kB blocks: 414.1 IO/s, 1.7 MB/s ( 13.6 Mbit/s) 8 kB blocks: 376.2 IO/s, 3.1 MB/s ( 24.7 Mbit/s) 16 kB blocks: 357.5 IO/s, 5.9 MB/s ( 46.9 Mbit/s) 32 kB blocks: 271.0 IO/s, 8.9 MB/s ( 71.0 Mbit/s) 65 kB blocks: 223.0 IO/s, 14.6 MB/s (116.9 Mbit/s) 131 kB blocks: 181.2 IO/s, 23.7 MB/s (190.0 Mbit/s) 262 kB blocks: 137.7 IO/s, 36.1 MB/s (288.9 Mbit/s) 524 kB blocks: 95.0 IO/s, 49.8 MB/s (398.6 Mbit/s) 1 MB blocks: 55.4 IO/s, 58.1 MB/s (465.0 Mbit/s) 2 MB blocks: 37.5 IO/s, 78.7 MB/s (629.9 Mbit/s) 4 MB blocks: 24.8 IO/s, 103.8 MB/s (830.6 Mbit/s) 8 MB blocks: 16.6 IO/s, 139.2 MB/s ( 1.1 Gbit/s) 16 MB blocks: 11.2 IO/s, 188.7 MB/s ( 1.5 Gbit/s) 33 MB blocks: 5.7 IO/s, 190.0 MB/s ( 1.5 Gbit/s)
可以看到，這個盤的iops大約在500左右，頻寬在1.5Gbps左右。
總結：
通過Azure的File Service可以通過檔案建立iscsi disk的方式把File Service中的檔案釋出成Disk，供iscsi客戶端掛載。在多個iscsi Server提供ha服務的情況下，通過multipath軟體可以實現HA的iscsi disk方案。
這個方案適合於一些叢集需要共享磁碟、仲裁盤的情況。