案例说明：
在KingbaseES Clusterware高可用的架构中，集群节点需要访问共享的存储设备，可以使用FC SAN、iscsi SAN、NAS等存储设备。本案例详细描述了，在Linux系统下构建 iscsi存储服务器及客户端访问的过程，为部署KingbaseES Clusterware做好相应的准备。
如下所示：KingbaseES Clusterware架构
系统架构节点信息：
ISCSI SAN存储架构：
一、iscsi server配置
Tips：
iscsi server提供共享存储，并接受client的访问。
1、系统环境信息
[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)
2、server端软件包安装及启动服务
#软件包安装
[root@node1 ~]# yum -y install targetd targetcli
#启动server端服务
[root@node1 ~]# systemctl start target
[root@node1 ~]# systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
[root@node1 ~]# systemctl status target
● target.service - Restore LIO kernel target configuration
Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled)
Active: active (exited) since Wed 2022-11-23 15:46:17 CST; 17s ago
Main PID: 22094 (code=exited, status=0/SUCCESS)
Nov 23 15:46:15 node1 systemd[1]: Starting Restore LIO kernel target configuration...
Nov 23 15:46:17 node1 target[22094]: No saved config file at /etc/target/saveconfig.json, ok, exiting
Nov 23 15:46:17 node1 systemd[1]: Started Restore LIO kernel target configuration.
3、iscsi server端配置
Tips：
当执行targetcli出现以下错误时，需要安装python-rtslib软件。
安装python-rtslib软件：
[root@node1 ~]# yum install -y python-rtslib
server端配置工具：
[root@node1 ~]# targetcli
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> help
AVAILABLE COMMANDS
==================
The following commands are available in the
current path:
- bookmarks action [bookmark]
- cd [path]
- clearconfig [confirm]
- exit
- get [group] [parameter...]
- help [topic]
- ls [path] [depth]
- pwd
- refresh
- restoreconfig [savefile] [clear_existing] [target] [storage_object]
- saveconfig [savefile]
- sessions [action] [sid]
- set [group] [parameter=value...]
- status
- version
4、server端添加共享磁盘
[root@node1 ~]# fdisk -l
.......
Disk /dev/sdb: 21.6 GB, 21579694080 bytes, 42147840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
5、server端iscsi配置
1）创建要发布的磁盘（块设备LUN)
[root@node1 ~]# targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 0]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 0]
o- loopback ......................................................................................................... [Targets: 0]
/> /backstores/block create server0.disk1 /dev/sdb
Created block storage object server0.disk1 using /dev/sdb.
#注：server0.disk1 是给/dev/sdb转化成块的名字。
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- server0.disk1 ................................................................. [/dev/sdb (20.1GiB) write-thru deactivated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 0]
o- loopback ......................................................................................................... [Targets: 0]
2）创建iqn名字即创建iscsi对象
iqn:iSCSI Qualified Name（ISCSI 限定名称）
/> /iscsi create iqn.test.com.bqdn:disk1
WWN not valid as: iqn, naa, eui
# 注意：iqn名称格式
/> /iscsi create iqn.2022-11.com.bqdn:disk1
Created target iqn.2022-11.com.bqdn:disk1.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- server0.disk1 ................................................................. [/dev/sdb (20.1GiB) write-thru deactivated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2022-11.com.bqdn:disk1 ........................................................................................ [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 0]
| o- luns .......................................................................................................... [LUNs: 0]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
3）设置ACL即将ISCSI对象与客户端IP或主机名绑定
/> /iscsi/iqn.2022-11.com.bqdn:disk1/tpg1/acls create iqn.2022-11.com.bdqn:client1
Created Node ACL for iqn.2022-11.com.bdqn:client1
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- server0.disk1 ................................................................. [/dev/sdb (20.1GiB) write-thru deactivated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2022-11.com.bqdn:disk1 ........................................................................................ [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 1]
| | o- iqn.2022-11.com.bdqn:client1 ......................................................................... [Mapped LUNs: 0]
| o- luns .......................................................................................................... [LUNs: 0]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
/>
#注：iqn.2022-11.com.bdqn:client1是 initiator 的名字，需要在客户端中设置的。
4）创建LUN并绑定块
/> /iscsi/iqn.2022-11.com.bqdn:disk1/tpg1/luns create /backstores/block/server0.disk1
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2022-11.com.bdqn:client1
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- server0.disk1 ................................................................... [/dev/sdb (20.1GiB) write-thru activated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2022-11.com.bqdn:disk1 ........................................................................................ [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 1]
| | o- iqn.2022-11.com.bdqn:client1 ......................................................................... [Mapped LUNs: 1]
| | o- mapped_lun0 ......................................................................... [lun0 block/server0.disk1 (rw)]
| o- luns .......................................................................................................... [LUNs: 1]
| | o- lun0 .............................................................. [block/server0.disk1 (/dev/sdb) (default_tg_pt_gp)]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
二、iscsi client访问iscsi server
1、系统环境信息
[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)
2、客户端软件包安装
[root@node2 ~]# rpm -qa |grep iscsi
libiscsi-1.9.0-6.el7.x86_64
iscsi-initiator-utils-6.2.0.873-32.el7.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64
3、客户端配置
# 配置iscsi client名称
[root@node2 ~]# cat /etc/iscsi/initiatorname.iscsi
#InitiatorName=iqn.1994-05.com.redhat:e43610d8fc4d
InitiatorName=iqn.2022-11.com.bdqn:client1
4、客户端访问iscsi服务器
1）客户端访问工具iscsiadm
[root@node2 ~]# iscsiadm --help
iscsiadm -m discoverydb [ -hV ] [ -d debug_level ] [-P printlevel] [ -t type -p ip:port -I ifaceN ... [ -Dl ] ] | [ [ -p ip:port -t type] [ -o operation ] [ -n name ] [ -v value ] [ -lD ] ]
iscsiadm -m discovery [ -hV ] [ -d debug_level ] [-P printlevel] [ -t type -p ip:port -I ifaceN ... [ -l ] ] | [ [ -p ip:port ] [ -l | -D ] ]
iscsiadm -m node [ -hV ] [ -d debug_level ] [ -P printlevel ] [ -L all,manual,automatic ] [ -U all,manual,automatic ] [ -S ] [ [ -T targetname -p ip:port -I ifaceN ] [ -l | -u | -R | -s] ] [ [ -o operation ] [ -n name ] [ -v value ] ]
iscsiadm -m session [ -hV ] [ -d debug_level ] [ -P printlevel] [ -r sessionid | sysfsdir [ -R | -u | -s ] [ -o operation ] [ -n name ] [ -v value ] ]
iscsiadm -m iface [ -hV ] [ -d debug_level ] [ -P printlevel ] [ -I ifacename | -H hostno|MAC ] [ [ -o operation ] [ -n name ] [ -v value ] ] [ -C ping [ -a ip ] [ -b packetsize ] [ -c count ] [ -i interval ] ]
iscsiadm -m fw [ -d debug_level ] [ -l ]
iscsiadm -m host [ -P printlevel ] [ -H hostno|MAC ] [ [ -C chap [ -x chap_tbl_idx ] ] | [ -C flashnode [ -A portal_type ] [ -x flashnode_idx ] ] | [ -C stats ] ] [ [ -o operation ] [ -n name ] [ -v value ] ]
iscsiadm -k priority
2）连接iscsi server
# discovery iscsi server
[root@node2 ~]# iscsiadm -m discovery -t st -p 192.168.8.200 -l
192.168.8.200:3260,1 iqn.2022-11.com.bqdn:disk1
Logging in to [iface: default, target: iqn.2022-11.com.bqdn:disk1, portal: 192.168.8.200,3260] (multiple)
Login to [iface: default, target: iqn.2022-11.com.bqdn:disk1, portal: 192.168.8.200,3260] successful.
# 查看共享存储
[root@node2 ~]# lsscsi
[0:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda
[2:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0
[3:0:0:0] disk LIO-ORG server0.disk1 4.0 /dev/sdb
# 如上所示，已经发现共享的存储/dev/sdb。
3）查看磁盘设备
Tips：
如下所示，可以通过fdisk查看到共享的存储，可以像本地磁盘一样去使用共享存储。
[root@node2 ~]# fdisk -l
.......
Disk /dev/sdb: 21.6 GB, 21579694080 bytes, 42147840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33550336 bytes
4）启动客户端iscsi服务
[root@node2 ~]# systemctl start iscsi
[root@node2 ~]# systemctl status iscsi
● iscsi.service - Login and scanning of iSCSI devices
Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; vendor preset: disabled)
Active: active (exited) since Wed 2022-11-23 16:17:53 CST; 1s ago
Docs: man:iscsid(8)
man:iscsiadm(8)
Process: 27430 ExecStart=/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=21)
Process: 27419 ExecStart=/usr/libexec/iscsi-mark-root-nodes (code=exited, status=0/SUCCESS)
Main PID: 27430 (code=exited, status=21)
Nov 23 16:17:53 node2 systemd[1]: Starting Login and scanning of iSCSI devices...
Nov 23 16:17:53 node2 iscsiadm[27430]: iscsiadm: No records found
Nov 23 16:17:53 node2 systemd[1]: Started Login and scanning of iSCSI devices.
三、iscsi client使用共享存储
1、创建磁盘分区
[root@node2 ~]# fdisk /dev/sdb
2、mount磁盘分区
[root@node2 ~]# mkdir /sharedsk
[root@node2 ~]# mount /dev/sdb1 /sharedsk
[root@node2 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 39G 23G 16G 61% /
devtmpfs 906M 0 906M 0% /dev
tmpfs 921M 84K 920M 1% /dev/shm
........
/dev/sdb1 21G 33M 21G 1% /sharedsk
#如上所示，在node2节点（iscsi client）已经可以像本地磁盘一样访问共享存储。
四、总结
通过iscsi存储可以构建高性价比的共享存储设备，在基于共享存储的高可用架构中被广泛使用。