目录
前言
为了探究 SAN存储及iscsi ，通过模拟的方式软件搭建服务。
iscsi 又称为 IP-SAN，是一种基于因特网及SCSI-3协议下的存储技术。
环境介绍
| 主机名 | 操作系统 | IP |
|---|---|---|
| server | CentOS7.9 | 10.0.0.10 |
| node01 | CentOS7.9 | 10.0.0.11 |
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─centos-root 253:0 0 95.1G 0 lvm /
└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk #将该磁盘做为LUNs使用。
sr0 11:0 1 1024M 0 rom
iscsi服务器配置
- 将磁盘分区
> fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x928654bb.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-41943039, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039):
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─centos-root 253:0 0 95.1G 0 lvm /
└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
└─sdb1 8:17 0 20G 0 part
sr0 11:0 1 1024M 0 rom
- 安装
scsi工具
> yum install -y targetcli
> systemctl enable --now target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
- 创建
iscsi磁盘（块设备）
> targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 0] #说明：block 中创建的是以硬盘形式的存储空间
| o- fileio ................................................................................................. [Storage Objects: 0] #说明：fileio 中创建的是以文件形式的存储空间
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 0]
o- loopback ......................................................................................................... [Targets: 0]
/> backstores/block create disk.node1 /dev/sdb1 #disk.node1 是磁盘名称，后面接磁盘本地存储路径
Created block storage object disk.node1 using /dev/sdb1.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- disk.node1 ................................................................... [/dev/sdb1 (20.0GiB) write-thru deactivated] #查看创建成功。
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 0]
o- loopback ......................................................................................................... [Targets: 0]
- 创建服务端程序（iscsi目标）
/> iscsi/ create iqn.2024-12.linux.iscsi.server:server #创建服务的程序，以时间命名，注意一定要遵循这个格式，否则无法创建。
Created target iqn.2024-12.linux.iscsi.server:server.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- disk.node1 ................................................................... [/dev/sdb1 (20.0GiB) write-thru deactivated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2024-12.linux.iscsi.server:server ............................................................................. [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 0]
| o- luns .......................................................................................................... [LUNs: 0]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
- 创建服务卷（逻辑单元LUN：为了让客户端识别）
/> iscsi/iqn.2024-12.linux.iscsi.server:server/tpg1/luns create /backstores/block/disk.node1 # 通过iscsi磁盘创建luns
Created LUN 0.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- disk.node1 ..................................................................... [/dev/sdb1 (20.0GiB) write-thru activated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2024-12.linux.iscsi.server:server ............................................................................. [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 0]
| o- luns .......................................................................................................... [LUNs: 1]
| | o- lun0 ................................................................ [block/disk.node1 (/dev/sdb1) (default_tg_pt_gp)]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
- 创建客户端连接器
/> iscsi/iqn.2024-12.linux.iscsi.server:server/tpg1/acls create iqn.2024-12.linux.iscsi.server:client
Created Node ACL for iqn.2024-12.linux.iscsi.server:client
Created mapped LUN 0.
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- disk.node1 ..................................................................... [/dev/sdb1 (20.0GiB) write-thru activated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2024-12.linux.iscsi.server:server ............................................................................. [TPGs: 1]
| o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
| o- acls .......................................................................................................... [ACLs: 1]
| | o- iqn.2024-12.linux.iscsi.server:client ................................................................ [Mapped LUNs: 1] #客户端连接器
| | o- mapped_lun0 ............................................................................ [lun0 block/disk.node1 (rw)]
| o- luns .......................................................................................................... [LUNs: 1]
| | o- lun0 ................................................................ [block/disk.node1 (/dev/sdb1) (default_tg_pt_gp)]
| o- portals .................................................................................................... [Portals: 1]
| o- 0.0.0.0:3260 ..................................................................................................... [OK]
o- loopback ......................................................................................................... [Targets: 0]
- 保存配置并重启target
/> saveconfig
Configuration saved to /etc/target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json
> systemctl restart target
#查询端口是否开启
> netstat -ntplu | egrep 3260
tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN -
客户端使用iscsi
- 安装程序包
> yum install -y iscsi*
- 修改配置文件
> vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2024-12.linux.iscsi.server:client #这里填写上面添加的客户端连接器
- 启动服务
> systemctl enable --now iscsid
Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service.
- 挂载iscsi
# 发现可用的iscsi
> iscsiadm -m discovery -t st -p 10.0.0.10
10.0.0.10:3260,1 iqn.2024-12.linux.iscsi.server:server
# login 该iscsi
> iscsiadm -m node -T iqn.2024-12.linux.iscsi.server:server -p 10.0.0.10:3260 --login
Logging in to [iface: default, target: iqn.2024-12.linux.iscsi.server:server, portal: 10.0.0.10,3260] (multiple)
Login to [iface: default, target: iqn.2024-12.linux.iscsi.server:server, portal: 10.0.0.10,3260] successful.
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─centos-root 253:0 0 95.1G 0 lvm /
└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk #该块设备就是通过iscsi发现
sr0 11:0 1 1024M 0 rom
#格式化并挂载
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─centos-root 253:0 0 95.1G 0 lvm /
└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
sr0 11:0 1 1024M 0 rom
> fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x6ae7176e.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (8192-41940991, default 8192):
Using default value 8192
Last sector, +sectors or +size{K,M,G} (8192-41940991, default 41940991):
Using default value 41940991
Partition 1 of type Linux and of size 20 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
> lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 99G 0 part
├─centos-root 253:0 0 95.1G 0 lvm /
└─centos-swap 253:1 0 3.9G 0 lvm [SWAP]
sdb 8:16 0 20G 0 disk
└─sdb1 8:17 0 20G 0 part
sr0 11:0 1 1024M 0 rom
> mkfs.xfs /dev/sdb1
meta-data=/dev/sdb1 isize=512 agcount=4, agsize=1310400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=5241600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
> mkdir -p /mnt/data
> mount /dev/sdb1 /mnt/data/
> df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs tmpfs 1.9G 12M 1.9G 1% /run
tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root xfs 96G 1.5G 94G 2% /
/dev/sda1 xfs 1014M 150M 865M 15% /boot
tmpfs tmpfs 378M 0 378M 0% /run/user/0
/dev/sdb1 xfs 20G 33M 20G 1% /mnt/data
注意：客户端挂载iscsi块存储使用时，当服务器需要重启，建议先停止业务服务，再umount 该块设备的挂载，然后重启主机。
- 卸载 iscsi
> iscsiadm -m node -T iqn.2024-12.linux.iscsi.server:server -p 10.0.0.10:3260 --logout