这是一篇关于GlusterFS入门文章，主要讲述GlusterFS环境搭建以及初步使用。
GlusterFS相关进程
glusterfs的进程结构
- glusterfs :用于客户端挂在的进程
- glusterfsd:用于数据读写的进程
- glusterd:物理节点上的glusterfsd的管理进程，glusterd可以管理节点上一个或者多个glusterfsd的进程
- gluster:用于和glusterd管理进程通信的交互的命令行工具
安装GlusterFS
建议使用ansible配置：yum -y install ansible
yum -y install glusterfs-server
systemctl enable glusterd 或 ansible glusterfs -m shell -a ' systemctl enable glusterd.service'
systemctl start glusterd
firewall-cmd --permanent --add-service=glusterfs
firewall-cmd --reload
配置hosts解析
cat >> /etc/hosts << EOF
192.168.23.11 centos7-1
192.168.23.12 centos7-2
192.168.23.13 centos7-3
192.168.23.14 centos7-4
EOF
配置ssh互信
ssh-copy-id centos7-1
ssh-copy-id centos7-2
ssh-copy-id centos7-3
ssh-copy-id centos7-4
建议关闭selinux
挂载本地磁盘
1.手动挂载
fdisk /dev/sdb 分别按：n 、p 、3个回车键 、w
ansible glusterfs -m shell -a 'lsblk'
ansible glusterfs -m shell -a 'mkfs.xfs -i size=512 /dev/sdb1'
ansible glusterfs -m shell -a 'mount -t xfs /dev/sdb1 /data/sdb1'
ansible glusterfs -m shell -a 'lsblk'
ansible glusterfs -m shell -a 'echo -e "/dev/sdb1 /data/sdb1 xfs defaults 0 0\n" >> /etc/fstab'
2.脚本挂载
vim /opt/fdisk.sh 编辑内容如下
#!/bin/bash
echo "the disks exist list:"
##grep出系统所带磁盘
fdisk -l |grep '磁盘 /dev/sd[a-z]'
echo "=================================================="
PS3="chose which disk you want to create:"
##选择需要创建的磁盘编号
select VAR in `ls /dev/sd*|grep -o 'sd[b-z]'|uniq` quit
do
case $VAR in
sda)
##本地磁盘就退出case语句
fdisk -l /dev/sda
break ;;
sd[b-z])
#create partitions
echo "n ##创建磁盘
p
w" | fdisk /dev/$VAR
#make filesystem
##格式化
mkfs.xfs -i size=512 /dev/${VAR}"1" &> /dev/null
#mount the system
mkdir -p /data/${VAR}"1" &> /dev/null
###永久挂载
echo -e "/dev/${VAR}"1" /data/${VAR}"1" xfs defaults 0 0\n" >> /etc/fstab
###使得挂载生效
mount -a &> /dev/null
break ;;
quit)
break;;
*)
echo "wrong disk,please check again";;
esac
done
chmod +x /opt/fdisk.sh
cd /opt/
sh -x fdisk.sh
配置GlusterFS卷，在任意一个节点执行都可以
- dis-volume 分布式卷
- stripe-volume 条带卷
- rep-volume 复制卷
- dis-stripe 分布式条带卷
- dis-rep 分布式复制卷
gluster peer probe centos7-1
gluster peer probe centos7-2
gluster peer probe centos7-3
gluster peer probe centos7-4
gluster peer status
创建分布式卷，没有指定类型，默认创建的是分布式卷
[root@centos7-1 ~]# gluster volume create dis-volume centos7-1:/data/sdb1 centos7-2:/data/sdb1 centos7-3:/data/sdb1 centos7-4:/data/sdb1 force
volume create: dis-volume: success: please start the volume to access data
查看类型
[root@centos7-1 ~]# gluster volume list
dis-volume
开启
[root@centos7-1 ~]# gluster volume start dis-volume
volume start: dis-volume: success
查看卷信息
[root@centos7-1 ~]# gluster volume info dis-volume
Volume Name: dis-volume
Type: Distribute
Volume ID: cbad7555-a018-4278-82d9-a81858470201
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: centos7-1:/data/sdb1
Brick2: centos7-2:/data/sdb1
Brick3: centos7-3:/data/sdb1
Brick4: centos7-4:/data/sdb1
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
创建其他类型卷，参考网上的例子：
创建条带卷，新版本已经不支持
gluster volume create stripe-volume stripe 2 node1:/data/sdc1 node2:/data/sdc1 force
创建复制卷
gluster volume create rep-volume replica 2 node3:/data/sdb1 node4:/data/sdb1 force#指定类型为replica，数值为2，且后面跟了2个Brick Server，所以创建的是复制卷gluster volume start rep-volumegluster volume info rep-volume
创建分布式条带卷
gluster volume create dis-stripe stripe 2 node1:/data/sdd1 node2:/data/sdd1 node3:/data/sdd1 node4:/data/sdd1 force#指定类型为stripe，数值为2，而且后面跟了4个Brick Server，是2的两倍，所以创建的是分布式条带卷gluster volume start dis-stripegluster volume info dis-stripe
创建分布式复制卷
gluster volume create dis-rep replica 2 node1:/data/sde1 node2:/data/sde1 node3:/data/sde1 node4:/data/sde1 force#指定类型为 replica，数值为 2，而且后面跟了 4 个 Brick Server，是 2 的两倍，所以创建的是分布式复制卷gluster volume start dis-repgluster volume info dis-repgluster volume list
GlusterFS客户端挂载
mkdir /glusterfs-client
mount.glusterfs centos7-1:dis-volume /glusterfs-client
也可以写到/etc/fstab文件：echo "centos7-1:dis-volume /glusterfs-client glusterfs defaults 0 0" >> /etc/fstab ,这个命令我没执行过
查看客户端文件
[root@centos7-1 glusterfs-client]# ls
10.c 1.c 2.c 3.c 4.c 5.c 6.c 7.c 8.c 9.c
查看GlusterFS文件存储位置
[root@centos7-1 glusterfs-client]# ansible glusterfs -m shell -a 'ls /data/sdb1'
centos7-3 | CHANGED | rc=0 >>
10.c
4.c
7.c
9.c
centos7-4 | CHANGED | rc=0 >>
1.c
2.c
8.c
centos7-2 | CHANGED | rc=0 >>
5.c
centos7-1 | CHANGED | rc=0 >>
3.c
6.c
删除卷，移除节点
gluster volume stop dis-volume 停止卷
gluster volume delete dis-volume 删除卷
gluster peer detach centos7-4 移除机器
停止卷
[root@centos7-1 glusterfs-client]# gluster volume stop dis-volume
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: dis-volume: success
[root@centos7-1 glusterfs-client]# gluster volume list
dis-volume
[root@centos7-1 glusterfs-client]# gluster volume info dis-volume
Volume Name: dis-volume
Type: Distribute
Volume ID: cbad7555-a018-4278-82d9-a81858470201
Status: Stopped
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: centos7-1:/data/sdb1
Brick2: centos7-2:/data/sdb1
Brick3: centos7-3:/data/sdb1
Brick4: centos7-4:/data/sdb1
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
删除卷
[root@centos7-1 glusterfs-client]# gluster volume delete dis-volume
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: dis-volume: success
[root@centos7-1 glusterfs-client]# gluster volume list
No volumes present in cluster
移除节点
[root@centos7-1 glusterfs-client]# gluster peer detach centos7-4
All clients mounted through the peer which is getting detached need to be remounted using one of the other active peers in the trusted storage pool to ensure client gets notification on any changes done on the gluster configuration and if the same has been done do you want to proceed? (y/n) y
peer detach: success
[root@centos7-1 glusterfs-client]# gluster peer status
Number of Peers: 2
Hostname: centos7-2
Uuid: 87788af3-ef7a-4b28-b179-906affbad8df
State: Peer in Cluster (Connected)
Hostname: centos7-3
Uuid: d751dc03-0265-4b6e-bb04-889ae8f7095f
State: Peer in Cluster (Connected)
查看客户端文件，已经看不到了
[root@centos7-1 glusterfs-client]# ls /glusterfs-client
ls: cannot access /glusterfs-client: Transport endpoint is not connected
查看节点中的文件，文件还在
[root@centos7-1 glusterfs-client]# ansible glusterfs -m shell -a 'ls /data/sdb1'
centos7-2 | CHANGED | rc=0 >>
5.c
centos7-4 | CHANGED | rc=0 >>
1.c
2.c
8.c
centos7-3 | CHANGED | rc=0 >>
10.c
4.c
7.c
9.c
centos7-1 | CHANGED | rc=0 >>
3.c
6.c
重新恢复
gluster peer probe centos7-4
gluster volume create dis-volume centos7-1:/data/sdb1 centos7-2:/data/sdb1 centos7-3:/data/sdb1 centos7-4:/data/sdb1 force
gluster volume start dis-volume
mkdir /test
mount.glusterfs centos7-1:dis-volume /test
[root@centos7-1 glusterfs-client]# ls /test/
10.c 1.c 2.c 3.c 4.c 5.c 6.c 7.c 8.c 9.c