ceph分布式存储部署（ceph-deploy）
ceph学习
ceph简介和特性
ceph时一个多版本存储系统，它把每一个待管理的数据量切分为一到多个固定带下的对象数据，并以其为原子
单元完成数据存取。
对象数据的地处存储服务是由多个主机组成的存储集群，该集群也被称为RADOS存储集群即可靠、自动化、分布式对象存储系统。ceph通过内部crush算法，实时方式计算除一个文件应该存储到那个存储对象里面，从而实现快速查找对象的一种方式。
librados是RADOS存储集群的api,它支持c、c++、java、python、ruby和PHP等编程语言
功能强大：ceph能够同时提供对象存储、块存储和文件存储三种存储服务的统一存储架构
可扩展性：ceph通过crush算法的寻址操作，有相当强大的扩展性。
高可用性：cpeh数据副本数量可以有管理员自定义，并通过crush算法指定副本的物理存储位置以分隔故障域，支持数据强一致性的特性也是ceph具有了高可靠性，可以忍受多种故障场景并自动尝试并进行修复
LIBRADOS --通过自编程的方式实现数据的存储能力
RADOSGW --通过标准的RESTFULL接口，提供一种云存储服务
RBD --将ceph提供的空间，模拟成一个个独立的块设备，当ceph环境部署完后，服务端准备好rbd接口
CFS --通过一个标准的文件系统接口来进行数据的存储ceph组件
组件 | 解析 |
Monitors | ceph Monitors守护进程（ceph-mon）维护集群状态的映射，包括监视器映射，管理器映射、OSD映射，MDS映射和CRUSH映射。监视器还负责管理守护进程和客户端的身份验证，通常至少需要三个监视器才能实现冗余和高可用性，基于paxos协议实现节点间的信息同步 |
Managers | ceph管理器（守护进程ceph-mgr）负责跟踪运行时指标和ceph集群的当前状态，包括存储利用率，当前性能指标和系统负载。高可用性通常至少需要两个管理器，基于raft协议实现节点间的信息同步 |
cephOSDS | ceph OSD（对象存储守护进程ceph-osd）存储数据，处理数据复制，回复，重新平衡，并通过检查其他osd守护进程的心跳来向ceph监视器和管理器提供一写监控信息，通常需要3个ceph OSD 来实现冗余和高可用性，本质上osd就是一个个host主机上存储磁盘 |
MDSs | ceph元数据服务器，代表ceph文件系统存储元数据。ceph元数据服务器允许POSIX文件系统用户执行基本命令（ls, find等），而不会给ceph集群带来巨大负担 |
ceph网络模型
ceph生产环境一般分为两个网段：
公有网络：用于用户的数据通信
集群网络：用于集群内部的管理通信
ceph版本
x.0.z-开发版
x.1.z-候选版
x.2.z-稳定、修正版
版本信息查看
https://docs.ceph.com/en/latest/releases/
https://docs.ceph.org.cn/start/intro/ceph部署
部署方式 | 特点解析 |
cephadm | 使用容器和systemd安装和管理ceph集群，并于CLI和仪表板GUI紧密集成，只支持octopus以后的新版本，当前官方推荐 |
ceph-deploy | 一个基于python实现的快速部署集群的工具，此工具从Nautilus版后不再支持和测试，建议使用此工具 Nautilus 之前的旧版本。不支持 RHEL8、CentOS 8 或更新的操作系统。 |
rook | 在k8s上运行ceph集群，同时还支持通过K8Sapi管理存储资源和配置，只支持 Nautilus 以后的新版本，不支持 RHEL8、CentOS 8 或更新的操作系统。 |
ceph-ansible | 使用ansible部署和管理ceph集群，应用较广，但是从Nautilus 和octopus版没有集成对应的api，所以较新的管理功能和仪表板集成不可用 |
ceph-salt | 使用salt和cephadm安装ceph |
ceph-mon | 使用Juju（模型驱动的k8s生命周期管理器OLM）安装ceph |
puppet-ceph | 通过puppet安装ceph |
二进制 | 手工安装 |
windows图形 | 在windows主机上通过图形操作就可以进行部署 |
环境准备
系统ubuntu18.04
下载连接
https://releases.ubuntu.com/18.04/ubuntu-18.04.6-desktop-amd64.iso
公有网络：10.0.0.0/24
集群网络：192.168.160.0/24
192.168.160.128
192.168.160.129
192.168.160.130
192.168.160.131
10.0.0.128
10.0.0.129
10.0.0.130
10.0.0.131
#导入key
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -
#添加apt源
echo deb https://download.ceph.com/debian-nautilus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
#更新源
sudo apt update
#用脚本实现批量创建用户
cat > create_cephadm.sh <<EOF
#!/bin/bash# 设定普通用户
useradd -m -s /bin/bash cephadm
echo cephadm:123456 | chpasswd
echo "cephadm ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/cephadm
chmod 0440 /etc/sudoers.d/cephadm
EOF
#批量执行
for i in {128..131}; do ssh root@192.168.160.$i bash < create_cephadm.sh ; done
#hosts用户名解析
所有节点都需要添加
192.168.160.128 node
192.168.160.129 node-1
192.168.160.130 node-2
192.168.160.131 mon
#ssh免密
所有节点都需要操作
root@node:~# su - ceph
ceph@node:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph/.ssh/id_rsa):
Created directory '/home/ceph/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ceph/.ssh/id_rsa.
Your public key has been saved in /home/ceph/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:uTyNJcrHhxWjSsqbwSrgernfbnxpZbqs9WTNkjoc5As cephadm@node
The key's randomart image is:
+---[RSA 2048]----+
| |
| |
| o |
| .o o |
| .oS o |
|. o +E=oOo+ |
|o .=.+oXBB o |
|..o. =o+OB . |
|oooo+o+++o. |
+----[SHA256]-----+
cephadm@node:~$ ssh-copy-id cephadm@192.168.160.128
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/ceph/.ssh/id_rsa.pub"
The authenticity of host '192.168.160.128 (192.168.160.128)' can't be established.
ECDSA key fingerprint is SHA256:XhYuJgB5QONz1yKl8gPQ9qwji3Mlcj+j8LU4D2/LFd4.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephadm@192.168.160.128's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'ceph@192.168.160.128'"
and check to make sure that only the key(s) you wanted were added.
cephadm@node:~$ ssh-copy-id ceph@192.168.160.129
cephadm@node:~$ ssh-copy-id ceph@192.168.160.130ceph-deploy工具安装
安装deploy工具
root@mon:~# apt install ceph-deploy
初始化mom节点准备
root@mon:~# su - ceph
cephadm@mon:~$ mkdir ceph-cluster/ #保存集群初始化配置信息
cephadm@mon:~$ cd ceph-cluster/
cephadm@mon:~/ceph-cluster$
使用
ceph-deploy --help
cpeh-deploy mon --help
new：开始部署一个新到ceph存储集群，并生成cluster.conf集群配置文件和keyring认证文件
install:在远程主机上安装ceph相关软件包，也可以通过--release指定安装版本
rgw：管理rgw守护程序（radosgw，对象存储网关）
mgr:管理mgr守护程序（ceph-mgr，ceph manager daemonceph管理器守护程序）
mds：管理mds守护程序（ceph metadata serverceph源数据服务器）
mon:管理mon守护程序（ceph-mon,ceph监视器）
gatherkeys：从指定获取提供新节点的验证keys,这些key会在添加新到mon/osd/md加入的时候使用
disk：管理远程主机磁盘
osd：在原从主机准备数据磁盘，即将指定远程主机的指定磁盘添加到ceph集群作为osd使用
repo：远程主机仓库
admin：推送ceph集群配置文件和client.admin文件到远程主机
config：将ceph.conf配置文件推送到远程主机或从远程主机拷贝
uninstall：从远程主机删除安装包
purgedata：从/var/lib/ceph 删除ceph数据，回删除/etc/ceph下的内容
forgetkeys：从本地主机删除所有的验证keying，包括client.admin，monitor，bootstrap等认证文件
pkg：管理远端主机的安装包
calamari：安装并配置一个calamari web节点，calamari是一个web监控节点
所有节点安装python2
apt install python2.7 -y
所有节点链接python2
root@nmon:~# ln -sv /usr/bin/python2.7 /usr/bin/python2
'/usr/bin/python2' -> '/usr/bin/python2.7'
初始化
cephadm@mon:~$ mkdir cluster && cd cluster
cephadm@mon:~/cluster$
cephadm@mon:~/cluster$ ls
cephadm@mon:~/cluster$ ls
cephadm@mon:~/cluster$
cephadm@mon:~/cluster$ ceph-deploy new --public-network 192.168.160.0/24 --cluster-network 10.0.0.0/24 mon
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.38): /usr/bin/ceph-deploy new --public-network 192.168.160.0/24 --cluster-network 10.0.0.0/24 mon
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fa94bfb6dc0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['mon']
[ceph_deploy.cli][INFO ] func : <function new at 0x7fa949522b50>
[ceph_deploy.cli][INFO ] public_network : 192.168.160.0/24
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : 10.0.0.0/24
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[mon][DEBUG ] connection detected need for sudo
[mon][DEBUG ] connected to host: mon
#初始化 mon 节点
cephadm@mon:~/cluster$ceph-deploy --overwrite-conf mon create-initial
[mon][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.mon.asok mon_status
[mon][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon/keyring auth get client.admin
[mon][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon/keyring auth get client.bootstrap-mds
[mon][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon/keyring auth get client.bootstrap-mgr
[mon][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon/keyring auth get client.bootstrap-osd
[mon][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-mon/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpi_co7T
cephadm@mon:~/cluster$
cephadm@mon:~/cluster$ ls -l
总用量 48
-rw------- 1 cephadm cephadm 113 12月 21 10:47 ceph.bootstrap-mds.keyring
-rw------- 1 cephadm cephadm 113 12月 21 10:47 ceph.bootstrap-mgr.keyring
-rw------- 1 cephadm cephadm 113 12月 21 10:47 ceph.bootstrap-osd.keyring
-rw------- 1 cephadm cephadm 113 12月 21 10:47 ceph.bootstrap-rgw.keyring
-rw------- 1 cephadm cephadm 151 12月 21 10:47 ceph.client.admin.keyring
-rw-rw-r-- 1 cephadm cephadm 260 12月 21 10:26 ceph.conf
-rw-rw-r-- 1 cephadm cephadm 16495 12月 21 10:47 ceph-deploy-ceph.log
-rw------- 1 cephadm cephadm 73 12月 21 10:26 ceph.mon.keyring
cephadm@mon:~/cluster$
#将密钥推送到/etc/ceph/目录下
cephadm@mon:~/cluster$ ceph-deploy admin mon
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.38): /usr/bin/ceph-deploy admin mon
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4366a0b050>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : ['mon']
[ceph_deploy.cli][INFO ] func : <function admin at 0x7f4367384ad0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to mon
[mon][DEBUG ] connection detected need for sudo
[mon][DEBUG ] connected to host: mon
[mon][DEBUG ] detect platform information from remote host
[mon][DEBUG ] detect machine type
[mon][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
#查看集群信息，无法找到密钥，因为没有权限
cephadm@mon:~/cluster$ ceph -s
2022-12-21T10:59:35.378+0800 7f87cab6f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2022-12-21T10:59:35.378+0800 7f87cab6f700 -1 AuthRegistry(0x7f87c405bc38) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2022-12-21T10:59:35.382+0800 7f87cab6f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2022-12-21T10:59:35.382+0800 7f87cab6f700 -1 AuthRegistry(0x7f87bc004a50) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2022-12-21T10:59:35.382+0800 7f87cab6f700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2022-12-21T10:59:35.382+0800 7f87cab6f700 -1 AuthRegistry(0x7f87cab6dff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
[errno 2] RADOS object not found (error connecting to the cluster)
#安装acl软件包
cephadm@mon:~/cluster$ sudo apt install acl -y
#设置cephadm给密钥文件读权限
cephadm@mon:~/cluster$ sudo setfacl -m u:cephadm:r /etc/ceph/ceph.client.admin.keyring
#查看集群
cephadm@mon:~/cluster$ ceph -s
cluster:
id: 87dcbebf-73ba-4d30-b620-9653b2446c76
health: HEALTH_WARN
mon is allowing insecure global_id reclaim
services:
mon: 1 daemons, quorum mon (age 14m)
mgr: no daemons active
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
#关掉集群告警
cephadm@mon:~/cluster$ ceph config set mon auth_allow_insecure_global_id_reclaim false
cephadm@mon:~/cluster$ ceph -s
cluster:
id: 87dcbebf-73ba-4d30-b620-9653b2446c76
health: HEALTH_OK
services:
mon: 1 daemons, quorum mon (age 16m)
mgr: no daemons active
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
cephadm@mon:~/cluster$添加mgr节点
#添加mgr节点
cephadm@mon:~/cluster$ ceph-deploy install --mgr node
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.38): /usr/bin/ceph-deploy install --mgr node
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] testing : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7ff306795910>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] dev_commit : None
[ceph_deploy.cli][INFO ] install_mds : False
[ceph_deploy.cli][INFO ] stable : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] adjust_repos : False
[ceph_deploy.cli][INFO ] func : <function install at 0x7ff3070c5a50>
[ceph_deploy.cli][INFO ] install_mgr : True
[ceph_deploy.cli][INFO ] install_all : False
[ceph_deploy.cli][INFO ] repo : False
[ceph_deploy.cli][INFO ] host : ['node']
[ceph_deploy.cli][INFO ] install_rgw : False
[ceph_deploy.cli][INFO ] install_tests : False
[ceph_deploy.cli][INFO ] repo_url : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] install_osd : False
[ceph_deploy.cli][INFO ] version_kind : stable
[ceph_deploy.cli][INFO ] install_common : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] dev : master
[ceph_deploy.cli][INFO ] nogpgcheck : False
[ceph_deploy.cli][INFO ] local_mirror : None
[ceph_deploy.cli][INFO ] release : None
[ceph_deploy.cli][INFO ] install_mon : False
[ceph_deploy.cli][INFO ] gpg_url : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts node
[ceph_deploy.install][DEBUG ] Detecting platform for host node ...
The authenticity of host 'node (192.168.160.128)' can't be established.
ECDSA key fingerprint is SHA256:XhYuJgB5QONz1yKl8gPQ9qwji3Mlcj+j8LU4D2/LFd4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node' (ECDSA) to the list of known hosts.
[node][DEBUG ] connection detected need for sudo
[node][DEBUG ] connected to host: node
[node][DEBUG ] detect platform information from remote host
[node][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: Ubuntu 18.04 bionic
[node][INFO ] installing Ceph on node
[node][INFO ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[node][DEBUG ] 正在读取软件包列表...
[node][DEBUG ] 正在分析软件包的依赖关系树...
[node][DEBUG ] 正在读取状态信息...
#mgr节点查看ceph包
root@node:~# dpkg -l |grep ceph
ii ceph-base 16.2.10-1bionic amd64 common ceph daemon libraries and management tools
ii ceph-common 16.2.10-1bionic amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-mgr 16.2.10-1bionic amd64 managerfor the ceph distributed storage system
ii ceph-mgr-modules-core 16.2.10-1bionic all ceph manager modules which are always enabled
ii libcephfs2 16.2.10-1bionic amd64 Ceph distributed file system client library
ii python3-ceph-argparse 16.2.10-1bionic all Python 3 utility libraries for Ceph CLI
ii python3-ceph-common 16.2.10-1bionic all Python 3 utility libraries for Ceph
ii python3-cephfs 16.2.10-1bionic amd64 Python 3 libraries for the Ceph libcephfs library
#查看集群
root@mon:~# ceph -s
cluster:
id: 87dcbebf-73ba-4d30-b620-9653b2446c76
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum mon (age 35s)
mgr: node(active, since 15s)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:添加osd节点
#安装osd
cephadm@mon:~/cluster$ ceph-deploy install --release pacific --osd node-1 node-2
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.38): /usr/bin/ceph-deploy install --release pacific --osd node-1 node-2
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] testing : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f1151dac910>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] dev_commit : None
[ceph_deploy.cli][INFO ] install_mds : False
[ceph_deploy.cli][INFO ] stable : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] adjust_repos : False
[ceph_deploy.cli][INFO ] func : <function install at 0x7f11526dca50>
[ceph_deploy.cli][INFO ] install_mgr : False
[ceph_deploy.cli][INFO ] install_all : False
[ceph_deploy.cli][INFO ] repo : False
[ceph_deploy.cli][INFO ] host : ['node-1', 'node-2']
[ceph_deploy.cli][INFO ] install_rgw : False
[ceph_deploy.cli][INFO ] install_tests : False
[ceph_deploy.cli][INFO ] repo_url : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] install_osd : True
[ceph_deploy.cli][INFO ] version_kind : stable
[ceph_deploy.cli][INFO ] install_common : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] dev : master
[ceph_deploy.cli][INFO ] nogpgcheck : False
[ceph_deploy.cli][INFO ] local_mirror : None
[ceph_deploy.cli][INFO ] release : pacific
[ceph_deploy.cli][INFO ] install_mon : False
[ceph_deploy.cli][INFO ] gpg_url : None
[ceph_deploy.install][DEBUG ] Installing stable version pacific on cluster ceph hosts node-1 node-2
[ceph_deploy.install][DEBUG ] Detecting platform for host node
#查看osd节点硬盘信息
cephadm@mon:~/cluster$ ceph-deploy disk list node-1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy disk list node-1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : list
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc56c9fa280>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : ['node-1']
[ceph_deploy.cli][INFO ] func : <function disk at 0x7fc56c9cf350>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[node-1][DEBUG ] connection detected need for sudo
[node-1][DEBUG ] connected to host: node-1
[node-1][DEBUG ] detect platform information from remote host
[node-1][DEBUG ] detect machine type
[node-1][DEBUG ] find the location of an executable
[node-1][INFO ] Running command: sudo fdisk -l
[node-1][INFO ] Disk /dev/sdb：20 GiB，21474836480 字节，41943040 个扇区
[node-1][INFO ] Disk /dev/sda：50 GiB，53687091200 字节，104857600 个扇区
[node-1][INFO ] Disk /dev/sdc：20 GiB，21474836480 字节，41943040 个扇区
cephadm@mon:~/cluster$ ceph-deploy disk list node-2
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy disk list node-2
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : list
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f79b9041280>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : ['node-2']
[ceph_deploy.cli][INFO ] func : <function disk at 0x7f79b9016350>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[node-2][DEBUG ] connection detected need for sudo
[node-2][DEBUG ] connected to host: node-2
[node-2][DEBUG ] detect platform information from remote host
[node-2][DEBUG ] detect machine type
[node-2][DEBUG ] find the location of an executable
[node-2][INFO ] Running command: sudo fdisk -l
[node-2][INFO ] Disk /dev/sda：50 GiB，53687091200 字节，104857600 个扇区
[node-2][INFO ] Disk /dev/sdc：20 GiB，21474836480 字节，41943040 个扇区
[node-2][INFO ] Disk /dev/sdb：20 GiB，21474836480 字节，41943040 个扇区
cephadm@mon:~/cluster$
#添加osd硬盘
cephadm@mon:~/cluster$ ceph-deploy --overwrite-conf osd create node-1 --data /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf osd create node-1 --data /dev/sdb
[node-1][WARNIN] Running command: /bin/systemctl start ceph-osd@0
[node-1][WARNIN] --> ceph-volume lvm activate successful for osd ID: 0
[node-1][WARNIN] --> ceph-volume lvm create successful for: /dev/sdb
[node-1][INFO ] checking OSD status...
[node-1][DEBUG ] find the location of an executable
[node-1][INFO ] Running command: sudo /usr/bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node-1 is now ready for osd use.
#查看osd磁盘
cephadm@mon:~/cluster$ ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 node-1 4896k 19.9G 0 0 0 0 exists,up
cephadm@mon:~/cluster$ ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 node-1 4896k 19.9G 0 0 0 0 exists,up
cephadm@mon:~/cluster$ ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 hdd 0.01949 1.00000 20 GiB 4.8 MiB 88 KiB 0 B 4.7 MiB 20 GiB 0.02 1.00 0 up
TOTAL 20 GiB 4.8 MiB 88 KiB 0 B 4.7 MiB 20 GiB 0.02
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
#继续添加硬盘
cephadm@mon:~/cluster$ ceph-deploy --overwrite-conf osd create node-1 --data /dev/sdc
cephadm@mon:~/cluster$ ceph-deploy --overwrite-conf osd create node-2 --data /dev/sdb
cephadm@mon:~/cluster$ ceph-deploy --overwrite-conf osd create node-2 --data /dev/sdc
cephadm@mon:~/cluster$ ceph -s
cluster:
id: 87dcbebf-73ba-4d30-b620-9653b2446c76
health: HEALTH_OK
services:
mon: 1 daemons, quorum mon (age 2h)
mgr: node(active, since 2h)
osd: 4 osds: 4 up (since 23s), 4 in (since 30s); 1 remapped pgs
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 21 MiB used, 80 GiB / 80 GiB avail
pgs: 1 active+clean
cephadm@mon:~/cluster$ ceph osd ls
0
1
2
3
cephadm@mon:~/cluster$
root@mon:~# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 node-1 5360k 19.9G 0 0 0 0 exists,up
1 node-1 5296k 19.9G 0 0 0 0 exists,up
2 node-2 5296k 19.9G 0 0 0 0 exists,up
3 node-2 5228k 19.9G 0 0 0 0 exists,up添加其他mon节点
cephadm@mon:~/cluster$ ceph-deploy mon add mon-1添加其他mgr节点
root@node-2:~# apt install ceph-mgr
cephadm@mon:~/cluster$ ceph-deploy mgr create node-2