Ceph块存储和对象存储

1 案例1：块存储应用案例
1.1 问题

延续Day04的实验内容，演示块存储在KVM虚拟化中的应用案例，实现以下功能：
Ceph创建块存储镜像
客户端安装部署ceph软件
客户端部署虚拟机
客户端创建secret
设置虚拟机配置文件，调用ceph存储
1.2 方案

使用Ceph存储创建镜像。
KVM虚拟机调用Ceph镜像作为虚拟机的磁盘。
1.3 步骤

实现此案例需要按照如下步骤进行。
1）创建磁盘镜像。
[root@node1 ~]# rbd create vm1-image –image-feature layering –size 10G
[root@node1 ~]# rbd create vm2-image –image-feature layering –size 10G
[root@node1 ~]# rbd list
[root@node1 ~]# rbd info vm1-image
[root@node1 ~]# qemu-img info rbd:rbd/vm1-image
image: rbd:rbd/vm1-image
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: unavailable
2）Ceph认证账户。
Ceph默认开启用户认证，客户端需要账户才可以访问，
默认账户名称为client.admin，key是账户的密钥，
可以使用ceph auth添加新账户（案例我们使用默认账户）。
[root@node1 ~]# cat /etc/ceph/ceph.conf //配置文件
[global]
mon_initial_members = node1, node2, node3
mon_host = 192.168.2.10,192.168.2.20,192.168.2.30
auth_cluster_required = cephx //开启认证
auth_service_required = cephx //开启认证
auth_client_required = cephx //开启认证
[root@node1 ~]# cat /etc/ceph/ceph.client.admin.keyring //账户文件
[client.admin]
key = AQBTsdRapUxBKRAANXtteNUyoEmQHveb75bISg==
代码如下：
注意：配置文件不用更改
[root@node1 ceph-cluster]# rbd list
demo-image
image
image-clone
vm1-image
vm2-image
[root@node1 ceph-cluster]#
[root@node1 ceph-cluster]# rbd info vm1-image
rbd image ‘vm1-image’:
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.105b238e1f29
format: 2
features: layering
flags:
[root@node1 ceph-cluster]#
[root@node1 ceph-cluster]#
[root@node1 ceph-cluster]# qemu-img info rbd:rbd/vm1-image
image: rbd:rbd/vm1-image
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: unavailable
[root@node1 ceph-cluster]#
[root@node1 ceph-cluster]#
[root@node1 ceph-cluster]# cat /etc/ceph/ceph.conf
[global]
fsid = 29908a48-7574-4aac-ac14-80a44b7cffbf
mon_initial_members = node1, node2, node3
mon_host = 192.168.4.11,192.168.4.12,192.168.4.13
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[root@node1 ceph-cluster]#
[root@node1 ceph-cluster]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQCXwL5bTwkVGxAA2ls+tuA8SMl2vmzQOPCdCg==
[root@node1 ceph-cluster]#

3）部署客户端环境。
注意：这里使用真实机当客户端！！！
客户端需要安装ceph-common软件包，拷贝配置文件（否则不知道集群在哪），
拷贝连接密钥（否则无连接权限）。
[root@room9pc01 ~]# yum -y install ceph-common
[root@room9pc01 ~]# scp 192.168.4.11:/etc/ceph/ceph.conf /etc/ceph/
[root@room9pc01 ~]# scp 192.168.4.11:/etc/ceph/ceph.client.admin.keyring \
/etc/ceph/

代码如下
注意，真机一样要先添加repo
[root@room9pc52 yum.repos.d]# vim ceph.repo
[root@room9pc52 yum.repos.d]# cat ceph.repo
[mon]
name=mon
baseurl=ftp://192.168.4.254/ceph/rhceph-2.0-rhel-7-x86_64/MON
gpgcheck=0
[osd]
name=osd
baseurl=ftp://192.168.4.254/ceph/rhceph-2.0-rhel-7-x86_64/OSD
gpgcheck=0
[tools]
name=tools
baseurl=ftp://192.168.4.254/ceph/rhceph-2.0-rhel-7-x86_64/Tools
gpgcheck=0
[root@room9pc52 yum.repos.d]# yum repolist
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* epel: mirrors.ustc.edu.cn
* extras: mirrors.163.com
* updates: mirrors.163.com
file:///cos7/repodata/repomd.xml: [Errno 14] curl#37 – “Couldn’t open file /cos7/repodata/repomd.xml”
正在尝试其它镜像。
mon | 4.1 kB 00:00
osd | 4.1 kB 00:00
tools | 3.8 kB 00:00
(1/6): mon/primary_db | 40 kB 00:00
(2/6): mon/group_gz | 489 B 00:00
(3/6): tools/primary_db | 31 kB 00:00
(4/6): osd/primary_db | 31 kB 00:00
(5/6): osd/group_gz | 447 B 00:00
(6/6): tools/group_gz | 459 B 00:00
源标识源名称状态
base/7/x86_64 CentOS-7 – Base 9,911
!cos7 cos7 9,591
epel/x86_64 Extra Packages for Enterprise Linux 7 – x86_6 12,726
extras/7/x86_64 CentOS-7 – Extras 432
librehat-shadowsocks/x86_64 Copr repo for shadowsocks owned by librehat 54
mon mon 41
osd osd 28
teamviewer/x86_64 TeamViewer – x86_64 18
tools tools 33
updates/7/x86_64 CentOS-7 – Updates 1,543
repolist: 34,377

安装软件并拷贝配置文件
[root@room9pc52 yum.repos.d]# yum install -y ceph-common
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* epel: mirrors.ustc.edu.cn
* extras: mirrors.163.com
* updates: mirrors.163.com
file:///cos7/repodata/repomd.xml: [Errno 14] curl#37 – “Couldn’t open file /cos7/repodata/repomd.xml”
正在尝试其它镜像。
正在解决依赖关系
–> 正在检查事务
—> 软件包 ceph-common.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在处理依赖关系 libcephfs1 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 librados2 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 librbd1 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 python-cephfs = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 python-rados = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 python-rbd = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libbabeltrace-ctf.so.1()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libbabeltrace.so.1()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libboost_iostreams-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libboost_program_options-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libboost_regex-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 librgw.so.2()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在检查事务
—> 软件包 boost-iostreams.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 boost-program-options.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 boost-regex.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 libbabeltrace.x86_64.0.1.2.4-3.el7cp 将被安装
—> 软件包 libcephfs1.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在处理依赖关系 libboost_random-mt.so.1.53.0()(64bit)，它被软件包 1:libcephfs1-10.2.2-38.el7cp.x86_64 需要
—> 软件包 librados2.x86_64.1.0.94.5-2.el7 将被升级
—> 软件包 librados2.x86_64.1.10.2.2-38.el7cp 将被更新
–> 正在处理依赖关系 liblttng-ust.so.0()(64bit)，它被软件包 1:librados2-10.2.2-38.el7cp.x86_64 需要
—> 软件包 librbd1.x86_64.1.0.94.5-2.el7 将被升级
—> 软件包 librbd1.x86_64.1.10.2.2-38.el7cp 将被更新
—> 软件包 librgw2.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在处理依赖关系 libfcgi.so.0()(64bit)，它被软件包 1:librgw2-10.2.2-38.el7cp.x86_64 需要
—> 软件包 python-cephfs.x86_64.1.10.2.2-38.el7cp 将被安装
—> 软件包 python-rados.x86_64.1.10.2.2-38.el7cp 将被安装
—> 软件包 python-rbd.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在检查事务
—> 软件包 boost-random.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 fcgi.x86_64.0.2.4.0-25.el7cp 将被安装
—> 软件包 lttng-ust.x86_64.0.2.4.1-4.el7 将被安装
–> 正在处理依赖关系 liburcu-cds.so.1()(64bit)，它被软件包 lttng-ust-2.4.1-4.el7.x86_64 需要
–> 正在处理依赖关系 liburcu-bp.so.1()(64bit)，它被软件包 lttng-ust-2.4.1-4.el7.x86_64 需要
–> 正在检查事务
—> 软件包 userspace-rcu.x86_64.0.0.7.16-1.el7 将被安装
–> 解决依赖关系完成

依赖关系解决

事务概要
================================================================================
安装 1 软件包 (+13 依赖软件包)
升级 ( 2 依赖软件包)

总下载量：26 M
Downloading packages:
No Presto metadata available for mon
(1/16): boost-iostreams-1.53.0-27.el7.x86_64.rpm | 61 kB 00:00
(2/16): boost-program-options-1.53.0-27.el7.x86_64.rpm | 156 kB 00:00
(3/16): boost-random-1.53.0-27.el7.x86_64.rpm | 39 kB 00:00
(4/16): fcgi-2.4.0-25.el7cp.x86_64.rpm | 47 kB 00:00
(5/16): libbabeltrace-1.2.4-3.el7cp.x86_64.rpm | 147 kB 00:00
(6/16): boost-regex-1.53.0-27.el7.x86_64.rpm | 300 kB 00:00
(7/16): libcephfs1-10.2.2-38.el7cp.x86_64.rpm | 1.9 MB 00:00
(8/16): librados2-10.2.2-38.el7cp.x86_64.rpm | 1.9 MB 00:00
(9/16): librbd1-10.2.2-38.el7cp.x86_64.rpm | 2.5 MB 00:00
(10/16): librgw2-10.2.2-38.el7cp.x86_64.rpm | 2.9 MB 00:00
(11/16): python-cephfs-10.2.2-38.el7cp.x86_64.rpm | 86 kB 00:00
(12/16): python-rados-10.2.2-38.el7cp.x86_64.rpm | 164 kB 00:00
(13/16): ceph-common-10.2.2-38.el7cp.x86_64.rpm | 16 MB 00:00
(14/16): python-rbd-10.2.2-38.el7cp.x86_64.rpm | 93 kB 00:00
(15/16): lttng-ust-2.4.1-4.el7.x86_64.rpm | 176 kB 00:00
(16/16): userspace-rcu-0.7.16-1.el7.x86_64.rpm | 73 kB 00:00
——————————————————————————–
总计 14 MB/s | 26 MB 00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : boost-iostreams-1.53.0-27.el7.x86_64 1/18
正在安装 : boost-random-1.53.0-27.el7.x86_64 2/18
正在安装 : 1:libcephfs1-10.2.2-38.el7cp.x86_64 3/18
正在安装 : boost-program-options-1.53.0-27.el7.x86_64 4/18
正在安装 : fcgi-2.4.0-25.el7cp.x86_64 5/18
正在安装 : userspace-rcu-0.7.16-1.el7.x86_64 6/18
正在安装 : lttng-ust-2.4.1-4.el7.x86_64 7/18
正在更新 : 1:librados2-10.2.2-38.el7cp.x86_64 8/18
正在安装 : 1:python-rados-10.2.2-38.el7cp.x86_64 9/18
正在更新 : 1:librbd1-10.2.2-38.el7cp.x86_64 10/18
正在安装 : 1:python-rbd-10.2.2-38.el7cp.x86_64 11/18
正在安装 : 1:python-cephfs-10.2.2-38.el7cp.x86_64 12/18
正在安装 : 1:librgw2-10.2.2-38.el7cp.x86_64 13/18
正在安装 : libbabeltrace-1.2.4-3.el7cp.x86_64 14/18
正在安装 : boost-regex-1.53.0-27.el7.x86_64 15/18
正在安装 : 1:ceph-common-10.2.2-38.el7cp.x86_64 16/18
清理 : 1:librbd1-0.94.5-2.el7.x86_64 17/18
清理 : 1:librados2-0.94.5-2.el7.x86_64 18/18
验证中 : 1:ceph-common-10.2.2-38.el7cp.x86_64 1/18
验证中 : 1:python-cephfs-10.2.2-38.el7cp.x86_64 2/18
验证中 : 1:librbd1-10.2.2-38.el7cp.x86_64 3/18
验证中 : boost-regex-1.53.0-27.el7.x86_64 4/18
验证中 : libbabeltrace-1.2.4-3.el7cp.x86_64 5/18
验证中 : userspace-rcu-0.7.16-1.el7.x86_64 6/18
验证中 : lttng-ust-2.4.1-4.el7.x86_64 7/18
验证中 : boost-iostreams-1.53.0-27.el7.x86_64 8/18
验证中 : 1:librados2-10.2.2-38.el7cp.x86_64 9/18
验证中 : fcgi-2.4.0-25.el7cp.x86_64 10/18
验证中 : boost-random-1.53.0-27.el7.x86_64 11/18
验证中 : boost-program-options-1.53.0-27.el7.x86_64 12/18
验证中 : 1:python-rbd-10.2.2-38.el7cp.x86_64 13/18
验证中 : 1:python-rados-10.2.2-38.el7cp.x86_64 14/18
验证中 : 1:libcephfs1-10.2.2-38.el7cp.x86_64 15/18
验证中 : 1:librgw2-10.2.2-38.el7cp.x86_64 16/18
验证中 : 1:librbd1-0.94.5-2.el7.x86_64 17/18
验证中 : 1:librados2-0.94.5-2.el7.x86_64 18/18

已安装:
ceph-common.x86_64 1:10.2.2-38.el7cp

作为依赖被升级:
librados2.x86_64 1:10.2.2-38.el7cp librbd1.x86_64 1:10.2.2-38.el7cp

完毕！

[root@room9pc52 ~]# scp 192.168.4.11:/etc/ceph/ceph.conf /etc/ce
centos-release centos-release-upstream ceph/
[root@room9pc52 ~]# scp 192.168.4.11:/etc/ceph/ceph.conf /etc/ce
centos-release centos-release-upstream ceph/
[root@room9pc52 ~]# scp 192.168.4.11:/etc/ceph/ceph.conf /etc/ceph/
[email protected]’s password:
ceph.conf 100% 235 670.5KB/s 00:00
[root@room9pc52 ~]# scp 192.168.4.11:/etc/ceph/ceph.client.admin.keyring /etc/ceph/
[email protected]’s password:
ceph.client.admin.keyring 100% 63 141.1KB/s 00:00
[root@room9pc52 ~]# ll /etc/ceph
总用量 12
-rw——- 1 root root 63 10月 12 09:31 ceph.client.admin.keyring
-rw-r–r– 1 root root 235 10月 12 09:30 ceph.conf
-rwxr-xr-x 1 root root 92 8月 10 2016 rbdmap

4）创建KVM虚拟机。
使用virt-manager创建2台普通的KVM虚拟机。
我创建了2台新的虚拟机，1台14，1台15

5）配置libvirt secret。
编写账户信息文件（真实机操作）
[root@room9pc01 ~]# vim secret.xml //新建临时文件，内容如下
<secret ephemeral=’no’ private=’no’>
<usage type=’ceph’>
<name>client.admin secret</name>
</usage>
</secret>
#使用XML配置文件创建secret
代码如下
[root@room9pc52 ~]# vim secret.xml
[root@room9pc52 ~]#
[root@room9pc52 ~]# cat secret.xml
<secret ephemeral=’no’ private=’no’>
<usage type=’ceph’>
<name>client.admin secret</name>
</usage>
</secret>

[root@room9pc01 ~]# virsh secret-define –file secret.xml
733f0fd1-e3d6-4c25-a69f-6681fc19802b
//随机的UUID，这个UUID对应的有账户信息
代码如下，注意这个UUID后面要用到
[root@room9pc52 ~]# virsh secret-define –file secret.xml
生成 secret 2962a576-d80c-48cc-bcdf-a807a8339c64

编写账户信息文件（真实机操作）
[root@room9pc01 ~]# ceph auth get-key client.admin
//获取client.admin的key，或者直接查看密钥文件
[root@room9pc01 ~]# cat /etc/ceph/ceph.client.admin.keyring
代码如下
[root@room9pc52 ~]# ceph auth get-key client.admin
AQCXwL5bTwkVGxAA2ls+tuA8SMl2vmzQOPCdCg==[root@room9pc52 ~]#
[root@room9pc52 ~]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQCXwL5bTwkVGxAA2ls+tuA8SMl2vmzQOPCdCg==

设置secret，添加账户的密钥
[root@room9pc01] virsh secret-set-value \
–secret 733f0fd1-e3d6-4c25-a69f-6681fc19802b \
–base64 AQBTsdRapUxBKRAANXtteNUyoEmQHveb75bISg
//这里secret后面是之前创建的secret的UUID
//base64后面是client.admin账户的密码
//现在secret中既有账户信息又有密钥信息
代码如下
[root@room9pc52 ~]# virsh secret-set-value –secret 2962a576-d80c-48cc-bcdf-a807a8339c64 –base64 AQCXwL5bTwkVGxAA2ls+tuA8SMl2vmzQOPCdCg==
secret 值设定
（注意，本次实验中，secret是上面生成的UUID,而base是后面生成的key）

6）虚拟机的XML配置文件。
每个虚拟机都会有一个XML配置文件，包括：
虚拟机的名称、内存、CPU、磁盘、网卡等信息
[root@room9pc01 ~]# vim /etc/libvirt/qemu/vm1.xml
//修改前内容如下
<disk type=’file’ device=’disk’>
<driver name=’qemu’ type=’qcow2’/>
<source file=’/var/lib/libvirt/images/vm1.qcow2’/>
<target dev=’vda’ bus=’virtio’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x07′ function=’0x0’/>
</disk>
不推荐直接使用vim修改配置文件，推荐使用virsh edit修改配置文件，效果如下：
[root@room9pc01] virsh edit vm1 //vm1为虚拟机名称
<disk type=’network’ device=’disk’>
<driver name=’qemu’ type=’raw’/>
<auth username=’admin’>
<secret type=’ceph’ uuid=’733f0fd1-e3d6-4c25-a69f-6681fc19802b’/>
</auth>
<source protocol=’rbd’ name=’rbd/vm1′> <host name=’192.168.4.11′ port=’6789’/> </source>
<target dev=’vda’ bus=’virtio’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x07′ function=’0x0’/>
</disk>

代码如下
（注意就是修改UUID要保持一直）
[root@room9pc52 ~]# virsh edit a14
没有更改域 a14 XML 配置。

[root@room9pc52 ~]#
(修改以下信息)
<disk type=’network’ device=’disk’>
<driver name=’qemu’ type=’raw’/>
<auth username=’admin’>
<secret type=’ceph’ uuid=’2962a576-d80c-48cc-bcdf-a807a8339c64’/>
</auth>
<source protocol=’rbd’ name=’rbd/vm1-image’>
<host name=’192.168.4.11′ port=’6789’/>
</source>
<target dev=’vda’ bus=’virtio’/>
<address type=’pci’ domain=’0x0000′ bus=’0x00′ slot=’0x07′ function=’0x0’/>
</disk>
注意，上面有一个坑，name=’rbd/vm1-iamge’ ceph创建的是什么镜像名，就写什么名，不要直接复制粘贴
完成后启动虚拟机，正常安装系统即可

拓展
真机上面也可以正常使用
[root@room9pc52 ~]# rbd ls
demo-image
image
image-clone
vm1-image
vm2-image
[root@room9pc52 ~]# rbd info vm1-image
rbd image ‘vm1-image’:
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.105b238e1f29
format: 2
features: layering
flags:
[root@room9pc52 ~]# qemu-img info rbd:rbd/vm1-image
image: rbd:rbd/vm1-image
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: unavailable
查看密钥
[root@room9pc52 ~]# virsh secret-list
UUID 用量
——————————————————————————–
2962a576-d80c-48cc-bcdf-a807a8339c64 ceph client.admin secret

2 案例2：Ceph文件系统
2.1 问题
延续前面的实验，实现Ceph文件系统的功能。具体实现有以下功能：
部署MDSs节点
创建Ceph文件系统
客户端挂载文件系统
2.2 方案
添加一台虚拟机，部署MDS节点。
主机的主机名及对应的IP地址如表-1所示。
node4 192.168.4.15
先配置好免密登陆
[root@node1 ceph-cluster]# ssh-copy-id 192.168.4.14
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: “/root/.ssh/id_rsa.pub”
The authenticity of host ‘192.168.4.14 (192.168.4.14)’ can’t be established.
ECDSA key fingerprint is SHA256:59Byua15gv4TAVEm7YutMtmgXfxGTYbzSfO84kICu0E.
ECDSA key fingerprint is MD5:7e:6e:95:ed:84:fe:5b:8e:26:94:73:9b:c3:4b:90:cb.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed — if you are prompted now it is to install the new keys
[email protected]’s password:

Number of key(s) added: 1

Now try logging into the machine, with: “ssh ‘192.168.4.14’”
and check to make sure that only the key(s) you wanted were added.

2.3 步骤

实现此案例需要按照如下步骤进行。
1）添加一台新的虚拟机，要求如下：
IP地址:192.168.4.14
主机名:node4
配置yum源（包括rhel、ceph的源）
与Client主机同步时间
node1允许无密码远程node4
2）部署元数据服务器
登陆node4，安装ceph-mds软件包
[root@node4 ~]# yum -y install ceph-mds
登陆node1部署节点操作
[root@node1 ~]# cd /root/ceph-cluster
//该目录，是最早部署ceph集群时，创建的目录
[root@node1 ceph-cluster]# ceph-deploy mds create node4
//给nod4拷贝配置文件，启动mds服务

以下为代码
NODE4先配置好，这里是直接再NODE4上面安装MDS，而不是在管理主机上面安装，同时配置好时间同步NTP
[root@node4 ~]# vim /etc/chrony.conf
[root@node4 ~]# systemctl restart chronyd
[root@node4 ~]# yum -y install ceph-mds
已加载插件：langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
mon | 4.1 kB 00:00
osd | 4.1 kB 00:00
tools | 3.8 kB 00:00
(1/6): mon/group_gz | 489 B 00:00
(2/6): mon/primary_db | 40 kB 00:00
(3/6): osd/group_gz | 447 B 00:00
(4/6): tools/primary_db | 31 kB 00:00
(5/6): osd/primary_db | 31 kB 00:00
(6/6): tools/group_gz | 459 B 00:00
正在解决依赖关系
–> 正在检查事务
—> 软件包 ceph-mds.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在处理依赖关系 ceph-base = 1:10.2.2-38.el7cp，它被软件包 1:ceph-mds-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libboost_iostreams-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-mds-10.2.2-38.el7cp.x86_64 需要
–> 正在检查事务
—> 软件包 boost-iostreams.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 ceph-base.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在处理依赖关系 ceph-common = 1:10.2.2-38.el7cp，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 ceph-selinux = 1:10.2.2-38.el7cp，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libcephfs1 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 librados2 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 librbd1 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 librgw2 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 hdparm，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 redhat-lsb-core，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libboost_random-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libcephfs.so.1()(64bit)，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 liblttng-ust.so.0()(64bit)，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
–> 正在检查事务
—> 软件包 boost-random.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 ceph-common.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在处理依赖关系 python-cephfs = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 python-rados = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 python-rbd = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libbabeltrace-ctf.so.1()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libbabeltrace.so.1()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libboost_program_options-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
–> 正在处理依赖关系 libboost_regex-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
—> 软件包 ceph-selinux.x86_64.1.10.2.2-38.el7cp 将被安装
—> 软件包 hdparm.x86_64.0.9.43-5.el7 将被安装
—> 软件包 libcephfs1.x86_64.1.10.2.2-38.el7cp 将被安装
—> 软件包 librados2.x86_64.1.0.94.5-2.el7 将被升级
—> 软件包 librados2.x86_64.1.10.2.2-38.el7cp 将被更新
—> 软件包 librbd1.x86_64.1.0.94.5-2.el7 将被升级
—> 软件包 librbd1.x86_64.1.10.2.2-38.el7cp 将被更新
—> 软件包 librgw2.x86_64.1.10.2.2-38.el7cp 将被安装
–> 正在处理依赖关系 libfcgi.so.0()(64bit)，它被软件包 1:librgw2-10.2.2-38.el7cp.x86_64 需要
—> 软件包 lttng-ust.x86_64.0.2.4.1-1.el7cp 将被安装
–> 正在处理依赖关系 liburcu-bp.so.1()(64bit)，它被软件包 lttng-ust-2.4.1-1.el7cp.x86_64 需要
–> 正在处理依赖关系 liburcu-cds.so.1()(64bit)，它被软件包 lttng-ust-2.4.1-1.el7cp.x86_64 需要
—> 软件包 redhat-lsb-core.x86_64.0.4.1-27.el7 将被安装
–> 正在处理依赖关系 redhat-lsb-submod-security(x86-64) = 4.1-27.el7，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
–> 正在处理依赖关系 /usr/bin/m4，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
–> 正在处理依赖关系 /usr/bin/patch，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
–> 正在处理依赖关系 spax，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
–> 正在检查事务
—> 软件包 boost-program-options.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 boost-regex.x86_64.0.1.53.0-27.el7 将被安装
—> 软件包 fcgi.x86_64.0.2.4.0-25.el7cp 将被安装
—> 软件包 libbabeltrace.x86_64.0.1.2.4-3.el7cp 将被安装
—> 软件包 m4.x86_64.0.1.4.16-10.el7 将被安装
—> 软件包 patch.x86_64.0.2.7.1-8.el7 将被安装
—> 软件包 python-cephfs.x86_64.1.10.2.2-38.el7cp 将被安装
—> 软件包 python-rados.x86_64.1.10.2.2-38.el7cp 将被安装
—> 软件包 python-rbd.x86_64.1.10.2.2-38.el7cp 将被安装
—> 软件包 redhat-lsb-submod-security.x86_64.0.4.1-27.el7 将被安装
—> 软件包 spax.x86_64.0.1.5.2-13.el7 将被安装
—> 软件包 userspace-rcu.x86_64.0.0.7.9-2.el7rhgs 将被安装
–> 解决依赖关系完成
192.168.4.254_rhel7/group_gz | 137 kB 00:00

依赖关系解决

================================================================================
Package 架构版本源大小
================================================================================
正在安装:
ceph-mds x86_64 1:10.2.2-38.el7cp tools 2.8 M
为依赖而安装:
boost-iostreams x86_64 1.53.0-27.el7 192.168.4.254_rhel7 61 k
boost-program-options x86_64 1.53.0-27.el7 192.168.4.254_rhel7 156 k
boost-random x86_64 1.53.0-27.el7 192.168.4.254_rhel7 39 k
boost-regex x86_64 1.53.0-27.el7 192.168.4.254_rhel7 300 k
ceph-base x86_64 1:10.2.2-38.el7cp mon 4.2 M
ceph-common x86_64 1:10.2.2-38.el7cp mon 16 M
ceph-selinux x86_64 1:10.2.2-38.el7cp mon 38 k
fcgi x86_64 2.4.0-25.el7cp mon 47 k
hdparm x86_64 9.43-5.el7 192.168.4.254_rhel7 83 k
libbabeltrace x86_64 1.2.4-3.el7cp mon 147 k
libcephfs1 x86_64 1:10.2.2-38.el7cp mon 1.9 M
librgw2 x86_64 1:10.2.2-38.el7cp mon 2.9 M
lttng-ust x86_64 2.4.1-1.el7cp mon 176 k
m4 x86_64 1.4.16-10.el7 192.168.4.254_rhel7 256 k
patch x86_64 2.7.1-8.el7 192.168.4.254_rhel7 110 k
python-cephfs x86_64 1:10.2.2-38.el7cp mon 86 k
python-rados x86_64 1:10.2.2-38.el7cp mon 164 k
python-rbd x86_64 1:10.2.2-38.el7cp mon 93 k
redhat-lsb-core x86_64 4.1-27.el7 192.168.4.254_rhel7 37 k
redhat-lsb-submod-security x86_64 4.1-27.el7 192.168.4.254_rhel7 15 k
spax x86_64 1.5.2-13.el7 192.168.4.254_rhel7 260 k
userspace-rcu x86_64 0.7.9-2.el7rhgs mon 70 k
为依赖而更新:
librados2 x86_64 1:10.2.2-38.el7cp mon 1.9 M
librbd1 x86_64 1:10.2.2-38.el7cp mon 2.5 M

事务概要
================================================================================
安装 1 软件包 (+22 依赖软件包)
升级 ( 2 依赖软件包)

总下载量： : userspace-rcu-0.7.9-2.el7rhgs.x86_64.rpm | 70 kB 00:00
——————————————————————————–
总计 23 MB/s | 34 MB 00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : boost-iostreams-1.53.0-27.el7.x86_64 1/27
正在安装 : boost-random-1.53.0-27.el7.x86_64 2/27
正在安装 : 1:libcephfs1-10.2.2-38.el7cp.x86_64 3/27
正在安装 : fcgi-2.4.0-25.el7cp.x86_64 4/27
正在安装 : spax-1.5.2-13.el7.x86_64 5/27
正在安装 : redhat-lsb-submod-security-4.1-27.el7.x86_64 6/27
正在安装 : patch-2.7.1-8.el7.x86_64 7/27
正在安装 : boost-program-options-1.53.0-27.el7.x86_64 8/27
正在安装 : hdparm-9.43-5.el7.x86_64 9/27
正在安装 : m4-1.4.16-10.el7.x86_64 10/27
正在安装 : redhat-lsb-core-4.1-27.el7.x86_64 11/27
正在安装 : libbabeltrace-1.2.4-3.el7cp.x86_64 12/27
正在安装 : boost-regex-1.53.0-27.el7.x86_64 13/27
正在安装 : userspace-rcu-0.7.9-2.el7rhgs.x86_64 14/27
正在安装 : lttng-ust-2.4.1-1.el7cp.x86_64 15/27
正在更新 : 1:librados2-10.2.2-38.el7cp.x86_64 16/27
正在更新 : 1:librbd1-10.2.2-38.el7cp.x86_64 17/27
正在安装 : 1:python-rados-10.2.2-38.el7cp.x86_64 18/27
正在安装 : 1:librgw2-10.2.2-38.el7cp.x86_64 19/27
正在安装 : 1:python-cephfs-10.2.2-38.el7cp.x86_64 20/27
正在安装 : 1:python-rbd-10.2.2-38.el7cp.x86_64 21/27
正在安装 : 1:ceph-common-10.2.2-38.el7cp.x86_64 22/27
正在安装 : 1:ceph-selinux-10.2.2-38.el7cp.x86_64 23/27
正在安装 : 1:ceph-base-10.2.2-38.el7cp.x86_64 24/27
正在安装 : 1:ceph-mds-10.2.2-38.el7cp.x86_64 25/27
清理 : 1:librbd1-0.94.5-2.el7.x86_64 26/27
清理 : 1:librados2-0.94.5-2.el7.x86_64 27/27
192.168.4.254_rhel7/productid | 1.6 kB 00:00
mon/productid | 1.6 kB 00:00
osd/productid | 1.6 kB 00:00
验证中 : userspace-rcu-0.7.9-2.el7rhgs.x86_64 1/27
验证中 : boost-regex-1.53.0-27.el7.x86_64 2/27
验证中 : libbabeltrace-1.2.4-3.el7cp.x86_64 3/27
验证中 : 1:librados2-10.2.2-38.el7cp.x86_64 4/27
验证中 : m4-1.4.16-10.el7.x86_64 5/27
验证中 : hdparm-9.43-5.el7.x86_64 6/27
验证中 : 1:libcephfs1-10.2.2-38.el7cp.x86_64 7/27
验证中 : 1:ceph-common-10.2.2-38.el7cp.x86_64 8/27
验证中 : 1:librbd1-10.2.2-38.el7cp.x86_64 9/27
验证中 : boost-iostreams-1.53.0-27.el7.x86_64 10/27
验证中 : boost-random-1.53.0-27.el7.x86_64 11/27
验证中 : boost-program-options-1.53.0-27.el7.x86_64 12/27
验证中 : patch-2.7.1-8.el7.x86_64 13/27
验证中 : 1:ceph-selinux-10.2.2-38.el7cp.x86_64 14/27
验证中 : 1:librgw2-10.2.2-38.el7cp.x86_64 15/27
验证中 : 1:python-cephfs-10.2.2-38.el7cp.x86_64 16/27
验证中 : redhat-lsb-submod-security-4.1-27.el7.x86_64 17/27
验证中 : 1:ceph-base-10.2.2-38.el7cp.x86_64 18/27
验证中 : redhat-lsb-core-4.1-27.el7.x86_64 19/27
验证中 : 1:python-rados-10.2.2-38.el7cp.x86_64 20/27
验证中 : lttng-ust-2.4.1-1.el7cp.x86_64 21/27
验证中 : spax-1.5.2-13.el7.x86_64 22/27
验证中 : 1:ceph-mds-10.2.2-38.el7cp.x86_64 23/27
验证中 : fcgi-2.4.0-25.el7cp.x86_64 24/27
验证中 : 1:python-rbd-10.2.2-38.el7cp.x86_64 25/27
验证中 : 1:librbd1-0.94.5-2.el7.x86_64 26/27
验证中 : 1:librados2-0.94.5-2.el7.x86_64 27/27

已安装:
ceph-mds.x86_64 1:10.2.2-38.el7cp

作为依赖被安装:
boost-iostreams.x86_64 0:1.53.0-27.el7
boost-program-options.x86_64 0:1.53.0-27.el7
boost-random.x86_64 0:1.53.0-27.el7
boost-regex.x86_64 0:1.53.0-27.el7
ceph-base.x86_64 1:10.2.2-38.el7cp
ceph-common.x86_64 1:10.2.2-38.el7cp
ceph-selinux.x86_64 1:10.2.2-38.el7cp
fcgi.x86_64 0:2.4.0-25.el7cp
hdparm.x86_64 0:9.43-5.el7
libbabeltrace.x86_64 0:1.2.4-3.el7cp
libcephfs1.x86_64 1:10.2.2-38.el7cp
librgw2.x86_64 1:10.2.2-38.el7cp
lttng-ust.x86_64 0:2.4.1-1.el7cp
m4.x86_64 0:1.4.16-10.el7
patch.x86_64 0:2.7.1-8.el7
python-cephfs.x86_64 1:10.2.2-38.el7cp
python-rados.x86_64 1:10.2.2-38.el7cp
python-rbd.x86_64 1:10.2.2-38.el7cp
redhat-lsb-core.x86_64 0:4.1-27.el7
redhat-lsb-submod-security.x86_64 0:4.1-27.el7
spax.x86_64 0:1.5.2-13.el7
userspace-rcu.x86_64 0:0.7.9-2.el7rhgs

作为依赖被升级:
librados2.x86_64 1:10.2.2-38.el7cp librbd1.x86_64 1:10.2.2-38.el7cp

完毕！
##################
以上，NODE4配置完毕，现在返回管理节点去操作
##############

[root@node1 ceph-cluster]# pwd
/root/ceph-cluster
[root@node1 ceph-cluster]# vim /etc/hosts
(把node4写入hosts)
下面进行免密操作
[root@node1 ceph-cluster]# for i in node{2..4}
> do
> scp /etc/hosts $i:/etc/
> done
hosts 100% 269 419.6KB/s 00:00
hosts 100% 269 401.4KB/s 00:00
The authenticity of host ‘node4 (192.168.4.14)’ can’t be established.
ECDSA key fingerprint is SHA256:59Byua15gv4TAVEm7YutMtmgXfxGTYbzSfO84kICu0E.
ECDSA key fingerprint is MD5:7e:6e:95:ed:84:fe:5b:8e:26:94:73:9b:c3:4b:90:cb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node4’ (ECDSA) to the list of known hosts.
hosts 100% 269 406.9KB/s 00:00
[root@node1 ceph-cluster]# ceph-deploy mds create node4
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.33): /usr/bin/ceph-deploy mds create node4
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fb47e755b48>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mds at 0x7fb47e72c7d0>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] mds : [(‘node4’, ‘node4’)]
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node4:node4
[node4][DEBUG ] connected to host: node4
[node4][DEBUG ] detect platform information from remote host
[node4][DEBUG ] detect machine type
[ceph_deploy.mds][INFO ] Distro info: Red Hat Enterprise Linux Server 7.4 Maipo
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node4
[node4][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node4][WARNIN] mds keyring does not exist yet, creating one
[node4][DEBUG ] create a keyring file
[node4][DEBUG ] create path if it doesn’t exist
[node4][INFO ] Running command: ceph –cluster ceph –name client.bootstrap-mds –keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node4 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node4/keyring
[node4][INFO ] Running command: systemctl enable ceph-mds@node4
[node4][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/[email protected] to /usr/lib/systemd/system/[email protected].
[node4][INFO ] Running command: systemctl start ceph-mds@node4
[node4][INFO ] Running command: systemctl enable ceph.target

同步配置文件和key
[root@node1 ceph-cluster]# ceph-deploy admin node4
代码如下
[root@node1 ceph-cluster]# ceph-deploy admin node4
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.33): /usr/bin/ceph-deploy admin node4
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fcdc9d38440>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : [‘node4’]
[ceph_deploy.cli][INFO ] func : <function admin at 0x7fcdca993140>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node4
[node4][DEBUG ] connected to host: node4
[node4][DEBUG ] detect platform information from remote host
[node4][DEBUG ] detect machine type
[node4][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

3）创建存储池
[root@fdfs_storage1 FastDFS]# mkdir -pv /data/fastdfs
4）修改配置文件。
[root@node4 ~]# ceph osd pool create cephfs_data 128
//创建存储池，对应128个PG
[root@node4 ~]# ceph osd pool create cephfs_metadata 128
//创建存储池，对应128个PG

[root@node4 ~]# ceph osd pool create cephfs_data 128
pool ‘cephfs_data’ created
[root@node4 ~]# ceph osd pool create cephfs_metadata 128
pool ‘cephfs_metadata’ created

[root@node4 ~]# ceph mds stat
e2:, 1 up:standby

5）创建Ceph文件系统
[root@node4 ~]# ceph mds stat //查看mds状态
e2:, 1 up:standby
[root@node4 ~]# ceph fs new myfs1 cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1
//注意，现在medadata池，再写data池
//默认，只能创建1个文件系统，多余的会报错
[root@node4 ~]# ceph fs ls
name: myfs1, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[root@node4 ~]# ceph mds stat
e4: 1/1/1 up {0=node4=up:creating}

代码如下
[root@node4 ~]# ceph mds stat
e2:, 1 up:standby
[root@node4 ~]#
[root@node4 ~]# ceph osd pool ls
rbd
cephfs_data
cephfs_metadata
[root@node4 ~]#
[root@node4 ~]#
[root@node4 ~]# ceph fs new myfs1 cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1
[root@node4 ~]#
[root@node4 ~]# ceph fs ls
name: myfs1, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
[root@node4 ~]#
[root@node4 ~]# ceph mds stat
e5: 1/1/1 up {0=node4=up:active}

客户端挂载
[root@client ~]# mount -t ceph 192.168.4.11:6789:/ /mnt/cephfs/ \
-o name=admin,secret=AQBTsdRapUxBKRAANXtteNUyoEmQHveb75bISg==
//注意:文件系统类型为ceph
//192.168.4.11为MON节点的IP（不是MDS节点）
//admin是用户名,secret是密钥
//密钥可以在/etc/ceph/ceph.client.admin.keyring中找到
[root@client ~]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQCXwL5bTwkVGxAA2ls+tuA8SMl2vmzQOPCdCg==
[root@client ~]#
[root@client ~]# mkdir /mnt/cephfs
[root@client ~]# mount -t ceph 192.168.4.11:6789:/ /mnt/cephfs -o name=admin,secret=AQCXwL5bTwkVGxAA2ls+tuA8SMl2vmzQOPCdCg==
[root@client ~]# ll /mnt/cephfs/
总用量 0

3 案例3：创建对象存储服务器
3.1 问题
延续前面的实验，实现Ceph对象存储的功能。具体实现有以下功能：
安装部署Rados Gateway
启动RGW服务
设置RGW的前端服务与端口
客户端测试
3.2 步骤
步骤一：部署对象存储服务器
1）准备实验环境，要求如下：
IP地址:192.168.4.15
主机名:node5
配置yum源（包括rhel、ceph的源）
与Client主机同步时间
node1允许无密码远程node5
修改node1的/etc/hosts，并同步到所有node主机
代码如下：
配置好yum,改主机名，写hosts，免密登陆
[root@node5 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.4.10 client
192.168.4.11 node1
192.168.4.12 node2
192.168.4.13 node3
192.168.4.14 node4
192.168.4.15 node5
配置时间同步NTP
[root@client ~]# vim /etc/chrony.conf
[root@client ~]# systemctl restart chronyd
[root@client ~]# cat /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 192.168.4.10 iburst

2）部署RGW软件包
[root@node1 ~]# ceph-deploy install –rgw node5
代码如下
[root@node1 ceph-cluster]# ceph-deploy install –rgw node5
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.33): /usr/bin/ceph-deploy install –rgw node5
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] testing : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f05af807f80>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] dev_commit : None
[ceph_deploy.cli][INFO ] install_mds : False
[ceph_deploy.cli][INFO ] stable : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] adjust_repos : True
[ceph_deploy.cli][INFO ] func : <function install at 0x7f05b04777d0>
[ceph_deploy.cli][INFO ] install_all : False
[ceph_deploy.cli][INFO ] repo : False
[ceph_deploy.cli][INFO ] host : [‘node5’]
[ceph_deploy.cli][INFO ] install_rgw : True
[ceph_deploy.cli][INFO ] install_tests : False
[ceph_deploy.cli][INFO ] repo_url : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] install_osd : False
[ceph_deploy.cli][INFO ] version_kind : stable
[ceph_deploy.cli][INFO ] install_common : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] dev : master
[ceph_deploy.cli][INFO ] local_mirror : None
[ceph_deploy.cli][INFO ] release : None
[ceph_deploy.cli][INFO ] install_mon : False
[ceph_deploy.cli][INFO ] gpg_url : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts node5
[ceph_deploy.install][DEBUG ] Detecting platform for host node5 …
The authenticity of host ‘node5 (192.168.4.15)’ can’t be established.
ECDSA key fingerprint is SHA256:59Byua15gv4TAVEm7YutMtmgXfxGTYbzSfO84kICu0E.
ECDSA key fingerprint is MD5:7e:6e:95:ed:84:fe:5b:8e:26:94:73:9b:c3:4b:90:cb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘node5’ (ECDSA) to the list of known hosts.
root@node5’s password:
root@node5’s password:
[node5][DEBUG ] connected to host: node5
[node5][DEBUG ] detect platform information from remote host
[node5][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: Red Hat Enterprise Linux Server 7.4 Maipo
[node5][INFO ] installing Ceph on node5
[node5][INFO ] Running command: yum clean all
[node5][DEBUG ] 已加载插件：langpacks, product-id, search-disabled-repos, subscription-manager
[node5][DEBUG ] This system is not registered with an entitlement server. You can use subscription-manager to register.
[node5][DEBUG ] 正在清理软件源： 192.168.4.254_rhel7 mon osd tools
[node5][DEBUG ] Cleaning up everything
[node5][DEBUG ] Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
[node5][INFO ] Running command: yum -y install ceph-radosgw
[node5][DEBUG ] 已加载插件：langpacks, product-id, search-disabled-repos, subscription-manager
[node5][DEBUG ] This system is not registered with an entitlement server. You can use subscription-manager to register.
[node5][DEBUG ] 正在解决依赖关系
[node5][DEBUG ] –> 正在检查事务
[node5][DEBUG ] —> 软件包 ceph-radosgw.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] –> 正在处理依赖关系 ceph-common = 1:10.2.2-38.el7cp，它被软件包 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 ceph-selinux = 1:10.2.2-38.el7cp，它被软件包 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 librados2 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 librgw2 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 mailcap，它被软件包 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 libfcgi.so.0()(64bit)，它被软件包 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 librgw.so.2()(64bit)，它被软件包 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在检查事务
[node5][DEBUG ] —> 软件包 ceph-common.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] –> 正在处理依赖关系 libcephfs1 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 librbd1 = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 python-cephfs = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 python-rados = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 python-rbd = 1:10.2.2-38.el7cp，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 libbabeltrace-ctf.so.1()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 libbabeltrace.so.1()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 libboost_iostreams-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 libboost_program_options-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 libboost_regex-mt.so.1.53.0()(64bit)，它被软件包 1:ceph-common-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] —> 软件包 ceph-selinux.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] –> 正在处理依赖关系 ceph-base = 1:10.2.2-38.el7cp，它被软件包 1:ceph-selinux-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] —> 软件包 fcgi.x86_64.0.2.4.0-25.el7cp 将被安装
[node5][DEBUG ] —> 软件包 librados2.x86_64.1.0.94.5-2.el7 将被升级
[node5][DEBUG ] —> 软件包 librados2.x86_64.1.10.2.2-38.el7cp 将被更新
[node5][DEBUG ] –> 正在处理依赖关系 libboost_random-mt.so.1.53.0()(64bit)，它被软件包 1:librados2-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 liblttng-ust.so.0()(64bit)，它被软件包 1:librados2-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] —> 软件包 librgw2.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] —> 软件包 mailcap.noarch.0.2.1.41-2.el7 将被安装
[node5][DEBUG ] –> 正在检查事务
[node5][DEBUG ] —> 软件包 boost-iostreams.x86_64.0.1.53.0-27.el7 将被安装
[node5][DEBUG ] —> 软件包 boost-program-options.x86_64.0.1.53.0-27.el7 将被安装
[node5][DEBUG ] —> 软件包 boost-random.x86_64.0.1.53.0-27.el7 将被安装
[node5][DEBUG ] —> 软件包 boost-regex.x86_64.0.1.53.0-27.el7 将被安装
[node5][DEBUG ] —> 软件包 ceph-base.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] –> 正在处理依赖关系 hdparm，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 redhat-lsb-core，它被软件包 1:ceph-base-10.2.2-38.el7cp.x86_64 需要
[node5][DEBUG ] —> 软件包 libbabeltrace.x86_64.0.1.2.4-3.el7cp 将被安装
[node5][DEBUG ] —> 软件包 libcephfs1.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] —> 软件包 librbd1.x86_64.1.0.94.5-2.el7 将被升级
[node5][DEBUG ] —> 软件包 librbd1.x86_64.1.10.2.2-38.el7cp 将被更新
[node5][DEBUG ] —> 软件包 lttng-ust.x86_64.0.2.4.1-1.el7cp 将被安装
[node5][DEBUG ] –> 正在处理依赖关系 liburcu-bp.so.1()(64bit)，它被软件包 lttng-ust-2.4.1-1.el7cp.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 liburcu-cds.so.1()(64bit)，它被软件包 lttng-ust-2.4.1-1.el7cp.x86_64 需要
[node5][DEBUG ] —> 软件包 python-cephfs.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] —> 软件包 python-rados.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] —> 软件包 python-rbd.x86_64.1.10.2.2-38.el7cp 将被安装
[node5][DEBUG ] –> 正在检查事务
[node5][DEBUG ] —> 软件包 hdparm.x86_64.0.9.43-5.el7 将被安装
[node5][DEBUG ] —> 软件包 redhat-lsb-core.x86_64.0.4.1-27.el7 将被安装
[node5][DEBUG ] –> 正在处理依赖关系 redhat-lsb-submod-security(x86-64) = 4.1-27.el7，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 /usr/bin/m4，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 /usr/bin/patch，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
[node5][DEBUG ] –> 正在处理依赖关系 spax，它被软件包 redhat-lsb-core-4.1-27.el7.x86_64 需要
[node5][DEBUG ] —> 软件包 userspace-rcu.x86_64.0.0.7.9-2.el7rhgs 将被安装
[node5][DEBUG ] –> 正在检查事务
[node5][DEBUG ] —> 软件包 m4.x86_64.0.1.4.16-10.el7 将被安装
[node5][DEBUG ] —> 软件包 patch.x86_64.0.2.7.1-8.el7 将被安装
[node5][DEBUG ] —> 软件包 redhat-lsb-submod-security.x86_64.0.4.1-27.el7 将被安装
[node5][DEBUG ] —> 软件包 spax.x86_64.0.1.5.2-13.el7 将被安装
[node5][DEBUG ] –> 解决依赖关系完成
[node5][DEBUG ]
[node5][DEBUG ] 依赖关系解决
[node5][DEBUG ]
[node5][DEBUG ] ================================================================================
[node5][DEBUG ] Package 架构版本源大小
[node5][DEBUG ] ================================================================================
[node5][DEBUG ] 正在安装:
[node5][DEBUG ] ceph-radosgw x86_64 1:10.2.2-38.el7cp tools 265 k
[node5][DEBUG ] 为依赖而安装:
[node5][DEBUG ] boost-iostreams x86_64 1.53.0-27.el7 192.168.4.254_rhel7 61 k
[node5][DEBUG ] boost-program-options x86_64 1.53.0-27.el7 192.168.4.254_rhel7 156 k
[node5][DEBUG ] boost-random x86_64 1.53.0-27.el7 192.168.4.254_rhel7 39 k
[node5][DEBUG ] boost-regex x86_64 1.53.0-27.el7 192.168.4.254_rhel7 300 k
[node5][DEBUG ] ceph-base x86_64 1:10.2.2-38.el7cp mon 4.2 M
[node5][DEBUG ] ceph-common x86_64 1:10.2.2-38.el7cp mon 16 M
[node5][DEBUG ] ceph-selinux x86_64 1:10.2.2-38.el7cp mon 38 k
[node5][DEBUG ] fcgi x86_64 2.4.0-25.el7cp mon 47 k
[node5][DEBUG ] hdparm x86_64 9.43-5.el7 192.168.4.254_rhel7 83 k
[node5][DEBUG ] libbabeltrace x86_64 1.2.4-3.el7cp mon 147 k
[node5][DEBUG ] libcephfs1 x86_64 1:10.2.2-38.el7cp mon 1.9 M
[node5][DEBUG ] librgw2 x86_64 1:10.2.2-38.el7cp mon 2.9 M
[node5][DEBUG ] lttng-ust x86_64 2.4.1-1.el7cp mon 176 k
[node5][DEBUG ] m4 x86_64 1.4.16-10.el7 192.168.4.254_rhel7 256 k
[node5][DEBUG ] mailcap noarch 2.1.41-2.el7 192.168.4.254_rhel7 31 k
[node5][DEBUG ] patch x86_64 2.7.1-8.el7 192.168.4.254_rhel7 110 k
[node5][DEBUG ] python-cephfs x86_64 1:10.2.2-38.el7cp mon 86 k
[node5][DEBUG ] python-rados x86_64 1:10.2.2-38.el7cp mon 164 k
[node5][DEBUG ] python-rbd x86_64 1:10.2.2-38.el7cp mon 93 k
[node5][DEBUG ] redhat-lsb-core x86_64 4.1-27.el7 192.168.4.254_rhel7 37 k
[node5][DEBUG ] redhat-lsb-submod-security x86_64 4.1-27.el7 192.168.4.254_rhel7 15 k
[node5][DEBUG ] spax x86_64 1.5.2-13.el7 192.168.4.254_rhel7 260 k
[node5][DEBUG ] userspace-rcu x86_64 0.7.9-2.el7rhgs mon 70 k
[node5][DEBUG ] 为依赖而更新:
[node5][DEBUG ] librados2 x86_64 1:10.2.2-38.el7cp mon 1.9 M
[node5][DEBUG ] librbd1 x86_64 1:10.2.2-38.el7cp mon 2.5 M
[node5][DEBUG ]
[node5][DEBUG ] 事务概要
[node5][DEBUG ] ================================================================================
[node5][DEBUG ] 安装 1 软件包 (+23 依赖软件包)
[node5][DEBUG ] 升级 ( 2 依赖软件包)
[node5][DEBUG ]
[node5][DEBUG ] 总下载量：31 M
[node5][DEBUG ] Downloading packages:
[node5][DEBUG ] No Presto metadata available for mon
[node5][DEBUG ] ——————————————————————————–
[node5][DEBUG ] 总计 29 MB/s | 31 MB 00:01
[node5][DEBUG ] Running transaction check
[node5][DEBUG ] Running transaction test
[node5][DEBUG ] Transaction test succeeded
[node5][DEBUG ] Running transaction
[node5][DEBUG ] 正在安装 : boost-iostreams-1.53.0-27.el7.x86_64 1/28
[node5][DEBUG ] 正在安装 : boost-random-1.53.0-27.el7.x86_64 2/28
[node5][DEBUG ] 正在安装 : 1:libcephfs1-10.2.2-38.el7cp.x86_64 3/28
[node5][DEBUG ] 正在安装 : fcgi-2.4.0-25.el7cp.x86_64 4/28
[node5][DEBUG ] 正在安装 : spax-1.5.2-13.el7.x86_64 5/28
[node5][DEBUG ] 正在安装 : redhat-lsb-submod-security-4.1-27.el7.x86_64 6/28
[node5][DEBUG ] 正在安装 : patch-2.7.1-8.el7.x86_64 7/28
[node5][DEBUG ] 正在安装 : boost-program-options-1.53.0-27.el7.x86_64 8/28
[node5][DEBUG ] 正在安装 : hdparm-9.43-5.el7.x86_64 9/28
[node5][DEBUG ] 正在安装 : m4-1.4.16-10.el7.x86_64 10/28
[node5][DEBUG ] 正在安装 : redhat-lsb-core-4.1-27.el7.x86_64 11/28
[node5][DEBUG ] 正在安装 : libbabeltrace-1.2.4-3.el7cp.x86_64 12/28
[node5][DEBUG ] 正在安装 : boost-regex-1.53.0-27.el7.x86_64 13/28
[node5][DEBUG ] 正在安装 : mailcap-2.1.41-2.el7.noarch 14/28
[node5][DEBUG ] 正在安装 : userspace-rcu-0.7.9-2.el7rhgs.x86_64 15/28
[node5][DEBUG ] 正在安装 : lttng-ust-2.4.1-1.el7cp.x86_64 16/28
[node5][DEBUG ] 正在更新 : 1:librados2-10.2.2-38.el7cp.x86_64 17/28
[node5][DEBUG ] 正在更新 : 1:librbd1-10.2.2-38.el7cp.x86_64 18/28
[node5][DEBUG ] 正在安装 : 1:librgw2-10.2.2-38.el7cp.x86_64 19/28
[node5][DEBUG ] 正在安装 : 1:python-rados-10.2.2-38.el7cp.x86_64 20/28
[node5][DEBUG ] 正在安装 : 1:python-cephfs-10.2.2-38.el7cp.x86_64 21/28
[node5][DEBUG ] 正在安装 : 1:python-rbd-10.2.2-38.el7cp.x86_64 22/28
[node5][DEBUG ] 正在安装 : 1:ceph-common-10.2.2-38.el7cp.x86_64 23/28
[node5][DEBUG ] 正在安装 : 1:ceph-selinux-10.2.2-38.el7cp.x86_64 24/28
[node5][DEBUG ] 正在安装 : 1:ceph-base-10.2.2-38.el7cp.x86_64 25/28
[node5][DEBUG ] 正在安装 : 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 26/28
[node5][DEBUG ] 清理 : 1:librbd1-0.94.5-2.el7.x86_64 27/28
[node5][DEBUG ] 清理 : 1:librados2-0.94.5-2.el7.x86_64 28/28
[node5][DEBUG ] 验证中 : userspace-rcu-0.7.9-2.el7rhgs.x86_64 1/28
[node5][DEBUG ] 验证中 : mailcap-2.1.41-2.el7.noarch 2/28
[node5][DEBUG ] 验证中 : boost-regex-1.53.0-27.el7.x86_64 3/28
[node5][DEBUG ] 验证中 : libbabeltrace-1.2.4-3.el7cp.x86_64 4/28
[node5][DEBUG ] 验证中 : 1:librados2-10.2.2-38.el7cp.x86_64 5/28
[node5][DEBUG ] 验证中 : m4-1.4.16-10.el7.x86_64 6/28
[node5][DEBUG ] 验证中 : hdparm-9.43-5.el7.x86_64 7/28
[node5][DEBUG ] 验证中 : 1:libcephfs1-10.2.2-38.el7cp.x86_64 8/28
[node5][DEBUG ] 验证中 : 1:ceph-common-10.2.2-38.el7cp.x86_64 9/28
[node5][DEBUG ] 验证中 : 1:librbd1-10.2.2-38.el7cp.x86_64 10/28
[node5][DEBUG ] 验证中 : boost-iostreams-1.53.0-27.el7.x86_64 11/28
[node5][DEBUG ] 验证中 : boost-random-1.53.0-27.el7.x86_64 12/28
[node5][DEBUG ] 验证中 : boost-program-options-1.53.0-27.el7.x86_64 13/28
[node5][DEBUG ] 验证中 : patch-2.7.1-8.el7.x86_64 14/28
[node5][DEBUG ] 验证中 : 1:ceph-selinux-10.2.2-38.el7cp.x86_64 15/28
[node5][DEBUG ] 验证中 : 1:librgw2-10.2.2-38.el7cp.x86_64 16/28
[node5][DEBUG ] 验证中 : 1:python-cephfs-10.2.2-38.el7cp.x86_64 17/28
[node5][DEBUG ] 验证中 : redhat-lsb-submod-security-4.1-27.el7.x86_64 18/28
[node5][DEBUG ] 验证中 : 1:ceph-base-10.2.2-38.el7cp.x86_64 19/28
[node5][DEBUG ] 验证中 : redhat-lsb-core-4.1-27.el7.x86_64 20/28
[node5][DEBUG ] 验证中 : 1:python-rados-10.2.2-38.el7cp.x86_64 21/28
[node5][DEBUG ] 验证中 : lttng-ust-2.4.1-1.el7cp.x86_64 22/28
[node5][DEBUG ] 验证中 : spax-1.5.2-13.el7.x86_64 23/28
[node5][DEBUG ] 验证中 : 1:ceph-radosgw-10.2.2-38.el7cp.x86_64 24/28
[node5][DEBUG ] 验证中 : fcgi-2.4.0-25.el7cp.x86_64 25/28
[node5][DEBUG ] 验证中 : 1:python-rbd-10.2.2-38.el7cp.x86_64 26/28
[node5][DEBUG ] 验证中 : 1:librbd1-0.94.5-2.el7.x86_64 27/28
[node5][DEBUG ] 验证中 : 1:librados2-0.94.5-2.el7.x86_64 28/28
[node5][DEBUG ]
[node5][DEBUG ] 已安装:
[node5][DEBUG ] ceph-radosgw.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ]
[node5][DEBUG ] 作为依赖被安装:
[node5][DEBUG ] boost-iostreams.x86_64 0:1.53.0-27.el7
[node5][DEBUG ] boost-program-options.x86_64 0:1.53.0-27.el7
[node5][DEBUG ] boost-random.x86_64 0:1.53.0-27.el7
[node5][DEBUG ] boost-regex.x86_64 0:1.53.0-27.el7
[node5][DEBUG ] ceph-base.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] ceph-common.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] ceph-selinux.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] fcgi.x86_64 0:2.4.0-25.el7cp
[node5][DEBUG ] hdparm.x86_64 0:9.43-5.el7
[node5][DEBUG ] libbabeltrace.x86_64 0:1.2.4-3.el7cp
[node5][DEBUG ] libcephfs1.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] librgw2.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] lttng-ust.x86_64 0:2.4.1-1.el7cp
[node5][DEBUG ] m4.x86_64 0:1.4.16-10.el7
[node5][DEBUG ] mailcap.noarch 0:2.1.41-2.el7
[node5][DEBUG ] patch.x86_64 0:2.7.1-8.el7
[node5][DEBUG ] python-cephfs.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] python-rados.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] python-rbd.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ] redhat-lsb-core.x86_64 0:4.1-27.el7
[node5][DEBUG ] redhat-lsb-submod-security.x86_64 0:4.1-27.el7
[node5][DEBUG ] spax.x86_64 0:1.5.2-13.el7
[node5][DEBUG ] userspace-rcu.x86_64 0:0.7.9-2.el7rhgs
[node5][DEBUG ]
[node5][DEBUG ] 作为依赖被升级:
[node5][DEBUG ] librados2.x86_64 1:10.2.2-38.el7cp librbd1.x86_64 1:10.2.2-38.el7cp
[node5][DEBUG ]
[node5][DEBUG ] 完毕！
[node5][INFO ] Running command: ceph –version
[node5][DEBUG ] ceph version 10.2.2-38.el7cp (119a68752a5671253f9daae3f894a90313a6b8e4)
[root@node1 ceph-cluster]#
[root@node1 ceph-cluster]#

添加免密登陆
[root@node1 ceph-cluster]# ssh-copy-id 192.168.4.15

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: “/root/.ssh/id_rsa.pub”
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed — if you are prompted now it is to install the new keys
[email protected]’s password:

Number of key(s) added: 1

Now try logging into the machine, with: “ssh ‘192.168.4.15’”
and check to make sure that only the key(s) you wanted were added.

同步配置文件与密钥到node5
[root@node1 ~]# cd /root/ceph-cluster
[root@node1 ~]# ceph-deploy admin node5
[root@node1 ceph-cluster]# ceph-deploy admin node5
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.33): /usr/bin/ceph-deploy admin node5
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f08adf35440>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : [‘node5’]
[ceph_deploy.cli][INFO ] func : <function admin at 0x7f08aeb90140>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node5
[node5][DEBUG ] connected to host: node5
[node5][DEBUG ] detect platform information from remote host
[node5][DEBUG ] detect machine type
[node5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

3）新建网关实例
启动一个rgw服务
[root@node1 ~]# ceph-deploy rgw create node5

[root@node1 ceph-cluster]# ceph-deploy rgw create node5
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.33): /usr/bin/ceph-deploy rgw create node5
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] rgw : [(‘node5’, ‘rgw.node5’)]
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc0a32e7fc8>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function rgw at 0x7fc0a3f50050>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts node5:rgw.node5
[node5][DEBUG ] connected to host: node5
[node5][DEBUG ] detect platform information from remote host
[node5][DEBUG ] detect machine type
[ceph_deploy.rgw][INFO ] Distro info: Red Hat Enterprise Linux Server 7.4 Maipo
[ceph_deploy.rgw][DEBUG ] remote host will use systemd
[ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to node5
[node5][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node5][WARNIN] rgw keyring does not exist yet, creating one
[node5][DEBUG ] create a keyring file
[node5][DEBUG ] create path recursively if it doesn’t exist
[node5][INFO ] Running command: ceph –cluster ceph –name client.bootstrap-rgw –keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.node5 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.node5/keyring
[node5][INFO ] Running command: systemctl enable [email protected]
[node5][WARNIN] Created symlink from /etc/systemd/system/ceph-radosgw.target.wants/[email protected] to /usr/lib/systemd/system/[email protected].
[node5][INFO ] Running command: systemctl start [email protected]
[node5][INFO ] Running command: systemctl enable ceph.target
[ceph_deploy.rgw][INFO ] The Ceph Object Gateway (RGW) is now running on host node5 and default port 7480
[root@node1 ceph-cluster]#

到node5去确认以下
[root@node5 ~]# ps aux | grep radosgw
ceph 3395 0.5 2.0 2291248 21120 ? Ssl 15:59 0:00 /usr/bin/radosgw -f –cluster ceph –name client.rgw.node5 –setuser ceph –setgroup ceph
root 3618 0.0 0.0 112676 984 pts/1 S+ 16:00 0:00 grep –color=auto radosgw
[root@node5 ~]# netstat -antup | grep 7480
tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 3395/radosgw
[root@node5 ~]# netstat -antup | grep radosgw
tcp 0 0 0.0.0.0:7480 0.0.0.0:* LISTEN 3395/radosgw
tcp 0 0 192.168.4.15:40238 192.168.4.11:6800 ESTABLISHED 3395/radosgw
tcp 0 9 192.168.4.15:41050 192.168.4.13:6804 ESTABLISHED 3395/radosgw
tcp 0 0 192.168.4.15:45620 192.168.4.13:6800 ESTABLISHED 3395/radosgw
tcp 0 9 192.168.4.15:54810 192.168.4.11:6804 ESTABLISHED 3395/radosgw
tcp 0 0 192.168.4.15:51324 192.168.4.12:6789 ESTABLISHED 3395/radosgw
tcp 0 9 192.168.4.15:54852 192.168.4.12:6804 ESTABLISHED 3395/radosgw
tcp 0 0 192.168.4.15:49496 192.168.4.12:6800 ESTABLISHED 3395/radosgw

4）修改服务端口
登陆node5，RGW默认服务端口为7480，修改为8000或80更方便客户端记忆和使用
[root@node5 ~]# vim /etc/ceph/ceph.conf
[client.rgw.node5]
host = node5
rgw_frontends = “civetweb port=8000”
//node5为主机名
//civetweb是RGW内置的一个web服务

[root@node5 ~]# cat /etc/ceph/ceph.conf
[global]
fsid = 29908a48-7574-4aac-ac14-80a44b7cffbf
mon_initial_members = node1, node2, node3
mon_host = 192.168.4.11,192.168.4.12,192.168.4.13
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
[client.rgw.node5]
host = node5
rgw_frontends = “civetweb port=8000″

[root@node5 ~]# vim /etc/ceph/ceph.conf
[root@node5 ~]# systemctl restart ceph\*
[root@node5 ~]# netstat -antup | grep radosgw
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 3711/radosgw
tcp 0 0 192.168.4.15:54838 192.168.4.11:6804 ESTABLISHED 3711/radosgw
tcp 0 0 192.168.4.15:45638 192.168.4.13:6800 ESTABLISHED 3711/radosgw
tcp 0 0 192.168.4.15:49524 192.168.4.12:6800 ESTABLISHED 3711/radosgw
tcp 0 0 192.168.4.15:41082 192.168.4.13:6804 ESTABLISHED 3711/radosgw
tcp 0 0 192.168.4.15:54882 192.168.4.12:6804 ESTABLISHED 3711/radosgw
tcp 0 0 192.168.4.15:47214 192.168.4.13:6789 ESTABLISHED 3711/radosgw
tcp 0 0 192.168.4.15:40270 192.168.4.11:6800 ESTABLISHED 3711/radosgw

步骤二：客户端测试
1）curl测试
[root@client ~]# curl 192.168.4.15:8000
<?xml version=”1.0″ encoding=”UTF-8″?><ListAllMyBucketsResult xmlns=”http://s3.amazonaws.com/doc/2006-03-01/”><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

代码如下
[root@client ~]# curl 192.168.4.15:8000
<?xml version=”1.0″ encoding=”UTF-8″?><ListAllMyBucketsResult xmlns=”http://s3.amazonaws.com/doc/2006-03-01/”><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[root@client ~]#

2）使用第三方软件访问
登陆node5（RGW）创建账户
[root@node5 ~]# radosgw-admin user create \
–uid=”testuser” –display-name=”First User”
… …
“keys”: [
{
“user”: “testuser”,
“access_key”: “5E42OEGB1M95Y49IBG7B”,
“secret_key”: “i8YtM8cs7QDCK3rTRopb0TTPBFJVXdEryRbeLGK6″
}
],
… …
#
[root@node5 ~]# radosgw-admin user info –uid=testuser
//testuser为用户，key是账户访问密钥

[root@node5 ~]# radosgw-admin user create –uid=”testuser” –display-name=”First User”
{
“user_id”: “testuser”,
“display_name”: “First User”,
“email”: “”,
“suspended”: 0,
“max_buckets”: 1000,
“auid”: 0,
“subusers”: [],
“keys”: [
{
“user”: “testuser”,
“access_key”: “9ZQIAYP67UVVNG7LFMKC”,
“secret_key”: “ilIzMZ0GVGGWHnmd6Q3KppfRNJkDQoaFttHb3SLO”
}
],
“swift_keys”: [],
“caps”: [],
“op_mask”: “read, write, delete”,
“default_placement”: “”,
“placement_tags”: [],
“bucket_quota”: {
“enabled”: false,
“max_size_kb”: -1,
“max_objects”: -1
},
“user_quota”: {
“enabled”: false,
“max_size_kb”: -1,
“max_objects”: -1
},
“temp_url_keys”: []
}

[root@node5 ~]# radosgw-admin user info –uid=testuser
{
“user_id”: “testuser”,
“display_name”: “First User”,
“email”: “”,
“suspended”: 0,
“max_buckets”: 1000,
“auid”: 0,
“subusers”: [],
“keys”: [
{
“user”: “testuser”,
“access_key”: “9ZQIAYP67UVVNG7LFMKC”,
“secret_key”: “ilIzMZ0GVGGWHnmd6Q3KppfRNJkDQoaFttHb3SLO”
}
],
“swift_keys”: [],
“caps”: [],
“op_mask”: “read, write, delete”,
“default_placement”: “”,
“placement_tags”: [],
“bucket_quota”: {
“enabled”: false,
“max_size_kb”: -1,
“max_objects”: -1
},
“user_quota”: {
“enabled”: false,
“max_size_kb”: -1,
“max_objects”: -1
},
“temp_url_keys”: []
}

3）客户端安装软件
[root@client ~]# yum install s3cmd-2.0.1-1.el7.noarch.rpm

[root@client ~]# yum install -y s3cmd-2.0.1-1.el7.noarch.rpm
已加载插件：langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
正在检查 s3cmd-2.0.1-1.el7.noarch.rpm: s3cmd-2.0.1-1.el7.noarch
s3cmd-2.0.1-1.el7.noarch.rpm 将被安装
正在解决依赖关系
–> 正在检查事务
—> 软件包 s3cmd.noarch.0.2.0.1-1.el7 将被安装
–> 解决依赖关系完成

依赖关系解决

======================================================================================
Package 架构版本源大小
======================================================================================
正在安装:
s3cmd noarch 2.0.1-1.el7 /s3cmd-2.0.1-1.el7.noarch 734 k

事务概要
======================================================================================
安装 1 软件包

总计：734 k
安装大小：734 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : s3cmd-2.0.1-1.el7.noarch 1/1
验证中 : s3cmd-2.0.1-1.el7.noarch 1/1

已安装:
s3cmd.noarch 0:2.0.1-1.el7

修改软件配置
[root@client ~]# s3cmd –configure
Access Key: 5E42OEGB1M95Y49IBG7BSecret Key: i8YtM8cs7QDCK3rTRopb0TTPBFJVXdEryRbeLGK6
S3 Endpoint [s3.amazonaws.com]: 192.168.4.15:8000
[%(bucket)s.s3.amazonaws.com]: %(bucket)s.192.168.4.15:8000
Use HTTPS protocol [Yes]: No
Test access with supplied credentials? [Y/n] Y
Save settings? [y/N] y
//注意，其他提示都默认回车

代码如下
[root@client ~]# s3cmd –configure
（注意，因配置的时候出错了，所以中途重新配置了以下，没有显示的地方默认按回车）
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 9ZQIAYP67UVVNG7LFMKC
Secret Key: ilIzMZ0GVGGWHnmd6Q3KppfRNJkDQoaFttHb3SLO
Default Region [US]:

Use “s3.amazonaws.com” for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 192.168.4.15:8000

Use “%(bucket)s.s3.amazonaws.com” to the target Amazon S3. “%(bucket)s” and “%(location)s” vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: %(bucket)s.192.168.4.15:8000

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: No

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t connect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: 9ZQIAYP67UVVNG7LFMKC
Secret Key: ilIzMZ0GVGGWHnmd6Q3KppfRNJkDQoaFttHb3SLO
Default Region: US
S3 Endpoint: 192.168.4.15:8000
DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.192.168.4.15:8000
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: False
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets…
ERROR: Test failed: [Errno -2] Name or service not known

Retry configuration? [Y/n] y

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key [9ZQIAYP67UVVNG7LFMKC]:
Secret Key [ilIzMZ0GVGGWHnmd6Q3KppfRNJkDQoaFttHb3SLO]:
Default Region [US]:

Use “s3.amazonaws.com” for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [ 192.168.4.15:8000]: 192.168.4.15:8000

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can’t connect to S3 directly
HTTP Proxy server name:

Test access with supplied credentials? [Y/n]
Please wait, attempting to list all buckets…
Success. Your access key and secret key worked fine 🙂

Now verifying that encryption works…
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to ‘/root/.s3cfg’

4）创建存储数据的bucket（类似于存储数据的目录）
[root@client ~]# s3cmd ls
[root@client ~]# s3cmd mb s3://my_bucket
Bucket ‘s3://my_bucket/’ created
[root@client ~]# s3cmd ls
2018-05-09 08:14 s3://my_bucket
[root@client ~]# s3cmd put /var/log/messages s3://my_bucket/log/
[root@client ~]# s3cmd ls
2018-05-09 08:14 s3://my_bucket
[root@client ~]# s3cmd ls s3://my_bucket
DIR s3://my_bucket/log/
[root@client ~]# s3cmd ls s3://my_bucket/log/
2018-05-09 08:19 309034 s3://my_bucket/log/messages

代码如下
[root@client ~]# s3cmd ls
（没创建的时候显示为空）
[root@client ~]# s3cmd mb s3://my_bucket
Bucket ‘s3://my_bucket/’ created
[root@client ~]# s3cmd ls
2018-10-12 08:24 s3://my_bucket
(上传文件测试一下)
[root@client ~]# s3cmd put /var/log/messages s3://my_bucket/log/
upload: ‘/var/log/messages’ -> ‘s3://my_bucket/log/messages’ [1 of 1]
586080 of 586080 100% in 3s 151.45 kB/s done
[root@client ~]# s3cmd ls
2018-10-12 08:24 s3://my_bucket
[root@client ~]# s3cmd ls s3://my_bucket
DIR s3://my_bucket/log/
[root@client ~]# s3cmd ls s3://my_bucket/log
DIR s3://my_bucket/log/
[root@client ~]# s3cmd ls s3://my_bucket/log/
2018-10-12 08:32 586080 s3://my_bucket/log/messages

测试下载功能
[root@client ~]# s3cmd get s3://my_bucket/log/messages /tmp/

[root@client ~]# s3cmd get s3://my_bucket/log/messages /tmp/
download: ‘s3://my_bucket/log/messages’ -> ‘/tmp/messages’ [1 of 1]
586080 of 586080 100% in 0s 28.11 MB/s done
[root@client ~]# ll /tmp/
总用量 576
-rw-r–r–. 1 root root 586080 10月 12 08:32 messages

测试删除功能
[root@client ~]# s3cmd del s3://my_bucket/log/messages

[root@client ~]# s3cmd del s3://my_bucket/log/messages
delete: ‘s3://my_bucket/log/messages’

Ceph块存储和对象存储

发表回复取消回复

查找内容

分类

近期文章

归档

页面

点击

2024 年 7 月
一	二	三	四	五	六	日
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31

Ceph块存储和对象存储

发表回复 取消回复

查找内容

分类

近期文章

归档

页面

点击

标签

发表回复取消回复