nfs-ganesha将rgw导出为nfs文件接口

概述

我们知道ceph是统一的分布式存储系统，其可以提供给应用三种访问数据的接口：对象(RGW)、块(RBD)和文件(CEPHFS)接口。我们在使用对象接口的时候通常都是基于HTTP协议来存取数据。

下面介绍另外一种方式来使用ceph的对象(RGW)接口 – nfs-ganesha。这种方式可以通过基于文件的访问协议（如NFSv3和NFSv4）来访问Ceph对象网关命名空间。详细的介绍可以去ceph官网看看：http://docs.ceph.com/docs/master/radosgw/nfs/。

1、环境准备

1.1、准备虚拟机

单纯为了测试可用性，所以我这里就是使用一个虚拟机搭建ceph集群，然后配置nfs-ganesha将rgw导出为nfs文件接口

[root@ceph01 tmp]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[root@ceph01 tmp]# uname -a
Linux ceph01 4.17.3-1.el7.elrepo.x86_64 #1 SMP Tue Jun 26 10:14:25 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

1.2、配置yum源

[root@ceph01 tmp]# ll /etc/yum.repos.d/
total 12
-rw-r--r--. 1 root root 2523 Jun 16 06:22 CentOS-Base.repo
-rw-r--r--. 1 root root  180 Jul  3 11:38 ceph.repo
-rw-r--r--. 1 root root  664 May 11 11:35 epel.repo

1.2.1、base源

[root@ceph01 tmp]# cat /etc/yum.repos.d/CentOS-Base.repo 
[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/os/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#released updates 
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/updates/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/
        http://mirrors.aliyuncs.com/centos/$releasever/contrib/$basearch/
        http://mirrors.cloud.aliyuncs.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7

1.2.2、epel源

[root@ceph01 tmp]# cat /etc/yum.repos.d/epel.repo 
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

[root@ceph01 tmp]# cat /etc/yum.repos.d/epel.repo 
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0

1.2.3、ceph源

我这里配置的Jewel的ceph源

[root@ceph01 tmp]# cat /etc/yum.repos.d/ceph.repo 
[ceph]
name=ceph
baseurl=http://download.ceph.com/rpm-jewel/el7/x86_64/
gpgcheck=0
[ceph-noarch]
name=cephnoarch
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch/
gpgcheck=0

2、相关软件包准备

2.1、安装ceph相关包

注意这里的 librgw2-devel 包后面编译nfs-ganesha时需要，所以一定要安装

[root@ceph01 tmp]# yum install ceph librgw2-devel -y
[root@ceph01 tmp]# ceph -v
ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)

部署ceph集群，下面是我部署好的单机ceph集群

[root@ceph01 tmp]# ceph -s
    cluster 29978a95-87f1-4a0c-96fa-2369e985bfbe
     health HEALTH_OK
     monmap e1: 1 mons at {ceph01=192.168.10.20:6789/0}
            election epoch 4, quorum 0 ceph01
     osdmap e26: 2 osds: 2 up, 2 in
            flags sortbitwise,require_jewel_osds
      pgmap v61: 128 pgs, 9 pools, 2390 bytes data, 177 objects
            217 MB used, 30480 MB / 30697 MB avail
                 128 active+clean

2.2、源码编译安装nfs-ganesha

下载nfs-ganesha源代码：https://github.com/nfs-ganesha/nfs-ganesha/tree/3cf14c95d0a6efb5d366f97333d07a08d6f3b2a5

下载nfs-ganesha依赖的模块：https://github.com/nfs-ganesha/ntirpc/tree/5575ddb022b996c318d263e504273ee6a34499a3

因为我使用的ceph是Jewel 10.2.11，所以这里下载的nfs-ganesha是V2.5.0，高版本或低版本都会有点问题。

将上面下载好的两个包放到/tmp目录下

[root@ceph01 tmp]# ll
···
-rw-r--r--.  1 root root 3936969 Jul  3 15:08 nfs-ganesha.zip
-rw-r--r--.  1 root root  632189 Jul  3 15:08 ntirpc.zip

解压

[root@ceph01 tmp]# unzip nfs-ganesha.zip
[root@ceph01 tmp]# unzip ntirpc.zip

因为ntirpc是nfs-ganesha所依赖的子模块，所以这里将ntirpc源代码放到nfs-ganasha对应的目录中去

[root@ceph01 tmp]# cp -r ntirpc/* nfs-ganesha/src/libntirpc/

编译安装nfs-nanesha

[root@ceph01 tmp]# mkdir mybuild && cd mybuild
[root@ceph01 mybuild]# cmake -DUSE_FSAL_RGW=ON /tmp/nfs-ganesha/src/
[root@ceph01 mybuild]# make && make install

也可以制作rpm包

[root@ceph01 mybuild]# make rpm

上面的make rpm命令会制作nfs-ganesha相关的rpm包，并将这些rpm包放到下面的路径下

[root@ceph01 x86_64]# pwd
/root/rpmbuild/RPMS/x86_64
[root@ceph01 x86_64]# ll
total 2920
-rw-r--r-- 1 root root  698244 Jan 22 13:04 nfs-ganesha-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root   23768 Jan 22 13:04 nfs-ganesha-ceph-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root 2060020 Jan 22 13:04 nfs-ganesha-debuginfo-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root   48464 Jan 22 13:04 nfs-ganesha-gpfs-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root   23592 Jan 22 13:04 nfs-ganesha-mem-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root    5384 Jan 22 13:04 nfs-ganesha-mount-9P-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root   14300 Jan 22 13:04 nfs-ganesha-nullfs-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root   38192 Jan 22 13:04 nfs-ganesha-proxy-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root   21152 Jan 22 13:04 nfs-ganesha-rgw-2.5.0-0.el7.centos.x86_64.rpm
-rw-r--r-- 1 root root   40148 Jan 22 13:04 nfs-ganesha-vfs-2.5.0-0.el7.centos.x86_64.rpm

如果有报错，根据报错信息去处理下，每个环境配置不一样，报错可能不一样。

2.3 rpm安装nfs-ganesha

下面的几个rpm包google去下载就行

[root@ceph01 mybuild]# rpm -ivh dbus-libs-1.10.24-12.el7.x86_64.rpm --replacefiles
[root@ceph01 mybuild]# rpm -ivh dbus-1.10.24-12.el7.x86_64.rpm --replacefiles
[root@ceph01 mybuild]# rpm -ivh nfs-ganesha-2.5.0-0.el7.centos.x86_64.rpm
[root@ceph01 mybuild]# rpm -ivh nfs-ganesha-rgw-2.5.0-0.el7.centos.x86_64.rpm

3、开始配置

3.1、准备rgw用户

[root@ceph01 mybuild]# radosgw-admin user create --uid=testuser --display-name="testuser " --access-key=testuser --secret=testuser

3.2、准备nfs-ganesha配置文件

配置起来还是比较简单的（不过是最简配置，注意我们的nfs-ganesha服务不一定非要是在ceph集群节点上，是可以在单独一台机器安装配置的）。

[root@ceph01 mybuild]# cat /etc/ganesha/ganesha.conf
EXPORT
{
        Export_ID=1;
        Path = "/";
        Pseudo = "/";
        Access_Type = RW;
        NFS_Protocols = 4;
        Transport_Protocols = TCP;
        FSAL {
                # 下面的值就是开始创建的rgw用户信息
                Name = RGW;
                User_Id = "testuser";
                Access_Key_Id ="testuser";
                Secret_Access_Key = "testuser";
        }
}

RGW {
    ceph_conf = "/etc/ceph/ceph.conf";
}

3.3、启动nfs-ganesha服务

nfs-ganesha和nfs服务有冲突，启动nfs-ganesha之前需要停止nfs服务，也可以通过更改nfs-ganesha的配置去避免冲突（先不搞这么复杂）。

[root@ceph01 mybuild]# systemctl stop nfs

启动nfs-ganesha

[root@ceph01 mybuild]# ganesha.nfsd -L /var/log/ganesha.log

如果使用rpm安装就使用下面的命令

[root@ceph01 mybuild]# systemctl start nfs-ganesha

3.4、查看nfs-ganesha服务是否正常启动

[root@ceph01 tmp]# ps aux|grep ganesha
root      16055  0.2  1.0 5222380 42292 ?       Ssl  15:22   0:06 ganesha.nfsd -L /var/log/ganesha.log

可以看到nfs-ganesha服务已经正常启动了

4、使用

现在来到另外一台服务器上面

4.1、安装nfs-utils

[root@ceph07 ~]# yum install -y nfs-utils

4.2、挂载nfs

这里的192.168.10.20就是我们上面的服务器ip

[root@ceph07 ~]# mount -t nfs4 192.168.10.20:/ /mnt/
[root@ceph07 ~]# mount |grep mnt
192.168.10.20:/ on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.32,local_lock=none,addr=192.168.10.20)

4.3、使用nfs客户端挂载目录

在创建之前查看ceph rgw池里面的对象数

[root@ceph01 tmp]# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    30697M     30480M         217M          0.71 
POOLS:
    NAME                       ID     USED     %USED     MAX AVAIL     OBJECTS 
    rbd                        0         0         0        14472M           0 
    .rgw.root                  1      1588         0        14472M           4 
    default.rgw.control        2         0         0        14472M           8 
    default.rgw.data.root      3         0         0        14472M           0 
    default.rgw.gc             4         0         0        14472M          32 
    default.rgw.log            5         0         0        14472M         127 
    default.rgw.users.uid      6       202         0        14472M           1 
    default.rgw.users.keys     7         9         0        14472M           1

可以看到现在rgw data池里面的对象数是0个，现在在nfs挂载目录下创建一个目录试试

[root@ceph07 ~]# mkdir /mnt/bk001

再次查看rgw data池里面的对象数

[root@ceph01 ceph01 ]# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED 
    30697M     30480M         217M          0.71 
POOLS:
    NAME                          ID     USED     %USED     MAX AVAIL     OBJECTS 
    rbd                           0         0         0        14472M           0 
    .rgw.root                     1      1588         0        14472M           4 
    default.rgw.control           2         0         0        14472M           8 
    default.rgw.data.root         3       591         0        14472M           2 
    default.rgw.gc                4         0         0        14472M          32 
    default.rgw.log               5         0         0        14472M         127 
    default.rgw.users.uid         6       202         0        14472M           2 
    default.rgw.users.keys        7         9         0        14472M           1 
    default.rgw.buckets.index     8         0         0        14472M           1

可以看到rgw data池里面的对象数已经由0增加到2了，其实这里的目录bk001就是对应到rgw里面的bucket

总结

我们上面主要做了这些步骤：

  • 安装ceph并搭建ceph集群
  • 使用radosgw-admin创建rgw用户（在创建rgw用户的时候，会自动创建rgw所需的存储池）
  • 获取nfs-ganesha及其依赖的ntirpc模块
  • 编译、安装和配置nfs-ganesha
  • 最后启动nfs-ganesha服务并在其客户端挂载、测试使用
赏

谢谢你请我吃糖果

微信

扫一扫，分享到微信

微信分享二维码
每天一点点。<br><br>QQ：1041315735<br>Email：1041315735@qq.com