在本章中，我们将介绍以下食谱：

• Functional changes from Hammer federated configuration
• RGW multi-site v2 requirement
• Installing the Ceph RGW multi-site v2 environment
• Configuring Ceph RGW multi-site v2
• Testing user, bucket, and object sync between master and secondary sites

单个区域配置通常由一个区域组组成，其中包含一个区域和一个或多个 RGW 实例，您可以在其中对实例之间的网关客户端请求进行负载平衡。在单个区域配置中，通常多个网关实例指向单个 Ceph 存储集群。在 Jewel 版本中，Ceph 支持 Ceph 对象网关的多个多站点配置选项：

• Zone: In a zone, one or more Ceph Object Gateways are logically grouped.
• Zone group: A zone group is a container of multiple zones. In a multi-site configuration there should be a master zone group. All the changes to configurations are handled by the master zone group.
• Realm: A realm can have multiple zone groups. It allows separation of the zone groups themselves between clusters. There can be multiple realms for having different configurations in the same cluster.
• Period: Every realm has a corresponding current period. Each period is a container of an epoch and an unique id. A period holds the current state of configuration of the zone groups and object storage strategies. Each period's commit operation, as well as any configuration change for a non-master zone, will increment the period’s epoch.

以下是区域配置的示意图：

在 Jewel 中，您可以将每个 Ceph 对象网关配置为在主动-主动区域配置中工作，从而允许写入非主区域。

多站点配置存储在称为领域的容器中。它存储区域、区域组和具有多个时期的时间段，用于跟踪配置的更改。在 Jewel 中，ceph-radosgw 守护进程通过消除对单独同步代理的需要来处理同步。此外，新的同步方法允许 Ceph 对象网关以主动-主动配置而不是主动-被动配置运行。

在这个秘籍中，您至少需要两个 Ceph 集群和至少两个对象网关（每个 Ceph 集群一个）。这是 RGW 多站点的最低要求。

这个秘籍假设有两个 Ceph 对象网关服务器，分别名为 us-east-1 和 us-west-1，有两个各自的 Ceph 集群。

在多站点配置中，需要一个主区域组和一个主区域。此外，每个区域组都需要一个主区域。区域组可能有一个或多个辅助或非主区域。

在这个秘籍中，us-east-1 主机将作为主区域组的主区域； us-west-1 主机将作为主区域组的辅助区域。

您需要安装两个集群，并且在每个集群上，您需要为客户端 I/O 和 RGW 复制安装至少一个 RGW 守护程序。

在这个秘籍中，我们将为第一个集群使用 ceph-node1、ceph-node2 和 ceph-node3 虚拟机，在这个集群，我们将使用 us-east-1 虚拟机作为 RGW 节点。在第二个集群中，我们将 ceph-node4、ceph-node5 和 ceph-node5 虚拟机用于集群节点和RGW 节点的 us-west-1 虚拟机。

您应该使用 第 1 章，Ceph – 介绍和超越，用于安装集群和 第 4 章，使用 Ceph 对象存储，用于在两个集群中安装 RGW 节点：

1. Bring up all eight VMs with the help of the vagrant up command:

        $ vagrant up ceph-node1 ceph-node2 ceph-node3 ceph-node4 ceph-node5 
          ceph-node6 us-east-1 us-west-1

2. Once all VMs come up, log in and update to CentOS release 7.4 and then reboot to CentOS 7.4:

        # vagrant ssh <vm-name>
        #sudo su -
        # yum update -y
        # reboot

您应该配置防火墙设置和 NTP 设置，如 第 1 章，Ceph - 简介及其他，在多站点配置中使用的所有节点中。

创建主集群和主 RGW 节点 us-east-1：

1. Install the ceph-ansible and ansible package, as given in Chapter 1, Ceph – Introduction and Beyond, and then update /etc/ansible/hosts for cluster 1 VMs:

2. You should copy the ceph-node1 ssh key to all cluster 1 nodes, including RGW node us-east-1.
3. Update all.yml and osds.yml and other Ansible configuration files, as given in Chapter 1, Ceph – Introduction and Beyond. For the us-east-1 RGW node, you should enable the following options in all.yml:

        radosgw_civetweb_port: 8080
        radosgw_civetweb_bind_ip: 192.168.1.107

4. After this configuration, you should run the ansible-playbook to install cluster 1 with the us-east-1 RGW node:

        ansible-playbook site.yml

5. Once the ansible-playbook finishes installing the cluster, you will have the following cluster running; please notice the monitor names:

创建辅助集群和主 RGW 节点 us-west-1：

您应该将 ceph-node4 ssh 密钥复制到所有集群 2 节点，包括 RGW 节点 us-west-1。

2. Update all.yml and osds.yml and other Ansible configuration files, as given in Chapter 1, Ceph – Introduction and Beyond. For the us-west-1 RGW node, you should enable the following options in all.yml:

        radosgw_civetweb_port: 8080
        radosgw_civetweb_bind_ip: 192.168.1.108

3. After this configuration, you should run the ansible-playbook to install cluster 1 with the us-west-1 RGW node:

        ansible-playbook site.yml

4. Once the ansible-playbook finishes installing the cluster, you will have the following cluster running; please notice the monitor names:

在以下部分中，您将为 RGW 主动-主动多站点配置主区域和辅助区域；这意味着您可以在两个站点上写入数据，并将其复制到另一个站点集群。像用户创建这样的元数据操作只能在主站点上执行。

我们将使用以下步骤来配置 RGW 多站点 v2 主区域和辅助区域：

多站点 v2 配置中的所有 RADOS 网关都将从主区域组和主区域内节点上的 radosgw 守护程序获取其配置。要在多站点 v2 配置中配置 RADOS 网关，您需要选择一个 radosgw 实例来配置主区域组和主区域。您应该使用 us-east-1 RGW 实例来配置您的主区域：

1. Create an RGW keyring in the /etc/ceph path and check if you are able to access the cluster with user RGW Cephx:

        # cp /var/lib/ceph/radosgw/ceph-rgw.us-east-1/
          keyring /etc/ceph/ceph.client.rgw.us-east-1.keyring
        # cat /etc/ceph/ceph.client.rgw.us-east-1.keyring
        # ceph -s --id rgw.us-east-1

现在您应该可以使用这个 RGW Cephx 用户在集群 1 中运行 radosgw-admin 命令了。

2. Create the RGW multi-site v2 realm. Run the following command in the us-east-1 RGW node to create a realm:

        # radosgw-admin realm create --rgw-realm=cookbookv2 
                                     --default --id rgw.us-east-1

您可以忽略前面屏幕截图中给出的错误消息；它将在 Jewel 的未来版本中修复。这是一个已知问题；这不是错误，而是声明为错误消息的信息消息。这不会导致配置 RGW 多站点 v2 时出现任何问题。

3. Create a master zone group. An RGW realm must have at least one RGW zone group, which will serve as the master zone group for the realm.

在us-east-1 RGW节点运行以下命令创建master zone group：

        # radosgw-admin zonegroup create --rgw-zonegroup=us 
          --endpoints=http://us-east-1.cephcookbook.com:8080 
          --rgw-realm=cookbookv2 --master --default 
          --id rgw.us-east-1

4. Create a master zone. An RGW zone group must have at least one RGW zone. Run the following command in the us-east-1 RGW node to create a master zone:

        # radosgw-admin zone create --rgw-zonegroup=us 
          --rgw-zone=us-east-1 --master --default 
          --endpoints=http://us-east-1.cephcookbook.com:8080 
          --id rgw.us-east-1

5. Remove default zone group and zone information from cluster 1:

        # radosgw-admin zonegroup remove --rgw-zonegroup=default
          --rgw-zone=default --id rgw.us-east-1
        # radosgw-admin zone delete --rgw-zone=default 
          --id rgw.us-east-1
        # radosgw-admin zonegroup delete --rgw-zonegroup=default 
          --id rgw.us-east-1

最后，使用新的 us 区域组和 us-east-1 区域更新时间段，这将用于多站点 v2：

        # radosgw-admin period update --commit --id rgw.us-east-1

6. Remove the RGW default pools:

        # for i in `ceph osd pool ls --id rgw.us-east-1 | 
          grep default.rgw`; do ceph osd pool delete $i $i 
          --yes-i-really-really-mean-it --id rgw.us-east-1; done

7. Create an RGW multi-site v2 system user. In the master zone, create a system user to establish authentication between multi-site radosgw daemons:

        # radosgw-admin user create --uid="replication-user" 
          --display-name="Multisite v2 replication user" 
          --system --id rgw.us-east-1

8. Finally, update the period with this system user information:

      # radosgw-admin zone modify --rgw-zone=us-east-1 
        --access-key=ZYCDNTEASHKREV4X9BUJ 
        --secret=4JbC4OC4vC6fy6EY6Pfp8rPZMrpDnYmETZxNyyu9 
        --id rgw.us-east-1
      # radosgw-admin period update --commit --id rgw.us-east-1

9. You also need to update the [client.rgw.us-east-1] section of ceph.conf with the rgw_zone=us-east-1 option:

10. Restart the us-east-1 RGW daemon:

        #  systemctl restart ceph-radosgw.target

在 RGW 中，多站点区域复制区域组内的所有数据，以确保每个区域具有相同的集合数据。在本节中，您将配置次要区域。 您应该使用 us-west-1 RGW 实例来配置您的辅助区域：

1. Create an RGW keyring in the /etc/ceph path and check if you are able to access the cluster with user RGW Cephx:

        # cp /var/lib/ceph/radosgw/ceph-rgw.us-west-1/
        keyring /etc/ceph/ceph.client.rgw.us-west-1.keyring
        # cat /etc/ceph/ceph.client.rgw.us-west-1.keyring
        # ceph -s --id rgw.us-west-1

您应该在辅助站点 RGW 节点 us-west-1 中运行以下步骤：

2. First of all, you need to pull the RGW realm.

需要使用RGW端点URL路径和master zone组中master zone的access key和secret key将realm拉到secondary zone RGW节点：

        # radosgw-admin realm pull 
          --url=http://us-east-1.cephcookbook.com:8080 
          --access-key=ZYCDNTEASHKREV4X9BUJ 
          --secret=4JbC4OC4vC6fy6EY6Pfp8rPZMrpDnYmETZxNyyu9 
          --id rgw.us-west-1

3. As this is the default realm for this RGW multi-site setup, you need to make it default:

        # radosgw-admin realm default --rgw-realm=cookbookv2
                                      --id rgw.us-west-1

4. You need to pull the period from the master site, because you need to get the latest version of the zone group and zone configurations for the realm:

        # radosgw-admin period pull 
          --url=http://us-east-1.cephcookbook.com:8080 
          --access-key=ZYCDNTEASHKREV4X9BUJ 
          --secret=4JbC4OC4vC6fy6EY6Pfp8rPZMrpDnYmETZxNyyu9 
          --id rgw.us-west-1

5. Create a secondary zone. Your secondary zone RGW node is us-west-1 and you need to run the following command in the secondary zone RGW node us-west-1:

        # radosgw-admin zone create --rgw-zonegroup=us 
          --rgw-zone=us-west-1 --access-key=ZYCDNTEASHKREV4X9BUJ
          --secret=4JbC4OC4vC6fy6EY6Pfp8rPZMrpDnYmETZxNyyu9
          --endpoints=http://us-west-1.cephcookbook.com:8080
          --id rgw.us-west-1

6. Remove the default zone from the secondary site, as you are not using it:

        # radosgw-admin zone delete --rgw-zone=default
                                    --id rgw.us-west-1

7. Finally, update the period on the secondary site:

        # radosgw-admin period update --commit 
          --id rgw.us-west-1

8. Remove the RGW default pools:

        # for i in `ceph osd pool ls --id rgw.us-west-1 | 
          grep default.rgw`; do ceph osd pool delete $i $i 
          --yes-i-really-really-mean-it --id rgw.us-west-1; done

9. You also need to update the [client.rgw.us-west-1] section of ceph.conf with the rgw_zone=us-west-1 option:

10. Then, restart the us-west-1 RGW daemon:

        #  systemctl restart ceph-radosgw.target

这样，主站点和辅助站点都启动并运行主动-主动异步复制，现在您可以检查两个站点上的同步状态。

以下命令将用于检查主区域和辅助区域之间的状态：

1. Cluster 1 master zone us-east-1 synchronization status:

        # radosgw-admin sync status --id rgw.us-east-1

2. Cluster 2 secondary zone us-west-1 synchronization status:

        #  radosgw-admin sync status --id rgw.us-west-1

此时，您应该已经配置了 RGW 多站点 V2。在本秘籍中，您将测试主站点和辅助站点之间的用户、存储桶和对象同步。

我们将使用以下命令来测试主区域和辅助区域之间的用户、存储桶和对象同步：

1. Let's create an s3 user in the master site and check if it gets synced to the secondary site. You should run the following commands in the master site RGW node us-east-1:

        # radosgw-admin user create --uid=pratima 
                                    --display-name="Pratima Umrao"
                                    --id rgw.us-east-1

2. Check the number of users in the master site:

        # radosgw-admin metadata list user --id rgw.us-east-1

3. Let's verify that the user "pratima" got synced to the secondary site. You should run the following commands in the secondary site RGW node us-west-1:

        # radosgw-admin metadata list user --id rgw.us-west-1

        # radosgw-admin user info --uid=pratima --id rgw.us-west-1

我们将在两个站点中使用 s3cmd 应用程序来创建存储桶并将对象上传到这些存储桶，并查看存储桶和对象是否在两个站点上同步。

4. Install and configure s3cmd at the master site us-east-1 node:

5. Configure s3cmd at the master site in the us-east-1 RGW node:

6. Update .s3cfg file options host_base and host_bucket with the master site RGW node hostname us-east-1.cephcookbook.com:

7. Install and configure s3cmd at the secondary site us-west-1 node:

8. Configure s3cmd at the secondary site in the us-west-1 RGW node:

9. Update .s3cfg file options host_base and host_bucket with the master site RGW node hostname us-west-1.

10. Create the test-bucket-master and upload: '/etc/hosts' object in the bucket test-bucket-master:

11. You can now check in the secondary site that the bucket and objects are synced:

12. Now you can do the opposite to check the active-active replication; for this you need to create the test-bucket-secondary and upload: '/root/anaconda-ks.cfg' objects in the bucket test-bucket-secondary at the secondary RGW node us-west-1:

13. You can now check in the master site that the bucket test-bucket-secondary and anaconda-ks.cfg objects are synced:

您可以看到 bucket test-bucket-secondary 和 anaconda-ks.cfg 对象已同步到主站点 us-east-1< /kbd>。这是主动-主动复制的行为。