Deploying a multi-site Ceph Object Gateway

Ceph Orchestrator supports multi-site configuration options for the Ceph Object Gateway.

You can configure each object gateway to work in an active-active zone configuration allowing writes to a non-primary zone. The multi-site configuration is stored within a container called a realm.

The realm stores zone groups, zones, and a time period. The rgw daemons handle the synchronization eliminating the need for a separate synchronization agent, thereby operating with an active-active configuration.

You can also deploy multi-site zones using the command line interface (CLI).

Note: The following configuration assumes at least two IBM Storage Ceph clusters are in geographically separate locations. However, the configuration also works on the same site.

Prerequisites

At least two running IBM Storage Ceph clusters.
At least two Ceph Object Gateway instances, one for each IBM Storage Ceph cluster.
Root-level access to all the nodes.
Nodes or containers are added to the storage cluster.
All Ceph Manager, Monitor and OSD daemons are deployed.

Procedure

In the cephadm shell, configure the primary zone:

Create a realm:

Syntax

radosgw-admin realm create --rgw-realm=REALM_NAME --default

Example

[ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=test_realm --default

If the storage cluster has a single realm, then specify the --default flag.

Create a primary zone group:

Syntax

radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 --master --default

Example

[ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --master --default

Create a primary zone:

Syntax

radosgw-admin zone create --rgw-zonegroup=PRIMARY_ZONE_GROUP_NAME --rgw-zone=PRIMARY_ZONE_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:RGW_PRIMARY_PORT_NUMBER_1 --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY

Example

[ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-1 --endpoints=http://rgw1:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

Optional: Delete the default zone, zone group, and the associated pools.

Important: Do not delete the default zone and its pools if you are using the default zone and zone group to store data. Also, removing the default zone group deletes the system user.

To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands.

Example

[ceph: root@host01 /]# radosgw-admin zonegroup delete --rgw-zonegroup=default
[ceph: root@host01 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
[ceph: root@host01 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it

Create a system user:

Syntax

radosgw-admin user create --uid=USER_NAME --display-name="USER_NAME" --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY --system

Example

[ceph: root@host01 /]# radosgw-admin user create --uid=zone.user --display-name="Zone user" --system

Make a note of the access_key and secret_key.

Add the access key and system key to the primary zone:

Syntax

radosgw-admin zone modify --rgw-zone=PRIMARY_ZONE_NAME --access-key=ACCESS_KEY --secret=SECRET_KEY

Example

[ceph: root@host01 /]# radosgw-admin zone modify --rgw-zone=us-east-1 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

Commit the changes:

Syntax

radosgw-admin period update --commit

Example

[ceph: root@host01 /]# radosgw-admin period update --commit

Outside the cephadm shell, fetch the FSID of the storage cluster and the processes:

Example
```
[root@host01 ~]#  systemctl list-units | grep ceph
```

Start the Ceph Object Gateway daemon:

Syntax

systemctl start ceph-FSID@DAEMON_NAME
systemctl enable ceph-FSID@DAEMON_NAME

Example

[root@host01 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service
[root@host01 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service

In the cephadm shell, configure the secondary zone.

Pull the primary realm configuration from the host:

Syntax

radosgw-admin realm pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY

Example

[ceph: root@host04 /]# radosgw-admin realm pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

Pull the primary period configuration from the host:

Syntax

radosgw-admin period pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY

Example

[ceph: root@host04 /]# radosgw-admin period pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

Configure a secondary zone:

Syntax

radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME
              --rgw-zone=_SECONDARY_ZONE_NAME_
              --access-key=_SYSTEM_ACCESS_KEY_ --secret=SYSTEM_SECRET_KEY
              --endpoints=http://RGW_SECONDARY_HOSTNAME:PORT_NUMBER"
              [--read-only]

Example

[ceph: root@host04 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ --endpoints=http://rgw.example.com:80

Optional: Delete the default zone.

Important: Do not delete the default zone and its pools if you are using the default zone and zone group to store data.

To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands.

Example

[ceph: root@host04 /]# radosgw-admin zone rm --rgw-zone=default
[ceph: root@host04 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
[ceph: root@host04 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it

Update the Ceph configuration database:

Syntax

ceph config set SERVICE_NAME rgw_zone SECONDARY_ZONE_NAME

Example

[ceph: root@host04 /]# ceph config set rgw rgw_zone us-east-2

Commit the changes:

Syntax

radosgw-admin period update --commit

Example

[ceph: root@host04 /]# radosgw-admin period update --commit

Outside the cephadm shell, fetch the FSID of the storage cluster and the processes:

Example
```
[root@host04 ~]#  systemctl list-units | grep ceph
```

Start the Ceph Object Gateway daemon:

Syntax

systemctl start ceph-FSID@DAEMON_NAME
systemctl enable ceph-FSID@DAEMON_NAME

Example

[root@host04 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service
[root@host04 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service

Optional: Deploy multi-site Ceph Object Gateways using the placement specification:

Syntax

ceph orch apply rgw NAME --realm=REALM_NAME --zone=PRIMARY_ZONE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2"

Example

[ceph: root@host04 /]# ceph orch apply rgw east --realm=test_realm --zone=us-east-1 --placement="2 host01 host02"

Verification

Check the synchronization status to verify the deployment:

Example
```
[ceph: root@host04 /]# radosgw-admin sync status
```