Syncing a multi-site Ceph Object Gateway

A multi-site sync reads the change log from other zones.

To get a high-level view of the sync progress from the metadata and the data logs, you can use the following command:

radosgw-admin sync status

This command lists which log shards, if any, which are behind their source zone.

Note: Sometimes you might observe recovering shards when running the radosgw-admin sync status command. For data sync, there are 128 shards of replication logs that are each processed independently. If any of the actions triggered by these replication log events result in any error from the network, storage, or elsewhere, those errors get tracked so the operation can retry again later. While a given shard has errors that need a retry, radosgw-admin sync status command reports that shard as recovering. This recovery happens automatically, so the operator does not need to intervene to resolve them.

If the results of the sync status you have previously run reports log shards are behind, run the following command substituting the shard-id for X.

Buckets within a multi-site object can also be monitored on the Ceph dashboard. For more information, see Monitoring buckets of a multi-site object.

radosgw-admin data sync status --shard-id=X --source-zone=ZONE_NAME

For example,

[ceph: root@host01 /]# radosgw-admin data sync status --shard-id=27 --source-zone=us-east
{
  "shard_id": 27,
  "marker": {
         "status": "incremental-sync",
         "marker": "1_1534494893.816775_131867195.1",
         "next_step_marker": "",
         "total_entries": 1,
         "pos": 0,
         "timestamp": "0.000000"
   },
   "pending_buckets": [],
   "recovering_buckets": [
         "pro-registry:4ed07bb2-a80b-4c69-aa15-fdc17ae6f5f2.314303.1:26"
   ]
}

The output lists which buckets are next to sync and which buckets, if any, are going to be retried due to previous errors.

Inspect the status of individual buckets with the following command, substituting the bucket ID for X.

radosgw-admin bucket sync status --bucket=X

Replace X with the ID number of the bucket.

The result shows which bucket index log shards are behind their source zone.

A common error in sync is EBUSY, which means the sync is already in progress, often on another gateway. Read errors that are written to the sync error log, which can be read with the following command:

radosgw-admin sync error list

The syncing process tries again until it is successful. Errors can still occur that can require intervention.