Syncing between buckets

Sync data between source and destination buckets.

Before you begin

At least two running IBM Storage Ceph 8.1 clusters with Ceph Object Gateway configured.

About this task

You can sync data between the source and destination buckets across zones, but not within the same zone. Note that internally, data is still pulled from the source at the destination zone.

A wildcard bucket name means that the current bucket is in the context of the bucket sync policy.

There are two types of syncing between buckets:

Syncing from a different bucket, where you need to specify the source bucket.
Syncing to a different bucket, where you need to specify the destination bucket.

Syncing from a different bucket

Procedure

Create a sync group to pull data from a bucket of another zone.

radosgw-admin sync group create  --bucket=BUCKET_NAME --group-id=GROUP_ID --status=enabled

For example,

[ceph: root@host01 /]# radosgw-admin sync group create --bucket=buck4 --group-id=buck4-default --status=enabled

Pull the data.

radosgw-admin sync group pipe create --bucket-name=BUCKET_NAME --group-id=GROUP_ID --pipe-id=PIPE_ID --source-zones=SOURCE_ZONE_NAME --source-bucket=SOURCE_BUCKET_NAME --dest-zones=DESTINATION_ZONE_NAME

In the following example the source bucket is buck5.

[ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck4 \ 
                          --group-id=buck4-default --pipe-id=pipe1 \ 
                          --source-zones='*' --source-bucket=buck5 \ 
                          --dest-zones='*'

Optional: Sync from buckets in specific zones.

radosgw-admin sync group pipe modify --bucket=BUCKET_NAME

For example,

[ceph: root@host01 /]# radosgw-admin sync group pipe modify --bucket=buck4 \ 
                          --group-id=buck4-default --pipe-id=pipe1 \ 
                          --source-zones=us-west --source-bucket=buck5 \ 
                          --dest-zones='*'

Check sync status.

radosgw-admin sync info --bucket-name=BUCKET_NAME

For example,

[ceph: root@host01 /]# radosgw-admin sync info --bucket=buck5 
{ 
    "sources": [], 
    "dests": [], 
    "hints": { 
        "sources": [], 
        "dests": [ 
            "buck4:115b12b3-....14433.2" 
        ] 
    }, 
    "resolved-hints-1": { 
        "sources": [], 
        "dests": [ 
            { 
                "id": "pipe1", 
                "source": { 
                    "zone": "us-west", 
                    "bucket": "buck5" 
                }, 
                "dest": { 
                    "zone": "us-east", 
                    "bucket": "buck4:115b12b3-....14433.2" 
                }, 
                ... 
            }, 
            { 
                "id": "pipe1", 
                "source": { 
                    "zone": "us-west", 
                    "bucket": "buck5" 
                }, 
                "dest": { 
                    "zone": "us-west-2", 
                    "bucket": "buck4:115b12b3-....14433.2" 
                }, 
                ... 
            } 
        ] 
    }, 
    "resolved-hints": { 
        "sources": [], 
        "dests": [] 
    }

Note: Note that there are resolved-hints, which means that the bucket buck5 found about buck4 syncing from it indirectly, and not from its own policy. The policy for buck5 itself is empty.

Syncing to a different bucket

Procedure

Create a sync group.

radosgw-admin sync group create  --bucket=BUCKET_NAME --group-id=GROUP_ID --status=enabled

For example,

[ceph: root@host01 /]# radosgw-admin sync group create --bucket=buck6 --group-id=buck6-default --status=enabled

Push the data.

radosgw-admin sync group pipe create --bucket-name=BUCKET_NAME --group-id=GROUP_ID --pipe-id=PIPE_ID --source-zones=SOURCE_ZONE_NAME --dest-zones=DESTINATION_ZONE_NAME --dest-bucket=DESTINATION_BUCKET_NAME

In the following example the destination bucket is buck5.

[ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck6 \ 
                          --group-id=buck6-default --pipe-id=pipe1 \ 
                          --source-zones='*' --dest-zones='*' --dest-bucket=buck5

Optional: Sync to buckets in specific zones.

radosgw-admin sync group pipe modify --bucket=BUCKET_NAME

For example,

[ceph: root@host01 /]# radosgw-admin sync group pipe modify --bucket=buck6 \ 
                          --group-id=buck6-default --pipe-id=pipe1 \ 
                          --source-zones='*' --dest-zones='us-west' --dest-bucket=buck5

Check the sync status.

radosgw-admin sync info --bucket-name=BUCKET_NAME

For example,

[ceph: root@host01 /]#  radosgw-admin sync info --bucket buck5
{
  "sources": [],
  "dests": [
    {
      "id": "pipe1",
      "source": {
        "zone": "us-west",
        "bucket": "buck6:c7887c5b-f6ff-4d5f-9736-aa5cdb4a15e8.20493.4"
      },
      "dest": {
        "zone": "us-east",
        "bucket": "buck5"
      },
      "params": {
        "source": {
          "filter": {
            "tags": []
          }
        },
        "dest": {},
        "priority": 0,
        "mode": "system",
        "user": "s3cmd"
      }
    },
  ],
  "hints": {
    "sources": [],
    "dests": [
      "buck5"
    ]
  },
  "resolved-hints-1": {
    "sources": [],
    "dests": []
  },
  "resolved-hints": {
    "sources": [],
    "dests": []
  }
}