Transitioning an object from one storage class to another

The object lifecycle transition rule allows you to transition an object from one storage class to another class.

Before you begin

A running IBM Storage Ceph cluster.
Installation of the Ceph Object Gateway software.
Root-level access to the Ceph Object Gateway node.
An S3 user created with user access.

About this task

You can migrate data between replicated pools, erasure-coded pools, replicated to erasure-coded pools, or erasure-coded to replicated pools with the Ceph Object Gateway lifecycle transition policy.

Note: Starting with 7.1, the policy-based lifecycle transition and storage class model is extended to support two more S3-compatible cloud platforms for cloud archival - IBM COS and IBM Storage Ceph.

Procedure

Create a new data pool.

ceph osd pool create POOL_NAME

For example,

[ceph: root@host01 /]# ceph osd pool create test.hot.data

Add a new storage class.

radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

For example,

[ceph: root@host01 /]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class hot.test
{
        "key": "default-placement",
        "val": {
            "name": "default-placement",
            "tags": [],
            "storage_classes": [
                "STANDARD",
                "hot.test"
            ]
        }
    }

Provide the zone placement information for the new storage class.

radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

For example,

[ceph: root@host01 /]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class hot.test --data-pool test.hot.data
{
           "key": "default-placement",
           "val": {
               "index_pool": "test_zone.rgw.buckets.index",
               "storage_classes": {
                   "STANDARD": {
                       "data_pool": "test.hot.data"
                   },
                   "hot.test": {
                       "data_pool": "test.hot.data",
                  }
               },
               "data_extra_pool": "",
               "index_type": 0
 }

Note: Consider setting the compression_type when creating cold or archival data storage pools with write once.

Enable the rgw application on the data pool.

ceph osd pool application enable POOL_NAME rgw

For example,

[ceph: root@host01 /]# ceph osd pool application enable test.hot.data rgw
enabled application 'rgw' on pool 'test.hot.data'

Restart all the rgw daemons.

Create a bucket.

aws s3api create-bucket --bucket BUCKET_NAME --create-bucket-configuration BUCKET_CONFIGURATION --endpoint-url ENDPOINT_URL

For example,

[ceph: root@host01 /]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://10.0.0.80:8080

Add the object.

aws --endpoint-url ENDPOINT_URL s3api put-object --bucket BUCKET_NAME

For example,

[ceph: root@host01 /]# aws --endpoint=http://10.0.0.80:8080 s3api put-object --bucket testbucket10

Create a second data pool.

ceph osd pool create POOL_NAME

For example,

[ceph: root@host01 /]# ceph osd pool create test.cold.data

Add a new storage class.

radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

For example,

[ceph: root@host01 /]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class cold.test
{
        "key": "default-placement",
        "val": {
            "name": "default-placement",
            "tags": [],
            "storage_classes": [
                "STANDARD",
                "cold.test"
            ]
        }
}

Provide the zone placement information for the new storage class.

radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

For example,

[ceph: root@host01 /]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class cold.test --data-pool test.cold.data

Enable rgw application on the data pool.

ceph osd pool application enable POOL_NAME rgw

For example,

[ceph: root@host01 /]# ceph osd pool application enable test.cold.data rgw
enabled application 'rgw' on pool 'test.cold.data'

Restart all the rgw daemons.

View the zone group configuration.

radosgw-admin zonegroup get --rgw-zonegroup=ZONE_GROUP_NAME > JSON_FILE_NAME

For example,

radosgw-admin zonegroup get --rgw-zonegroup= us > rgw.json
{
    "id": "3019de59-ddde-4c5c-b532-7cdd29de09a1",
    "name": "default",
    "api_name": "default",
    "is_master": "true",
    "endpoints": [],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
    "zones": [
        {
            "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
            "name": "default",
            "endpoints": [],
            "log_meta": "false",
            "log_data": "false",
            "bucket_index_max_shards": 11,
            "read_only": "false",
            "tier_type": "",
            "sync_from_all": "true",
            "sync_from": [],
            "redirect_zone": ""
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": [],
            "storage_classes": [
                "hot.test",
                "cold.test",
                "STANDARD"
            ]
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "",
    "sync_policy": {
        "groups": []
    }
}

View the zone configuration.

radosgw-admin zone get [--rgw-zone=ZONE_NAME]

For example,

radosgw-admin zone get
{
    "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "roles_pool": "default.rgw.meta:roles",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "otp_pool": "default.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "storage_classes": {
                    "cold.test": {
                        "data_pool": "test.cold.data"
                    },
                    "hot.test": {
                        "data_pool": "test.hot.data"
                    },
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "realm_id": "",
    "notif_pool": "default.rgw.log:notif"
}

Create a bucket.

aws s3api create-bucket --bucket BUCKET_NAME --create-bucket-configuration BUCKET_CONFIGURATION --endpoint-url ENDPOINT_URL

For example,

[ceph: root@host01 /]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://10.0.0.80:8080

List the objects prior to transition.

radosgw-admin bucket list --bucket BUCKET_NAME

[ceph: root@host01 /]# radosgw-admin bucket list --bucket testbucket10
{
            "ETag": "\"211599863395c832a3dfcba92c6a3b90\"",
            "Size": 540,
            "StorageClass": "STANDARD",  
            "Key": "obj1",
            "VersionId": "W95teRsXPSJI4YWJwwSG30KxSCzSgk-",
            "IsLatest": true,
            "LastModified": "2023-11-23T10:38:07.214Z",
            "Owner": {
                "DisplayName": "test-user",
                "ID": "test-user"
            }
}

Create a JSON file for lifecycle configuration.

vi LICEYCLE_CONFGURATION_FILE.json

For example,

[ceph: root@host01 /]# vi lifecycle.json

Add the specific lifecycle configuration rule in the file.

{
    "Rules": [
        {
            "Filter": {
                "Prefix": ""
            },
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 5,
                    "StorageClass": "hot.test"
                },
 {
                    "Days": 20,
                    "StorageClass": "cold.test"
                }
            ],
            "Expiration": {
                "Days": 365
            },
            "ID": "double transition and expiration"
        }
    ]
}

The lifecycle configuration example shows an object that transitions from the default STANDARD storage class to the hot.test storage class after 5 days, again transitions after 20 days to the cold.test storage class, and finally expires after 365 days in the cold.test storage class.

Set the lifecycle configuration on the bucket.

aws s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration LICEYCLE_CONFGURATION_FILE.json

For example,

[ceph: root@host01 /]# aws s3api put-bucket-lifecycle-configuration --bucket testbucket10 --lifecycle-configuration file://lifecycle.json

Retrieve the lifecycle configuration on the bucket.

aws s3api get-bucket-lifecycle-configuration --bucket BUCKET_NAME

For example,

[ceph: root@host01 /]# aws s3api get-bucket-lifecycle-configuration --bucket testbucket10
{
   "Rules": [
       {
           "Expiration": {
               "Days": 365
           },
           "ID": "double transition and expiration",
           "Prefix": "",
           "Status": "Enabled",
           "Transitions": [
               {
                   "Days": 20,
                   "StorageClass": "cold.test"
               },
               {
                   "Days": 5,
                   "StorageClass": "hot.test"
               }
           ]
       }
   ]
}

Verify that the object is transitioned to the given storage class.

radosgw-admin bucket list --bucket BUCKET_NAME

For example,

[ceph: root@host01 /]# radosgw-admin bucket list --bucket testbucket10
{
            "ETag": "\"211599863395c832a3dfcba92c6a3b90\"",
            "Size": 540,
            "StorageClass": "cold.test",   
            "Key": "obj1",
            "VersionId": "W95teRsXPSJI4YWJwwSG30KxSCzSgk-",
            "IsLatest": true,
            "LastModified": "2023-11-23T10:38:07.214Z",
            "Owner": {
                "DisplayName": "test-user",
                "ID": "test-user"
            }
}