The blog post showcases how to migrate all buckets from one IBM Cloud Object Storage (COS) instance to another in the US region.
This is done by using a script in Python. The source IBM Cloud Object Storage (COS) instance will be the instance from which you are migrating, and the target/destination COS instance is the instance to which instance you will be migrating. This script uses ibm-cos and ibm-platform-services SDKs. An example of the architecture is below:
- Make sure you have at least two COS instances on the same IBM Cloud account
- Install Python
- Make sure you have the necessary permissions to do the following:
- Create buckets
- Modify buckets
- Create IAM policy for COS instances
- Install libraries for Python
- ibm-cos-sdk for python:
pip3 install ibm-cos-sdk
pip3 install ibm-platform-services
Set environment variables for the script
The following are the environment variables that the scripts use:
You can create and download your IBM Cloud API key in the IBM cloud console at Manage > Access (IAM) > API keys.
You can find GUID for the source and target instances in the cloud console resource list. Type in the name of each COS instance and click on the white part of the row of the instance to retrieve the GUID.
To find your US COS endpoint, click on your source COS instance from the Resource List in the navigation menu. Then, click on Endpoints and make sure the Selection Location dropdown says us-geo. Select the region that your buckets are in and make sure to prepend https:// in the environment variable.
Leave the values for the IAM_POLICY_MANAGEMENT_URL, IAM_POLICY_MANAGEMENT_AUTHTYPE and DISABLE_RULES as is.
The iam_account_id is the same value as your ibmcloud_api_key.
The suffix is used to append a name at the end of the newly created bucket since bucket names are globally unique.
Run the script
After the environment variables have been set, you may now run the script. You can find the code of the script below here.
COS instance migration script
This script was designed to help users migrate one COS instance to another instance on the same account for a US region. The function calls in the main function are executed in the following order.
migrateBuckets function: This function gathers all buckets from one source COS instance and creates them in the target COS instance. The newly created target bucket will have a suffix attached to it.
addReplicationRulesToMigratedBuckets function: The function enables replication rules to the source buckets so it can write data to the target buckets when data is added or removed after the rule is applied. This is achieved by enabling versioning on both source and target buckets. Versioning is required to enable replication. Versioning is a history of all files in a bucket. The script also creates an IAM policy on the entire source and destination instance to allow source buckets to write to their respective target buckets. Make sure DISABLE_RULES to false.
replicateExistingFiles function: I previously mentioned that replication applies to a bucket when newly adding or deleting files after the rule has been set. If you want to transfer files that existed before the rule was applied, make sure DISABLE_RULES to false to activate this function.
Disable replication rules
If you want to disable the replication rules for the buckets, set DISABLE_RULES to true and run the script again.
By following these steps, you will successfully migrate buckets from one US IBM Cloud Object Storage (COS) instance to another per region.
If you have any questions, you can reach out to me on LinkedIn.