In June of 2020, IBM Cloud VPC Gen 2 Infrastructure was announced, supporting more advanced IaaS capabilities.
Both the IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud can now leverage the next generation of IBM Cloud VPC virtual servers in creating Kubernetes-based clusters to run stateful applications.
This technical blog post will explain how to migrate a stateful application, including persistent volume’s data, from an IBM Cloud VPC Gen 1 (Classic) Kubernetes cluster to an IBM Cloud VPC Gen 2 Kubernetes cluster. There are a variety of tooling and mechanisms that exist in accomplishing the application migration from an older VPC Gen 1 cluster to a new VPC Gen 2 cluster. This blog will focus on describing using the following tooling in supporting the application and data migration:
The steps both to install and configure the migration tooling—including showcasing a sample WordPress application migration—will be shown.
- Set up the Velero CLI on an IBM Cloud VPC Kubernetes cluster.
- Back up an application and its data (PVC) using the Velero and restic.
- Migrate WordPress application from a VPC Gen 1 cluster to a VPC Gen 2 cluster, including its data.
The following prerequisites are required to migrate an application from one cluster to another (whether these clusters are in the same or different regions):
- The OC CLI (in case of Red Hat OpenShift cluster)
- The IBM Cloud and kubectl CLI (for set up, see Getting started with the IBM Cloud CLI)
- Pre-created IBM Cloud VPC Kubernetes clusters (Gen 1 and Gen 2)
- The IBM Cloud VPC CSI volume driver installed on both the Gen 1 and Gen 2 clusters
- Install the Velero CLI on your system
- Create an IBM Cloud Object Storage instance
- Set up the Velero server on the source (IBM Cloud VPC Gen 1) Kubernetes cluster
- Deploy the WordPress application on the source cluster
- [Optional] Access the WordPress application and write some data
- Set up Velero and restic for the WordPress application pods
- Take the WordPress application backup to IBM Cloud Object Storage by using the Velero CLI
- Verify backup
- Set up the Velero server on the target (IBM Cloud VPC Gen 2) Kubernetes cluster
- Restore the WordPress application backup on target (IBM Cloud VPC Gen 2) Kubernetes cluster
- Verify restore
- Uninstall Velero
Step 1: Install the Velero CLI on your system
You can install the Velero CLI on Mac OS by running the following command:
Step 2: Create an IBM Cloud Object Storage instance
You can follow this link to create an IBM Cloud Object Storage instance.
Then, create a credential for an IBM Cloud Object Storage instance. Make sure you select the
HMAC option on the UI as these
HMAC-based credentials will be required for setting up the Velero server in the cluster. More details.
After you create the HMAC credentials, copy the values of the
secret_access_key fields to use when installing Velero server on the clusters.
Step 3: Set up the Velero server on the source (IBM Cloud VPC Gen 1) Kubernetes cluster
Create a file name—
os-cred-for-velero—and add the
secret_access_key keys that you got from Step 2.
Install the Velero server on the cluster. Velero normally takes volume backup via snapshot, but that requires provider support. Therefore, we need to disable snapshot and enable restic to backup PVC data while installing the Velero server.
Use the following command to install the Velero server. Use this link to get the right set of inputs for the IBM Cloud Object Storage endpoint and region:
Verify that Velero is installed and that restic pods are equal to the number of nodes in the cluster:
Note: If your Kubernetes cluster is created on Red Hat OpenShift on IBM Cloud, you need to follow additional steps. Restic must run in privileged mode, so you must annotate the
velero namespace and create a custom SCC to allow restic to run properly.
Run the following command to add the security context in the restic:
Modify DaemonSet pod as privileged by adding securityContext:
Step 4: Deploy the WordPress application on the source cluster
You can either take a backup of a running application or just try to deploy the WordPress application for experimental purposes on the cluster and then take a backup.
Before beginning to deploy the WordPress application or taking backup of existing application, make sure that the IBM VPC CSI Driver is in a running state:
Deploy the WordPress application in a
wordpress namespace so that you can take a backup of namespace by using Velero. Follow the steps in this Kubernetes tutorial.
After you complete the tutorial, the following three resources are created:
These WordPress deployment files create two persistent volumes—
wp-pv-claim—that are provisioned by IBM VPC CSI Driver.
Copy the names of the
VolumeMounts in these deployment's pods, which will be used to take backup of volume data.
mysql-deployment.yaml, the pod's
VolumeMounts name is
wordpress-deployment.yaml, the pod's
VolumeMounts name is
Step 5: Access the WordPress application and write some data [optional]
You can verify the WordPress application by checking service and pod status:
To access the WordPress application, get the LoadBalancer details as follows:
Access the WordPress application by using the
http://32e01c88-us-south.lb.appdomain.cloud/ URL and add user or post comments in order to write data to the PVCs.
Step 6: Set up Velero and restic for the WordPress application pods
To take a correct set of backups along with persistent volume data so that the restore will work as expected, you need to set the annotation in the running application pod with the volume names that are used at the time of mounting in the pods. In the case of the WordPress application,
VolumeMounts names are
To add the annotation in the WordPress application's pods, get the running pods in the
Run following command to add the annotations in the pods:
Note: You can specify other volumes as per your application by using comma-separation.
Step 7: Take the WordPress application backup to IBM Cloud Object Storage by using the Velero CLI
The WordPress application backup will be taken by using the entire
wordpress namespace so that all Kubernetes objects that are related to the application are backed up.
Use the following command to take WordPress application backup as name
gen1clusterbackup (or any other name):
Step 8: Verify backup
To verify the backup, run the following Velero command:
This may take some time. Once the backup is finished, the status will show as
Completed. If you encounter any issues, follow the troubleshooting guide.
Step 9: Set up the Velero server on the target (IBM Cloud VPC Gen 2) Kubernetes cluster
For this step, you just need to repeat Step 3 on the target cluster (as you did with the source cluster). Then, your cluster is ready to restore the application and persistent volume data.
Verify that the IBM VPC CSI driver is in running state:
Verify that the Velero server is in running state:
Step 10: Restore the WordPress application backup on the target (IBM Cloud VPC Gen 2) Kubernetes cluster
You can restore the backup (i.e.,
gen1clusterbackup) that was taken on the VPC Gen 1 cluster by using following command:
This will take some time as it will create two PVCs, deploy the application, and restore the PVC's data.
Verify the restore by using the following command:
It should be in the
Step 11: Verify restore
To verify the WordPress application restoration on the target (IBM VPC Gen 2) cluster, run the following commands:
Check LoadBalancer details to access the WordPress application:
You can access the application by using the
Note: The WordPress application's MySQL database contains a LoadBalancer URL reference. The MySQL database is backed up as part of the persistent volume data. You must update the source cluster WordPress application's LoadBalancer reference to the target cluster WordPress application's LoadBalancer URL. This may not be the case for other applications.
Log in to MySQL pods:
Inside the MySQL pod, log in as a MySQL user. Get the MySQL password from the
kustomization.yaml file in Step 4:
Connect to the
Check the source cluster WordPress application's LoadBalancer URL in the
wp_options database table:
Update the LoadBalancer URL to the target cluster WordPress Application's URL, such as the
http://1724c482-eu-gb.lb.appdomain.cloud in the
Update the post metadata for the target cluster WordPress application's LoadBalancer URL:
Step 12: Uninstall Velero
The following commands will allow you to uninstall Velero from the clusters:
This blog post has shown the process of how to set up Velero with restic on an IBM Cloud Kubernetes Service cluster and how to migrate a WordPress application and its persistent volume data from an IBM Cloud VPC Gen 1 to IBM Cloud VPC Gen 2 cluster.
This overall process can be applied to any application or namespaces running on an IBM Cloud Kubernetes Service cluster. In future blogs, we'll show you how to backup and restore entire IBM Cloud Kubernetes Service clusters within and across regions.