In June of 2020, IBM Cloud VPC Gen 2 Infrastructure was announced, supporting more advanced IaaS capabilities.
Both the IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud can now leverage the next generation of IBM Cloud VPC virtual servers in creating Kubernetes-based clusters to run stateful applications.
This technical blog post will explain how to migrate a stateful application, including persistent volume’s data, from an IBM Cloud VPC Gen 1 (Classic) Kubernetes cluster to an IBM Cloud VPC Gen 2 Kubernetes cluster. There are a variety of tooling and mechanisms that exist in accomplishing the application migration from an older VPC Gen 1 cluster to a new VPC Gen 2 cluster. This blog will focus on describing using the following tooling in supporting the application and data migration:
The steps both to install and configure the migration tooling—including showcasing a sample WordPress application migration—will be shown.
In this document, persistent volume data is referred to PVC, which is created by the IBM Cloud VPC CSI volume driver.
Objectives
- Set up the Velero CLI on an IBM Cloud VPC Kubernetes cluster.
- Back up an application and its data (PVC) using the Velero and restic.
- Migrate WordPress application from a VPC Gen 1 cluster to a VPC Gen 2 cluster, including its data.
Prerequisites
The following prerequisites are required to migrate an application from one cluster to another (whether these clusters are in the same or different regions):
- The OC CLI (in case of Red Hat OpenShift cluster)
- The IBM Cloud and kubectl CLI (for set up, see Getting started with the IBM Cloud CLI)
- Pre-created IBM Cloud VPC Kubernetes clusters (Gen 1 and Gen 2)
- The IBM Cloud VPC CSI volume driver installed on both the Gen 1 and Gen 2 clusters
Steps
- Install the Velero CLI on your system
- Create an IBM Cloud Object Storage instance
- Set up the Velero server on the source (IBM Cloud VPC Gen 1) Kubernetes cluster
- Deploy the WordPress application on the source cluster
- [Optional] Access the WordPress application and write some data
- Set up Velero and restic for the WordPress application pods
- Take the WordPress application backup to IBM Cloud Object Storage by using the Velero CLI
- Verify backup
- Set up the Velero server on the target (IBM Cloud VPC Gen 2) Kubernetes cluster
- Restore the WordPress application backup on target (IBM Cloud VPC Gen 2) Kubernetes cluster
- Verify restore
- Uninstall Velero
Step 1: Install the Velero CLI on your system
You can install the Velero CLI on Mac OS by running the following command:
The Velero CLI uses your cluster’s Kubernetes context. Ensure that the Velero CLI uses the correct kubeconfig
for your cluster. More details can be found in the Velero CLI installation docs.
Step 2: Create an IBM Cloud Object Storage instance
You can follow this link to create an IBM Cloud Object Storage instance.
Create an S3 bucket. Make sure to create the bucket across regions so that you can take a backup from any region’s cluster on IBM Cloud. More details.
Then, create a credential for an IBM Cloud Object Storage instance. Make sure you select the HMAC
option on the UI as these HMAC
-based credentials will be required for setting up the Velero server in the cluster. More details.
After you create the HMAC credentials, copy the values of the access_key_id
and secret_access_key
fields to use when installing Velero server on the clusters.
Step 3: Set up the Velero server on the source (IBM Cloud VPC Gen 1) Kubernetes cluster
Create a file name—os-cred-for-velero
—and add the access_key_id
and secret_access_key
keys that you got from Step 2.
Install the Velero server on the cluster. Velero normally takes volume backup via snapshot, but that requires provider support. Therefore, we need to disable snapshot and enable restic to backup PVC data while installing the Velero server.
Use the following command to install the Velero server. Use this link to get the right set of inputs for the IBM Cloud Object Storage endpoint and region:
Verify that Velero is installed and that restic pods are equal to the number of nodes in the cluster:
Note: If your Kubernetes cluster is created on Red Hat OpenShift on IBM Cloud, you need to follow additional steps. Restic must run in privileged mode, so you must annotate the velero
namespace and create a custom SCC to allow restic to run properly.
Run the following command to add the security context in the restic:
Annotate the velero
namespace:
Modify DaemonSet pod as privileged by adding securityContext:
Step 4: Deploy the WordPress application on the source cluster
You can either take a backup of a running application or just try to deploy the WordPress application for experimental purposes on the cluster and then take a backup.
Before beginning to deploy the WordPress application or taking backup of existing application, make sure that the IBM VPC CSI Driver is in a running state:
Deploy the WordPress application in a wordpress
namespace so that you can take a backup of namespace by using Velero. Follow the steps in this Kubernetes tutorial.
After you complete the tutorial, the following three resources are created:
These WordPress deployment files create two persistent volumes—mysql-pv-claim
and wp-pv-claim
—that are provisioned by IBM VPC CSI Driver.
Copy the names of the VolumeMounts
in these deployment’s pods, which will be used to take backup of volume data.
In mysql-deployment.yaml
, the pod’s VolumeMounts
name is mysql-persistent-storage
. In wordpress-deployment.yaml
, the pod’s VolumeMounts
name is wordpress-persistent-storage
.
Step 5: Access the WordPress application and write some data [optional]
You can verify the WordPress application by checking service and pod status:
To access the WordPress application, get the LoadBalancer details as follows:
Access the WordPress application by using the http://32e01c88-us-south.lb.appdomain.cloud/
URL and add user or post comments in order to write data to the PVCs.
Step 6: Set up Velero and restic for the WordPress application pods
To take a correct set of backups along with persistent volume data so that the restore will work as expected, you need to set the annotation in the running application pod with the volume names that are used at the time of mounting in the pods. In the case of the WordPress application, VolumeMounts
names are mysql-persistent-storage
and wordpress-persistent-storage
.
To add the annotation in the WordPress application’s pods, get the running pods in the wordpress
namespace:
Run following command to add the annotations in the pods:
Note: You can specify other volumes as per your application by using comma-separation.
Step 7: Take the WordPress application backup to IBM Cloud Object Storage by using the Velero CLI
The WordPress application backup will be taken by using the entire wordpress
namespace so that all Kubernetes objects that are related to the application are backed up.
Use the following command to take WordPress application backup as name gen1clusterbackup
(or any other name):
Step 8: Verify backup
To verify the backup, run the following Velero command:
This may take some time. Once the backup is finished, the status will show as Completed
. If you encounter any issues, follow the troubleshooting guide.
Step 9: Set up the Velero server on the target (IBM Cloud VPC Gen 2) Kubernetes cluster
For this step, you just need to repeat Step 3 on the target cluster (as you did with the source cluster). Then, your cluster is ready to restore the application and persistent volume data.
Verify that the IBM VPC CSI driver is in running state:
Verify that the Velero server is in running state:
Step 10: Restore the WordPress application backup on the target (IBM Cloud VPC Gen 2) Kubernetes cluster
You can restore the backup (i.e., gen1clusterbackup
) that was taken on the VPC Gen 1 cluster by using following command:
This will take some time as it will create two PVCs, deploy the application, and restore the PVC’s data.
Verify the restore by using the following command:
It should be in the Completed
state.
Step 11: Verify restore
To verify the WordPress application restoration on the target (IBM VPC Gen 2) cluster, run the following commands:
Check LoadBalancer details to access the WordPress application:
You can access the application by using the http://1724c482-eu-gb.lb.appdomain.cloud/
URL.
Note: The WordPress application’s MySQL database contains a LoadBalancer URL reference. The MySQL database is backed up as part of the persistent volume data. You must update the source cluster WordPress application’s LoadBalancer reference to the target cluster WordPress application’s LoadBalancer URL. This may not be the case for other applications.
Log in to MySQL pods:
Inside the MySQL pod, log in as a MySQL user. Get the MySQL password from the kustomization.yaml
file in Step 4:
Connect to the wordpress
database:
Check the source cluster WordPress application’s LoadBalancer URL in the wp_options
database table:
Update the LoadBalancer URL to the target cluster WordPress Application’s URL, such as the http://1724c482-eu-gb.lb.appdomain.cloud
in the wp_options
table:
Update the post metadata for the target cluster WordPress application’s LoadBalancer URL:
Step 12: Uninstall Velero
The following commands will allow you to uninstall Velero from the clusters:
Conclusion
This blog post has shown the process of how to set up Velero with restic on an IBM Cloud Kubernetes Service cluster and how to migrate a WordPress application and its persistent volume data from an IBM Cloud VPC Gen 1 to IBM Cloud VPC Gen 2 cluster.
This overall process can be applied to any application or namespaces running on an IBM Cloud Kubernetes Service cluster. In future blogs, we’ll show you how to backup and restore entire IBM Cloud Kubernetes Service clusters within and across regions.