In June of 2020, IBM Cloud VPC Gen 2 Infrastructure was announced, supporting more advanced IaaS capabilities.

Both the IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud can now leverage the next generation of IBM Cloud VPC virtual servers in creating Kubernetes-based clusters to run stateful applications.

This technical blog post will explain how to migrate a stateful application, including persistent volume’s data, from an IBM Cloud VPC Gen 1 (Classic) Kubernetes cluster to an IBM Cloud VPC Gen 2 Kubernetes cluster. There are a variety of tooling and mechanisms that exist in accomplishing the application migration from an older VPC Gen 1 cluster to a new VPC Gen 2 cluster. This blog will focus on describing using the following tooling in supporting the application and data migration:

• Velero
• Restic

The steps both to install and configure the migration tooling—including showcasing a sample WordPress application migration—will be shown.

In this document, persistent volume data is referred to PVC, which is created by the IBM Cloud VPC CSI volume driver.

Objectives

1. Set up the Velero CLI on an IBM Cloud VPC Kubernetes cluster.
2. Back up an application and its data (PVC) using the Velero and restic.
3. Migrate WordPress application from a VPC Gen 1 cluster to a VPC Gen 2 cluster, including its data.

Prerequisites

The following prerequisites are required to migrate an application from one cluster to another (whether these clusters are in the same or different regions):

1. The OC CLI (in case of Red Hat OpenShift cluster)
2. The IBM Cloud and kubectl CLI (for set up, see Getting started with the IBM Cloud CLI)
3. Pre-created IBM Cloud VPC Kubernetes clusters (Gen 1 and Gen 2)
4. The IBM Cloud VPC CSI volume driver installed on both the Gen 1 and Gen 2 clusters

Steps

1. Install the Velero CLI on your system
2. Create an IBM Cloud Object Storage instance
3. Set up the Velero server on the source (IBM Cloud VPC Gen 1) Kubernetes cluster
4. Deploy the WordPress application on the source cluster
5. [Optional] Access the WordPress application and write some data
6. Set up Velero and restic for the WordPress application pods
7. Take the WordPress application backup to IBM Cloud Object Storage by using the Velero CLI
8. Verify backup
9. Set up the Velero server on the target (IBM Cloud VPC Gen 2) Kubernetes cluster
10. Restore the WordPress application backup on target (IBM Cloud VPC Gen 2) Kubernetes cluster
11. Verify restore
12. Uninstall Velero

Step 1: Install the Velero CLI on your system

You can install the Velero CLI on Mac OS by running the following command:

$brew install velero

The Velero CLI uses your cluster’s Kubernetes context. Ensure that the Velero CLI uses the correct kubeconfig for your cluster. More details can be found in the Velero CLI installation docs.

Step 2: Create an IBM Cloud Object Storage instance

You can follow this link to create an IBM Cloud Object Storage instance.

Create an S3 bucket. Make sure to create the bucket across regions so that you can take a backup from any region’s cluster on IBM Cloud. More details.

Then, create a credential for an IBM Cloud Object Storage instance. Make sure you select the HMAC option on the UI as these HMAC-based credentials will be required for setting up the Velero server in the cluster. More details.

After you create the HMAC credentials, copy the values of the access_key_id and secret_access_key fields to use when installing Velero server on the clusters.

Step 3: Set up the Velero server on the source (IBM Cloud VPC Gen 1) Kubernetes cluster

Create a file name—os-cred-for-velero—and add the access_key_id and secret_access_key keys that you got from Step 2.

[default]
aws_access_key_id=hfdi894f333
aws_secret_access_key=8dfj9fdndf

Install the Velero server on the cluster. Velero normally takes volume backup via snapshot, but that requires provider support. Therefore, we need to disable snapshot and enable restic to backup PVC data while installing the Velero server. 

Use the following command to install the Velero server. Use this link to get the right set of inputs for the IBM Cloud Object Storage endpoint and region:

$velero install --provider aws --bucket gen1togen2 --secret-file ./os-cred-for-velero --use-restic --use-volume-snapshots=false --plugins velero/velero-plugin-for-aws:v1.1.0 --backup-location-config region=eu-geo,s3ForcePathStyle="true",s3Url=https://s3.eu.cloud-object-storage.appdomain.cloud

Verify that Velero is installed and that restic pods are equal to the number of nodes in the cluster:

$ kubectl get pods -n velero
NAME                     READY   STATUS    RESTARTS   AGE
restic-5g2pk             1/1     Running   0          2m2s
restic-gfv8p             1/1     Running   0          2m2s
velero-944667869-5h5wb   1/1     Running   0          2m2s
$

Note: If your Kubernetes cluster is created on Red Hat OpenShift on IBM Cloud, you need to follow additional steps. Restic must run in privileged mode, so you must annotate the velero namespace and create a custom SCC to allow restic to run properly.

Run the following command to add the security context in the restic:

$oc adm policy add-scc-to-user privileged -z velero -n velero

Annotate the velero namespace:

$kubectl annotate namespace velero openshift.io/node-selector=""

Modify DaemonSet pod as privileged by adding securityContext:

$kubectl patch ds/restic --namespace velero --type json -p '[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value": { "privileged": true}}]'

Step 4: Deploy the WordPress application on the source cluster

You can either take a backup of a running application or just try to deploy the WordPress application for experimental purposes on the cluster and then take a backup. 

Before beginning to deploy the WordPress application or taking backup of existing application, make sure that the IBM VPC CSI Driver is in a running state:

$ibmcloud ks cluster addon ls --cluster <cluster-name> | grep vpc-block-csi-driver
$kubectl get pods -n kube-system | grep vpc

Deploy the WordPress application in a wordpress namespace so that you can take a backup of namespace by using Velero. Follow the steps in this Kubernetes tutorial.

After you complete the tutorial, the following three resources are created:

kustomization.yaml        mysql-deployment.yaml     wordpress-deployment.yaml

These WordPress deployment files create two persistent volumes—mysql-pv-claim and wp-pv-claim—that are provisioned by IBM VPC CSI Driver. 

Copy the names of the VolumeMounts in these deployment’s pods, which will be used to take backup of volume data.

In mysql-deployment.yaml, the pod’s VolumeMounts name is mysql-persistent-storage. In wordpress-deployment.yaml, the pod’s VolumeMounts name is wordpress-persistent-storage.

Step 5: Access the WordPress application and write some data [optional]

You can verify the WordPress application by checking service and pod status:

$ kubectl get pods -n wordpress
NAME                              READY   STATUS    RESTARTS   AGE
wordpress-6cfd4955b6-4nnkj        1/1     Running   0          14h
wordpress-mysql-8d6799546-wgqn9   1/1     Running   0          14h
$

To access the WordPress application, get the LoadBalancer details as follows:

$ kubectl get svc -n wordpress | grep LoadBalancer
wordpress         LoadBalancer   172.21.99.254  32e01c88-us-south.lb.appdomain.cloud  80:31854/TCP   14h
$

Access the WordPress application by using the http://32e01c88-us-south.lb.appdomain.cloud/ URL and add user or post comments in order to write data to the PVCs.

Step 6: Set up Velero and restic for the WordPress application pods

To take a correct set of backups along with persistent volume data so that the restore will work as expected, you need to set the annotation in the running application pod with the volume names that are used at the time of mounting in the pods. In the case of the WordPress application, VolumeMounts names are mysql-persistent-storage and wordpress-persistent-storage. 

To add the annotation in the WordPress application’s pods, get the running pods in the wordpress namespace:

$ kubectl get pods -n wordpress
NAME                              READY   STATUS    RESTARTS   AGE
wordpress-6cfd4955b6-4nnkj        1/1     Running   0          14h
wordpress-mysql-8d6799546-wgqn9   1/1     Running   0          14h
$

Run following command to add the annotations in the pods:

$kubectl -n wordpress annotate pod/wordpress-6cfd4955b6-4nnkj backup.velero.io/backup-volumes=wordpress-persistent-storage

$kubectl -n wordpress annotate pod/wordpress-mysql-8d6799546-wgqn9 backup.velero.io/backup-volumes=mysql-persistent-storage

Note: You can specify other volumes as per your application by using comma-separation.

Step 7: Take the WordPress application backup to IBM Cloud Object Storage by using the Velero CLI

The WordPress application backup will be taken by using the entire wordpress namespace so that all Kubernetes objects that are related to the application are backed up.

Use the following command to take WordPress application backup as name gen1clusterbackup (or any other name):

$velero backup create gen1clusterbackup --include-namespaces wordpress

Step 8: Verify backup

To verify the backup, run the following Velero command:

$velero backup get | grep gen1clusterbackup

This may take some time. Once the backup is finished, the status will show as Completed. If you encounter any issues, follow the troubleshooting guide.  

Step 9: Set up the Velero server on the target (IBM Cloud VPC Gen 2) Kubernetes cluster

For this step, you just need to repeat Step 3 on the target cluster (as you did with the source cluster). Then, your cluster is ready to restore the application and persistent volume data. 

Verify that the IBM VPC CSI driver is in running state:

$ibmcloud ks cluster addon ls --cluster <cluster-name> | grep vpc-block-csi-driver
$kubectl get pods -n kube-system | grep vpc

Verify that the Velero server is in running state:

$kubectl get pods -n velero

Step 10: Restore the WordPress application backup on the target (IBM Cloud VPC Gen 2) Kubernetes cluster

You can restore the backup (i.e., gen1clusterbackup) that was taken on the VPC Gen 1 cluster by using following command:

$velero restore create restoreongen2cluster --from-backup gen1clusterbackup

This will take some time as it will create two PVCs, deploy the application, and restore the PVC’s data. 

Verify the restore by using the following command:

$velero restore get | grep restoreongen2cluster

It should be in the Completed state.

Step 11: Verify restore

To verify the WordPress application restoration on the target (IBM VPC Gen 2) cluster, run the following commands:

$ kubectl get pods -n wordpress
NAME                              READY   STATUS    RESTARTS   AGE
wordpress-6cfd4955b6-4nnkj        1/1     Running   0          15h
wordpress-mysql-8d6799546-wgqn9   1/1     Running   0          15h
$

Check LoadBalancer details to access the WordPress application:

$ kubectl get svc -n wordpress
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP                         PORT(S)        AGE
wordpress         LoadBalancer   172.21.99.254   1724c482-eu-gb.lb.appdomain.cloud   80:31854/TCP   15h
wordpress-mysql   ClusterIP      None            <none>                              3306/TCP       15h
$

You can access the application by using the http://1724c482-eu-gb.lb.appdomain.cloud/ URL.

Note: The WordPress application’s MySQL database contains a LoadBalancer URL reference. The MySQL database is backed up as part of the persistent volume data. You must update the source cluster WordPress application’s LoadBalancer reference to the target cluster WordPress application’s LoadBalancer URL. This may not be the case for other applications.

Log in to MySQL pods:

$kubectl exec -it wordpress-mysql-8d6799546-wgqn9 -n wordpress sh

Inside the MySQL pod, log in as a MySQL user. Get the MySQL password from the kustomization.yaml file in Step 4:

$mysql -u root -p

Connect to the wordpress database:

mysql> connect wordpress

Check the source cluster WordPress application’s LoadBalancer URL in the wp_options database table:

mysql>select option_value from wp_options where option_value like '%32e01c88-us-south.lb.appdomain.cloud';

Update the LoadBalancer URL to the target cluster WordPress Application’s URL, such as the http://1724c482-eu-gb.lb.appdomain.cloud in the wp_options table:

mysql>UPDATE wp_options SET option_value = replace(option_value, 'http://32e01c88-us-south.lb.appdomain.cloud', 'http://1724c482-eu-gb.lb.appdomain.cloud') WHERE option_name = 'home' OR option_name = 'siteurl';UPDATE wp_posts SET guid = replace(guid, 'http://32e01c88-us-south.lb.appdomain.cloud', 'http://1724c482-eu-gb.lb.appdomain.cloud');UPDATE wp_posts SET post_content = replace(post_content, 'http://32e01c88-us-south.lb.appdomain.cloud', 'http://1724c482-eu-gb.lb.appdomain.cloud');

Update the post metadata for the target cluster WordPress application’s LoadBalancer URL:

mysql>UPDATE wp_postmeta SET meta_value = replace(meta_value, 'http://32e01c88-us-south.lb.appdomain.cloud', 'http://1724c482-eu-gb.lb.appdomain.cloud');

Step 12: Uninstall Velero

The following commands will allow you to uninstall Velero from the clusters:

$kubectl delete namespace/velero clusterrolebinding/velero
$kubectl delete crds -l component=velero

Conclusion

This blog post has shown the process of how to set up Velero with restic on an IBM Cloud Kubernetes Service cluster and how to migrate a WordPress application and its persistent volume data from an IBM Cloud VPC Gen 1 to IBM Cloud VPC Gen 2 cluster. 

This overall  process can be applied to any application or namespaces running on an IBM Cloud Kubernetes Service cluster. In future blogs, we’ll show you how to backup and restore entire IBM Cloud Kubernetes Service clusters within and across regions.

• IBM Cloud Kubernetes Service
• Red Hat OpenShift on IBM Cloud
• IBM Cloud VPC